Nan DUN Kenjiro Taura Akinori Yonezawa The University of Tokyo Sep 12, 2007 1dunnan/talk/070912.JSSST07.pptx.

HandyFS: A Simple Ad-hoc Distributed File System only

using FUSE and SSHFSNan DUN Kenjiro Taura Akinori Yonezawa

The University of Tokyo

Sep 12, 2007 1http://www.yl.is.s.u-tokyo.ac.jp/~dunnan/talk/070912.JSSST07.pptx

Background Related Work

◦SSHFS◦FUSE

HandyFS◦Designs and Implementation◦Evaluation and Problems

Future Work

Sep 12, 2007 http://www.yl.is.s.u-tokyo.ac.jp/~dunnan/talk/070912.JSSST07.pptx 2

Outline

User explicit sharing operation◦rcp, scp[OpenSSH]

◦FTP[J. Postel, et al. 1985.], GridFTP[Globus]

Conventional Distributed File System◦Inner-Cluster Sharing NFSv3, NFSv4

◦Inter-Cluster Sharing AFS[OpenAFS], Gfarm[Tatebe et al. ‘04]

Background: File Sharing on Grid

Sep 12, 2007 3http://www.yl.is.s.u-tokyo.ac.jp/~dunnan/talk/070912.JSSST07.pptx

Complexity of conventional DFSs◦ Client-Server Configurations◦ Administrative Privilege◦ Typical Firewall Setting of the Grid

Simplicity of SSHFS [http://fuse.sourceforge.net/sshfs.html]

◦ Only need available SSH connection◦ Client-side mount only◦ User-level Operations without privileges


SSHFS: A simple alternative

Create local mount point◦user@hongo000:~$ mkdir mnt_chiba

Mount remote directory to mount point◦user@hongo000:~$ sshfs chiba000.intrigger.nii.ac.jp:/sharedir mnt_chiba

Access remote files via local mount point◦user@hongo000:~$ [ls, rm, cp, mkdir] mnt_chiba


SSHFS: Usage


SSHFS: Framework

fuse

hostA_mountpoint hostB_mountpoint

SSH client

hostA

SSHFS

User Application

General File System Call

SSH Commands

Translationfuse

SSH client

hostB


SSHFS: Evaluation

chiba suzuk okubo imade0

5

10

15

20

25

30chiba; scp;

24.79

suzuk; scp; 27.33 okubo; scp;

24.92imade; scp;

20.63

chiba; sftp; 21.67

suzuk; sftp; 26.49

okubo; sftp; 17.71

imade; sftp; 12.85

chiba; sshfs (fuse2.6.5);

15.52

suzuk; sshfs (fuse2.6.5);

17.53 okubo; sshfs (fuse2.6.5);

12.86 imade; sshfs (fuse2.6.5);

9.45

scp sftpsshfs (fuse2.6.5) sshfs (fuse2.7.0)

Data Transfer Rate (Mbps)

SSHFS-FUSE overhead

Disadvantages of Merely Using SSHFS◦ Non-scalable

Users have to create many mount points for hosts◦ Non-unified namespace

Files under different mount points (hosts) should be manipulated separately

Motivation of HandyFS◦ Inherit simplicity from SSHFS◦ Create a scalable mount for many hosts◦ Ad-hoc create a unified namespace for files on

many hosts


HandyFS: Motivation


HandyFS: Usage Create ONE local mount point

◦user@hongo000:~$ mkdir mnt_all Mount MULTIPLE remote directories to

mount point◦user@hongo000:~$ handyfs chiba000.intrigger.nii.ac.jp:/sharedir1

chiba001.intrigger.nii.ac.jp:/sharedir2 ...... mnt_all

Access remote files via local mount point◦user@hongo000:~$ [ls, rm, cp] mnt_all


HandyFS: Framework

FUSE Kernel Module

User Application HandyFS SSHFS

Host1 Host2

User

Kernel

SSHFS

The trick is using FUSE twice1. SSHFS mount for remote host2. Local mount for directory merge

/handyfs_mnt /sshfs_mnt1

/sshfs_mnt1

Source directory Tree at HostB

Source directory Tree at HostA

Directories Merge◦ Source directories and Target (virtual) directory

Source directory: SSHFS mount points for each host Target directory: Directory in HandyFS namespace


HandyFS: File Operations

/mnt_hostA

/A /B

/B/C

/mnt_hostB

/B /C

/B/D

/

/A /B

/B/C

Virtual directory Tree

/C

/B/D

Directories Merge◦ Source directories and Target (virtual) directory◦ Directory Table

To remember virtual directory is merged from which source directory

◦ On-Demand Merge Operation Merge operation occurs only when request reach

non-merged path



/cmndir /srcdirA/cmndir

Virtual Directory Source Directories

/srcdirB/cmndir

Implemented by using BerkeleyDB to achieve good performance

File Lookup



Lookup a file with full pathname

Retrieve source directory entries from

directory table

Lookup file in each retrieved source directory entries

If target pathname has not been

merged, then merge and store into directory table Return result to user

On-demand Merge

Directory Split◦ Remove one or several source directories from

virtual directory, we have two alternatives Remove directory table entries

Heavy Operation: Search entire directory table Re-insert overhead if source become available again

Blind user view of source directories by filtering Use a in-memory filter to mark current available sources File lookup results are filtered before returning to user Efficient and re-usable



Directory Split



Lookup a file with full pathname

Retrieve source directory entries

from directory table

Lookup file in each retrieved source directory entries

If target pathname has not been

merged, then merge and store into directory table

Return result to user

Filtering out results that are from

removed source directories

File Creation◦ File is created at random source directory◦ For directory file, an entry is created in directory

table

Directory Read◦ Similar as file lookup

Lookup target directory Read all entries from source directories that

composed of target directory



Some Practical Problems◦ Same file from multiple sources

Not compatible with UNIX file semantics Define priority of source directory

Only files from source directory with high priority are visible to user

◦ Stale of Directory Table Remote directory tree may be modified by its local

operations Periodically update



Experimental Environments◦ InTrigger Platform:

https://www.logos.ic.i.u-tokyo.ac.jp/intrigger/


HandyFS: Evaluation

Hardware/Software Specification

CPU Core2 Duo 2.13Hz

Memory 4GB

Network Gigabit Ethernet

OS Linux 2.6.18

FUSE Kernel Interface 7.8

FUSE Library 2.7.0

SSHFS 1.8

https://www.logos.ic.i.u-tokyo.ac.jp/intrigger/

Experimental Results◦ Data Transfer Rate


HandyFS: Evaluation

chiba suzuk okubo imade0

10

20

30

40

50

60

70

223 222

ftp sftp sshfs HandyFS

SSHFS-FUSE overhead

TWICE-FUSE-MOUNT overhead

Experimental Results◦ Source Compilation


HandyFS: Evaluation

OpenSSH4.6 Source Compilation Benchmark0

50

100

150

200

250

300

350

Local FUSE-Local NFSSSHFS HandyFS

Wide-Area LatencySSHFS

Overhead

TWICE-FUSE-MOUNT overhead

Performance◦ HandyFS itself

To remove MOUNT-TWICE overhead◦ SSHFS performance

Need be tuned at source level

Scalability◦ “One-to-All” mount “All-to-All” mount


Future Work

Nan DUN Kenjiro Taura Akinori Yonezawa The University of Tokyo Sep 12, 2007 1dunnan/talk/070912.JSSST07.pptx.

Documents

sshfs mount points

local mount point user

scalable mount

chiba mount remote directory

grid simplicity of sshfs

mnt1 slide

directory merge handyfs

different mount points