CS3530 OPERATING SYSTEMS Summer 2014 File Management Chapter 8.

CS3530OPERATING SYSTEMS

Summer 2014

File ManagementChapter 8

Purpose of File Management

Data should be organized in some convenient and efficient manner. In particular, users should be able to:Store data into filesFind and use files that have previously been created

File System

A set of OS Services that provides Files and Directories for user applications

Files

A file is simply a sequence of bytes that have been stored in some device (storage) on the computer system

Files

The bytes contain data stored in the file, such as: A text file just containing characters that we are interested in A word processing document file that also contains data about

how to format the text A database contains data organized in multiple files.

In general, the File Management system does not have any knowledge about how the data in a file is organized. That is the responsibility of the application programs that create and use the file.

Permanent Storage Devices

Disk Drives Flash Memory (Memory stick) CDs and DVDs Magnetic tape drives

File Attributes

Name Symbolic (Human-readable) label of the file

Type Executable file, text print file, binary file, etc.

Location The physical address on disk of the file

Size in Mb, Gb, or Tb

Other File Attributes

Protection Permissions for who can read, write, and execute

the file, etc. Time, date

When file was created, modified, accessed

Folders

Name of folder Typically, a folder may contain Files and

other folders (commonly called sub-folders or sub-directories)

This results in a Tree Structure of Folder and Files.

Folder/Directory Tree Structure

Pathnames

The pathname of a file specifies the sequence of folders one must traverse to travel down the tree to the file.

This pathname actually describes the absolute path of the file, which is the sequence of folders one must travel from the root of the tree to the desired file.

A relative path describes the sequence of Folders one must traverse starting at some intermediate place on the absolute path.

File Links

Allow a directory entry to point to a file (or entry) that is not directly below it in the tree structureUnix: Symbolic LinkWindows: Shortcut

Link in Directory Tree Structure

Access Methods

An access method describes the manner and mechanisms by which a process accesses the data in a file.

There are two common access methods: Sequential Random (or Direct)

File Operations

When a process needs to use a file, there are a number of operations it can perform:

Open Close Read Write

Create File

Allocate space for file Make entry for file in the Directory

Open File

Make file accessible for read/write operations Locates file in Directory Returns internal ID for the file

Commonly called a Handle handle = open(filename, parameters)

File Open

Write File

System call specifies: Handle from Open call Location, length of information to be written Possibly, location in the file where data is to be

written write(file handle,buffer,length)

Write File

Use Handle to locate file on disk Use file’s Write pointer to determine the

position in the file to write to Update file’s Write Pointer

Read File

System call specifies: Handle from Open call Memory Location, length of information to be read Possibly, location in the file where data is to be

read from read(file handle, buffer) read(file handle, buffer, length)

Read File

Uses Handle to locate file on disk Uses file’s Read Pointer to determine the

position in the file to read from Update file’s Read Pointer

Close File

Makes file no longer accessible from application Deletes the Handle created by Open

File Close

Delete File

Deletes entry for file in Directory De-allocates disk space used by the file

Sequential Access

If the process has opened a file for sequential access, the File Management subsystem will keep track of the current file position for reading and writing.

To carry this out, the system will maintain a file pointer that will be the position of the next read or write.

File Pointer

The value of the file pointer will be initialized during Open to one of two possible values Normally, this value will be set to 0 to start the reading

or writing at the beginning of the file. If the file is being opened to append data to the file, the

File Position pointer will be set to the current size of the file.

After each read or write, the File Position Pointer will be incremented by the amount of data that was read or written.

Stream

A Stream is the flow of data bytes, one byte after another, into the process (for reading) and out of the process (for writing).

This concept applies to Sequential Access and was originally invented for network I/O, but several modern programming languages (e.g. C/C++, Java, C#) have also incorporated it.

Standard I/O

Standard Input Defaults to keyboard

Standard Output Defaults to console

I/O Redirection

Standard Input can come from a file app.exe < def.txt

Standard Output can go to a file App.exe > def.txt

Standard Output from one application can be Standard Input for another with a pipe App1.exe | app2.exe

Called a Pipe

A Pipe

Pipe

A Pipe is a connection that is dynamically established between two processes.

When a process reads data, the data will come from another process rather than a file. Thus, a pipe has a process at one end that is writing to the pipe and another process reading data at the other end of the pipe.

It is often the situation that one process will produce output that another process needs for input.

Pipe and Performance

Using a pipe can improve system performance in two ways:

By not using a file, the applications save time by not using disk I/O.

A pipe has the characteristic that the receiving process can read whatever data has already been written. Thus we do not need to wait until the first process has written all of the data before we start executing the second process.

Directory Functions

Search for a file Create a file Delete a file List a directory Rename a file Traverse the file system

Disk Space Allocation

Contiguous File is allocated contiguous disk space

Contiguous Allocation

Read/Write Disk Address Calculation

Contiguous Allocation

Advantages Simple to implement Good disk I/O performance

Disadvantages Need to know max file size ahead of time Probably will waste disk space Necessary space may not be available

Disk Space Allocation

Cluster Allocation Disk space allocated in blocks Space allocated as needed

Cluster Allocation

Cluster Allocation

Advantages Tends not to waste disk space

Disadvantages Additional overhead to keep track of clusters Can cause poor disk I/O performance May limit maximum size of File System

Cluster Performance

Clusters tend to be scattered around the disk This is called External Fragmentation Can cause poor performance as disk arm needs

to move a lot Requires De-fragmentation utility

Cluster Performance

Large clusters can reduce External Fragmentation If lots of small files, then space will be wasted

inside each cluster This is called Internal Fragmentation

Crossing Cluster Boundary

Break logical read into multiple physical reads

Managing Cluster Allocation

Linked Each cluster has a pointer to the next cluster

Indexed Single table has pointers to each of the clusters

Linked Blocks

Index Block

Windows File Systems

Fat16 (File Allocation Table) MS-Dos, Windows 95 Max 2GB space for a FileSystem Generally bad disk fragmentation

Fat32 Windows 98 Supported by Windows 2000, XP, 2003

NTFS (New Technology File System)

Windows FAT Table

Windows FAT Processing

Disk Space Allocation

Allocate a free cluster

Update FATSystem Failure

Free Space Management

Linked Linked list of free clusters

Bit Map Special File with a vector of bits Each bit corresponds to a cluster

Linked Free Blocks

Windows FAT Table

DirectoryEntry

FATCluster 1

Cluster 2

Cluster 3

FreeCluster

Free List

FreeCluster

Windows FAT Processing

Unreliable!!! Need to run Scandisk after reboot to attempt

to fix any problems

Windows NTFS File System

Available on Windows 2000, XP, 2003 Maintains transaction log to recover after

reboot Support for file protection Large (64 bit) cluster pointers

Allows small clusters Avoids internal fragmentation

Windows NTFS File System

Disk File System Types

FAT16, FAT32, NTFS, UFS Unix Journaling File System Windows Encrypting File System Network File System (NFS)

Multiple File System Types

Disk File Systems Other File System Organizations

CD-Rom DVD Zip Disk

Multiple File System Types

Each Disk Partition has a single File System A given computer can have a number of

different File System types Modern systems support this capability with a

Virtual File System

Virtual File System

Removable Devices

Connecting Media Called Mounting the FileSystem Can be

Physical Media Logical (across the network)

Journaling

Is only used when writing to a disk and it acts as a sort of punch clock for all writes.

This fixes the problem of disk corruption when data is written to the hard drive and then the computer crashes or power is lost.

Without a journal the operating system would have no way to know if the file was completely written to disk.

Journaling

With a journal the file is first written to the journal, punch-in, and then the journal writes the file to disk when ready.

Once it has successfully written to the disk, it is removed from the journal, punch-out, and the operation is complete.

If power is lost while being written to disk the file system can check the journal for all operations that have not yet been completed and remember where it left off.

Linux File Systems

Ext stands for Extended file system and was the first created specifically for Linux. It has had four revisions and each one has added fairly significant features.

The first version of Ext was a major upgrade from the Minix file system used at the time, but it lacks major features used in today’s computing.

At this time you probably should not use Ext in any machine due to its limitation and age. It also is no longer supported in many distributions.

Linux File Systems

Ext2 is not a journaling file system, and when introduced was the first to allow for extended file attributes and 2 terabyte drives. Because Ext2 does not use a journal it has significantly less writes applied to the disk.Due to lower write requirements, and hence lower erases, it is ideal for flash memory especially on USB flash drives.Modern SSDs have a increased life span and additional features that can negate the need for using a non-journaling file systems.

Linux File Systems

Ext3 is basically just Ext2 with journaling. The aim of Ext3 was to be backwards compatible with Ext2 and therefore disks can be converted between the two without needing to format the drive. The problem with keeping compatibility is many of the limitations of Ext2 still exist

in Ext3. The benefit of keeping backwards compatibility is the fact that most of the testing, bug fixes, and use cases for Ext2 also apply to Ext3 making it stable and fast.

Use if you need to upgrade a previous Ext2 file system to have journaling. You will probably get the best database performance from Ext3 due to years

of optimizations. Not the best choice for file servers because it lacks disk snapshots and file

recovery is very difficult if deleted.

Linux File Systems

Ext4, just like Ext3 before it, keeps backwards compatibility with its predecessors. As a matter of fact, you can mount Ext2 and Ext3 as an Ext4 file system in Linux and that alone can increase performance under certain conditions. You can also mount an Ext4 file system as Ext3 without ill effects. Ext4 reduces file fragmentation, allows for larger volumes and files, and employs delayed

allocation which helps with flash memory life as well as fragmentation. Although it is used in other file systems, delayed allocation has potential for data loss and has come under some scrutiny.

A better choice for SSDs than Ext3 and improves on general performance over both previous Ext versions. If this is your distro’s default supported file system, you should probably stick with it for any desktop or laptop you set up.

It also shows promising performance numbers for database servers, but hasn’t been around as long as Ext3.

Unix File System

Supports protection More reliable than Windows FAT system Need to run fsck (File System Check) utility

on boot-up (similar to Windows Scandisk)

Unix File System

Inodes In a file system, a file is represented by an inode, a data structure

containing information about the actual data that make up the file. Every partition has its own set of inodes. Each inode describes a data structure on the hard disk, storing the

attributes of a file, including the physical location of the file data. When a hard disk is initialized to accept data storage, usually during

the initial system installation process or when adding extra disks to an existing system, a fixed number of inodes per partition is created.

This number will be the maximum amount of files, of all types (including directories, special files, links etc.) that can exist at the same time on the partition.

The typically count is 1 inode per 2 to 8 kilobytes of storage.

Inodes At the time a new file is created, it also creates new inode. In that inode is

the following information:• Owner and group owner of the file.• File type (regular, directory, ...)• Permissions on the file Section 3.4.1• Date and time of creation, last read and change.• Date and time this information has been changed in the inode.• Number of links to this file (see later in this chapter).• File size• An address defining the actual location of the file data.

The only information not included in an inode, is the file name and directory. These are stored in the special directory files. By comparing file names and

inode numbers, the system can make up a tree-structure that the user understands.

Users can display inode numbers using the -i option to ls. The inodes have their own separate space on the disk.