FiST: Stackable File System Language and Templates

File System development is very difficult and time consuming. Even small changes to existing file systems require deep understanding of kernel internals, making the barrier to entry for new developers high. Moreover, porting file system code from one operating system to another is almost as difficult as the first port. Past proposals to provide extensible (stackable) file system interfaces would have simplified the development of new file systems. These proposals, however, advocated massive changes to existing operating system interfaces and existing file systems; operating system vendors and maintainers resist making any large changes to their kernels because of stability and performance concerns. As a result, file system development is still a difficult, long, and non-portable process.

The FiST (File System Translator) system combines two methods to solve the above problems in a novel way: a set of stackable file system templates for each operating system, and a high-level language that can describe stackable file systems in a cross-platform portable fashion. Using FiST, stackable file systems need only be described once. FiST's code generation tool, fistgen, compiles a single file system description into loadable kernel modules for several operating systems (currently Solaris, Linux, and FreeBSD). The project demonstrates that with FiST, code size and development time are reduced significantly, while imposing a small performance overhead of only 1-2%. These benefits are achieved, as well as portability, without changing existing operating systems or file system.

Public Software:

Mailing Lists:

Click on the list name below to access the subscription Web page. That page will also tell you how to post to the list, access the lists' archives, change your subscription status, etc.

List Name Purpose Who Can Subscribe Who Can Post Comments
fistgen For users' and developers' questions, announcements, bug reports, and general. Anyone Subscribers only This list used to be called fist at cs.columbia.edu. Use new name please.
fistgen-announce For announcements only Anyone Developers only  

Miscellaneous:

If you are interested in a stackable file systems in general, I have a collection of old stacking papers.

If you find a problem, please describe it in detail and submit a bug report via Bugzilla.

Download software.

Journal Articles:

# Title (click for html version) Formats Published In Date Comments
1 The Case for Specialized File Systems, or, Fighting File System Obesity PDF BibTeX ;login: The USENIX Magazine Feb 2010  
2 On Incremental File System Development PS PDF BibTeX ACM Transactions on Storage (TOS) May 2006  
3 Versatility and Unix Semantics in Namespace Unification PS PDF BibTeX ACM Transactions on Storage (TOS) Feb 2006  
4 Unionfs: Bringing File Systems Together BibTeX Linux Journal Dec 2004  
5 Writing Stackable Filesystems BibTeX Linux Journal May 2003 Software described in article

Conference and Workshop Papers:

# Title (click for html version) Formats Published In Date Comments
1 RAIF: Redundant Array of Independent Filesystems PS PDF BibTeX IEEE Conference on Mass Storage Systems and Technologies (MSST 2007) Sep 2007  
2 Kernel Support for Stackable File Systems PS PDF BibTeX 2007 Ottawa Linux Symposium Jun 2007  
3 Making Linux Stacking-Friendly BibTeX First Linux Storage and Filesystem Workshop, held in conjuction with the fifth USENIX Conference on File and Storage Technologies (FAST 2007) Feb 2007  
4 UnionFS: User- and Community-oriented Development of a Unification Filesystem PS PDF BibTeX 2006 Ottawa Linux Symposium Jul 2006  
5 Accurate and Efficient Replaying of File System Traces PS PDF BibTeX Fourth USENIX Conference on File and Storage Technologies (FAST 2005) Dec 2005  
6 Adding Secure Deletion to Your Favorite File System PS PDF BibTeX Third IEEE Security In Storage Workshop (SISW 2005) Dec 2005  
7 Stackable File Systems and Methods Thereof PDF BibTeX US Patent Application 10/862,212

Dec 2005 US Patent Application 10/862,212 publication
8 Increasing Distributed Storage Survivability with a Stackable RAID-like File System PS PDF BibTeX First IEEE/ACM Workshop on Cluster Security, in conjunction with the Fifth IEEE/ACM International Symposium on Cluster Computing and the Grid (CCGrid 2005) May 2005 Won Best Paper Award
9 I3FS: An In-Kernel Integrity Checker and Intrusion Detection File System PS PDF BibTeX Usenix Large Installation Systems Administration Conference (LISA 18) Nov 2004  
10 Avfs: An On-Access Anti-Virus File System PS PDF BibTeX 13th USENIX Security Symposium (Security 2004) Aug 2004  
11 A Versatile and User-Oriented Versioning File System PS PDF BibTeX Third USENIX Conference on File and Storage Technologies (FAST 2004) Apr 2004  
12 Tracefs: A File System to Trace Them All PS PDF BibTeX Third USENIX Conference on File and Storage Technologies (FAST 2004) Apr 2004  
13 Reducing Storage Management Costs via Informed User-Based Policies PS PDF BibTeX NASA/IEEE Conference on Mass Storage Systems and Technologies (MSST 2004) Apr 2004  
14 Cryptographic File Systems Performance: What You Don't Know Can Hurt You PS PDF BibTeX 2003 IEEE Security In Storage Workshop (SISW 2003) Oct 2003  
15 NCryptfs: A Secure and Convenient Cryptographic File System PS PDF BibTeX Slides Usenix Technical Conference, General Track Jun 2003  
16 Fast Indexing: Support for Size-Changing Algorithms in Stackable File System PS PDF BibTeX Usenix Technical Conference Jun 2001  
17 FiST: A Language for Stackable File Systems PS PDF BibTeX Usenix Technical Conference Jun 2000  
18 Extending File Systems Using Stackable Templates PS PDF BibTeX Usenix Technical Conference Jun 1999  
19 A Stackable File System Interface For Linux PS PDF BibTeX LinuxExpo 99 May 1999  

Technical Reports:

# Title (click for html version) Formats Published In Date Comments
1 Storage Virtualization with a Stackable File System PDF BibTeX Stony Brook U. CS TechReport FSL-05-03 Dec 2005 M.S. Thesis
2 Versatility and Unix Semantics in a Fan-Out Unification File System PS PDF BibTeX Stony Brook U. CS TechReport FSL-04-01b Oct 2004  
3 Versatile File System Tracing with Tracefs PS PDF BibTeX Stony Brook U. CS TechReport FSL-04-05 Aug 2004 M.S. Thesis
4 Enhancing File System Integrity Through Checksums PS PDF BibTeX Stony Brook U. CS TechReport FSL-04-04 May 2004  
5 Operating System Support for Extensible Secure File Systems PS PDF BibTeX Stony Brook U. CS TechReport FSL-04-02 May 2004 Ph.D. Research Proficiency Exam (RPE)
6 Cryptographic File Systems Performance: What You Don't Know Can Hurt You PS PDF BibTeX Stony Brook U. CS TechReport FSL-03-02 Aug 2003 Slightly expanded version of SISW'03 paper with same title.
7 Reducing Storage Management Costs via Informed User-Based Policies PS PDF BibTeX Stony Brook U. CS TechReport FSL-03-01 Mar 2003  
8 The Design and Implementation of Elastic Quotas: A System for Flexible File System Management PS PDF BibTeX Columbia U. CS TechReport CUCS-014-02 Jun 2002  
9 FiST: A System for Stackable File System Code Generation PS PDF BibTeX Columbia University May 2001 Ph.D. Dissertation
10 Performance of Size-Changing Algorithms in Stackable File Systems PDF BibTeX Columbia U. CS TechReport CUCS-023-00 Dec 2000  
11 Stackable File Systems As a Security Tool PS PDF BibTeX Columbia U. CS TechReport CUCS-036-99 Dec 1999  
12 Usenetfs: A Stackable File System for Large Article Directories PS PDF BibTeX Columbia U. CS TechReport CUCS-022-98 Jul 1998  
13 Cryptfs: A Stackable Vnode Level Encryption File System PS PDF BibTeX Columbia U. CS TechReport CUCS-021-98 Jun 1998 This tech-report had been cited more than 15 times before the new version of the work was published -- NCryptfs.
14 FiST: A File System Component Compiler PS PDF BibTeX Columbia University Apr 1997 Ph.D. Thesis Proposal

Works In Progress:

# Title (click for html version) Formats Published In Date Comments
1 Fast Indexing: Support for Size-Changing Algorithms in Stackable File System PS PDF BibTeX   Oct 2001 Expanded version of Usenix 2001 paper

Past Students:

# Name (click for home page) Program Period Current Location
1 Charles P. Wright PhD May 2003 - May 2006 Partner, Senior Software Architect, Illumon (New York, NY)
2 Shrikar Archak MS Sep 2009 - Dec 2010 Senior Software Engineer, Core Deduplication Filesystem group, Datadomain/EMC
3 Puja Gupta MS Jan 2003 - Dec 2003 Software Engineering Manager, Darwin Runtime, Core OS, Apple Inc. (Cupertino, CA)
4 Harikesavan Pathangi Krishnan MS Jan 2003 - Dec 2003 Software Engineer, Engineering department for WAFS product line, Packeteer, Inc. (South Plainfield, NJ)
5 Rohit Kumar MS Jan 2018 - Dec 2018 Software Development Engineer II, Amazon Services, Amazon Services (Seattle, WA)
6 Kiran-Kumar Muniswamy-Reddy MS Jan 2002 - May 2004 Consulting Member of Technical Staff, Oracle Corp (Seattle, WA)
7 Sunil Satnur MS Sep 2004 - Dec 2005 Staff Engineer, Storage and Avaliability Group, VMware Inc. (Palo Alto, CA)
8 Kiron Vijayasankar MS Dec 2006 - Dec 2007 Member of Technical Staff, Engineering Development, Riverbed Technology (Sunnyvale, CA)
9 Charles P. Wright BS (Honors) Dec 2001 - May 2003 Application Software Developer, Walleye Software (New York, NY)
10 Josef "Jeff" Sipek BS Jul 2005 - May 2008 Senior Member of Technical Staff, VMware (Boston, MA)

Sponsors:

# Sponsor Amount Period Type Title (click for award abstract)
1 HP/Intel $131,529 2002-2003 Sole PI Linux Application Performance and File System Security
2 HP/Intel $22,490 2001-2002 Sole PI Linux Network Scalability and File System Reliability