Storage Systems

Course Objective

Slow storage has been the Achilles’ heel in building fast and efficient systems. However, in the last 10-15 years we have witnessed the rise of Non-Volatile Memory (NVM) storage technologies in commodity computing. These technologies such as NAND Flash and Optane offer multiple orders of magnitude performance improvements over hard-disk drives (HDD). This advancement represents one of the most fundamental shifts in storage technology since the emergence of the HDD in the 1960s. The objective of this course is to educate students about the impact and implications of NVM storage on system design (architecture, operating system), distributed systems, storage services, and application designs.

Upon completion of this course students will be able to:

• Understand the fundamental difference between NVM storage (Flash SSDs) and HDDs [Knowledge and understanding]
• Learn about the internals of SSDs (FTL, GC, wear leveling, over-provisioning) and the NVM Express (NVMe) host interface [Knowledge and understanding]
• Explain the impact that the emergence of NVM storage has on the design of the Linux block layer, file systems, and key value store designs [Applying knowledge and understanding]
• Learn about emerging Byte-addressable Persistent Memories (pmem) and their impact on operating systems and applications development [Knowledge and understanding]
• Analyze and explain advanced storage topics: Programmable storage, Storage disaggregation, virtualization, scheduling, and Distributed transaction systems [Applying knowledge and understanding] [Making judgements]
• Apply the knowledge learned in a practical setting by developing a complete storage stack using NVMe ZNS storage devices [Applying knowledge and understanding] [Making judgements] [Lifelong learning skills]
• Explain and present design and implementation of a complex storage stack [Communication]

Course Content

We will have weekly lectures that will cover topics such as introduction to NVM storage technology (NAND flash), internals of NVM storage (FTL design, garbage collection, wear leveling), design tradeoffs, emergence of new storage host controllers and protocols (NVM Express), operating systems implications of new storage devices, new flash-optimized file systems (Log-structured file systems, F2FS, DFS), key value stores (B+ trees, Hash tables, and LSM trees), emerging Byte-addressable Persistent Memories (pmem) and persistent heaps and data structures, programmability and application-specific storage, virtualization, and modern NVM-specific distributed systems designs like distributed transactions on flash etc.

Apart from lectures, there will be programming assignments. The course will also link lectures and assignments to modern storage system features and offer insights into state-of-the-art research efforts.

Additional Information Teaching Methods

2x Lectures / week, Group coding project work with 1 lab session / week for Q&A (deliverables: code and report). Optional coding bonuses available.

Method of Assessment

Exam (50%, 50 points), Practical Assignments (50%, 50 points over 5 assignments) -- resits only available for the exam. Both exam and practical assignment need to be passed separately. Their cut off is 25 points for both. A total of (25 + 25 =) 50 / 100 points will give you a grade of 6.0/10.

Practical assignment is done in a group of 2 students. Students are graded on both the completion of the assignment as well as their understanding of the material (based on the oral discussion/interview and report/code submitted that pass various design units tests).

[FRAUD] The course is made fraud resistant by having specialized group projects with interviews to assess the knowledge of the students. Furthermore, the exam is taken individually in-person on the campus.

Literature

Slides (provided in the class), lectures will be recorded and uploaded to Canvas, project handbook, and further online reading material/links.

Additional Information Target Audience

Recommended background knowledge

Operating systems concepts (kernel, userspace, multithreading programming, device management), basic storage systems concepts (file systems, inode, trees) computer organization (CPU caches, instructions, interrupts), distributed systems, knowledge of C/C++. For practical assignments (QEMU, Linux development environment - gcc, cmake, gdb, and friends).

We will provide a set of background reading literature on storage before the course starts (available on Canvas). For help with the programming project please join the lab session and ask the TAs.