LSM in a Week

Build a simple key-value storage engine in a week!

Tutorial

The tutorial is available at https://skyzh.github.io/mini-lsm. You can use the provided starter code to kick off your project, and follow the tutorial to implement the LSM tree.

Community

You may join skyzh's Discord server and study with the mini-lsm community.

Development

cargo x install-tools
cargo x check
cargo x book

If you changed public API in the reference solution, you might also need to synchronize it to the starter crate. To do this, use cargo x sync.

Progress

We are working on a new version of the mini-lsm tutorial that is split into 3 weeks.

• Week 1: Storage Format + Engine Skeleton
• Week 2: Compaction and Persistence
• Week 3: Multi-Version Concurrency Control
• The Extra Week / Rest of Your Life: Optimizations (unlikely to be available in 2024...)

Week + Chapter	Topic	Solution	Starter Code	Writeup
1.1	Memtables	✅	✅	✅
1.2	Block Format	✅	✅	✅
1.3	Table Format	✅	✅	✅
1.4	Merge Iterators	✅	✅	✅
1.5	Storage Engine - Read Path	✅	✅	✅
1.6	Storage Engine - Write Path	✅	✅	✅
1.7	Bloom Filter and Key Compression
2.1	Compaction Implementation	✅	🚧	🚧
2.2	Compaction Strategy - Simple	✅	🚧	🚧
2.3	Compaction Strategy - Tiered	✅
2.4	Compaction Strategy - Leveled	✅
2.5	Manifest	✅
2.6	Write-Ahead Log	✅
2.7	Batch Write + Checksum
3.1	Timestamp Key Encoding + New Block Format
3.2	Prefix Bloom Filter
3.3	Snapshot Read
3.4	Watermark and Garbage Collection
3.5	Transactions and Optimistic Concurrency Control
3.6	Serializable Snapshot Isolation
3.7	TTL (Time-to-Live) Entries
4.1	Benchmarking
4.2	Block Compression
4.3	Trivial Move and Parallel Compaction
4.4	Alternative Block Encodings
4.5	Rate Limiter and I/O Optimizations
4.6	Build Your Own Block Cache
4.7	Build Your Own SkipList
4.8	Async Engine
4.9	Key-Value Separation
4.10	Column Families
4.11	SQL over Mini-LSM

License

The Mini-LSM starter code and solution are under Apache 2.0 license. The author reserves the full copyright of the tutorial materials (markdown files and figures).