@@ -2,12 +2,11 @@
|
||||
|
||||
[](https://github.com/skyzh/mini-lsm/actions/workflows/main.yml)
|
||||
|
||||
Build a simple key-value storage engine in a week!
|
||||
Build a simple key-value storage engine in a week! And extend your LSM engine on the second + third week.
|
||||
|
||||
## Tutorial
|
||||
## [Tutorial](https://skyzh.github.io/mini-lsm)
|
||||
|
||||
The tutorial is available at [https://skyzh.github.io/mini-lsm](https://skyzh.github.io/mini-lsm). You can use the provided starter
|
||||
code to kick off your project, and follow the tutorial to implement the LSM tree.
|
||||
The Mini-LSM book is available at [https://skyzh.github.io/mini-lsm](https://skyzh.github.io/mini-lsm).
|
||||
|
||||
## Community
|
||||
|
||||
|
@@ -34,4 +34,6 @@ cargo x copy-test --week 1 --day 1
|
||||
cargo x scheck
|
||||
```
|
||||
|
||||
Now, you can go ahead and start [Week 1: Mini-LSM](./week1-overview.md).
|
||||
|
||||
{{#include copyright.md}}
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Mini-LSM Overview
|
||||
# Mini-LSM Course Overview
|
||||
|
||||
## Overview of LSM
|
||||
|
||||
@@ -55,4 +55,10 @@ There are two types of read: lookup and scan. Lookup finds one key in the LSM tr
|
||||
|
||||
We have 3 parts (weeks) for this tutorial. In the first week, we will focus on the storage structure and the storage format of an LSM storage engine. In the second week, we will dive into compactions in depth and implement persistence support for the storage engine. In the third week, we will implement multi-version concurrency control.
|
||||
|
||||
* [The First Week: Mini-LSM](./week1-overview.md)
|
||||
* [The Second Week: Compaction and Persistence](./week2-overview.md)
|
||||
* [The Third Week: Multi-Version Concurrency Control](./week3-overview.md)
|
||||
|
||||
To set up the environment, please take a look at [Environment Setup](./00-get-started.md).
|
||||
|
||||
{{#include copyright.md}}
|
||||
|
@@ -84,6 +84,9 @@ You may join skyzh's Discord server and study with the mini-lsm community.
|
||||
|
||||
[](https://skyzh.dev/join/discord)
|
||||
|
||||
## Get Started
|
||||
|
||||
Now, you may go ahead and get an overview of the LSM structure in [Mini-LSM Course Overview](./00-overview.md).
|
||||
|
||||
## About the Author
|
||||
|
||||
|
@@ -4,13 +4,13 @@
|
||||
|
||||
In the first week of the tutorial, you will build necessary storage formats for the storage engine, the read path and the write path of the system, and have a working implementation of an LSM-based key-value store. There are 7 chapters (days) for this part.
|
||||
|
||||
* Day 1: Memtable. You will implement the in-memory read and write path of the system.
|
||||
* Day 2: Merge iterator. You will extend what you have built in day 1 and implement a `scan` interface for your system.
|
||||
* Day 3: Block encoding. Now we start the first step of the on-disk structure and build the encoding/decoding of the blocks.
|
||||
* Day 4: SST encoding. SSTs are composed of blocks and at the end of the day, you will have the basic building blocks of the LSM on-disk structure.
|
||||
* Day 5: Read path. Now that we have both in-memory and on-disk structures, we can combine them together and have a fully-working read path for the storage engine.
|
||||
* Day 6: Write path. In day 5, the test harness generates the structures, and in day 6, you will control the SST flushes by yourself. You will implement flush to level-0 SST and the storage engine is complete.
|
||||
* Day 7: SST optimizations. We will implement several SST format optimizations and improve the performance of the system.
|
||||
* [Day 1: Memtable](./week1-01-memtable.md). You will implement the in-memory read and write path of the system.
|
||||
* [Day 2: Merge Iterator](./week1-02-merge-iterator.md). You will extend what you have built in day 1 and implement a `scan` interface for your system.
|
||||
* [Day 3: Block Encoding](./week1-03-block.md). Now we start the first step of the on-disk structure and build the encoding/decoding of the blocks.
|
||||
* [Day 4: SST Encoding](./week1-04-sst.md). SSTs are composed of blocks and at the end of the day, you will have the basic building blocks of the LSM on-disk structure.
|
||||
* [Day 5: Read Path](./week1-05-read-path.md). Now that we have both in-memory and on-disk structures, we can combine them together and have a fully-working read path for the storage engine.
|
||||
* [Day 6: Write Path](./week1-06-write-path.md). In day 5, the test harness generates the structures, and in day 6, you will control the SST flushes by yourself. You will implement flush to level-0 SST and the storage engine is complete.
|
||||
* [Day 7: SST Optimizations](./week1-07-sst-optimizations.md). We will implement several SST format optimizations and improve the performance of the system.
|
||||
|
||||
At the end of the week, your storage engine should be able to handle all get/scan/put requests. The only missing parts are persisting the LSM state to disk and a more efficient way of organizing the SSTs on the disk. You will have a working **Mini-LSM** storage engine.
|
||||
|
||||
|
@@ -7,13 +7,13 @@ In the last week, you have implemented all necessary structures for an LSM stora
|
||||
We have 7 chapters (days) in this part:
|
||||
|
||||
|
||||
* Day 1: Compaction Implementation. You will merge all L0 SSTs into a sorted run.
|
||||
* Day 2: Simple Leveled Compaction. You will implement a classic leveled compaction algorithm and use compaction simulator to see how well it works.
|
||||
* Day 3: Tiered/Universal Compaction. You will implement the RocksDB universal compaction algorithm and understand the pros/cons.
|
||||
* Day 4: Leveled Compaction. You will implement the RocksDB leveled compaction algorithm. This compaction algorithm also supports partial compaction, so as to reduce peak space usage.
|
||||
* Day 5: Manifest. You will store the LSM state on the disk and recover from the state.
|
||||
* Day 6: WAL. User requests will be routed to both memtable and WAL.
|
||||
* Day 7: Write batch interface and checksums.
|
||||
* [Day 1: Compaction Implementation](./week2-01-compaction.md). You will merge all L0 SSTs into a sorted run.
|
||||
* [Day 2: Simple Leveled Compaction](./week2-02-simple.md). You will implement a classic leveled compaction algorithm and use compaction simulator to see how well it works.
|
||||
* [Day 3: Tiered/Universal Compaction](./week2-03-tiered.md). You will implement the RocksDB universal compaction algorithm and understand the pros/cons.
|
||||
* [Day 4: Leveled Compaction](./week2-04-leveled.md). You will implement the RocksDB leveled compaction algorithm. This compaction algorithm also supports partial compaction, so as to reduce peak space usage.
|
||||
* [Day 5: Manifest](./week2-05-manifest.md). You will store the LSM state on the disk and recover from the state.
|
||||
* [Day 6: Write-Ahead Log (WAL)](./week2-06-wal.md). User requests will be routed to both memtable and WAL so that all operations will be persisted.
|
||||
* [Day 7: Write Batch and Checksums](./week2-07-snacks.md). You will implement write batch API (for preparation for week 3 MVCC) and checksums for all of your storage formats.
|
||||
|
||||
## Compaction and Read Amplification
|
||||
|
||||
|
@@ -1,3 +1,3 @@
|
||||
# Week 3 Solution
|
||||
|
||||
This is the solution of Mini-LSM week 3.
|
||||
This is the solution of Mini-LSM week 3 with MVCC implementation.
|
||||
|
3
mini-lsm-starter/README.md
Normal file
3
mini-lsm-starter/README.md
Normal file
@@ -0,0 +1,3 @@
|
||||
# mini-lsm-starter
|
||||
|
||||
Starter code for Mini-LSM.
|
@@ -1,3 +1,3 @@
|
||||
# Week 2 Solution
|
||||
|
||||
This is the solution of Mini-LSM week 1 + week 2.
|
||||
This is the solution for Mini-LSM week 1 + week 2.
|
||||
|
Reference in New Issue
Block a user