maokaihe/mini_lsm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(book): finish overview

Browse Source 
Signed-off-by: Alex Chi <iskyzh@gmail.com>
This commit is contained in:
Alex Chi
2022-12-23 18:44:59 -05:00
parent bf38b99f5a
commit c09a54e796
15 changed files with 1060 additions and 94 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
Cargo.lock generated
View File

@@ -178,6 +178,13 @@ dependencies = [
"bytes", "bytes",
] ]
[[package]]
name = "mini-lsm-starter"
version = "0.1.0"
dependencies = [
"bytes",
]
[[package]] [[package]]
name = "mini-lsm-xtask" name = "mini-lsm-xtask"
version = "0.1.0" version = "0.1.0"

1
Cargo.toml
View File

@@ -1,6 +1,7 @@
[workspace] [workspace]
members = [ members = [
"mini-lsm", "mini-lsm",
"mini-lsm-starter",
"xtask" "xtask"
] ]

89
mini-lsm-book/src/00-overview.md
View File

@@ -4,4 +4,91 @@ In this tutorial, you will learn how to build a simple LSM-Tree storage engine i
## What is LSM, and Why LSM? ## What is LSM, and Why LSM?
## Roadmap Log-structured merge tree is a data structure to maintain key-value pairs. This data structure is widely used in
distributed database systems like [TiDB](https://www.pingcap.com) and [CockroachDB](https://www.cockroachlabs.com) as
their underlying storage engine. [RocksDB](http://rocksdb.org), based on [LevelDB](https://github.com/google/leveldb),
is an implementation of LSM-Tree storage engine. It provides a wide range of key-value access functionalities and is
used in a lot of production systems.
Generally speaking, LSM Tree is an append-friendly data structure. It is more intuitive to compare LSM to other
key-value data structure like RB-Tree and B-Tree. For RB-Tree and B-Tree, all data operations are in-place. That is to
say, when you update the value corresponding to the key, the value will be overwritten at its original memory or disk
space. But in an LSM Tree, all write operations, i.e., insertions, updates, deletions, are performed in somewhere else.
These operations will be batched into SST (sorted string table) files and be written to the disk. Once written to the
disk, the file will not be changed. These operations are applied lazily on disk with a special task called compaction.
The compaction job will merge multiple SST files and remove unused data.
This architectural design makes LSM tree easy to work with.
1. Data are immutable on persistent storage, which means that it is easier to offload the background tasks (compaction)
to remote servers. It is also feasible to directly store and serve data from cloud-native storage systems like S3.
2. An LSM tree can balance between read, write and space amplification by changing the compaction algorithm. The data
structure itself is super versatile and can be optimized for different workloads.
In this tutorial, we will learn how to build an LSM-Tree-based storage engine in the Rust programming language.
## Overview of LSM
An LSM storage engine generally contains 3 parts:
1. Write-ahead log to persist temporary data for recovery.
2. SSTs on the disk for maintaining a tree structure.
3. Mem-tables in memory for batching small writes.
The storage engine generally provides the following interfaces:
* `Put(key, value)`: store a key-value pair in the LSM tree.
* `Delete(key)`: remove a key and its corresponding value.
* `Get(key)`: get the value corresponding to a key.
To ensure persistence,
* `Sync()`: ensure all the operations before `sync` are persisted to the disk.
Some engines choose to combine `Put` and `Delete` into a single operation called `WriteBatch`, which accepts a batch
of key value pairs.
In this tutorial, we assume the LSM tree is using leveled compaction algorithm, which is commonly used in real-world
systems.
## Write Flow
![Write Flow](figures/lsm-tutorial/00-lsm-write-flow.svg)
The write flow of LSM contains 4 steps:
1. Write the key-value pair to write-ahead log, so that it can be recovered after the storage engine crashes.
2. Write the key-value pair to memtable. After (1) and (2) completes, we can notify the user that the write operation
is completed.
3. When a memtable is full, we will flush it to the disk as an SST file in the background.
4. We will compact some files in some level into lower levels to maintain a good shape for the LSM tree, so that read
amplification is low.
## Read Flow
![Read Flow](figures/lsm-tutorial/00-lsm-read-flow.svg)
When we want to read a key,
1. We will first probe all the memtables from latest to oldest.
2. If the key is not found, we will then search the entire LSM tree containing SSTs to find the data.
## Tutorial Overview
![Tutorial Overview](figures/lsm-tutorial/00-lsm-tutorial-overview.svg)
In this tutorial, we will build the LSM tree structure in 7 days:
* Day 1: Block encoding. SSTs are composed of multiple data blocks. We will implement the block encoding.
* Day 2: SST encoding.
* Day 3: Engine. In this day we will get a functional (but not persistent) key-value engine with `get`, `put`, `delete`
API.
* Day 4: Block cache. To reduce disk I/O and maximize performance, we will use moka-rs to build a block cache for the
LSM tree.
* Day 5: Compaction. Now it's time to maintain a leveled structure for SSTs.
* Day 6: Recovery. We will implement WAL and manifest so that the engine can recover after restart.
* Day 7: Bloom filter and key compression. They are widely-used optimizations in LSM tree structures.
We provide you starter code (see `mini-lsm-starter` crate), where we simply replace all function body with
`unimplemented!()`. You can start your project based on this starter code. We provide test cases, but they are very
simple. We recommend you to think carefully about your implementation and write test cases by yourself.

185
mini-lsm-book/src/figures/lsm-tutorial/00-lsm-read-flow.svg Normal file
View File

@@ -0,0 +1,185 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xl="http://www.w3.org/1999/xlink" xmlns:dc="http://purl.org/dc/elements/1.1/" version="1.1" viewBox="6 194 693 361" width="693" height="361">
<defs>
<marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="FilledArrow_Marker" stroke-linejoin="miter" stroke-miterlimit="10" viewBox="-1 -4 10 8" markerWidth="10" markerHeight="8" color="black">
<g>
<path d="M 8 0 L 0 -3 L 0 3 Z" fill="currentColor" stroke="currentColor" stroke-width="1"/>
</g>
</marker>
<marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="FilledArrow_Marker_2" stroke-linejoin="miter" stroke-miterlimit="10" viewBox="-1 -4 10 8" markerWidth="10" markerHeight="8" color="#7f8080">
<g>
<path d="M 8 0 L 0 -3 L 0 3 Z" fill="currentColor" stroke="currentColor" stroke-width="1"/>
</g>
</marker>
<marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="FilledArrow_Marker_3" stroke-linejoin="miter" stroke-miterlimit="10" viewBox="-1 -3 6 6" markerWidth="6" markerHeight="6" color="#7f8080">
<g>
<path d="M 3.7333333 0 L 0 -1.4 L 0 1.4 Z" fill="currentColor" stroke="currentColor" stroke-width="1"/>
</g>
</marker>
</defs>
<g id="00-lsm-read-flow" fill-opacity="1" stroke="none" stroke-dasharray="none" fill="none" stroke-opacity="1">
<title>00-lsm-read-flow</title>
<rect fill="white" x="6" y="194" width="693" height="361"/>
<g id="00-lsm-read-flow_Layer_1">
<title>Layer 1</title>
<g id="Graphic_30">
<rect x="81.5" y="329" width="386" height="188" fill="white"/>
<rect x="81.5" y="329" width="386" height="188" stroke="gray" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
</g>
<g id="Graphic_29">
<rect x="128.292" y="358.5" width="62.81032" height="41.5" fill="white"/>
<rect x="128.292" y="358.5" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(133.292 370.026)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="11.445159" y="15">SST</tspan>
</text>
</g>
<g id="Graphic_28">
<text transform="translate(90 370.026)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="6394885e-19" y="15">L0</tspan>
</text>
</g>
<g id="Graphic_27">
<rect x="128.5" y="410.026" width="62.81032" height="41.5" fill="white"/>
<rect x="128.5" y="410.026" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(133.5 421.552)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="11.445159" y="15">SST</tspan>
</text>
</g>
<g id="Graphic_26">
<rect x="201.09484" y="410.026" width="62.81032" height="41.5" fill="white"/>
<rect x="201.09484" y="410.026" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(206.09484 421.552)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="11.445159" y="15">SST</tspan>
</text>
</g>
<g id="Graphic_25">
<text transform="translate(90.208 421.552)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="6394885e-19" y="15">L1</tspan>
</text>
</g>
<g id="Graphic_24">
<rect x="200.88684" y="461.552" width="62.81032" height="41.5" fill="white"/>
<rect x="200.88684" y="461.552" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(205.88684 473.078)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="11.445159" y="15">SST</tspan>
</text>
</g>
<g id="Graphic_23">
<rect x="273.48168" y="461.552" width="62.81032" height="41.5" fill="white"/>
<rect x="273.48168" y="461.552" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(278.48168 473.078)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="11.445159" y="15">SST</tspan>
</text>
</g>
<g id="Graphic_22">
<text transform="translate(90 473.078)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="6394885e-19" y="15">Ln</tspan>
</text>
</g>
<g id="Graphic_21">
<rect x="128.292" y="461.552" width="62.81032" height="41.5" fill="white"/>
<rect x="128.292" y="461.552" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(133.292 473.078)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="11.445159" y="15">SST</tspan>
</text>
</g>
<g id="Graphic_20">
<rect x="346.07652" y="461.552" width="62.81032" height="41.5" fill="white"/>
<rect x="346.07652" y="461.552" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(351.07652 473.078)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="11.445159" y="15">SST</tspan>
</text>
</g>
<g id="Line_19">
<line x1="7" y1="319" x2="697.696" y2="319" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-dasharray="4.0,4.0" stroke-width="1"/>
</g>
<g id="Graphic_18">
<text transform="translate(12 325)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="0" y="15">On Disk</tspan>
</text>
</g>
<g id="Graphic_17">
<text transform="translate(12 296.552)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="0" y="15">In Memory</tspan>
</text>
</g>
<g id="Graphic_16">
<rect x="128.5" y="245.974" width="62.81032" height="41.5" fill="white"/>
<rect x="128.5" y="245.974" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(133.5 248.276)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="8.317159" y="15">Mem </tspan>
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="7.589159" y="33.448">Table</tspan>
</text>
</g>
<g id="Graphic_15">
<rect x="201.09484" y="245.974" width="62.81032" height="41.5" fill="white"/>
<rect x="201.09484" y="245.974" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(206.09484 248.276)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="8.317159" y="15">Mem </tspan>
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="7.589159" y="33.448">Table</tspan>
</text>
</g>
<g id="Graphic_14">
<rect x="273.68968" y="245.974" width="62.81032" height="41.5" fill="white"/>
<rect x="273.68968" y="245.974" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(278.68968 248.276)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="8.317159" y="15">Mem </tspan>
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="7.589159" y="33.448">Table</tspan>
</text>
</g>
<g id="Graphic_13">
<rect x="530.95384" y="365.026" width="62.81032" height="41.5" fill="white"/>
<rect x="530.95384" y="365.026" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(535.95384 376.552)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="9.557159" y="15">WAL</tspan>
</text>
</g>
<g id="Graphic_12">
<rect x="500.31726" y="252.052" width="124.08348" height="28.447998" fill="white"/>
<rect x="500.31726" y="252.052" width="124.08348" height="28.447998" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(505.31726 257.052)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="16.52974" y="15">key + value</tspan>
</text>
</g>
<g id="Line_11">
<line x1="562.359" y1="280.5" x2="562.359" y2="355.126" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
</g>
<g id="Line_10">
<line x1="159.8668" y1="287.474" x2="159.75381" y2="348.6" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
</g>
<g id="Line_9">
<line x1="500.31726" y1="266.38404" x2="346.4" y2="266.65207" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
</g>
<g id="Line_5">
<path d="M 436.92997 383.00897 C 436.92997 383.00897 456.93 404.82856 456.93 434.00897 C 456.93 452.75967 448.67184 468.88394 442.76857 478.01486" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
</g>
<g id="Graphic_3">
<rect x="511.8359" y="475.5" width="101.04616" height="41.5" fill="white"/>
<rect x="511.8359" y="475.5" width="101.04616" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(516.8359 487.026)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="14.851079" y="15">Manifest</tspan>
</text>
</g>
<g id="Line_2">
<line x1="467.5" y1="472.1117" x2="502.2417" y2="480.95225" marker-end="url(#FilledArrow_Marker_2)" stroke="#7f8080" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
</g>
<g id="Line_31">
<line x1="110.41992" y1="227.974" x2="338.68008" y2="227.974" marker-end="url(#FilledArrow_Marker_3)" stroke="#7f8080" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/>
</g>
<g id="Graphic_32">
<text transform="translate(45 199)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="216005e-17" y="15">1. Find the key-value pair in all memtables (new to old)</tspan>
</text>
</g>
<g id="Line_33">
<line x1="63" y1="364.5" x2="63" y2="501.1" marker-end="url(#FilledArrow_Marker_3)" stroke="#7f8080" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/>
</g>
<g id="Graphic_34">
<text transform="translate(36.340004 531)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="34674486e-19" y="15">2. Find the key-value pair in SSTs (top layer to bottom layer)</tspan>
</text>
</g>
</g>
</g>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 12 KiB

370
mini-lsm-book/src/figures/lsm-tutorial/00-lsm-tutorial-overview.svg Normal file
View File

@@ -0,0 +1,370 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xl="http://www.w3.org/1999/xlink" xmlns:dc="http://purl.org/dc/elements/1.1/" version="1.1" viewBox="-35 181 814 588" width="814" height="588">
<defs>
<marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="FilledArrow_Marker" stroke-linejoin="miter" stroke-miterlimit="10" viewBox="-1 -4 10 8" markerWidth="10" markerHeight="8" color="black">
<g>
<path d="M 8 0 L 0 -3 L 0 3 Z" fill="currentColor" stroke="currentColor" stroke-width="1"/>
</g>
</marker>
<marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="FilledArrow_Marker_2" stroke-linejoin="miter" stroke-miterlimit="10" viewBox="-1 -4 10 8" markerWidth="10" markerHeight="8" color="#7f8080">
<g>
<path d="M 8 0 L 0 -3 L 0 3 Z" fill="currentColor" stroke="currentColor" stroke-width="1"/>
</g>
</marker>
</defs>
<g id="00-lsm-tutorial-overview" fill-opacity="1" stroke="none" stroke-dasharray="none" fill="none" stroke-opacity="1">
<title>00-lsm-tutorial-overview</title>
<rect fill="white" x="-35" y="181" width="814" height="588"/>
<g id="00-lsm-tutorial-overview_Layer_1">
<title>Layer 1</title>
<g id="Graphic_57">
<rect x="467.5" y="182" width="208.47" height="328.7561" fill="white"/>
<path d="M 467.5 182 L 675.97 182 L 675.97 510.7561 L 467.5 510.7561 Z" stroke="gray" stroke-linecap="round" stroke-linejoin="round" stroke-dasharray="4.0,4.0" stroke-width="1"/>
</g>
<g id="Graphic_55">
<rect x="-34.5" y="182" width="494.55256" height="328.7561" fill="white"/>
<path d="M -34.5 182 L 460.05256 182 L 460.05256 510.7561 L -34.5 510.7561 Z" stroke="gray" stroke-linecap="round" stroke-linejoin="round" stroke-dasharray="4.0,4.0" stroke-width="1"/>
</g>
<g id="Graphic_54">
<rect x="-25" y="231.802" width="223.20984" height="174.724" fill="white"/>
<path d="M -25 231.802 L 198.20984 231.802 L 198.20984 406.526 L -25 406.526 Z" stroke="gray" stroke-linecap="round" stroke-linejoin="round" stroke-dasharray="4.0,4.0" stroke-width="1"/>
</g>
<g id="Graphic_52">
<rect x="61.5" y="558.03406" width="475.82584" height="92.448" fill="white"/>
<path d="M 61.5 558.03406 L 537.32584 558.03406 L 537.32584 650.48206 L 61.5 650.48206 Z" stroke="gray" stroke-linecap="round" stroke-linejoin="round" stroke-dasharray="4.0,4.0" stroke-width="1"/>
</g>
<g id="Graphic_51">
<text transform="translate(373.76584 626.01794)" fill="black">
<tspan font-family="Helvetica Neue" font-weight="bold" font-size="16" fill="black" x="0" y="16">Day 2: SST Encoding</tspan>
</text>
</g>
<g id="Graphic_48">
<rect x="-3.6741593" y="675.5681" width="541" height="92.448" fill="white"/>
<path d="M -3.6741593 675.5681 L 537.32584 675.5681 L 537.32584 768.0161 L -3.6741593 768.0161 Z" stroke="gray" stroke-linecap="round" stroke-linejoin="round" stroke-dasharray="4.0,4.0" stroke-width="1"/>
</g>
<g id="Graphic_47">
<rect x="128.292" y="358.5" width="62.81032" height="41.5" fill="white"/>
<rect x="128.292" y="358.5" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(133.292 370.026)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="11.445159" y="15">SST</tspan>
</text>
</g>
<g id="Graphic_46">
<text transform="translate(90 370.026)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="6394885e-19" y="15">L0</tspan>
</text>
</g>
<g id="Graphic_45">
<rect x="128.5" y="410.026" width="62.81032" height="41.5" fill="white"/>
<rect x="128.5" y="410.026" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(133.5 421.552)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="11.445159" y="15">SST</tspan>
</text>
</g>
<g id="Graphic_44">
<rect x="201.09484" y="410.026" width="62.81032" height="41.5" fill="white"/>
<rect x="201.09484" y="410.026" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(206.09484 421.552)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="11.445159" y="15">SST</tspan>
</text>
</g>
<g id="Graphic_43">
<text transform="translate(90.208 421.552)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="6394885e-19" y="15">L1</tspan>
</text>
</g>
<g id="Graphic_42">
<rect x="200.88684" y="461.552" width="62.81032" height="41.5" fill="white"/>
<rect x="200.88684" y="461.552" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(205.88684 473.078)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="11.445159" y="15">SST</tspan>
</text>
</g>
<g id="Graphic_41">
<rect x="273.48168" y="461.552" width="62.81032" height="41.5" fill="white"/>
<rect x="273.48168" y="461.552" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(278.48168 473.078)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="11.445159" y="15">SST</tspan>
</text>
</g>
<g id="Graphic_40">
<text transform="translate(90 473.078)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="6394885e-19" y="15">Ln</tspan>
</text>
</g>
<g id="Graphic_39">
<rect x="128.292" y="461.552" width="62.81032" height="41.5" fill="white"/>
<rect x="128.292" y="461.552" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(133.292 473.078)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="11.445159" y="15">SST</tspan>
</text>
</g>
<g id="Graphic_38">
<rect x="346.07652" y="461.552" width="62.81032" height="41.5" fill="white"/>
<rect x="346.07652" y="461.552" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(351.07652 473.078)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="11.445159" y="15">SST</tspan>
</text>
</g>
<g id="Line_37">
<line x1="7" y1="319" x2="697.696" y2="319" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-dasharray="4.0,4.0" stroke-width="1"/>
</g>
<g id="Graphic_36">
<text transform="translate(12 325)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="0" y="15">On Disk</tspan>
</text>
</g>
<g id="Graphic_35">
<text transform="translate(12 296.552)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="0" y="15">In Memory</tspan>
</text>
</g>
<g id="Graphic_34">
<rect x="128.5" y="245.974" width="62.81032" height="41.5" fill="white"/>
<rect x="128.5" y="245.974" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(133.5 248.276)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="8.317159" y="15">Mem </tspan>
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="7.589159" y="33.448">Table</tspan>
</text>
</g>
<g id="Graphic_33">
<rect x="201.09484" y="245.974" width="62.81032" height="41.5" fill="white"/>
<rect x="201.09484" y="245.974" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(206.09484 248.276)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="8.317159" y="15">Mem </tspan>
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="7.589159" y="33.448">Table</tspan>
</text>
</g>
<g id="Graphic_32">
<rect x="273.68968" y="245.974" width="62.81032" height="41.5" fill="white"/>
<rect x="273.68968" y="245.974" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(278.68968 248.276)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="8.317159" y="15">Mem </tspan>
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="7.589159" y="33.448">Table</tspan>
</text>
</g>
<g id="Graphic_31">
<rect x="530.95384" y="365.026" width="62.81032" height="41.5" fill="white"/>
<rect x="530.95384" y="365.026" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(535.95384 376.552)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="9.557159" y="15">WAL</tspan>
</text>
</g>
<g id="Graphic_30">
<rect x="500.31726" y="252.052" width="124.08348" height="28.447998" fill="white"/>
<rect x="500.31726" y="252.052" width="124.08348" height="28.447998" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(505.31726 257.052)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="16.52974" y="15">key + value</tspan>
</text>
</g>
<g id="Line_29">
<line x1="562.359" y1="280.5" x2="562.359" y2="355.126" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
</g>
<g id="Line_28">
<line x1="159.8668" y1="287.474" x2="159.75381" y2="348.6" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
</g>
<g id="Line_27">
<line x1="500.31726" y1="266.38404" x2="346.4" y2="266.65207" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
</g>
<g id="Graphic_26">
<text transform="translate(568 289.75)" fill="black">
<tspan font-family="Helvetica Neue" font-size="14" fill="black" x="0" y="15">1. </tspan>
<tspan font-family="PingFang SC" font-size="14" fill="black" y="15">Write to WAL</tspan>
</text>
</g>
<g id="Graphic_25">
<text transform="translate(351.07652 272.88794)" fill="black">
<tspan font-family="Helvetica Neue" font-size="14" fill="black" x="0" y="15">2. </tspan>
<tspan font-family="PingFang SC" font-size="14" fill="black" y="15">Write to MemTable</tspan>
</text>
</g>
<g id="Graphic_24">
<text transform="translate(170.88384 329.25)" fill="black">
<tspan font-family="Helvetica Neue" font-size="14" fill="black" x="0" y="15">3. </tspan>
<tspan font-family="PingFang SC" font-size="14" fill="black" y="15">Flush to Disk (Background)</tspan>
</text>
</g>
<g id="Line_23">
<path d="M 436.92997 383.00897 C 436.92997 383.00897 456.93 404.82856 456.93 434.00897 C 456.93 452.75967 448.67184 468.88394 442.76857 478.01486" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
</g>
<g id="Graphic_22">
<text transform="translate(270.372 426.304)" fill="black">
<tspan font-family="Helvetica Neue" font-size="14" fill="black" x="0" y="13">4. Compaction (Background)</tspan>
</text>
</g>
<g id="Graphic_21">
<rect x="80.01552" y="571.13" width="62.81032" height="41.5" fill="white"/>
<rect x="80.01552" y="571.13" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(85.01552 573.432)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="9.661159" y="15">Data </tspan>
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="6.109159" y="33.448">Block</tspan>
</text>
</g>
<g id="Graphic_20">
<rect x="142.82584" y="571.13" width="62.81032" height="41.5" fill="white"/>
<rect x="142.82584" y="571.13" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(147.82584 573.432)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="9.661159" y="15">Data </tspan>
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="6.109159" y="33.448">Block</tspan>
</text>
</g>
<g id="Graphic_19">
<rect x="204.82584" y="571.13" width="62.81032" height="41.5" fill="white"/>
<rect x="204.82584" y="571.13" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(209.82584 573.432)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="9.661159" y="15">Data </tspan>
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="6.109159" y="33.448">Block</tspan>
</text>
</g>
<g id="Graphic_18">
<rect x="266.82584" y="571.13" width="62.81032" height="41.5" fill="white"/>
<rect x="266.82584" y="571.13" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(271.82584 573.432)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="9.661159" y="15">Data </tspan>
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="6.109159" y="33.448">Block</tspan>
</text>
</g>
<g id="Graphic_17">
<rect x="328.82584" y="571.13" width="62.81032" height="41.5" fill="white"/>
<rect x="328.82584" y="571.13" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(333.82584 573.432)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="9.661159" y="15">Data </tspan>
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="6.109159" y="33.448">Block</tspan>
</text>
</g>
<g id="Graphic_16">
<rect x="390.82584" y="571.13" width="62.81032" height="41.5" fill="white"/>
<rect x="390.82584" y="571.13" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(395.82584 582.656)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="6.701159" y="15">Index</tspan>
</text>
</g>
<g id="Line_15">
<line x1="128.292" y1="503.052" x2="80.01552" y2="571.13" stroke="#7f8080" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
</g>
<g id="Line_14">
<line x1="191.10232" y1="503.052" x2="453.63616" y2="571.13" stroke="#7f8080" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
</g>
<g id="Graphic_13">
<rect x="7" y="697.76" width="44.48448" height="26.391998" fill="white"/>
<rect x="7" y="697.76" width="44.48448" height="26.391998" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(12 701.732)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="3.610239" y="15">Key</tspan>
</text>
</g>
<g id="Graphic_12">
<rect x="51.48448" y="697.76" width="62.81032" height="26.391998" fill="white"/>
<rect x="51.48448" y="697.76" width="62.81032" height="26.391998" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(56.48448 701.732)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="7.141159" y="15">Value</tspan>
</text>
</g>
<g id="Graphic_11">
<rect x="109.56808" y="697.76" width="44.48448" height="26.391998" fill="white"/>
<rect x="109.56808" y="697.76" width="44.48448" height="26.391998" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(114.56808 701.732)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="3.610239" y="15">Key</tspan>
</text>
</g>
<g id="Graphic_10">
<rect x="154.05256" y="697.76" width="62.81032" height="26.391998" fill="white"/>
<rect x="154.05256" y="697.76" width="62.81032" height="26.391998" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(159.05256 701.732)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="7.141159" y="15">Value</tspan>
</text>
</g>
<g id="Graphic_9">
<rect x="211.56808" y="697.76" width="44.48448" height="26.391998" fill="white"/>
<rect x="211.56808" y="697.76" width="44.48448" height="26.391998" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(216.56808 701.732)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="3.610239" y="15">Key</tspan>
</text>
</g>
<g id="Graphic_8">
<rect x="256.05256" y="697.76" width="62.81032" height="26.391998" fill="white"/>
<rect x="256.05256" y="697.76" width="62.81032" height="26.391998" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(261.05256 701.732)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="7.141159" y="15">Value</tspan>
</text>
</g>
<g id="Graphic_7">
<rect x="313.56808" y="697.76" width="44.48448" height="26.391998" fill="white"/>
<rect x="313.56808" y="697.76" width="44.48448" height="26.391998" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(318.56808 701.732)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="3.610239" y="15">Key</tspan>
</text>
</g>
<g id="Graphic_6">
<rect x="358.05256" y="697.76" width="62.81032" height="26.391998" fill="white"/>
<rect x="358.05256" y="697.76" width="62.81032" height="26.391998" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(363.05256 701.732)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="7.141159" y="15">Value</tspan>
</text>
</g>
<g id="Graphic_5">
<rect x="415.5681" y="697.76" width="44.48448" height="26.391998" fill="white"/>
<rect x="415.5681" y="697.76" width="44.48448" height="26.391998" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(420.5681 701.732)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="3.610239" y="15">Key</tspan>
</text>
</g>
<g id="Graphic_4">
<rect x="460.05256" y="697.76" width="62.81032" height="26.391998" fill="white"/>
<rect x="460.05256" y="697.76" width="62.81032" height="26.391998" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(465.05256 701.732)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="7.141159" y="15">Value</tspan>
</text>
</g>
<g id="Line_3">
<line x1="80.01552" y1="612.63" x2="7" y2="697.76" stroke="#7f8080" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
</g>
<g id="Line_2">
<line x1="142.82584" y1="612.63" x2="522.8629" y2="697.76" stroke="#7f8080" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
</g>
<g id="Graphic_49">
<text transform="translate(361.15398 743.552)" fill="black">
<tspan font-family="Helvetica Neue" font-weight="bold" font-size="16" fill="black" x="32969183e-19" y="16">Day 1: Block Encoding</tspan>
</text>
</g>
<g id="Graphic_53">
<text transform="translate(-19.052517 236.802)" fill="black">
<tspan font-family="Helvetica Neue" font-weight="bold" font-size="16" fill="black" x="0" y="16">Day 3 &amp; 4: Engine</tspan>
</text>
</g>
<g id="Graphic_56">
<text transform="translate(-29.499995 187)" fill="black">
<tspan font-family="Helvetica Neue" font-weight="bold" font-size="16" fill="black" x="0" y="16">Day 5: Compaction</tspan>
</text>
</g>
<g id="Graphic_58">
<text transform="translate(547.658 488.052)" fill="black">
<tspan font-family="Helvetica Neue" font-weight="bold" font-size="16" fill="black" x="0" y="16">Day 6: Recovery</tspan>
</text>
</g>
<g id="Graphic_59">
<rect x="511.8359" y="421.304" width="101.04616" height="41.5" fill="white"/>
<rect x="511.8359" y="421.304" width="101.04616" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(516.8359 432.83)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="14.851079" y="15">Manifest</tspan>
</text>
</g>
<g id="Graphic_60">
<text transform="translate(573.656 594.526)" fill="black">
<tspan font-family="Helvetica Neue" font-weight="bold" font-size="16" fill="black" x="0" y="16">Day 7.1: Bloom Filter</tspan>
</text>
</g>
<g id="Line_61">
<line x1="568.656" y1="604.25806" x2="547.22584" y2="604.25806" marker-end="url(#FilledArrow_Marker_2)" stroke="#7f8080" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
</g>
<g id="Graphic_62">
<text transform="translate(573.656 712.0601)" fill="black">
<tspan font-family="Helvetica Neue" font-weight="bold" font-size="16" fill="black" x="3240075e-18" y="16">Day 7.2: Key Compression</tspan>
</text>
</g>
<g id="Line_63">
<line x1="568.656" y1="721.7921" x2="547.22584" y2="721.7921" marker-end="url(#FilledArrow_Marker_2)" stroke="#7f8080" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
</g>
</g>
</g>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 23 KiB

187
mini-lsm-book/src/figures/lsm-tutorial/00-lsm-write-flow.svg Normal file
View File

@@ -0,0 +1,187 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xl="http://www.w3.org/1999/xlink" xmlns:dc="http://purl.org/dc/elements/1.1/" version="1.1" viewBox="6 245 693 273" width="693" height="273">
<defs>
<marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="FilledArrow_Marker" stroke-linejoin="miter" stroke-miterlimit="10" viewBox="-1 -4 10 8" markerWidth="10" markerHeight="8" color="black">
<g>
<path d="M 8 0 L 0 -3 L 0 3 Z" fill="currentColor" stroke="currentColor" stroke-width="1"/>
</g>
</marker>
<marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="FilledArrow_Marker_2" stroke-linejoin="miter" stroke-miterlimit="10" viewBox="-1 -4 10 8" markerWidth="10" markerHeight="8" color="#7f8080">
<g>
<path d="M 8 0 L 0 -3 L 0 3 Z" fill="currentColor" stroke="currentColor" stroke-width="1"/>
</g>
</marker>
</defs>
<g id="00-lsm-write-flow" fill-opacity="1" stroke="none" stroke-dasharray="none" fill="none" stroke-opacity="1">
<title>00-lsm-write-flow</title>
<rect fill="white" x="6" y="245" width="693" height="273"/>
<g id="00-lsm-write-flow_Layer_1">
<title>Layer 1</title>
<g id="Graphic_112">
<rect x="81.5" y="329" width="386" height="188" fill="white"/>
<rect x="81.5" y="329" width="386" height="188" stroke="gray" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
</g>
<g id="Graphic_105">
<rect x="128.292" y="358.5" width="62.81032" height="41.5" fill="white"/>
<rect x="128.292" y="358.5" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(133.292 370.026)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="11.445159" y="15">SST</tspan>
</text>
</g>
<g id="Graphic_104">
<text transform="translate(90 370.026)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="6394885e-19" y="15">L0</tspan>
</text>
</g>
<g id="Graphic_103">
<rect x="128.5" y="410.026" width="62.81032" height="41.5" fill="white"/>
<rect x="128.5" y="410.026" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(133.5 421.552)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="11.445159" y="15">SST</tspan>
</text>
</g>
<g id="Graphic_102">
<rect x="201.09484" y="410.026" width="62.81032" height="41.5" fill="white"/>
<rect x="201.09484" y="410.026" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(206.09484 421.552)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="11.445159" y="15">SST</tspan>
</text>
</g>
<g id="Graphic_101">
<text transform="translate(90.208 421.552)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="6394885e-19" y="15">L1</tspan>
</text>
</g>
<g id="Graphic_100">
<rect x="200.88684" y="461.552" width="62.81032" height="41.5" fill="white"/>
<rect x="200.88684" y="461.552" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(205.88684 473.078)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="11.445159" y="15">SST</tspan>
</text>
</g>
<g id="Graphic_99">
<rect x="273.48168" y="461.552" width="62.81032" height="41.5" fill="white"/>
<rect x="273.48168" y="461.552" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(278.48168 473.078)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="11.445159" y="15">SST</tspan>
</text>
</g>
<g id="Graphic_98">
<text transform="translate(90 473.078)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="6394885e-19" y="15">Ln</tspan>
</text>
</g>
<g id="Graphic_97">
<rect x="128.292" y="461.552" width="62.81032" height="41.5" fill="white"/>
<rect x="128.292" y="461.552" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(133.292 473.078)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="11.445159" y="15">SST</tspan>
</text>
</g>
<g id="Graphic_96">
<rect x="346.07652" y="461.552" width="62.81032" height="41.5" fill="white"/>
<rect x="346.07652" y="461.552" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(351.07652 473.078)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="11.445159" y="15">SST</tspan>
</text>
</g>
<g id="Line_95">
<line x1="7" y1="319" x2="697.696" y2="319" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-dasharray="4.0,4.0" stroke-width="1"/>
</g>
<g id="Graphic_94">
<text transform="translate(12 325)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="0" y="15">On Disk</tspan>
</text>
</g>
<g id="Graphic_93">
<text transform="translate(12 296.552)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="0" y="15">In Memory</tspan>
</text>
</g>
<g id="Graphic_92">
<rect x="128.5" y="245.974" width="62.81032" height="41.5" fill="white"/>
<rect x="128.5" y="245.974" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(133.5 248.276)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="8.317159" y="15">Mem </tspan>
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="7.589159" y="33.448">Table</tspan>
</text>
</g>
<g id="Graphic_91">
<rect x="201.09484" y="245.974" width="62.81032" height="41.5" fill="white"/>
<rect x="201.09484" y="245.974" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(206.09484 248.276)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="8.317159" y="15">Mem </tspan>
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="7.589159" y="33.448">Table</tspan>
</text>
</g>
<g id="Graphic_90">
<rect x="273.68968" y="245.974" width="62.81032" height="41.5" fill="white"/>
<rect x="273.68968" y="245.974" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(278.68968 248.276)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="8.317159" y="15">Mem </tspan>
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="7.589159" y="33.448">Table</tspan>
</text>
</g>
<g id="Graphic_89">
<rect x="530.95384" y="365.026" width="62.81032" height="41.5" fill="white"/>
<rect x="530.95384" y="365.026" width="62.81032" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(535.95384 376.552)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="9.557159" y="15">WAL</tspan>
</text>
</g>
<g id="Graphic_88">
<rect x="500.31726" y="252.052" width="124.08348" height="28.447998" fill="white"/>
<rect x="500.31726" y="252.052" width="124.08348" height="28.447998" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(505.31726 257.052)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="16.52974" y="15">key + value</tspan>
</text>
</g>
<g id="Line_87">
<line x1="562.359" y1="280.5" x2="562.359" y2="355.126" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
</g>
<g id="Line_86">
<line x1="159.8668" y1="287.474" x2="159.75381" y2="348.6" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
</g>
<g id="Line_85">
<line x1="500.31726" y1="266.38404" x2="346.4" y2="266.65207" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
</g>
<g id="Graphic_84">
<text transform="translate(568 289.75)" fill="black">
<tspan font-family="Helvetica Neue" font-size="14" fill="black" x="0" y="15">1. </tspan>
<tspan font-family="PingFang SC" font-size="14" fill="black" y="15">Write to WAL</tspan>
</text>
</g>
<g id="Graphic_83">
<text transform="translate(351.07652 272.88794)" fill="black">
<tspan font-family="Helvetica Neue" font-size="14" fill="black" x="0" y="15">2. </tspan>
<tspan font-family="PingFang SC" font-size="14" fill="black" y="15">Write to MemTable</tspan>
</text>
</g>
<g id="Graphic_82">
<text transform="translate(170.88384 329.25)" fill="black">
<tspan font-family="Helvetica Neue" font-size="14" fill="black" x="0" y="15">3. </tspan>
<tspan font-family="PingFang SC" font-size="14" fill="black" y="15">Flush to Disk (Background)</tspan>
</text>
</g>
<g id="Line_81">
<path d="M 436.92997 383.00897 C 436.92997 383.00897 456.93 404.82856 456.93 434.00897 C 456.93 452.75967 448.67184 468.88394 442.76857 478.01486" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
</g>
<g id="Graphic_80">
<text transform="translate(270.372 426.304)" fill="black">
<tspan font-family="Helvetica Neue" font-size="14" fill="black" x="0" y="13">4. Compaction (Background)</tspan>
</text>
</g>
<g id="Graphic_55">
<rect x="511.8359" y="475.5" width="101.04616" height="41.5" fill="white"/>
<rect x="511.8359" y="475.5" width="101.04616" height="41.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(516.8359 487.026)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="14.851079" y="15">Manifest</tspan>
</text>
</g>
<g id="Line_113">
<line x1="467.5" y1="472.1117" x2="502.2417" y2="480.95225" marker-end="url(#FilledArrow_Marker_2)" stroke="#7f8080" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
</g>
</g>
</g>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB

12
mini-lsm-starter/Cargo.toml Normal file
View File

@@ -0,0 +1,12 @@
[package]
name = "mini-lsm-starter"
version = { workspace = true }
edition = { workspace = true }
homepage = { workspace = true }
keywords = { workspace = true }
license = { workspace = true }
repository = { workspace = true }
publish = false
[dependencies]
bytes = "1"

26
mini-lsm-starter/src/block.rs Normal file
View File

@@ -0,0 +1,26 @@
#![allow(unused_variables)] // TODO(you): remove this lint after implementing this mod
#![allow(dead_code)] // TODO(you): remove this lint after implementing this mod
mod builder;
mod iterator;
use bytes::Bytes;
pub use builder::BlockBuilder;
pub use iterator::BlockIterator;
/// A block is the smallest unit of read and caching in LSM tree. It is a collection of sorted key-value pairs.
pub struct Block {}
impl Block {
pub fn encode(&self) -> Bytes {
unimplemented!()
}
pub fn decode(data: &[u8]) -> Self {
unimplemented!()
}
}
#[cfg(test)]
mod tests;

29
mini-lsm-starter/src/block/builder.rs Normal file
View File

@@ -0,0 +1,29 @@
#![allow(unused_variables)] // TODO(you): remove this lint after implementing this mod
#![allow(dead_code)] // TODO(you): remove this lint after implementing this mod
use super::Block;
/// Builds a block
pub struct BlockBuilder {}
impl BlockBuilder {
/// Creates a new block builder
pub fn new(target_size: usize) -> Self {
unimplemented!()
}
/// Adds a key-value pair to the block
#[must_use]
pub fn add(&mut self, key: &[u8], value: &[u8]) -> bool {
unimplemented!()
}
pub fn is_empty(&self) -> bool {
unimplemented!()
}
/// Builds a block
pub fn build(self) -> Block {
unimplemented!()
}
}

62
mini-lsm-starter/src/block/iterator.rs Normal file
View File

@@ -0,0 +1,62 @@
#![allow(unused_variables)] // TODO(you): remove this lint after implementing this mod
#![allow(dead_code)] // TODO(you): remove this lint after implementing this mod
use std::sync::Arc;
use super::Block;
pub struct BlockIterator {}
impl BlockIterator {
fn new(block: Arc<Block>) -> Self {
unimplemented!()
}
pub fn create_and_seek_to_first(block: Arc<Block>) -> Self {
unimplemented!()
}
pub fn create_and_seek_to_key(block: Arc<Block>, key: &[u8]) -> Self {
unimplemented!()
}
pub fn key(&self) -> &[u8] {
unimplemented!()
}
pub fn value(&self) -> &[u8] {
unimplemented!()
}
pub fn is_valid(&self) -> bool {
unimplemented!()
}
pub fn seek_to_first(&mut self) {
unimplemented!()
}
pub fn seek_to_last(&mut self) {
unimplemented!()
}
pub fn len(&self) -> usize {
unimplemented!()
}
pub fn is_empty(&self) -> bool {
unimplemented!()
}
pub fn seek_to(&mut self, idx: usize) {
unimplemented!()
}
pub fn next(&mut self) {
unimplemented!()
}
pub fn seek_to_key(&mut self, key: &[u8]) {
unimplemented!()
}
}

1
mini-lsm-starter/src/block/tests.rs Normal file
View File

@@ -0,0 +1 @@
//! Please copy `mini-lsm/src/block/tests.rs` here so that you can run tests.

1
mini-lsm-starter/src/lib.rs Normal file
View File

@@ -0,0 +1 @@
pub mod block;

92
mini-lsm/src/block.rs
View File

@@ -39,94 +39,4 @@ impl Block {
} }
#[cfg(test)] #[cfg(test)]
mod tests { mod tests;
use std::sync::Arc;
use super::{builder::BlockBuilder, iterator::BlockIterator, *};
#[test]
fn test_block_build_single_key() {
let mut builder = BlockBuilder::new(16);
assert!(builder.add(b"233", b"233333"));
builder.build();
}
#[test]
fn test_block_build_full() {
let mut builder = BlockBuilder::new(16);
assert!(builder.add(b"11", b"11"));
assert!(!builder.add(b"22", b"22"));
builder.build();
}
fn key_of(idx: usize) -> Vec<u8> {
format!("key_{:03}", idx).into_bytes()
}
fn value_of(idx: usize) -> Vec<u8> {
format!("value_{:010}", idx).into_bytes()
}
fn num_of_keys() -> usize {
100
}
fn generate_block() -> Block {
let mut builder = BlockBuilder::new(10000);
for idx in 0..num_of_keys() {
let key = key_of(idx);
let value = value_of(idx);
assert!(builder.add(&key[..], &value[..]));
}
builder.build()
}
#[test]
fn test_block_build_all() {
generate_block();
}
#[test]
fn test_block_encode() {
let block = generate_block();
block.encode();
}
#[test]
fn test_block_decode() {
let block = generate_block();
let encoded = block.encode();
let decoded_block = Block::decode(&encoded);
assert_eq!(block.offsets, decoded_block.offsets);
assert_eq!(block.data, decoded_block.data);
}
fn as_bytes(x: &[u8]) -> Bytes {
Bytes::copy_from_slice(x)
}
#[test]
fn test_block_iterator() {
let block = Arc::new(generate_block());
let mut iter = BlockIterator::create_and_seek_to_first(block);
for i in 0..num_of_keys() {
let key = iter.key();
let value = iter.value();
assert_eq!(
key,
key_of(i),
"expected key: {:?}, actual key: {:?}",
as_bytes(&key_of(i)),
as_bytes(key)
);
assert_eq!(
value,
value_of(i),
"expected value: {:?}, actual value: {:?}",
as_bytes(&value_of(i)),
as_bytes(value)
);
iter.next();
}
}
}

89
mini-lsm/src/block/tests.rs Normal file
View File

@@ -0,0 +1,89 @@
use std::sync::Arc;
use super::{builder::BlockBuilder, iterator::BlockIterator, *};
#[test]
fn test_block_build_single_key() {
let mut builder = BlockBuilder::new(16);
assert!(builder.add(b"233", b"233333"));
builder.build();
}
#[test]
fn test_block_build_full() {
let mut builder = BlockBuilder::new(16);
assert!(builder.add(b"11", b"11"));
assert!(!builder.add(b"22", b"22"));
builder.build();
}
fn key_of(idx: usize) -> Vec<u8> {
format!("key_{:03}", idx).into_bytes()
}
fn value_of(idx: usize) -> Vec<u8> {
format!("value_{:010}", idx).into_bytes()
}
fn num_of_keys() -> usize {
100
}
fn generate_block() -> Block {
let mut builder = BlockBuilder::new(10000);
for idx in 0..num_of_keys() {
let key = key_of(idx);
let value = value_of(idx);
assert!(builder.add(&key[..], &value[..]));
}
builder.build()
}
#[test]
fn test_block_build_all() {
generate_block();
}
#[test]
fn test_block_encode() {
let block = generate_block();
block.encode();
}
#[test]
fn test_block_decode() {
let block = generate_block();
let encoded = block.encode();
let decoded_block = Block::decode(&encoded);
assert_eq!(block.offsets, decoded_block.offsets);
assert_eq!(block.data, decoded_block.data);
}
fn as_bytes(x: &[u8]) -> Bytes {
Bytes::copy_from_slice(x)
}
#[test]
fn test_block_iterator() {
let block = Arc::new(generate_block());
let mut iter = BlockIterator::create_and_seek_to_first(block);
for i in 0..num_of_keys() {
let key = iter.key();
let value = iter.value();
assert_eq!(
key,
key_of(i),
"expected key: {:?}, actual key: {:?}",
as_bytes(&key_of(i)),
as_bytes(key)
);
assert_eq!(
value,
value_of(i),
"expected value: {:?}, actual value: {:?}",
as_bytes(&value_of(i)),
as_bytes(value)
);
iter.next();
}
}

3
xtask/Cargo.toml
View File

@@ -6,8 +6,7 @@ homepage = { workspace = true }
keywords = { workspace = true } keywords = { workspace = true }
license = { workspace = true } license = { workspace = true }
repository = { workspace = true } repository = { workspace = true }
publish = false
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies] [dependencies]
anyhow = "1" anyhow = "1"