update copy test command

Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
This commit is contained in:
Alex Chi Z
2024-01-21 14:47:20 +08:00
parent b67c22611d
commit 99da8855b8
13 changed files with 99 additions and 59 deletions

View File

@@ -28,7 +28,7 @@ cargo x install-tools
## Run tests
```
cargo x copy-test day1
cargo x copy-test --week 1 --day 1
cargo x scheck
```

View File

@@ -8,6 +8,13 @@ In this chapter, you will:
* Implement freezing memtable logic.
* Implement LSM read path `get` for memtables.
To copy the test cases into the starter code and run them,
```
cargo x copy-test --week 1 --day 1
cargo x scheck
```
## Task 1: SkipList Memtable
In this task, you will need to modify:

View File

@@ -8,6 +8,14 @@ In this chapter, you will:
* Implement merge iterator.
* Implement LSM read path `scan` for memtables.
To copy the test cases into the starter code and run them,
```
cargo x copy-test --week 1 --day 2
cargo x scheck
```
## Task 1: Memtable Iterator
In this chapter, we will implement the LSM `scan` interface. `scan` returns a range of key-value pairs in order using an iterator API. In the previous chapter, you have implemented the `get` API and the logic to create immutable memtables, and your LSM state should now have multiple memtables. You will need to first create iterators on a single memtable, then create a merge iterator on all memtables, and finally implement the range limit for the iterators.

View File

@@ -7,6 +7,14 @@ In this chapter, you will:
* Implement SST block encoding.
* Implement SST block decoding and block iterator.
To copy the test cases into the starter code and run them,
```
cargo x copy-test --week 1 --day 3
cargo x scheck
```
## Task 1: Block Builder
You have already implemented all in-memory structures for an LSM storage engine in the previous two chapters. Now it's time to build the on-disk structures. The basic unit of the on-disk structure is blocks. Blocks are usually of 4-KB size (the size may vary depending on the storage medium), which is equivalent to the page size in the operating system and the page size on an SSD. A block stores ordered key-value pairs. An SST is composed of multiple blocks. When the number of memtables exceed the system limit, it will flush the memtable as an SST. In this chapter, you will implement the encoding and decoding of a block.

View File

@@ -7,6 +7,14 @@ In this chapter, you will:
* Implement SST encoding and metadata encoding.
* Implement SST decoding and iterator.
To copy the test cases into the starter code and run them,
```
cargo x copy-test --week 1 --day 4
cargo x scheck
```
## Task 1: SST Builder
In this task, you will need to modify:

View File

@@ -8,6 +8,14 @@ In this chapter, you will:
* Implement LSM read path `get` with SSTs.
* Implement LSM read path `scan` with SSTs.
To copy the test cases into the starter code and run them,
```
cargo x copy-test --week 1 --day 5
cargo x scheck
```
## Task 1: Two Merge Iterator
## Task 2: Read Path - Get

View File

@@ -7,6 +7,14 @@ In this chapter, you will:
* Implement the LSM write path with L0 flush.
* Implement the logic to correctly update the LSM state.
To copy the test cases into the starter code and run them,
```
cargo x copy-test --week 1 --day 6
cargo x scheck
```
## Task 1: Flush Memtable to SST
## Task 2: Update the LSM State

View File

@@ -9,6 +9,14 @@ In this chapter, you will:
* Implement bloom filter on SSTs and integrate into the LSM read path `get`.
* Implement key compression in SST block format.
To copy the test cases into the starter code and run them,
```
cargo x copy-test --week 1 --day 7
cargo x scheck
```
## Task 1: Bloom Filters
## Task 2: Integrate Bloom Filter on the Read Path

View File

@@ -1,16 +1,16 @@
use std::path::PathBuf;
use std::path::{Path, PathBuf};
use anyhow::{anyhow, Result};
use clap::Parser;
use console::style;
use duct::cmd;
#[derive(clap::Subcommand, Debug)]
enum CopyTestAction {
Day1,
Day2,
Day3,
Day4,
#[derive(clap::Parser, Debug)]
struct CopyTestAction {
#[arg(long)]
week: usize,
#[arg(long)]
day: usize,
}
#[derive(clap::Subcommand, Debug)]
@@ -30,7 +30,6 @@ enum Action {
/// Check starter code
Scheck,
/// Copy test cases
#[command(subcommand)]
CopyTest(CopyTestAction),
}
@@ -126,58 +125,44 @@ fn sync() -> Result<()> {
}
fn copy_test_case(test: CopyTestAction) -> Result<()> {
match test {
CopyTestAction::Day1 => {
cmd!(
"cp",
"mini-lsm/src/block/tests.rs",
"mini-lsm-starter/src/block/tests.rs"
)
.run()?;
}
CopyTestAction::Day2 => {
cmd!(
"cp",
"mini-lsm/src/table/tests.rs",
"mini-lsm-starter/src/table/tests.rs"
)
.run()?;
}
CopyTestAction::Day3 => {
cmd!(
"cp",
"mini-lsm/src/mem_table/tests.rs",
"mini-lsm-starter/src/mem_table/tests.rs"
)
.run()?;
cmd!(
"cp",
"mini-lsm/src/iterators/tests/merge_iterator_test.rs",
"mini-lsm-starter/src/iterators/tests/merge_iterator_test.rs"
)
.run()?;
cmd!(
"cp",
"mini-lsm/src/iterators/tests/two_merge_iterator_test.rs",
"mini-lsm-starter/src/iterators/tests/two_merge_iterator_test.rs"
)
.run()?;
cmd!(
"cp",
"mini-lsm/src/iterators/tests.rs",
"mini-lsm-starter/src/iterators/tests.rs"
)
.run()?;
}
CopyTestAction::Day4 => {
cmd!(
"cp",
"mini-lsm/src/tests/day4_tests.rs",
"mini-lsm-starter/src/tests/day4_tests.rs"
)
.run()?;
use std::fmt::Write;
let src_dir = "mini-lsm/src/tests";
let target_dir = "mini-lsm-starter/src/tests";
if !Path::new(target_dir).exists() {
std::fs::create_dir(target_dir)?;
}
let test_filename = format!("week{}_day{}.rs", test.week, test.day);
let src = format!("{}/{}", src_dir, test_filename);
let target = format!("{}/{}", target_dir, test_filename);
cmd!("cp", src, target).run()?;
let test_filename = "harness.rs";
let src = format!("{}/{}", src_dir, test_filename);
let target = format!("{}/{}", target_dir, test_filename);
cmd!("cp", src, target).run()?;
let mut test_file = Vec::new();
for file in Path::new(&target_dir).read_dir()? {
let file = file?;
let fname = file.file_name();
let fnamestr = fname
.as_os_str()
.to_str()
.ok_or_else(|| anyhow!("invalid filename?"))?;
if let Some((mod_name, _)) = fnamestr.split_once(".rs") {
test_file.push(mod_name.to_string());
}
}
let mut tests_mod = String::new();
writeln!(tests_mod, "//! DO NOT MODIFY -- Mini-LSM tests modules")?;
writeln!(
tests_mod,
"//! This file will be automatically rewritten by the copy-test command."
)?;
writeln!(tests_mod)?;
for tf in test_file {
writeln!(tests_mod, "mod {};", tf)?;
}
println!("{}", tests_mod);
std::fs::write("mini-lsm-starter/src/tests.rs", tests_mod)?;
Ok(())
}