[thin_metadata_pack/unpack] Replace C++ implementation with a Rust one.
The Rust implementation is multithreaded, performs better in general and does custom compression of btree nodes to achieve much better compression ratios. unpack also checksums expanded metadata to validate it. Format version has jumped to 3, no backwards compatibility, but I think that's ok since we never made a release that contained the C++ version of these tools. Benchmarks ========== On an 8 core, 16 hyperthread machine. metadata 1G, full: Pack size pack time unpack time ------------------------------------------------------ C++ 193M 50.3s 6.9s (no verify) Rust 70M 1.4s 1.8s (verify) metadata 16G, sparse: Pack size pack time unpack time ------------------------------------------------------ C++ 21M 68s 1s (no verify) Rust 4M 8.6s 0.5s (verify)
This commit is contained in:
11
README.md
11
README.md
@@ -13,6 +13,12 @@ The [expat](http://expat.sourceforge.net/) xml parser library (version 1).
|
||||
The libaio library (note this is not the same as the aio library that you get by linking -lrt)
|
||||
make, autoconf etc.
|
||||
|
||||
A couple of non-essential tools are written in rust, and will
|
||||
require cargo and rustcc to be installed:
|
||||
|
||||
thin_metadata_pack
|
||||
thin_metadata_unpack
|
||||
|
||||
There are more requirements for testing, detailed below.
|
||||
|
||||
Building
|
||||
@@ -23,6 +29,11 @@ Building
|
||||
make
|
||||
sudo make install
|
||||
|
||||
Building Rust tools
|
||||
===================
|
||||
|
||||
sudo make install-rust-tools
|
||||
|
||||
Quick examples
|
||||
==============
|
||||
|
||||
|
Reference in New Issue
Block a user