[thin_metadata_pack/unpack] Replace C++ implementation with a Rust one.

The Rust implementation is multithreaded, performs better in general and
does custom compression of btree nodes to achieve much better compression
ratios.  unpack also checksums expanded metadata to validate it.

Format version has jumped to 3, no backwards compatibility, but I think
that's ok since we never made a release that contained the C++ version
of these tools.

Benchmarks
==========

On an 8 core, 16 hyperthread machine.

metadata 1G, full:

      Pack size    pack time     unpack time
------------------------------------------------------
C++      193M        50.3s          6.9s (no verify)
Rust      70M         1.4s          1.8s (verify)

metadata 16G, sparse:

       Pack size    pack time     unpack time
------------------------------------------------------
C++      21M          68s           1s   (no verify)
Rust      4M           8.6s         0.5s (verify)
This commit is contained in:
Joe Thornber
2020-06-09 09:15:00 +01:00
parent c48851e747
commit 61de3f9287
21 changed files with 1818 additions and 376 deletions

View File

@@ -13,6 +13,12 @@ The [expat](http://expat.sourceforge.net/) xml parser library (version 1).
The libaio library (note this is not the same as the aio library that you get by linking -lrt)
make, autoconf etc.
A couple of non-essential tools are written in rust, and will
require cargo and rustcc to be installed:
thin_metadata_pack
thin_metadata_unpack
There are more requirements for testing, detailed below.
Building
@@ -23,6 +29,11 @@ Building
make
sudo make install
Building Rust tools
===================
sudo make install-rust-tools
Quick examples
==============