Commit Graph

2 Commits

Author SHA1 Message Date
Joe Thornber
7a85e47a20 [rust] squash clippy (linter) warnings 2020-06-10 16:46:38 +01:00
Joe Thornber
61de3f9287 [thin_metadata_pack/unpack] Replace C++ implementation with a Rust one.
The Rust implementation is multithreaded, performs better in general and
does custom compression of btree nodes to achieve much better compression
ratios.  unpack also checksums expanded metadata to validate it.

Format version has jumped to 3, no backwards compatibility, but I think
that's ok since we never made a release that contained the C++ version
of these tools.

Benchmarks
==========

On an 8 core, 16 hyperthread machine.

metadata 1G, full:

      Pack size    pack time     unpack time
------------------------------------------------------
C++      193M        50.3s          6.9s (no verify)
Rust      70M         1.4s          1.8s (verify)

metadata 16G, sparse:

       Pack size    pack time     unpack time
------------------------------------------------------
C++      21M          68s           1s   (no verify)
Rust      4M           8.6s         0.5s (verify)
2020-06-09 09:15:00 +01:00