Commit Graph

25 Commits

Author SHA1 Message Date
Joe Thornber
024554c987 Merge rust tools into a single pdata_tools exe 2021-10-11 12:07:26 +01:00
Ming-Hung Tsai
b7c3969747 [build] Update dependencies
- Update fixedbitset to 0.4
- Update indicatif to 0.16
- Update libc to 0.2.*
- Update nix to 0.21
- Update nom to 6.2.*
- Update dependencies with cargo-update
2021-06-26 00:59:37 +08:00
Kay Lin
c18cd42d35 [build] Update dependencies
- Update crc32c to 0.6, which allows it to be built on AArch64.

  - Update base64 to 0.13
  - Update byteorder to 0.14
  - Update io-uring to 0.4
  - Update libc to 0.2.83
  - Update nix to 0.19
  - Update nom to 6.0.1
  - Update quick-xml to 0.20
  - Update rand to 0.8
  - Update tempfile to 3.2
  - Update tui to 0.14

Signed-off-by: Kay Lin <i@v2bv.net>
2021-06-26 00:08:58 +08:00
Ming-Hung Tsai
bfc7f96d9f [tests] Enable testing the rust targets
This is a temporary solution for development.
Usage: cargo test --features rust_tests
2021-06-21 23:27:51 +08:00
Joe Thornber
7e869bb8e0 Code review of cache_dump/check
Added support for metadata format2 to cache_dump.  Nothing tested.
2021-03-04 11:13:08 +00:00
Joe Thornber
a4ba01cacd
Merge pull request #160 from mingnus/rust-cache-tools
First draft of cache_check in Rust
2021-02-24 09:53:51 +00:00
Joe Thornber
763b2d14b2 [io_engine] Add Block::zeroed() constructor 2021-02-09 14:34:26 +00:00
Ming-Hung Tsai
faa371c208 [cache (rust)] Implement cache primitives 2021-02-03 15:18:47 +08:00
Ming-Hung Tsai
becdbbdb49 [build] Update Rust package version 2020-11-24 16:03:58 +08:00
Joe Thornber
819fc6d54c [thin_explore] accept a node path on the command line
Helpful to examine thin_check failures.
2020-09-22 10:47:04 +01:00
Joe Thornber
8493cf7081 [thin_explore] First code drop 2020-09-16 15:10:01 +01:00
Joe Thornber
44142f657a [thin_check (rust)] Add error handling to io_engine interface 2020-09-02 12:57:47 +01:00
Joe Thornber
e8d7e5cf1e [thin_check (rust)] move report creation to top level 2020-08-17 16:05:06 +01:00
Joe Thornber
e65d2dec6f [thin_check (rust)] Add progress bar 2020-08-11 13:44:33 +01:00
Joe Thornber
fa4ea3e2d9 [functional-tests] port some of the cache_check tests to Rust 2020-08-07 14:30:00 +01:00
Joe Thornber
fcfcc60b89 [functional-tests] Move thin_check functional tests to Rust.
They'll be run as part of 'cargo test' now.
2020-07-31 11:04:12 +01:00
Joe Thornber
4f120911d2 [thin_check (rust)] First attempt at multithreading the tree walking.
Still using a single io engine, so little benefit.
2020-07-29 16:38:52 +01:00
Joe Thornber
e9abdd9c88 [thin_check (rust)] Switch to a different io_uring crate.
This one works.
2020-07-28 12:57:30 +01:00
Joe Thornber
3cf6307762 Merge branch 'master' into 2020-06-13-thin-check-rewrite 2020-07-27 15:53:26 +01:00
Joe Thornber
d03dac8f75 [thin_shrink] write test harness 2020-06-26 16:44:47 +01:00
Joe Thornber
259eef9eee [thin_shrink] calculate remaps 2020-06-25 10:44:57 +01:00
Joe Thornber
861b2f21ff [thin_shrink] Write thinp xml format reader and writer.
Still need to tidy up the error handling.
2020-06-24 13:32:47 +01:00
Joe Thornber
fdf641aff3 [thin_metadata_{pack,unpak}] use anyhow in toplevel. 2020-06-21 11:50:24 +01:00
Joe Thornber
380632565d [thin_metadata_{pack/unpack}] Fix get_nr_blocks()
It didn't work with block devices.
2020-06-10 14:38:42 +01:00
Joe Thornber
61de3f9287 [thin_metadata_pack/unpack] Replace C++ implementation with a Rust one.
The Rust implementation is multithreaded, performs better in general and
does custom compression of btree nodes to achieve much better compression
ratios.  unpack also checksums expanded metadata to validate it.

Format version has jumped to 3, no backwards compatibility, but I think
that's ok since we never made a release that contained the C++ version
of these tools.

Benchmarks
==========

On an 8 core, 16 hyperthread machine.

metadata 1G, full:

      Pack size    pack time     unpack time
------------------------------------------------------
C++      193M        50.3s          6.9s (no verify)
Rust      70M         1.4s          1.8s (verify)

metadata 16G, sparse:

       Pack size    pack time     unpack time
------------------------------------------------------
C++      21M          68s           1s   (no verify)
Rust      4M           8.6s         0.5s (verify)
2020-06-09 09:15:00 +01:00