Joe Thornber
bc058f8baf
[thin_check (rust)] BTree values must now implement Copy
2020-09-18 10:06:33 +01:00
Joe Thornber
bcfb9a73a1
[thin_explore] display ranges where possible
2020-09-18 09:12:51 +01:00
Joe Thornber
8493cf7081
[thin_explore] First code drop
2020-09-16 15:10:01 +01:00
Joe Thornber
5168621f02
[thin_check (rust)] Optimise SyncIoEngine
...
*_many no longer get/put for each block
2020-09-02 13:28:16 +01:00
Joe Thornber
44142f657a
[thin_check (rust)] Add error handling to io_engine interface
2020-09-02 12:57:47 +01:00
Joe Thornber
b82307d8a5
[thin_check (rust)] drop O_DIRECT for the sync_io engine.
...
O_DIRECT slows us down, and there's no correctness reason for having it.
2020-08-21 11:39:41 +01:00
Joe Thornber
cda92de441
[thin_check (rust)] Add a threaded version of btree walk.
...
Bottom level mappings use this if there are few devices. Performance
is a bit slower for io_uring, and much slower for sync io (which I think
is due to io scheduling which I can't do much about).
2020-08-21 10:10:49 +01:00
Joe Thornber
b01a0a46d1
[thin_metadata_pack/unpack] Use Vec::with_capacity() to avoid reallocs.
...
Gives a small speed boost to both pack and unpack.
2020-08-21 09:14:54 +01:00
Joe Thornber
c9a759b4e8
[thin_check (rust)] Use vec::with_capacity() to avoid reallocations.
2020-08-21 09:00:21 +01:00
Joe Thornber
2cc2dffab5
[thin_check (rust)] Make NodeVisitor::visit non mut.
...
Preparation for making btree_walk multithreaded
2020-08-20 11:05:14 +01:00
Joe Thornber
a1c206b774
[thin_check (rust)] NodeVisitor only needs to see leaf nodes
2020-08-20 10:55:38 +01:00
Joe Thornber
936e06e132
[thin_check (rust)] Remove some unused params from NodeVisitor::visit
2020-08-20 10:46:06 +01:00
Joe Thornber
1999343d2f
[thin_check (rust)] squash a couple of warnings
2020-08-20 10:33:02 +01:00
Joe Thornber
7834d661e2
[thin_check (rust)] auto repair space map leaks
2020-08-19 14:31:01 +01:00
Joe Thornber
cdd0beb527
[thin_check (rust)] Change io_engine trait to use slices rather than Vecs
2020-08-18 12:57:05 +01:00
Joe Thornber
2aa6859502
[thin_check (rust)] add write support to io_engine
2020-08-18 12:52:16 +01:00
Joe Thornber
67a54b4ebc
[thin_check (rust)] add --auto-repair switch
2020-08-18 11:47:42 +01:00
Joe Thornber
8eec84fbec
[thin_check (rust)] introduce ASpaceMap type alias
2020-08-18 11:06:15 +01:00
Joe Thornber
04f3ba5a33
[thin_check (rust)] Pass ctx to check_space_map
2020-08-18 10:59:04 +01:00
Joe Thornber
4beb2db337
[thin_check (rust)] Factor out check_mapping_bottom_level
2020-08-18 10:53:11 +01:00
Joe Thornber
239ae6b6ec
[thin_check (rust)] factor out spawn_progress_thread
2020-08-18 09:48:51 +01:00
Joe Thornber
e8d7e5cf1e
[thin_check (rust)] move report creation to top level
2020-08-17 16:05:06 +01:00
Joe Thornber
5743e3e9ba
[thin_check (rust)] Add title method to reports
2020-08-17 15:36:21 +01:00
Joe Thornber
9995751dde
[thin_check (rust)] Provide 3 different report types.
...
ProgressBar for interactive use.
Simple for when not a tty
and quiet for -q
2020-08-17 13:10:32 +01:00
Joe Thornber
e1cfc3866b
[thin_check (rust)] Mappings top level weren't being ref counted
2020-08-13 14:43:19 +01:00
Joe Thornber
092447d17a
[thin_check (rust)] remove some dead code
2020-08-13 14:30:04 +01:00
Joe Thornber
2fa732a93c
[functional tests] Port some of Hank's sh tests to Rust.
...
Run with 'cargo test'
2020-08-13 14:20:29 +01:00
Joe Thornber
b0e7520fbf
[thin/superblock (rust)] Unpack flags
2020-08-13 13:46:07 +01:00
Joe Thornber
afa3f2f04d
[thin_check (rust)] Rename Spinner -> Reporter
2020-08-12 10:25:06 +01:00
Joe Thornber
544335ae4a
[thin_check (rust)] Send all reporting through the Spinner.
...
This means the spinner doesn't overwrite messages.
2020-08-12 09:35:21 +01:00
Joe Thornber
3757e1d947
[thin_check (rust)] check metadata space map
2020-08-12 08:02:29 +01:00
Joe Thornber
e65d2dec6f
[thin_check (rust)] Add progress bar
2020-08-11 13:44:33 +01:00
Joe Thornber
34425521e2
[thin_check (rust)] change BTreeWalker to use a space map rather than seen bitset
2020-08-11 10:50:43 +01:00
Joe Thornber
50bde693a1
[thin_check (rust)] Factor out pdata/unpack
2020-08-10 15:42:10 +01:00
Joe Thornber
55ee4bfad8
[thin_check (rust)] replace IndexVisitor with a call to btree_to_map
2020-08-10 14:56:39 +01:00
Joe Thornber
e28c602c3d
[thin_check (rust)] factor out btree_to_map() fn
2020-08-10 14:45:35 +01:00
Joe Thornber
cbc9c2c72a
[thin_check (rust)] Improve data_sm handling
2020-08-10 12:56:41 +01:00
Joe Thornber
d5444d2255
[thin_check (rust)] sm bitmap entries were being unpacked incorrectly.
2020-08-10 12:55:05 +01:00
Joe Thornber
b915257e10
[thin_check (rust)] Fix race in btree walking.
...
The seen bitset was locked once to test, and separately to insert.
2020-08-10 12:30:12 +01:00
Joe Thornber
4e4b7ca2b1
[thin_check (rust)] add --sync-io flag
...
Makes it easier to switch between engines
2020-08-10 11:24:50 +01:00
Joe Thornber
0f865856ed
[thin_check (rust)] Improve SyncIoEngine.
...
Now opens the file multiple times so different threads can do io in parallel.
2020-08-10 10:44:47 +01:00
Joe Thornber
f0df17af9e
[thin_check (rust)] Get SyncIoEngine working again.
2020-08-10 08:59:02 +01:00
Joe Thornber
08e3ea948e
[thin_check (rust)] rename block_manager.rs -> io_engine.rs
2020-08-10 08:29:32 +01:00
Joe Thornber
fd0c0ffc1d
[thin_check (rust)] data space map now checked.
2020-08-08 16:42:32 +01:00
Joe Thornber
4054b1be4c
[thin_check (rust)] Switch to BTreeMap.
...
It's faster.
2020-08-08 14:58:13 +01:00
Joe Thornber
1e4a038b41
[thin_check (rust)] Reimplement CoreSpaceMap
...
We now use a simple vector of elements that can hold 'nr thin devs'. Much faster.
2020-08-08 13:29:30 +01:00
Joe Thornber
7cf239b878
[thin_check (rust)] speed up CoreSpaceMap
2020-08-08 12:36:13 +01:00
Joe Thornber
ec8f7b7fa8
[thin_check (rust)] Keep track of data block ref counts
...
as we walk the mapping tree.
2020-08-08 09:54:16 +01:00
Joe Thornber
8f76371bb2
[functional-tests] Fix clippy warnings
2020-08-07 15:41:21 +01:00
Joe Thornber
fa4ea3e2d9
[functional-tests] port some of the cache_check tests to Rust
2020-08-07 14:30:00 +01:00
Joe Thornber
4a0582bb5d
[thin_check (rust)] start decoding the space maps.
2020-08-06 07:51:48 +01:00
Joe Thornber
197e4ffbfd
[thin_check (rust)] Rename ValueType trait to Unpack
2020-08-05 08:01:02 +01:00
Joe Thornber
1d44025584
[thin_check (rust)] Walk the top level and bottom level of the mapping tree separately
2020-08-04 12:11:36 +01:00
Joe Thornber
f56ea2d031
[thin_check (rust)] walk devices tree.
2020-08-03 16:22:08 +01:00
Joe Thornber
1368227a71
[thin_check (rust)] add btree node checks
2020-08-03 15:04:59 +01:00
Joe Thornber
cc2582b8b1
[thin_check (rust)] factor out pdata/btree.rs
2020-08-03 12:37:32 +01:00
Joe Thornber
7243f95380
[thin_check (rust)] Drop nr threads down to 4
...
We get very little benefit from threads atm. Need to improve IO handling first.
2020-07-30 10:17:36 +01:00
Joe Thornber
de172147d3
[thin_check (rust)] squash a lot of warnings
2020-07-30 10:12:51 +01:00
Joe Thornber
f7623e6264
[thin_check (rust)] remove spurious mutex.
2020-07-30 09:59:02 +01:00
Joe Thornber
8146fba9d2
[thin_check (rust)] Move mutex inside IoEngines.
...
Makes it easier to share an engine between different threads.
2020-07-30 08:47:16 +01:00
Joe Thornber
4f120911d2
[thin_check (rust)] First attempt at multithreading the tree walking.
...
Still using a single io engine, so little benefit.
2020-07-29 16:38:52 +01:00
Joe Thornber
d5597d5d36
[thin_check (rust)] Factor out tree walking code
2020-07-29 11:12:03 +01:00
Joe Thornber
e9abdd9c88
[thin_check (rust)] Switch to a different io_uring crate.
...
This one works.
2020-07-28 12:57:30 +01:00
Joe Thornber
a90294e279
[thin_check (rust)] read many blocks at once.
...
We need to switch to io_uring to really get the benefit of this.
2020-07-28 11:45:25 +01:00
Joe Thornber
062a1b8a2c
[thin_check (rust)] Walk mapping tree.
...
Sync IO, no checks beyond checksumming.
2020-07-28 10:51:48 +01:00
Joe Thornber
1398cf31d1
[thin_check (Rust)] work in progress
2020-07-27 15:53:42 +01:00
Joe Thornber
3cf6307762
Merge branch 'master' into 2020-06-13-thin-check-rewrite
2020-07-27 15:53:26 +01:00
Joe Thornber
e9055f5d7d
Merge branch 'master' of github.com:jthornber/thin-provisioning-tools
2020-07-22 16:09:33 +01:00
Joe Thornber
b7e02d0ae4
[thin_metadata_pack] FIx bug where pack would try and read too much.
2020-07-09 13:53:46 +01:00
Joe Thornber
19138dbd81
[thin_shrink] add unit tests for build_remaps
2020-07-06 11:44:49 +01:00
Joe Thornber
470d0b7532
[thin_shrink] more test cases
2020-07-01 07:52:53 +01:00
Joe Thornber
4d7f9c7ddc
[thin_shrink] add some more test cases
2020-06-30 08:32:04 +01:00
Joe Thornber
3618da3a12
[thin_shrink] Get test framework built
...
This verifies data has been moved to the correct place.
2020-06-29 10:49:40 +01:00
Joe Thornber
d03dac8f75
[thin_shrink] write test harness
2020-06-26 16:44:47 +01:00
Joe Thornber
07da5704d5
[thin_shrink] add --no-copy
2020-06-26 08:57:26 +01:00
Joe Thornber
abf06236b1
[thin_shrink] Tidy up some messages
2020-06-26 08:47:40 +01:00
Joe Thornber
d8a0805753
[thin_shrink] fix some clippy warnings
2020-06-26 08:31:02 +01:00
Joe Thornber
7df56a5a04
[thin_shrink] Squash some warnings
2020-06-26 08:00:53 +01:00
Joe Thornber
27c6e36f5c
[thin_shrink] Very simple copier implementation
2020-06-25 16:50:11 +01:00
Joe Thornber
fcf44d46b7
[thin_shrink] remove some debug printlns
2020-06-25 16:12:35 +01:00
Joe Thornber
31abc468be
[thin_shrink] Stub the copier
2020-06-25 15:28:45 +01:00
Joe Thornber
d8957e3d86
[thin_shrink] Rewrites xml
...
Just need to do copying now
2020-06-25 14:57:37 +01:00
Joe Thornber
259eef9eee
[thin_shrink] calculate remaps
2020-06-25 10:44:57 +01:00
Joe Thornber
3f1b776359
[thin_shrink] Add NoopVisitor
2020-06-24 13:55:08 +01:00
Joe Thornber
861b2f21ff
[thin_shrink] Write thinp xml format reader and writer.
...
Still need to tidy up the error handling.
2020-06-24 13:32:47 +01:00
Joe Thornber
5e19029e65
Merge branch '2020-06-19-use-anyhow' into 2020-06-13-thin-check-rewrite
2020-06-22 10:16:27 +01:00
Joe Thornber
fdf641aff3
[thin_metadata_{pack,unpak}] use anyhow in toplevel.
2020-06-21 11:50:24 +01:00
Joe Thornber
7d625eb6ff
[src/file_utils.rs] Don't print to stderr in file_exists.
2020-06-15 12:48:19 +01:00
Joe Thornber
29d56f62a5
wip
2020-06-14 08:17:46 +01:00
Joe Thornber
7a85e47a20
[rust] squash clippy (linter) warnings
2020-06-10 16:46:38 +01:00
Joe Thornber
f90010e22a
[pack/unpack] fix bug in file_utils::file_size()
2020-06-10 15:07:40 +01:00
Joe Thornber
380632565d
[thin_metadata_{pack/unpack}] Fix get_nr_blocks()
...
It didn't work with block devices.
2020-06-10 14:38:42 +01:00
Joe Thornber
821628b004
[pack/unpack] add missing file.
2020-06-09 14:59:13 +01:00
Joe Thornber
409a660082
[pack/unpack] Get the functional tests working again.
...
There's some hard coded version numbers in the tests, but I'm
leaving for now since I'll rewrite in Rust to avoid too much
of a proliferation of languages.
2020-06-09 13:03:39 +01:00
Joe Thornber
db5a71a53c
[rust tools] Get the tools version from autoconf
2020-06-09 10:23:52 +01:00
Joe Thornber
61de3f9287
[thin_metadata_pack/unpack] Replace C++ implementation with a Rust one.
...
The Rust implementation is multithreaded, performs better in general and
does custom compression of btree nodes to achieve much better compression
ratios. unpack also checksums expanded metadata to validate it.
Format version has jumped to 3, no backwards compatibility, but I think
that's ok since we never made a release that contained the C++ version
of these tools.
Benchmarks
==========
On an 8 core, 16 hyperthread machine.
metadata 1G, full:
Pack size pack time unpack time
------------------------------------------------------
C++ 193M 50.3s 6.9s (no verify)
Rust 70M 1.4s 1.8s (verify)
metadata 16G, sparse:
Pack size pack time unpack time
------------------------------------------------------
C++ 21M 68s 1s (no verify)
Rust 4M 8.6s 0.5s (verify)
2020-06-09 09:15:00 +01:00