Commit Graph

215 Commits

Author SHA1 Message Date
Ming-Hung Tsai
0d87619a93 [thin_repair/thin_dump (rust)] Show repairing progress 2021-08-27 17:30:56 +08:00
Ming-Hung Tsai
71d47ef58b [thin/cache (rust)] Validate superblock checksum 2021-08-27 11:17:27 +08:00
Ming-Hung Tsai
8a822cebec [thin_repair/thin_dump (rust)] Support rebuilding superblock 2021-08-27 02:17:57 +08:00
Ming-Hung Tsai
213442bffd [all (rust)] Fix cosmetic names for argument values 2021-08-27 02:17:57 +08:00
Ming-Hung Tsai
434d24d10a [thin_repair (rust)] Support setting missing superblock fields automatically
While rebuilding a superblock, the transaction_id, data_block_size, and
nr_data_blocks could be inherited from the current superblock if available
(assumed it's a multiple-activated copy, so supposed partially valid),
or derived from the probed data mappings and device details, that saves
the hassle of manual input.

Note that either the current superblock or user-overrides must be compatible
with the probed values, thus the filters are applied.
2021-08-27 02:17:57 +08:00
Ming-Hung Tsai
5dad1097c3 [thin_repair (rust)] Add searching for missing roots
Based on the method of commit 9e20465
2021-08-27 02:17:57 +08:00
Ming-Hung Tsai
b7132440d0 [space_map (rust)] Encapsulate implementations 2021-08-27 00:08:21 +08:00
Ming-Hung Tsai
ec44d043e8 [space_map (rust)] Restrict metadata space map size 2021-08-06 13:22:13 +08:00
Ming-Hung Tsai
c167403212 [space_map (rust)] Fix the maximum value of reference counts 2021-08-06 13:22:13 +08:00
Ming-Hung Tsai
e158dc7601 [btree_builder] Fix issues with under populated shared nodes
A pre-built node could be under populated (less than half-full) due to
following reasons:

- A single shared leaf generated by dm-thin key removal. The residency
  could drop below 50%, until it reaches the merge threshold (33% or 44%,
  depends on its context).
- A shared root, which could have any possible nr_entries.
- Underfull shared nodes (less than 33% residency) caused by kernel issues.

To avoid producing under populated nodes, those kinds of pre-built nodes,
except the roots, will be merged into their siblings.
2021-08-06 13:22:13 +08:00
Ming-Hung Tsai
bd39b570ef [btree_builder] Fix reference counts of unshifted child values
The ref counts now are handled in a straightforward method:
The pre-built subtrees serve as snapshots of potentially shared nodes.
They hold an initial ref count on the pre-built nodes and their children.
The temporary ref counts are later dropped at the end of device building.

This way fixes the ref counts of unshifted nodes, thus the 'reserve'
operation introduced by commit 6d16c58 is reverted.
2021-08-05 23:41:23 +08:00
Ming-Hung Tsai
052c9f90ea [thin/cache_repair (rust)] Fix file open options 2021-08-05 23:41:23 +08:00
Ming-Hung Tsai
7239204b01 [cache_repair (rust)] First pass at cache_repair 2021-07-27 23:51:20 +08:00
Ming-Hung Tsai
4b9766846e [cache (rust)] Prepare for cache_repair
- Adapt function interfaces for repairing purpose
- Finalize the metadata in the Restorer
- Make the Restorer public
2021-07-27 23:51:20 +08:00
Ming-Hung Tsai
85581cdf5a [thin_repair (rust)] First pass at thin_repair 2021-07-27 23:51:20 +08:00
Ming-Hung Tsai
dccd844714 [thin_restore (rust)] Prepare for thin_repair
- Finalize the metadata in the Restorer
- Make the Restorer public
2021-07-27 23:51:20 +08:00
Ming-Hung Tsai
d42bde371f [thin_dump (rust)] Prepare for thin_repair
- Adapt function interfaces for repairing purpose
- Pull out reuseable structures
2021-07-27 23:51:20 +08:00
Ming-Hung Tsai
56d7da66aa [thin_dump (rust)] Tidy up
- Do not use shared internals as Defs
- Build Defs with common leaf sequences only, that could forms
  more common Defs between partially shared subtrees.
2021-07-27 23:45:02 +08:00
Ming-Hung Tsai
7002a8ae8d [all (rust)] Pull out structures for intermediate representation
Also fix the data type for thin timestamp
2021-07-21 18:46:06 +08:00
Ming-Hung Tsai
a50c9d97e2 [thin_metadata_unpack] Fix truncated output file on invalid input
Check the input header before creating or truncating the output file
2021-07-09 01:22:22 +08:00
Ming-Hung Tsai
d00388f68a [thin_shrink] Support short options 2021-07-07 16:06:04 +08:00
Ming-Hung Tsai
1526ab3472 [all] Apply cargo fmt, and fix clippy warning of branches_sharing_code 2021-07-02 16:17:52 +08:00
Kay Lin
c18cd42d35 [build] Update dependencies
- Update crc32c to 0.6, which allows it to be built on AArch64.

  - Update base64 to 0.13
  - Update byteorder to 0.14
  - Update io-uring to 0.4
  - Update libc to 0.2.83
  - Update nix to 0.19
  - Update nom to 6.0.1
  - Update quick-xml to 0.20
  - Update rand to 0.8
  - Update tempfile to 3.2
  - Update tui to 0.14

Signed-off-by: Kay Lin <i@v2bv.net>
2021-06-26 00:08:58 +08:00
Ming-Hung Tsai
2cb84236d4 [all (rust)] Tidy command line options 2021-06-23 14:33:28 +08:00
Ming-Hung Tsai
cd48f00191 [all (rust)] Make sync-io the default
Multithreaded sync-io has performance similar to async-io. Also,
sync-io saves the hassle of setting ulimits to get io_uring working
on some systems (commit ba7fd7b). Now we default to sync-io, and
leave async-io as a hidden option for testing and benchmarking.
2021-06-23 14:33:28 +08:00
Ming-Hung Tsai
361d19adaa [space_map (rust)] Fix cache hit with async-io 2021-06-23 14:33:28 +08:00
Ming-Hung Tsai
fba2adbe56 [tests] Fix minor test errors
- Fix command line arguments
- Adapt error messages for rust implementations
2021-06-21 23:27:51 +08:00
Ming-Hung Tsai
7daff7350a [thin/cache_dump (rust)] Add buffered outputs, and fix command line options 2021-06-21 23:27:51 +08:00
Ming-Hung Tsai
c71132c056 [space_map (rust)] Do not use an extra block set for building metadata sm
The blocks storing metadata itself are located continuously within
a certain reserved range, hence there's no need to use a block set
as the representation.
2021-06-21 23:11:57 +08:00
Ming-Hung Tsai
7ab97a9aae [space_map (rust)] Fix nr_free in index entries 2021-06-16 01:39:31 +08:00
Ming-Hung Tsai
4b7b3658ff [thin/cache_restore (rust)] Build the metadata space map in-place
That avoids cloning the source space map
2021-06-16 01:39:31 +08:00
Ming-Hung Tsai
9ab8dfa283 [space_map (rust)] Fix bitmap packing
- Allow packing unaligned number of bitmap entries that could happen
  with the last bitmap. Unused trailing entries are zeroed by memset.
- Fix none_free_before in index_entry
2021-06-16 01:39:31 +08:00
Ming-Hung Tsai
3fda9cc1f8 [thin_restore (rust)] Do not iterate mapping tree leaves twice 2021-06-16 01:39:31 +08:00
Ming-Hung Tsai
de7e79fc06 [btree_builder] Rename Builder to BTreeBuilder for clarity 2021-06-16 01:39:31 +08:00
Ming-Hung Tsai
6d16c5816e [btree_builder] Fix reference counts of btree nodes
A leaf node should be counted only if it is referenced by some internal
nodes, since the leaves generated by <def> sections might be unshifted
and merged with exclusive mappings or other shared defs, or they might
not even be used by any of the devices if the xml was tampered. The
internal nodes should be handled in the same manner.

The new 'reserve' operation is designed for this purpose. Applications
could reserve a block for later use, without setting its ref count
immediately. This way saves the hassles of tracking unused leaves.
2021-06-16 01:39:31 +08:00
Ming-Hung Tsai
9e061a03a8 [space_map (rust)] Do not reset search position 2021-06-11 20:51:49 +08:00
Ming-Hung Tsai
88e7f8fd69 [array_builder] Simplify array building process 2021-06-11 20:51:49 +08:00
Ming-Hung Tsai
60b65ebe7a [space_map (rust)] Fix uninitialized bytes in index block 2021-06-08 20:17:24 +08:00
Ming-Hung Tsai
b12530f580 [space_map (rust)] Fix nr_allocated tracking in CoreSpaceMap 2021-05-28 20:20:30 +08:00
Ming-Hung Tsai
3a653eaa5f [thin_restore (rust)] Build metadata and data space maps 2021-05-28 03:19:42 +08:00
Ming-Hung Tsai
c142cd0d48 [space_map (rust)] Fix space map building
- Fix out-of-bounds index
- Automatically flush queued writes before function return
- Track allocated blocks in write_batcher (might be space consuming)
2021-05-28 03:19:42 +08:00
Ming-Hung Tsai
13d6c72ad9 [cache_restore (rust)] Build metadata space map 2021-05-28 03:19:42 +08:00
Ming-Hung Tsai
48d4fc51ed [space_map (rust)] Factor out space_map_metadata 2021-05-28 03:19:42 +08:00
Ming-Hung Tsai
d5e6a69af6 [thin (rust)] Fix the unit of metadata_block_size in superblock 2021-05-28 03:19:42 +08:00
Ming-Hung Tsai
6a29f6a41a [cache_check (rust)] Fix discard bitset availability checking 2021-05-28 03:19:42 +08:00
Ming-Hung Tsai
ce94ba73a5 [cache_restore (rust)] First draft 2021-05-28 03:19:42 +08:00
Ming-Hung Tsai
e336b3a63f [math (rust)] Make the functions generic 2021-05-28 03:19:42 +08:00
Ming-Hung Tsai
1198a3f713 [cache (rust)] Implement Pack for superblock 2021-05-28 03:19:42 +08:00
Ming-Hung Tsai
511ae9e908 [checksum] Support cache and era superblock 2021-05-28 03:19:42 +08:00
Ming-Hung Tsai
159dda9659 [thin_restore (rust)] Tidy up with the builder pattern 2021-05-28 03:19:42 +08:00