emo
/
thin-provisioning-tools
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Compare commits

merge into: emo:master
Branches Tags
emo:master
emo:broken
...
pull from: emo:broken
Branches Tags
emo:master
emo:broken

378 Commits
master ... broken

Author SHA1 Message Date
pepe 77d2feb230 :) 2023-05-13 00:20:51 +00:00
Ming-Hung Tsai 8806dfe4f4 [cache_writeback] Fix space map commits 
Fix github issue #138
 2022-01-10 23:31:10 +08:00
Joe Thornber 8f9f9c74f6
Merge pull request #193 from arrikto/feature-fix-era-invalidate 
[era_invalidate] Don't read the live metadata when the --metadata-snapshot option is provided
 2022-01-05 11:45:08 +00:00
Nikos Tsironis 9f30793355 [era_invalidate] Don't read the live metadata when the --metadata-snapshot option is provided 
Until now, 'era_invalidate' read the live metadata (superblock), instead
of the metadata snapshot, when using the --metadata-snapshot parameter.

Fix this by passing the location of the metadata snapshot to
'open_metadata()', when a metadata snapshot is used.

Signed-off-by: Nikos Tsironis <ntsironis@arrikto.com>
 2022-01-05 12:30:02 +02:00
Joe Thornber cab57534c6
Merge pull request #191 from mingnus/2021-10-20-era-tools-rebase 
era-tools in Rust
 2021-11-01 09:51:59 +00:00
Ming-Hung Tsai f3c2ade90a [thin_ll_dump] Fix potential segfault while reading invalid subtree roots 2021-10-21 21:58:09 +08:00
Ming-Hung Tsai e5f0acd288 [thin_metadata_size (rust)] First code drop 2021-10-21 17:51:28 +08:00
Ming-Hung Tsai 9ea75ba113 [tests] Move cache_metadata_size tests to Rust 
Changes for the Rust version in corresponding to command line changes:
- Disable tests 'all_args_agree' and 'conradictory_info_fails'
- Test conflicts between --nr-blocks and {--device-size|--block-size}
 2021-10-21 17:51:28 +08:00
Ming-Hung Tsai 34f6b6fc62 [cache_metadata_size (rust)] First code drop 2021-10-21 17:51:28 +08:00
Ming-Hung Tsai cc89cef43f [era_dump (rust)] Dump the current_writeset that has not been archived 
This patch is added in corresponding to the dm-era patch de89afc1 in kernel:
dm era: Recover committed writeset after crash
 2021-10-21 17:51:28 +08:00
Ming-Hung Tsai 533e174051 [era_dump (rust)] Support wrapped-around era in logical dump 2021-10-21 17:51:28 +08:00
Ming-Hung Tsai 89e568b897 [era_dump (rust)] Implement logical dump 2021-10-21 17:51:28 +08:00
Ming-Hung Tsai cbc3baba45 [tests] Move era_dump tests to Rust 2021-10-21 17:51:28 +08:00
Ming-Hung Tsai f6eb5173c9 [era_restore] Avoid touching the output file by checking input file earlier 
The output file has been checked by the caller, so there's no need
to check the output file again.
 2021-10-21 17:51:28 +08:00
Ming-Hung Tsai 8d3f65244d [tests] Move era_restore tests to Rust 2021-10-21 17:51:28 +08:00
Ming-Hung Tsai 4cfe93570c [tests] Add era test fixtures 2021-10-21 17:51:28 +08:00
Ming-Hung Tsai 321fce882f [tests] Move era_check tests to Rust 2021-10-21 17:51:28 +08:00
Ming-Hung Tsai 0215b5fecd [era_check (rust)] Remove unused option 2021-10-21 17:51:28 +08:00
Ming-Hung Tsai 8fa7f96dfd [era_dump/restore (rust)] Support displaying writeset blocks in ranges 
- Group adjacent marked blocks into ranges
- Show marked blocks only, and ignore untouched blocks
 2021-10-21 17:49:36 +08:00
Ming-Hung Tsai 4559039066 [era_invalidate (rust)] First code drop 2021-10-21 17:04:54 +08:00
Ming-Hung Tsai 0791208ca4 [io_engine (rust)] Open file exclusively 2021-10-21 16:18:14 +08:00
Ming-Hung Tsai 9ab4172790 [cache_restore (rust)] Tidy up 2021-10-21 16:18:14 +08:00
Ming-Hung Tsai 36767bcda6 [era_repair (rust)] First code drop 2021-10-21 16:18:14 +08:00
Ming-Hung Tsai 3a8dc8da2d [xml (rust)] Log invalid attribute name 2021-10-21 16:18:14 +08:00
Ming-Hung Tsai ed7480e96d [era_restore (rust)] First code drop 2021-10-21 16:18:14 +08:00
Ming-Hung Tsai 55a81c0b9f [era_dump (rust)] First code drop 2021-10-21 16:18:14 +08:00
Ming-Hung Tsai 8a1399e3bb [cache_dump (rust)] Make use of the --repair option 
Ignore non-fatal errors, and regenerate unavailable dirty bits if --repair was applied.
 2021-10-21 16:18:14 +08:00
Ming-Hung Tsai 3d36456a36 [era_check (rust)] First code drop 2021-10-21 16:18:14 +08:00
Ming-Hung Tsai c8a1da1df9 [all] Apply cargo fmt, and fix clippy warnings 2021-10-20 20:23:27 +08:00
Ming-Hung Tsai 13aeefcdeb [thin] Fix typo 2021-10-20 20:23:27 +08:00
Joe Thornber 959b04ecb5 [thin_dump] fix overrides 2021-10-20 13:06:15 +01:00
Joe Thornber 56ea130650 [tests/thin_dump] Update help string 2021-10-19 15:24:16 +01:00
Joe Thornber a568da8c71 [tests/thin_dump] remove some extraneous output to stderr 2021-10-19 15:23:17 +01:00
Joe Thornber c5645d798f [tests/thin_delta] fix bad_option check 2021-10-19 14:25:56 +01:00
Joe Thornber 44d29b0467 [tests/thin_check] Clean up test dir 2021-10-19 14:25:25 +01:00
Joe Thornber 98be38dc4c [tests] Improve stdout/stderr logging when running sub processes 2021-10-19 14:20:47 +01:00
Joe Thornber 13400c2465 [tests/thin_delta] thin_delta should be a c++ command 2021-10-19 14:11:56 +01:00
Joe Thornber 6c5405ccf8 [tests] Get the run_* functions to log what they're doing. 2021-10-18 16:59:17 +01:00
Joe Thornber 024554c987 Merge rust tools into a single pdata_tools exe 2021-10-11 12:07:26 +01:00
Joe Thornber c9b47437f2
Merge pull request #189 from mingnus/2021-08-30-functional-tests 
Functional tests and misc fixes
 2021-09-30 15:15:21 +01:00
Ming-Hung Tsai d712190db4 [io_engine (rust)] Open file exclusively 2021-09-22 17:07:56 +08:00
Ming-Hung Tsai 6bd7741dfa [cache_repair] Avoid touching the output file by checking input file earlier 
The output file has been checked by the caller, so there's no need
to check the output file again.
 2021-09-22 17:07:56 +08:00
Ming-Hung Tsai 5abb92838c [tests] Port the remaining cache tests 2021-09-22 17:07:56 +08:00
Ming-Hung Tsai 66c1d629a4 [all (rust)] Keep silent if --quiet was applied 2021-09-22 17:07:56 +08:00
Ming-Hung Tsai d436f35ed3 [thin_dump/thin_repair/thin_restore (rust)] Fix errors in tests 
- Move error messages to stderr
- Fix the transaction_id and data_block_size in test fixture,
  and check the data_block_size in thin_restore.
- Disable the missing_something tests for Rust. The transaction_id
  and nr_data_blocks now could be recovered automatically.
- Limit the use of override options in test cases. Only broken
  superblocks could be overridden, and the provided values
  should be compatible to the original metadata.
- Remove unused option
 2021-09-22 17:06:28 +08:00
Ming-Hung Tsai c133e62353 [thin_dump] Emit superblock flags 2021-09-22 17:05:59 +08:00
Ming-Hung Tsai 34f927d989 [thin_check (rust)] Make better use of Rust's Result type 
Replace the bail_out checking with the returned Result, which helps
decoupling the internal state of Report from application logic.
 2021-09-22 17:05:59 +08:00
Ming-Hung Tsai 438730951e [thin_check (rust)] Allow ignoring non-fatal errors in mapping tree 2021-09-22 17:05:59 +08:00
Ming-Hung Tsai a18fd60f3f [thin_check (rust)] Fix auto-repair related errors 
- Returns error on metadata leaks
- Clear needs_check flag on success
- Check auto-repair write errors
- Fix file open flags, and correct spelling
 2021-09-22 17:05:59 +08:00
Ming-Hung Tsai e7fa012701 [btree] Fix ref-counting on overwritten values 2021-09-22 17:05:59 +08:00
Ming-Hung Tsai f8c40a1fda [thin_check (rust)] Set compatibility between options 2021-09-22 17:05:59 +08:00
Ming-Hung Tsai bb53083271 [array (rust)] Fix building uncontiguous array 2021-09-22 17:05:59 +08:00
Ming-Hung Tsai 2f22a8c55d [all (rust)] Fix errors in testing input option 
- Fix file mode bits checking
- Return error reason from stat
 2021-09-22 17:05:59 +08:00
Ming-Hung Tsai 4ed2348b36 [cache_restore (rust)] Fix errors in tests 
- Move error messages to stderr (fixes unwritable_output_file,
  input_file_not_found, and missing_input_option)
- Validate XML structures implicitly (fixes corrupted_input_data)
- Check output file size (fixes tiny_output_file)
- Allow restoring XML without the hints (for creating test fixtures)
- Provide XML error context
- Remove unused option
 2021-09-22 17:05:29 +08:00
Ming-Hung Tsai 59e44667a9 [tests] Port cache_restore tests 2021-09-16 21:55:46 +08:00
Ming-Hung Tsai 9253117132 [tests] Add cache fixtures 
- Generate temp xml and metadata files
- Correct existing tests to use cache fixtures
- Fix cache xml generation
 2021-09-16 21:55:46 +08:00
Ming-Hung Tsai 0acc57d17f [thin_restore (rust)] Fix errors in tests 
- Move error messages to stderr (fixes unwritable_output_file,
  input_file_not_found, and missing_input_option)
- Validate XML structures implicitly (fixes corrupted_input_data)
- Check output file size (fixes tiny_output_file)
- Provide XML error context
- Fix mk_valid_input()
 2021-09-16 21:55:46 +08:00
Ming-Hung Tsai aaa84aa35a [tests] Enable tests for rust thin_repair 2021-09-16 21:55:46 +08:00
Ming-Hung Tsai ea8dcb54d0 [tests] Fix the file size for testing 2021-09-16 21:55:46 +08:00
Ming-Hung Tsai 737cf2f67d [tests] Refine trait names 2021-09-16 21:55:20 +08:00
Ming-Hung Tsai 47d39d1efa [tests] Do not assume no stderr with thin_dump 2021-09-10 23:50:36 +08:00
Joe Thornber 9aa36f017a
Merge pull request #188 from mingnus/2021-08-05-thin-repair-find-roots 
Support broken superblock in thin_repair
 2021-09-10 13:59:16 +01:00
Ming-Hung Tsai 0d87619a93 [thin_repair/thin_dump (rust)] Show repairing progress 2021-08-27 17:30:56 +08:00
Ming-Hung Tsai 71d47ef58b [thin/cache (rust)] Validate superblock checksum 2021-08-27 11:17:27 +08:00
Ming-Hung Tsai 8a822cebec [thin_repair/thin_dump (rust)] Support rebuilding superblock 2021-08-27 02:17:57 +08:00
Ming-Hung Tsai 213442bffd [all (rust)] Fix cosmetic names for argument values 2021-08-27 02:17:57 +08:00
Ming-Hung Tsai 434d24d10a [thin_repair (rust)] Support setting missing superblock fields automatically 
While rebuilding a superblock, the transaction_id, data_block_size, and
nr_data_blocks could be inherited from the current superblock if available
(assumed it's a multiple-activated copy, so supposed partially valid),
or derived from the probed data mappings and device details, that saves
the hassle of manual input.

Note that either the current superblock or user-overrides must be compatible
with the probed values, thus the filters are applied.
 2021-08-27 02:17:57 +08:00
Ming-Hung Tsai 5dad1097c3 [thin_repair (rust)] Add searching for missing roots 
Based on the method of commit 9e20465
 2021-08-27 02:17:57 +08:00
Ming-Hung Tsai b7132440d0 [space_map (rust)] Encapsulate implementations 2021-08-27 00:08:21 +08:00
Joe Thornber 15189aa28b
Merge pull request #187 from mingnus/2021-08-24-thin-repair-find-roots-fixes 
Fix issues in finding roots
 2021-08-26 16:23:24 +01:00
Ming-Hung Tsai e6f17d4b4f [thin_repair/thin_dump] Exclude unwanted btree nodes 2021-08-24 22:26:22 +08:00
Ming-Hung Tsai d2d6ab7926 [thin_repair/thin_dump] Check consistency of thin_ids before running a regular dump 2021-08-24 22:26:22 +08:00
Ming-Hung Tsai d1e8168fb6 [thin_repair/thin_dump] Change the label type for empty leaves 
Empty leaves now are treated as bottom-level leaves, so that empty
devices could be recovered.
 2021-08-24 22:26:22 +08:00
Ming-Hung Tsai c2e6db74b9 [thin_repair/thin_dump] Remove unused function parameter 2021-08-24 22:26:22 +08:00
Ming-Hung Tsai c286041f25 [thin_repair/thin_dump] Fix sorting of data mapping candidates 
- Fix the references for sorting. The timestamp statistics is stored
  in node_info corresponding to the second element.
- Fix the timestamp comparison routine. The mapping root with more recent
  blocks should have higher priority.
 2021-08-24 22:26:15 +08:00
Joe Thornber b58e42bb95
Merge pull request #185 from mingnus/2021-07-29-btree-builder-fixes 
Fix reference counting in btree construction
 2021-08-06 08:35:46 +01:00
Ming-Hung Tsai ec44d043e8 [space_map (rust)] Restrict metadata space map size 2021-08-06 13:22:13 +08:00
Ming-Hung Tsai c167403212 [space_map (rust)] Fix the maximum value of reference counts 2021-08-06 13:22:13 +08:00
Ming-Hung Tsai e158dc7601 [btree_builder] Fix issues with under populated shared nodes 
A pre-built node could be under populated (less than half-full) due to
following reasons:

- A single shared leaf generated by dm-thin key removal. The residency
  could drop below 50%, until it reaches the merge threshold (33% or 44%,
  depends on its context).
- A shared root, which could have any possible nr_entries.
- Underfull shared nodes (less than 33% residency) caused by kernel issues.

To avoid producing under populated nodes, those kinds of pre-built nodes,
except the roots, will be merged into their siblings.
 2021-08-06 13:22:13 +08:00
Ming-Hung Tsai bd39b570ef [btree_builder] Fix reference counts of unshifted child values 
The ref counts now are handled in a straightforward method:
The pre-built subtrees serve as snapshots of potentially shared nodes.
They hold an initial ref count on the pre-built nodes and their children.
The temporary ref counts are later dropped at the end of device building.

This way fixes the ref counts of unshifted nodes, thus the 'reserve'
operation introduced by commit 6d16c58 is reverted.
 2021-08-05 23:41:23 +08:00
Ming-Hung Tsai 052c9f90ea [thin/cache_repair (rust)] Fix file open options 2021-08-05 23:41:23 +08:00
Joe Thornber 63b8b9fc40 Clarify a namespace. 
See issue 184.
 2021-08-04 13:46:30 +01:00
Joe Thornber 03d0651d0a
Merge pull request #183 from mingnus/2021-07-08-thin-repair 
First pass at thin_repair and cache_repair
 2021-07-29 13:22:36 +01:00
Ming-Hung Tsai 3dc01bf962 [tests] Use IntoIterator on the array argument (requires Rust 1.53) 2021-07-27 23:51:20 +08:00
Ming-Hung Tsai 7239204b01 [cache_repair (rust)] First pass at cache_repair 2021-07-27 23:51:20 +08:00
Ming-Hung Tsai 4b9766846e [cache (rust)] Prepare for cache_repair 
- Adapt function interfaces for repairing purpose
- Finalize the metadata in the Restorer
- Make the Restorer public
 2021-07-27 23:51:20 +08:00
Ming-Hung Tsai 85581cdf5a [thin_repair (rust)] First pass at thin_repair 2021-07-27 23:51:20 +08:00
Ming-Hung Tsai dccd844714 [thin_restore (rust)] Prepare for thin_repair 
- Finalize the metadata in the Restorer
- Make the Restorer public
 2021-07-27 23:51:20 +08:00
Ming-Hung Tsai d42bde371f [thin_dump (rust)] Prepare for thin_repair 
- Adapt function interfaces for repairing purpose
- Pull out reuseable structures
 2021-07-27 23:51:20 +08:00
Ming-Hung Tsai 56d7da66aa [thin_dump (rust)] Tidy up 
- Do not use shared internals as Defs
- Build Defs with common leaf sequences only, that could forms
  more common Defs between partially shared subtrees.
 2021-07-27 23:45:02 +08:00
Ming-Hung Tsai 7002a8ae8d [all (rust)] Pull out structures for intermediate representation 
Also fix the data type for thin timestamp
 2021-07-21 18:46:06 +08:00
Joe Thornber 6d31d4def2
Merge pull request #182 from mingnus/2021-06-23-functional-tests 
Clean up function test commons
 2021-07-21 11:27:05 +01:00
Ming-Hung Tsai 222b4f0902 [tests] Do not perform Path to str conversion for compatibility 2021-07-21 18:00:20 +08:00
Ming-Hung Tsai 16190f0f9a [tests] Pull out common submodules 
New modules: fixture, process, program, target, and thin
 2021-07-21 17:50:34 +08:00
Ming-Hung Tsai 66b49e6f3d [tests] Change the path type to OsString for compatibility 2021-07-21 17:27:13 +08:00
Joe Thornber b92151d527
Merge pull request #181 from mingnus/2021-06-23-functional-tests 
Functional tests revised
 2021-07-19 14:01:37 +01:00
Ming-Hung Tsai a50c9d97e2 [thin_metadata_unpack] Fix truncated output file on invalid input 
Check the input header before creating or truncating the output file
 2021-07-09 01:22:22 +08:00
Ming-Hung Tsai 6cecf0f673 [file_utils] Check the file type to prevent unexpected writes by thin_repair 2021-07-09 01:22:22 +08:00
Ming-Hung Tsai 87ada9b493 [tests] Add basic tests for cache_dump 2021-07-09 01:22:22 +08:00
Ming-Hung Tsai f395bab7be [tests] Use traits to specify test parameters 
To deal with variety in target attributes and their expected outputs,
the test parameters are categorized into traits, thus the test program
could define test parameters in a more structured way, without having
to pass multiple tightly-coupled parameters to test functions.
 2021-07-09 01:22:22 +08:00
Ming-Hung Tsai 12ef69c31b [tests] Pull out common tests on i/o options into reusable modules 
- Introduce modules for testing input/output options
- Provide macros for generating test cases
- Hide details of subprocess execution
 2021-07-08 22:02:44 +08:00
Ming-Hung Tsai 6660fde3c4 [tests] Refine the test naming and error messages 
- Make the naming of test cases less ambiguous, e.g., rename
  "missing_input_file" to "missing_input_arg" or "input_file_not_found"
- Unify the error messages on input/output options
 2021-07-08 01:05:15 +08:00
Ming-Hung Tsai d00388f68a [thin_shrink] Support short options 2021-07-07 16:06:04 +08:00
Ming-Hung Tsai 1526ab3472 [all] Apply cargo fmt, and fix clippy warning of branches_sharing_code 2021-07-02 16:17:52 +08:00
Joe Thornber 5ac9ae2dae
Merge pull request #180 from mingnus/2021-06-25-update-deps 
Update config scripts and Rust dependencies
 2021-06-28 15:44:59 +01:00
Ming-Hung Tsai 8bb4aaef8f [build] Create the destdir for solely installing rust tools 2021-06-28 08:19:30 +08:00
Ming-Hung Tsai 725ad1d9b0 [manpages] Make the footer backward compatible 2021-06-28 08:19:30 +08:00
Ming-Hung Tsai 857e3a7d3d [manpages] Make the header and footer backward compatible 2021-06-28 08:19:30 +08:00
Kay Lin 6f1a6a59dd [manpages] Update txt2man to 1.7.1 
Signed-off-by: Kay Lin <i@v2bv.net>
 2021-06-26 13:40:56 +08:00
Ming-Hung Tsai 2d201c5483 [build] Update config.guess and config.sub to version 2021-05-24 2021-06-26 13:38:56 +08:00
Ming-Hung Tsai b7c3969747 [build] Update dependencies 
- Update fixedbitset to 0.4
- Update indicatif to 0.16
- Update libc to 0.2.*
- Update nix to 0.21
- Update nom to 6.2.*
- Update dependencies with cargo-update
 2021-06-26 00:59:37 +08:00
Kay Lin c18cd42d35 [build] Update dependencies 
- Update crc32c to 0.6, which allows it to be built on AArch64.

  - Update base64 to 0.13
  - Update byteorder to 0.14
  - Update io-uring to 0.4
  - Update libc to 0.2.83
  - Update nix to 0.19
  - Update nom to 6.0.1
  - Update quick-xml to 0.20
  - Update rand to 0.8
  - Update tempfile to 3.2
  - Update tui to 0.14

Signed-off-by: Kay Lin <i@v2bv.net>
 2021-06-26 00:08:58 +08:00
Ming-Hung Tsai 4905c0eb73 [build] Update .gitignore 2021-06-26 00:08:58 +08:00
Joe Thornber cf27a2cf4f
Merge pull request #179 from mingnus/2021-06-03-cache-restore-fixes 
Minor fixes
 2021-06-23 07:55:59 +01:00
Ming-Hung Tsai 2cb84236d4 [all (rust)] Tidy command line options 2021-06-23 14:33:28 +08:00
Ming-Hung Tsai cd48f00191 [all (rust)] Make sync-io the default 
Multithreaded sync-io has performance similar to async-io. Also,
sync-io saves the hassle of setting ulimits to get io_uring working
on some systems (commit ba7fd7b). Now we default to sync-io, and
leave async-io as a hidden option for testing and benchmarking.
 2021-06-23 14:33:28 +08:00
Ming-Hung Tsai 361d19adaa [space_map (rust)] Fix cache hit with async-io 2021-06-23 14:33:28 +08:00
Joe Thornber 5dd2e81bf0
Merge pull request #178 from jwakely/boost-iostreams 
[build] Remove -lboost_iostreams linker flag
 2021-06-23 07:27:19 +01:00
Jonathan Wakely 6b7e66d8f9 [build] Remove -lboost_iostreams linker flag 
This was previously needed for thin-provisioning/thin_metadata_pack.cc
but that file was rewritten in Rust and no longer needs Boost. The flag
causes every binary to have a completely redundant depedency on
libboost_iostream.so, which is an issue for RHEL packaging.
 2021-06-22 21:23:54 +01:00
Joe Thornber 8e609458c2
Merge pull request #177 from mingnus/2021-06-03-cache-restore-fixes 
Fix restoration tools
 2021-06-22 09:50:49 +01:00
Ming-Hung Tsai fba2adbe56 [tests] Fix minor test errors 
- Fix command line arguments
- Adapt error messages for rust implementations
 2021-06-21 23:27:51 +08:00
Ming-Hung Tsai 2bd3c17578 [tests] Fix version string checking 2021-06-21 23:27:51 +08:00
Ming-Hung Tsai bfc7f96d9f [tests] Enable testing the rust targets 
This is a temporary solution for development.
Usage: cargo test --features rust_tests
 2021-06-21 23:27:51 +08:00
Ming-Hung Tsai 7daff7350a [thin/cache_dump (rust)] Add buffered outputs, and fix command line options 2021-06-21 23:27:51 +08:00
Ming-Hung Tsai c71132c056 [space_map (rust)] Do not use an extra block set for building metadata sm 
The blocks storing metadata itself are located continuously within
a certain reserved range, hence there's no need to use a block set
as the representation.
 2021-06-21 23:11:57 +08:00
Ming-Hung Tsai 7ab97a9aae [space_map (rust)] Fix nr_free in index entries 2021-06-16 01:39:31 +08:00
Ming-Hung Tsai 4b7b3658ff [thin/cache_restore (rust)] Build the metadata space map in-place 
That avoids cloning the source space map
 2021-06-16 01:39:31 +08:00
Ming-Hung Tsai 9ab8dfa283 [space_map (rust)] Fix bitmap packing 
- Allow packing unaligned number of bitmap entries that could happen
  with the last bitmap. Unused trailing entries are zeroed by memset.
- Fix none_free_before in index_entry
 2021-06-16 01:39:31 +08:00
Ming-Hung Tsai 3fda9cc1f8 [thin_restore (rust)] Do not iterate mapping tree leaves twice 2021-06-16 01:39:31 +08:00
Ming-Hung Tsai de7e79fc06 [btree_builder] Rename Builder to BTreeBuilder for clarity 2021-06-16 01:39:31 +08:00
Ming-Hung Tsai 6d16c5816e [btree_builder] Fix reference counts of btree nodes 
A leaf node should be counted only if it is referenced by some internal
nodes, since the leaves generated by <def> sections might be unshifted
and merged with exclusive mappings or other shared defs, or they might
not even be used by any of the devices if the xml was tampered. The
internal nodes should be handled in the same manner.

The new 'reserve' operation is designed for this purpose. Applications
could reserve a block for later use, without setting its ref count
immediately. This way saves the hassles of tracking unused leaves.
 2021-06-16 01:39:31 +08:00
Ming-Hung Tsai 9e061a03a8 [space_map (rust)] Do not reset search position 2021-06-11 20:51:49 +08:00
Ming-Hung Tsai 88e7f8fd69 [array_builder] Simplify array building process 2021-06-11 20:51:49 +08:00
Joe Thornber 101028ed5f
Merge pull request #176 from mingnus/2021-04-28-coverity-fixes 
Clear thin superblock flags in restored metadata
 2021-06-09 10:10:44 +01:00
Ming-Hung Tsai 60b65ebe7a [space_map (rust)] Fix uninitialized bytes in index block 2021-06-08 20:17:24 +08:00
Ming-Hung Tsai c32517f827 [thin] Clear superblock flags in restored metadata 
The needs_check flag is unnecessary for a restored metadata since
it is assumed clean and has no errors
 2021-06-07 19:06:56 +08:00
Joe Thornber 01aac6c1c1
Merge pull request #175 from mingnus/2021-04-28-coverity-fixes 
Fix resource leak
 2021-06-04 15:22:24 +01:00
Ming-Hung Tsai 429e7f01d7 [file_utils] Fix resource leak 2021-06-04 21:37:02 +08:00
Joe Thornber 2413b5d31f
Merge pull request #174 from mingnus/2021-04-28-coverity-fixes 
Fix issues detected by Coverity
 2021-06-03 12:11:54 +01:00
Ming-Hung Tsai 8014643b9e [build] Allow running tests without the --enable-testing configuration 
- Make the test targets always available (test, unit-test, functional-test)
- Cleanup include paths, and bring back the CPPFLAGS in commit 3e24cff that
  was overwritten by the merge ca8f6df (PR #60)
- Introduce the dev-tools target for building testing/dbg tools individually
- Leave the --enable-testing option for backward compatibility
 2021-06-03 18:54:26 +08:00
Ming-Hung Tsai 050eacf4cb [all] Remove unreachable code 2021-06-03 18:54:26 +08:00
Ming-Hung Tsai db52308d85 [build] Remove unused sources from the regular build 2021-06-03 18:54:26 +08:00
Ming-Hung Tsai 25ed2dfc9a [thin_dump] Fix warnings on potential NULL pointer 2021-06-03 18:54:26 +08:00
Ming-Hung Tsai 2e62363446 [all] Fix uninitialized class members 2021-06-03 18:54:26 +08:00
Ming-Hung Tsai 3145a1f4de [thin_metadata_size] Fix potential string overflow 2021-06-03 18:54:26 +08:00
Ming-Hung Tsai 759407445f [thin_show_duplicates] Fix potential errors 
- Fix error if no --block-sector provided
- Fix errors on pools without mappings, or zero-length file
 2021-06-03 18:50:20 +08:00
Ming-Hung Tsai 75c0a3656c [thin_dump] Fix leaked shared object handle 2021-06-02 03:43:58 +08:00
Ming-Hung Tsai 041ed7858c [build] Fix customized emitter linkage 2021-06-02 00:51:39 +08:00
Ming-Hung Tsai 0004dced93 [thin_show_metadata] Fix out-of-bounds access 2021-06-02 00:51:39 +08:00
Ming-Hung Tsai f7e4a8faa9 [all] Fix resource leaks 2021-06-02 00:51:39 +08:00
Joe Thornber b9df99fd6a
Merge pull request #173 from mingnus/2021-05-12-cache-restore 
cache_restore wip
 2021-06-01 09:15:38 +01:00
Ming-Hung Tsai b12530f580 [space_map (rust)] Fix nr_allocated tracking in CoreSpaceMap 2021-05-28 20:20:30 +08:00
Ming-Hung Tsai 3a653eaa5f [thin_restore (rust)] Build metadata and data space maps 2021-05-28 03:19:42 +08:00
Ming-Hung Tsai c142cd0d48 [space_map (rust)] Fix space map building 
- Fix out-of-bounds index
- Automatically flush queued writes before function return
- Track allocated blocks in write_batcher (might be space consuming)
 2021-05-28 03:19:42 +08:00
Ming-Hung Tsai 13d6c72ad9 [cache_restore (rust)] Build metadata space map 2021-05-28 03:19:42 +08:00
Ming-Hung Tsai 5ecae3ad88 [tests] Fix numeric literal annotation 2021-05-28 03:19:42 +08:00
Ming-Hung Tsai 48d4fc51ed [space_map (rust)] Factor out space_map_metadata 2021-05-28 03:19:42 +08:00
Ming-Hung Tsai d5e6a69af6 [thin (rust)] Fix the unit of metadata_block_size in superblock 2021-05-28 03:19:42 +08:00
Ming-Hung Tsai 6a29f6a41a [cache_check (rust)] Fix discard bitset availability checking 2021-05-28 03:19:42 +08:00
Ming-Hung Tsai ce94ba73a5 [cache_restore (rust)] First draft 2021-05-28 03:19:42 +08:00
Ming-Hung Tsai e336b3a63f [math (rust)] Make the functions generic 2021-05-28 03:19:42 +08:00
Ming-Hung Tsai 1198a3f713 [cache (rust)] Implement Pack for superblock 2021-05-28 03:19:42 +08:00
Ming-Hung Tsai 511ae9e908 [checksum] Support cache and era superblock 2021-05-28 03:19:42 +08:00
Ming-Hung Tsai 159dda9659 [thin_restore (rust)] Tidy up with the builder pattern 2021-05-28 03:19:42 +08:00
Ming-Hung Tsai 1907dab5ee [cache (rust)] Implement Pack and Default for restoration 2021-05-28 03:19:42 +08:00
Ming-Hung Tsai 86e2db3a1a [cache (rust)] Add visitor traits for cache_restore 2021-05-28 03:19:42 +08:00
Ming-Hung Tsai 2a77036fa8 [array_builder] First draft 
Not tested yet
 2021-05-28 03:19:42 +08:00
Ming-Hung Tsai a6e1870b2b [array (rust)] Implement Pack for restoration 2021-05-28 03:19:42 +08:00
Ming-Hung Tsai 7e2d69ede9 [cache_dump (rust)] Allow partially broken dirty bitset 2021-05-28 03:19:42 +08:00
Joe Thornber db48f51049
Merge pull request #172 from mingnus/2021-05-04-thin-dump-fixes 
Make thin_dump runnable
 2021-05-27 19:03:05 +01:00
Ming-Hung Tsai 2a9e7cf74f [thin_dump (rust)] Split runs at the position with multiple references 
That is, an element with multiple references will serve as the beginning
of an atomic run.
 2021-05-28 00:18:27 +08:00
Ming-Hung Tsai 30cfcd9a88 [thin_dump (rust)] Use common leaf sequences to pack metadata 2021-05-28 00:07:32 +08:00
Ming-Hung Tsai 4c47fcabbf [thin_dump (rust)] Fix missing outputs 2021-05-27 23:51:37 +08:00
Joe Thornber 4f192cea0f
Merge pull request #171 from mingnus/rust-cache-tools 
Fix bugs in array iteration and text outputs
 2021-05-13 15:02:41 +01:00
Ming-Hung Tsai 7e53c36d6b [cache (rust)] Fix bugs in array iteration and text outputs 
- Fix array indexing
- Fix panic on empty array
- Remove trailing null characters from the policy name
- Change XML tag naming for backward compatibility
 2021-05-13 21:36:52 +08:00
Joe Thornber 11c354b3b1
Merge pull request #170 from mingnus/rust-cache-tools 
Merge recent changes in cache-tools
 2021-05-12 09:29:49 +01:00
Ming-Hung Tsai 1bbb63f06b [cache_check (rust)] Fix discard bitset size checking 2021-05-12 15:50:14 +08:00
Ming-Hung Tsai b7bf82b8f2 [all] Fix newline in version string 2021-05-12 12:03:52 +08:00
Ming-Hung Tsai 965fbb6e8f [all] Apply cargo fmt, and fix clippy warnings 2021-05-11 23:53:31 +08:00
Ming-Hung Tsai 0553a78c04 Add pre-commit hooks 2021-05-11 23:18:43 +08:00
Ming-Hung Tsai 5baeab4a5c Merge branch 'main' into rust-cache-tools 2021-05-11 23:16:08 +08:00
Joe Thornber d9a96758b0
Merge pull request #168 from mingnus/2020-10-09-thin-restore-rewrite 
Merge recent changes in thin_restore
 2021-05-04 09:21:40 +01:00
Ming-Hung Tsai 43e433149b [all] Apply cargo fmt 2021-05-04 16:10:23 +08:00
Ming-Hung Tsai 4b4584c830 [thin_restore (rust)] Apply several fixes 
- Fix reading queued blocks
- Fix unnecessary block shadowing when there's no remaining values
- Prevent superblock from overwritten
- Flush queued writes before updating superblock
 2021-05-03 00:07:34 +08:00
Ming-Hung Tsai e9899ac610 Add missing math.rs 2021-04-26 12:47:05 +08:00
Ming-Hung Tsai cf4b937ade [cache_check (rust)] Check space map counts 
- Support space map checking and auto-repair
 2021-04-23 16:23:21 +08:00
Ming-Hung Tsai 636d50a38d [thin_check (rust)] Pull out space map checking routines 2021-04-23 16:04:56 +08:00
Ming-Hung Tsai e1628f9004 [cache_check (rust)] Add more checks 
- Check array indices continuity
- Support ignore_non_fatal
- Report blocknr of IoErrors
- Report array block indeices
 2021-04-21 23:35:22 +08:00
Ming-Hung Tsai 239ff7dfa1 [cache_check (rust)] Add more checks 
- Report errors
- Support reading partially broken bitset
  - The output is a bitmap of 2-bit entries, indicating availability of bits
 2021-04-21 14:09:38 +08:00
Ming-Hung Tsai 3279d8381b [array_walker] Read multiple array blocks at once 2021-04-17 00:10:14 +08:00
Ming-Hung Tsai c17559791f [bitset] Rename bitset module 2021-04-17 00:10:14 +08:00
Ming-Hung Tsai 1964015d81 [array_walker] Handle the whole array block at once 
That gives the visitor more controls over the data processing and locking,
and also improves the performance by 10-15%.
 2021-04-17 00:06:09 +08:00
Ming-Hung Tsai 95dee9f66d [cache_check (rust)] Do not remap ArrayErrors to keep the error context 2021-04-16 21:46:30 +08:00
Ming-Hung Tsai 9b4a0607ea [cache_check (rust)] Detect structural errors on arrays 
- Define error types for arrays
- Propagate low-level errors to tools
- Rename array_block.rs to array.rs
 2021-04-14 16:02:50 +08:00
Ming-Hung Tsai ace9c1d1e3 [cache_check (rust)] Add more checks 
- check version-2 dirty bitset size and consistency
- check discard bitset size
 2021-04-11 22:48:10 +08:00
Ming-Hung Tsai ae630f1fd8 [btree_walker] Fix error returning 2021-04-11 20:18:44 +08:00
Ming-Hung Tsai 860b3ca7d2 [cache_check (rust)] Add more checks 
- Check the version-1 dirty flag
- Check mappings against the origin device size, if the cache is clean
- Check superblock fields
 2021-04-11 20:18:25 +08:00
Joe Thornber 0119b51a9c Merge branch 'main' of github.com:jthornber/thin-provisioning-tools 2021-04-08 10:47:18 +01:00
Joe Thornber c41b2e8ec3 Some interface changes to let dm-unit use the thin checking code more easily. 2021-04-08 10:45:50 +01:00
Joe Thornber 89372c3fa7
Merge pull request #167 from mingnus/main 
btree minor fixes
 2021-03-25 09:31:21 +00:00
Ming-Hung Tsai c496e8a4c8 [btree] Remove FIXMEs 2021-03-25 16:42:36 +08:00
Ming-Hung Tsai 8bfe7ee6f3 [btree] Fix rebalancing checks 2021-03-25 15:06:52 +08:00
Joe Thornber 040e3bfc2d Lot's of work on thin_restore 2021-03-24 14:20:20 +00:00
Joe Thornber 12c1c6e1f5 Merge branch 'main' into 2020-10-09-thin-restore-rewrite 2021-03-23 11:22:03 +00:00
Ming-Hung Tsai e0eb8fea87 [btree (rust)] Show out-of-order keys 2021-03-12 12:37:07 +08:00
Joe Thornber 0d46c606dd
Merge pull request #166 from mingnus/main 
Fix a syntax error
 2021-03-11 17:13:28 +00:00
Ming-Hung Tsai 0ff72374f8 [btree_builder (rust)] Fix the max_entries 2021-03-12 00:42:54 +08:00
Joe Thornber 1b7f43ef9f
Merge pull request #165 from mingnus/main 
Fix the max_entries in node_header
 2021-03-11 10:54:44 +00:00
Ming-Hung Tsai 7d983e3155 [btree_builder (rust)] Fix the max_entries 2021-03-11 18:24:10 +08:00
Joe Thornber d0675dd7bf Add missing header. 
So many errors from one issue
 2021-03-04 11:19:04 +00:00
Joe Thornber 7e869bb8e0 Code review of cache_dump/check 
Added support for metadata format2 to cache_dump.  Nothing tested.
 2021-03-04 11:13:08 +00:00
Joe Thornber e6c6275aea Code review of src/pdata/array* 2021-03-03 10:27:57 +00:00
Joe Thornber fdcc09c27e [cache_dump] Squash some clippy warnings 2021-03-03 09:48:15 +00:00
Joe Thornber 3cf0dba469 [cache_dump] fix how needs_check flag is checked 2021-03-03 09:47:42 +00:00
Ming-Hung Tsai eb3d181f95 [cache_dump (rust)] First draft of cache_dump 2021-03-03 12:27:51 +08:00
Ming-Hung Tsai fde0e0e2b8 [cache (rust)] Add Mapping::is_dirty() 2021-03-03 12:14:46 +08:00
Ming-Hung Tsai 74fcb9d505 [cache (rust)] Fix data types 2021-02-26 23:31:12 +08:00
Joe Thornber 3d9b32e509 Merge branch 'main' of github.com:jthornber/thin-provisioning-tools into main 2021-02-26 15:26:33 +00:00
Joe Thornber d4299a00d0 Make pack_node() and calc_max_entries() public for dm-unit 2021-02-26 15:25:53 +00:00
Joe Thornber 441a000c52
Merge pull request #163 from mingnus/rust-cache-tools 
[cache (rust)] Fix merge conflicts
 2021-02-24 12:33:50 +00:00
Ming-Hung Tsai a7ecfba2b4 [cache (rust)] Fix merge conflicts 2021-02-24 20:21:10 +08:00
Joe Thornber a4ba01cacd
Merge pull request #160 from mingnus/rust-cache-tools 
First draft of cache_check in Rust
 2021-02-24 09:53:51 +00:00
Ming-Hung Tsai 2bb3bf65b7 [cache_check (rust)] Implement basic functions 2021-02-24 17:39:11 +08:00
Joe Thornber 78c93ce09b
Merge pull request #162 from mingnus/main-thin-debug 
[dbg] Enhance debugging tools
 2021-02-22 08:28:07 +00:00
Ming-Hung Tsai e8410dec04 [dbg] Add missing commands to cache/era_debug 2021-02-21 19:16:51 +08:00
Ming-Hung Tsai 29f9182078 [dbg] Remove nested function template to reduce code size 2021-02-21 01:07:55 +08:00
Ming-Hung Tsai a81cef4467 [dbg] Pull out common code into dbg-lib 
- Modularize common routines
- Extract the block_dumper interface for displaying blocks
- Remove inheritance from show_traits
 2021-02-21 01:04:35 +08:00
Ming-Hung Tsai c95e31bef6 [era_debug] Display bitset entries in run-length fashion 2021-02-21 00:02:57 +08:00
Ming-Hung Tsai afbd913e22 [era] Add era_debug 2021-02-18 23:40:39 +08:00
Ming-Hung Tsai 1bc88bfde8 [cache_debug] Simplify the trait types 2021-02-18 18:20:48 +08:00
Ming-Hung Tsai e3a646e4d2 [dbg] Improve error resilience of debugging tools 
Do not open the metadata. The tool interprets user specified blocks one-by-one.
Thus, there's no need to preload the metadata structures.

Also remove the unused --ignore-metadata-space-map option. It was designed
to control the loading of space maps with the old metadata constructors.
 2021-02-18 00:00:55 +08:00
Joe Thornber 58cd881340 Fix regression where era_restore wouldn't work with devices. 
check_file_exists() had an extra parameter added with a default, which was
the wrong default for era_restore.
 2021-02-17 15:15:57 +00:00
Joe Thornber 763b2d14b2 [io_engine] Add Block::zeroed() constructor 2021-02-09 14:34:26 +00:00
Joe Thornber c3c6d37aea Fix a lot of clippy warnings 2021-02-08 10:38:21 +00:00
Joe Thornber e62fa3e835 Merge branch 'main' of github.com:jthornber/thin-provisioning-tools into main 2021-02-04 10:01:17 +00:00
Joe Thornber 6f30d17f04 version.rs includes VERSION directly. 
So you no longer need to run ./configure to build the rust tools
 2021-02-04 10:00:32 +00:00
Joe Thornber 8dbbe7fe27
Merge pull request #161 from mingnus/main-thin-debug 
[dbg] Integrate thin_debug, and add cache_debug
 2021-02-04 10:00:09 +00:00
Ming-Hung Tsai 08bd2f47bd [cache_debug] Fork thin_debug into cache_debug 2021-02-04 14:29:18 +08:00
Ming-Hung Tsai a05ac553b6 [dbg] Hide implementations of shared components 2021-02-04 14:28:17 +08:00
Ming-Hung Tsai b9b04dc872 [thin_debug] Factor out reusable componments 2021-02-04 14:19:28 +08:00
Ming-Hung Tsai e046bbf5c4 [thin_debug] Simplify the output format 
Turn nested fields into attributes. Also replace spaces in field names
by underscores.
 2021-02-03 15:51:52 +08:00
Ming-Hung Tsai 0816430ba0 [thin_debug] Remove the using boost directive to avoid namespace pollution 
boost::uint32_t might conflict with the toolchain defined uint32_t.
which causes template argument deduction failed.
 2021-02-03 15:51:52 +08:00
Ming-Hung Tsai ab3b2cbda2 [thin_debug] Refine the outputs 
- Show the blocknr from node header
- Display the index of node entries
 2021-02-03 15:51:52 +08:00
Ming-Hung Tsai 6dc5086643 [thin_debug] Add commands to show space maps 2021-02-03 15:51:52 +08:00
Ming-Hung Tsai 3bfa775887 [thin_debug] Show space map roots in superblock 2021-02-03 15:51:52 +08:00
Ming-Hung Tsai 127f44c66b [thin_debug] Enhance error handling 
- Handle exception thrown by commands
- Add help and exit commands
 2021-02-03 15:51:52 +08:00
Ming-Hung Tsai 62d09c6752 [thin_debug] Reduce code size by eliminating duplicated types 
This patch doesn't have a significant effect - only a few KBs of code
is reduced. However, it's still a nice have.
 2021-02-03 15:51:52 +08:00
Ming-Hung Tsai 0ce026caf5 [thin_debug] Integrate thin_debug into the main program 2021-02-03 15:51:52 +08:00
Ming-Hung Tsai faa371c208 [cache (rust)] Implement cache primitives 2021-02-03 15:18:47 +08:00
Ming-Hung Tsai 087a4711a5 [array (rust)] Implement array basis 2021-02-03 15:18:41 +08:00
Joe Thornber 9733ceb949
Merge pull request #158 from mingnus/main 
[cache_writeback] Support offset within the source and destination devices
 2021-01-20 11:06:21 +00:00
Ming-Hung Tsai 4c17076f09 [cache_writeback] Support offset within the source and destination devices 2021-01-20 12:02:13 +08:00
Joe Thornber 76ac202463
Merge pull request #157 from mingnus/main 
[build] Enable building the dev-tools for functional tests
 2021-01-12 09:02:43 +00:00
Ming-Hung Tsai 6a2fa73924 [build] Enable building the dev-tools for functional tests 
- Factor out the dev-tools into a stand-alone, no-installed program
- Built the dev-tools if --enable-testing is specified
- Remove the --enable-dev-tools configure option
- Allow suffix on the binary name
- Update symlinks
- Cleanup Makefile
 2021-01-12 16:08:51 +08:00
Ming-Hung Tsai 8d59a83f7c [functional-tests] Update thin_delta help text 2021-01-12 02:50:46 +08:00
Ming-Hung Tsai ca7e79a828 Merge commit 'b67b587' into main 2021-01-12 02:50:12 +08:00
Joe Thornber 04e0eb3a66 [thin_restore (rust)] rewrite the btree_builder 
Now copes with adding shared leaves.
 2020-12-09 13:22:32 +00:00
Joe Thornber 443b3c8f0b [io_engine (rust)] get_nr_blocks() wasn't handling block devices. 
Now calls file_utils::file_size()
 2020-12-02 15:20:14 +00:00
Joe Thornber ba7fd7bd2b [thin_check (rust)] Make --sync-io the default. 
For some systems you have to adjust the ulimits to get io_uring
to work, so we now default to using sync io.

Also added --async-io flag.
 2020-12-02 11:33:05 +00:00
Joe Thornber 0e4622f337 [Rust tools] squash lots of warnings 2020-12-01 11:50:32 +00:00
Joe Thornber 327fc80fb0 Merge branch 'main' into 2020-10-09-thin-restore-rewrite 2020-11-25 11:20:28 +00:00
Joe Thornber 83eea11d12
Merge pull request #155 from mingnus/thin-check-fix-metadata-leaks 
thin_check fixes
 2020-11-24 10:30:35 +00:00
Ming-Hung Tsai 565c656ed2 [thin_generate_damage] Do not open a new transaction to prevent ref-count underflow 
There's a chance that thin_generate_damage tries to change ref-counts of
space map blocks due to its random nature, which could lead to problems.
If the ref-counts of metadata space map blocks (shadow source) is changed
to zero, then the ref-counts will become underflow after a shadow operation.

In-place space map modification is a way to prevent that value underflow.
An alternative approach is to avoid changing ref-counts of space map blocks.
 2020-11-24 18:18:21 +08:00
Ming-Hung Tsai c932a76f08 [unit-tests] Add underfull nodes counting tests 2020-11-24 18:18:21 +08:00
Ming-Hung Tsai 61f07573e1 [metadata_counter] Count under populated nodes if the option is provided 2020-11-24 18:18:21 +08:00
Ming-Hung Tsai 1fe8a0dbde [thin_check] Allow using --clear-needs-check and --skip-mappings together 
Although it is not recommended to clear the flag without a full
examination, however, the usage has been documented as an approach
to reduce lvchange run time [1]. For the purpose of backward
compatibility and avoiding boot failure after upgrading thin_check [2],
the limitation is now removed.

[1] https://wiki.archlinux.org/index.php/LVM#Thinly-provisioned_root_volume_device_times_out
[2] Community feedback on previous commit:
    https://github.com/jthornber/thin-provisioning-tools/commit/b278f4f
 2020-11-24 18:17:36 +08:00
Ming-Hung Tsai f364de35bc [unit-tests] Fix unflushed trashed blocks and variable referencing 2020-11-24 17:55:17 +08:00
Joe Thornber 5af95167b4
Merge pull request #154 from mingnus/main 
thin development tools update
 2020-11-24 09:31:23 +00:00
Ming-Hung Tsai becdbbdb49 [build] Update Rust package version 2020-11-24 16:03:58 +08:00
Ming-Hung Tsai 7ceb500fc8 [thin_delta] Support comparing two specific subtrees 2020-11-24 15:58:01 +08:00
Ming-Hung Tsai 1d5b52b0dd [thin_delta] Clean up duplicated code 2020-11-24 15:17:35 +08:00
Ming-Hung Tsai b42408ef41 [thin] Introduce thin_patch_superblock to override superblock fields 2020-11-24 14:57:05 +08:00
Joe Thornber 1ae62adec6 work in progress 2020-11-18 14:33:56 +00:00
Joe Thornber 37ea0280df [thin_restore] first pass at btree_builder. 
No tests yet
 2020-10-26 12:05:27 +00:00
Joe Thornber f60ae770c2 [thin_explore] Explore devices tree, including path support. 2020-10-15 11:53:09 +01:00
Joe Thornber c42b623e39 Merge branch '2020-08-13-thin-check-rewrite' into main 2020-10-09 11:21:12 +01:00
Joe Thornber e9fbcc31de [thin_dump (rust)] First pass at thin_dump. 
Doesn't include --repair.

This includes <def> and <ref> sections for shared regions.
 2020-09-28 15:45:13 +01:00
Joe Thornber a88ae3ca18 [thin_check (rust)] factor out device detail 2020-09-25 09:59:16 +01:00
Joe Thornber 66b6a1ba48 [thin_check (rust)] --superblock-only, --skip-mappings, INFO fields 2020-09-24 13:55:58 +01:00
Joe Thornber 34052c540c [thin_check (rust)] Reinstate walk_node_threaded 2020-09-24 09:40:38 +01:00
Joe Thornber b67b587a10 [thin_shrink] Add comment pointing people at Nikhil's PoC 2020-09-22 12:01:17 +01:00
Joe Thornber f4c3098e02 [thin_check (rust)] fix bug in key range splitting. 
Ranges were not being ommitted when a block was ommitted due to
being shared and already visited.
 2020-09-22 10:47:23 +01:00
Joe Thornber 819fc6d54c [thin_explore] accept a node path on the command line 
Helpful to examine thin_check failures.
 2020-09-22 10:47:04 +01:00
Joe Thornber b193d19603 [thin_check (rust)] output complete node paths with errors. 
This can be used with thin_explore
 2020-09-18 11:16:09 +01:00
Joe Thornber bc058f8baf [thin_check (rust)] BTree values must now implement Copy 2020-09-18 10:06:33 +01:00
Joe Thornber bcfb9a73a1 [thin_explore] display ranges where possible 2020-09-18 09:12:51 +01:00
Joe Thornber 8493cf7081 [thin_explore] First code drop 2020-09-16 15:10:01 +01:00
Joe Thornber 5168621f02 [thin_check (rust)] Optimise SyncIoEngine 
*_many no longer get/put for each block
 2020-09-02 13:28:16 +01:00
Joe Thornber 44142f657a [thin_check (rust)] Add error handling to io_engine interface 2020-09-02 12:57:47 +01:00
Joe Thornber b82307d8a5 [thin_check (rust)] drop O_DIRECT for the sync_io engine. 
O_DIRECT slows us down, and there's no correctness reason for having it.
 2020-08-21 11:39:41 +01:00
Joe Thornber cda92de441 [thin_check (rust)] Add a threaded version of btree walk. 
Bottom level mappings use this if there are few devices.  Performance
is a bit slower for io_uring, and much slower for sync io (which I think
is due to io scheduling which I can't do much about).
 2020-08-21 10:10:49 +01:00
Joe Thornber b01a0a46d1 [thin_metadata_pack/unpack] Use Vec::with_capacity() to avoid reallocs. 
Gives a small speed boost to both pack and unpack.
 2020-08-21 09:14:54 +01:00
Joe Thornber c9a759b4e8 [thin_check (rust)] Use vec::with_capacity() to avoid reallocations. 2020-08-21 09:00:21 +01:00
Joe Thornber 2cc2dffab5 [thin_check (rust)] Make NodeVisitor::visit non mut. 
Preparation for making btree_walk multithreaded
 2020-08-20 11:05:14 +01:00
Joe Thornber a1c206b774 [thin_check (rust)] NodeVisitor only needs to see leaf nodes 2020-08-20 10:55:38 +01:00
Joe Thornber 936e06e132 [thin_check (rust)] Remove some unused params from NodeVisitor::visit 2020-08-20 10:46:06 +01:00
Joe Thornber 1999343d2f [thin_check (rust)] squash a couple of warnings 2020-08-20 10:33:02 +01:00
Joe Thornber 0372e689e5 Merge branch '2020-08-13-thin-check-rewrite' 2020-08-19 14:32:40 +01:00
Joe Thornber 7834d661e2 [thin_check (rust)] auto repair space map leaks 2020-08-19 14:31:01 +01:00
Joe Thornber cdd0beb527 [thin_check (rust)] Change io_engine trait to use slices rather than Vecs 2020-08-18 12:57:05 +01:00
Joe Thornber 2aa6859502 [thin_check (rust)] add write support to io_engine 2020-08-18 12:52:16 +01:00
Joe Thornber 67a54b4ebc [thin_check (rust)] add --auto-repair switch 2020-08-18 11:47:42 +01:00
Joe Thornber 8eec84fbec [thin_check (rust)] introduce ASpaceMap type alias 2020-08-18 11:06:15 +01:00
Joe Thornber 04f3ba5a33 [thin_check (rust)] Pass ctx to check_space_map 2020-08-18 10:59:04 +01:00
Joe Thornber 4beb2db337 [thin_check (rust)] Factor out check_mapping_bottom_level 2020-08-18 10:53:11 +01:00
Joe Thornber 239ae6b6ec [thin_check (rust)] factor out spawn_progress_thread 2020-08-18 09:48:51 +01:00
Joe Thornber 65799f9a14
Merge pull request #150 from mingnus/thin-check-fix-metadata-leaks 
Add space-map counting tests
 2020-08-18 08:50:32 +01:00
Joe Thornber e8d7e5cf1e [thin_check (rust)] move report creation to top level 2020-08-17 16:05:06 +01:00
Ming-Hung Tsai 44d025be0c [unit-tests] Add space map counting tests 2020-08-17 22:57:46 +08:00
Ming-Hung Tsai 3c49949796 [space-maps/disk] Support ignoring broken bitmaps on counting index_store 2020-08-17 22:57:26 +08:00
Joe Thornber 5743e3e9ba [thin_check (rust)] Add title method to reports 2020-08-17 15:36:21 +01:00
Joe Thornber 9995751dde [thin_check (rust)] Provide 3 different report types. 
ProgressBar for interactive use.
Simple for when not a tty
and quiet for -q
 2020-08-17 13:10:32 +01:00
Ming-Hung Tsai 27ca8cc009 [block_counter] Add block_counter::clear() 2020-08-17 15:43:09 +08:00
Ming-Hung Tsai de843991e3 [transaction_manager] Add transaction_manager::commit() 
It should be called by metadata::commit() and reserve_metadata_snap()
(issue #73)
 2020-08-17 15:43:04 +08:00
Ming-Hung Tsai 9f3823c97d [metadata_checker] Rename function to reflect command line changes 2020-08-14 18:49:41 +08:00
Joe Thornber 7466cd7182 [functional-tests (rust)] TestDir now creates dir in ./ 
O_DIRECT doesn't work with /tmp/
 2020-08-14 10:54:31 +01:00
Joe Thornber e1cfc3866b [thin_check (rust)] Mappings top level weren't being ref counted 2020-08-13 14:43:19 +01:00
Joe Thornber 092447d17a [thin_check (rust)] remove some dead code 2020-08-13 14:30:04 +01:00
Joe Thornber 2fa732a93c [functional tests] Port some of Hank's sh tests to Rust. 
Run with 'cargo test'
 2020-08-13 14:20:29 +01:00
Joe Thornber a8a2f560ec [build] add symlinks for thin_generate_{mappings/damage} 2020-08-13 13:47:51 +01:00
Joe Thornber b0e7520fbf [thin/superblock (rust)] Unpack flags 2020-08-13 13:46:07 +01:00
Joe Thornber c254ebe384 [functional-tests] Port 3 --auto-repair tests from Scheme to Rust 2020-08-12 13:03:01 +01:00
Joe Thornber 5a16f21199 [functional tests] Fixup a few tests that broke when we bumped the version nr. 2020-08-12 12:33:16 +01:00
Joe Thornber bf202d076b Merge branch '2020-06-13-thin-check-rewrite' 2020-08-12 11:18:28 +01:00
Joe Thornber afa3f2f04d [thin_check (rust)] Rename Spinner -> Reporter 2020-08-12 10:25:06 +01:00
Joe Thornber 544335ae4a [thin_check (rust)] Send all reporting through the Spinner. 
This means the spinner doesn't overwrite messages.
 2020-08-12 09:35:21 +01:00
Joe Thornber 3757e1d947 [thin_check (rust)] check metadata space map 2020-08-12 08:02:29 +01:00
Joe Thornber e65d2dec6f [thin_check (rust)] Add progress bar 2020-08-11 13:44:33 +01:00
Joe Thornber 34425521e2 [thin_check (rust)] change BTreeWalker to use a space map rather than seen bitset 2020-08-11 10:50:43 +01:00
Joe Thornber 50bde693a1 [thin_check (rust)] Factor out pdata/unpack 2020-08-10 15:42:10 +01:00
Joe Thornber 55ee4bfad8 [thin_check (rust)] replace IndexVisitor with a call to btree_to_map 2020-08-10 14:56:39 +01:00
Joe Thornber e28c602c3d [thin_check (rust)] factor out btree_to_map() fn 2020-08-10 14:45:35 +01:00
Joe Thornber cbc9c2c72a [thin_check (rust)] Improve data_sm handling 2020-08-10 12:56:41 +01:00
Joe Thornber d5444d2255 [thin_check (rust)] sm bitmap entries were being unpacked incorrectly. 2020-08-10 12:55:05 +01:00
Joe Thornber b915257e10 [thin_check (rust)] Fix race in btree walking. 
The seen bitset was locked once to test, and separately to insert.
 2020-08-10 12:30:12 +01:00
Joe Thornber 4e4b7ca2b1 [thin_check (rust)] add --sync-io flag 
Makes it easier to switch between engines
 2020-08-10 11:24:50 +01:00
Joe Thornber 0f865856ed [thin_check (rust)] Improve SyncIoEngine. 
Now opens the file multiple times so different threads can do io in parallel.
 2020-08-10 10:44:47 +01:00
Joe Thornber f0df17af9e [thin_check (rust)] Get SyncIoEngine working again. 2020-08-10 08:59:02 +01:00
Joe Thornber 08e3ea948e [thin_check (rust)] rename block_manager.rs -> io_engine.rs 2020-08-10 08:29:32 +01:00
Joe Thornber fd0c0ffc1d [thin_check (rust)] data space map now checked. 2020-08-08 16:42:32 +01:00
Joe Thornber 4054b1be4c [thin_check (rust)] Switch to BTreeMap. 
It's faster.
 2020-08-08 14:58:13 +01:00
Joe Thornber 1e4a038b41 [thin_check (rust)] Reimplement CoreSpaceMap 
We now use a simple vector of elements that can hold 'nr thin devs'.  Much faster.
 2020-08-08 13:29:30 +01:00
Joe Thornber 7cf239b878 [thin_check (rust)] speed up CoreSpaceMap 2020-08-08 12:36:13 +01:00
Joe Thornber ec8f7b7fa8 [thin_check (rust)] Keep track of data block ref counts 
as we walk the mapping tree.
 2020-08-08 09:54:16 +01:00
Joe Thornber 8f76371bb2 [functional-tests] Fix clippy warnings 2020-08-07 15:41:21 +01:00
Joe Thornber fa4ea3e2d9 [functional-tests] port some of the cache_check tests to Rust 2020-08-07 14:30:00 +01:00
Joe Thornber 4a0582bb5d [thin_check (rust)] start decoding the space maps. 2020-08-06 07:51:48 +01:00
Joe Thornber 904d9b0c84 [functional-tests] port thin_metadata_unpack tests to Rust. 
cargo test
 2020-08-05 10:00:44 +01:00
Joe Thornber d2678fdf27 [functional-tests] port thin_metadata_pack tests to Rust. 
cargo test
 2020-08-05 09:09:18 +01:00
Joe Thornber 23568aaa11 [functional-tests] port thin_delta tests to rust. 
cargo test
 2020-08-05 08:24:52 +01:00
Joe Thornber 197e4ffbfd [thin_check (rust)] Rename ValueType trait to Unpack 2020-08-05 08:01:02 +01:00
Joe Thornber 1d44025584 [thin_check (rust)] Walk the top level and bottom level of the mapping tree separately 2020-08-04 12:11:36 +01:00
Joe Thornber 4ac428128a [functional-tests (rust)] port thin_repair tests to rust. 
cargo test
 2020-08-04 11:30:46 +01:00
Joe Thornber f56ea2d031 [thin_check (rust)] walk devices tree. 2020-08-03 16:22:08 +01:00
Joe Thornber 1368227a71 [thin_check (rust)] add btree node checks 2020-08-03 15:04:59 +01:00
Joe Thornber cc2582b8b1 [thin_check (rust)] factor out pdata/btree.rs 2020-08-03 12:37:32 +01:00
Joe Thornber cdf19b2454 [functional-tests (rust)] remove comment 2020-08-03 11:00:09 +01:00
Joe Thornber 39822a7165 [functional-tests (rust)] squash warnings 2020-08-03 10:59:19 +01:00
Joe Thornber 9552cb4817 [functional-tests] Port thin_rmap tests to Rust. 
cargo test
 2020-08-03 10:47:03 +01:00
Joe Thornber ad29fe65fa [functional-tests] Use thin_restore macro 2020-07-31 16:34:04 +01:00
Joe Thornber baf1fe325f [functional-tests] Move thin_dump tests to Rust. 
cargo test
 2020-07-31 16:31:10 +01:00
Joe Thornber 78db9a24fa [functional-tests (rust)] factor out TestDir 2020-07-31 14:26:22 +01:00
Joe Thornber 084a26bf85 [functional-tests] Recode thin_restore tests in Rust. 
Now part of 'cargo test'
 2020-07-31 12:12:40 +01:00
Joe Thornber fcfcc60b89 [functional-tests] Move thin_check functional tests to Rust. 
They'll be run as part of 'cargo test' now.
 2020-07-31 11:04:12 +01:00
Joe Thornber 7243f95380 [thin_check (rust)] Drop nr threads down to 4 
We get very little benefit from threads atm.  Need to improve IO handling first.
 2020-07-30 10:17:36 +01:00
Joe Thornber de172147d3 [thin_check (rust)] squash a lot of warnings 2020-07-30 10:12:51 +01:00
Joe Thornber f7623e6264 [thin_check (rust)] remove spurious mutex. 2020-07-30 09:59:02 +01:00
Joe Thornber 8146fba9d2 [thin_check (rust)] Move mutex inside IoEngines. 
Makes it easier to share an engine between different threads.
 2020-07-30 08:47:16 +01:00
Joe Thornber 4f120911d2 [thin_check (rust)] First attempt at multithreading the tree walking. 
Still using a single io engine, so little benefit.
 2020-07-29 16:38:52 +01:00
Joe Thornber d5597d5d36 [thin_check (rust)] Factor out tree walking code 2020-07-29 11:12:03 +01:00
Joe Thornber e9abdd9c88 [thin_check (rust)] Switch to a different io_uring crate. 
This one works.
 2020-07-28 12:57:30 +01:00
Joe Thornber a90294e279 [thin_check (rust)] read many blocks at once. 
We need to switch to io_uring to really get the benefit of this.
 2020-07-28 11:45:25 +01:00
Joe Thornber 062a1b8a2c [thin_check (rust)] Walk mapping tree. 
Sync IO, no checks beyond checksumming.
 2020-07-28 10:51:48 +01:00
Joe Thornber 1398cf31d1 [thin_check (Rust)] work in progress 2020-07-27 15:53:42 +01:00
Joe Thornber 3cf6307762 Merge branch 'master' into 2020-06-13-thin-check-rewrite 2020-07-27 15:53:26 +01:00
Joe Thornber 5e19029e65 Merge branch '2020-06-19-use-anyhow' into 2020-06-13-thin-check-rewrite 2020-06-22 10:16:27 +01:00
Joe Thornber fdf641aff3 [thin_metadata_{pack,unpak}] use anyhow in toplevel. 2020-06-21 11:50:24 +01:00
Joe Thornber 29d56f62a5 wip 2020-06-14 08:17:46 +01:00
165 changed files with 9817 additions and 7431 deletions
Show Stats Expand all files Collapse all files

1
.gitignore vendored
View File

@ -20,6 +20,7 @@ core
googletest/
bin/pdata_tools
bin/pdata_tools_dev
thin_check
thin_dump
thin_restore

13
.pre-commit-config.yaml Normal file
View File

@ -0,0 +1,13 @@
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v3.4.0
hooks:
- id: check-merge-conflict
- id: end-of-file-fixer
- id: mixed-line-ending
- id: trailing-whitespace
- repo: https://github.com/doublify/pre-commit-rust
rev: master
hooks:
- id: fmt
- id: clippy

567
Cargo.lock generated
View File

@ -1,567 +0,0 @@
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
[[package]]
name = "adler32"
version = "1.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "aho-corasick"
version = "0.7.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"memchr 2.3.3 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "ansi_term"
version = "0.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"winapi 0.3.8 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "anyhow"
version = "1.0.31"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "arrayvec"
version = "0.4.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"nodrop 0.1.14 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "atty"
version = "0.2.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"hermit-abi 0.1.13 (registry+https://github.com/rust-lang/crates.io-index)",
"libc 0.2.71 (registry+https://github.com/rust-lang/crates.io-index)",
"winapi 0.3.8 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "autocfg"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "bitflags"
version = "1.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "byteorder"
version = "1.3.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "cc"
version = "1.0.54"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "cfg-if"
version = "0.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "clap"
version = "2.33.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"ansi_term 0.11.0 (registry+https://github.com/rust-lang/crates.io-index)",
"atty 0.2.14 (registry+https://github.com/rust-lang/crates.io-index)",
"bitflags 1.2.1 (registry+https://github.com/rust-lang/crates.io-index)",
"strsim 0.8.0 (registry+https://github.com/rust-lang/crates.io-index)",
"textwrap 0.11.0 (registry+https://github.com/rust-lang/crates.io-index)",
"unicode-width 0.1.7 (registry+https://github.com/rust-lang/crates.io-index)",
"vec_map 0.8.2 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "crc32c"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "crc32fast"
version = "1.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"cfg-if 0.1.9 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "env_logger"
version = "0.7.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"log 0.4.8 (registry+https://github.com/rust-lang/crates.io-index)",
"regex 1.3.9 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "fixedbitset"
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "flate2"
version = "1.0.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"cfg-if 0.1.9 (registry+https://github.com/rust-lang/crates.io-index)",
"crc32fast 1.2.0 (registry+https://github.com/rust-lang/crates.io-index)",
"libc 0.2.71 (registry+https://github.com/rust-lang/crates.io-index)",
"miniz_oxide 0.3.6 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "getrandom"
version = "0.1.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"cfg-if 0.1.9 (registry+https://github.com/rust-lang/crates.io-index)",
"libc 0.2.71 (registry+https://github.com/rust-lang/crates.io-index)",
"wasi 0.9.0+wasi-snapshot-preview1 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "hermit-abi"
version = "0.1.13"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"libc 0.2.71 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "lazy_static"
version = "1.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "lexical-core"
version = "0.6.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"arrayvec 0.4.12 (registry+https://github.com/rust-lang/crates.io-index)",
"bitflags 1.2.1 (registry+https://github.com/rust-lang/crates.io-index)",
"cfg-if 0.1.9 (registry+https://github.com/rust-lang/crates.io-index)",
"rustc_version 0.2.3 (registry+https://github.com/rust-lang/crates.io-index)",
"ryu 1.0.5 (registry+https://github.com/rust-lang/crates.io-index)",
"static_assertions 0.3.4 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "libc"
version = "0.2.71"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "log"
version = "0.4.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"cfg-if 0.1.9 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "memchr"
version = "2.3.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "miniz_oxide"
version = "0.3.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"adler32 1.0.4 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "nix"
version = "0.17.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"bitflags 1.2.1 (registry+https://github.com/rust-lang/crates.io-index)",
"cc 1.0.54 (registry+https://github.com/rust-lang/crates.io-index)",
"cfg-if 0.1.9 (registry+https://github.com/rust-lang/crates.io-index)",
"libc 0.2.71 (registry+https://github.com/rust-lang/crates.io-index)",
"void 1.0.2 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "nodrop"
version = "0.1.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "nom"
version = "5.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"lexical-core 0.6.7 (registry+https://github.com/rust-lang/crates.io-index)",
"memchr 2.3.3 (registry+https://github.com/rust-lang/crates.io-index)",
"version_check 0.9.2 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "num-derive"
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"proc-macro2 1.0.18 (registry+https://github.com/rust-lang/crates.io-index)",
"quote 1.0.6 (registry+https://github.com/rust-lang/crates.io-index)",
"syn 1.0.30 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "num-traits"
version = "0.2.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"autocfg 1.0.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "num_cpus"
version = "1.13.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"hermit-abi 0.1.13 (registry+https://github.com/rust-lang/crates.io-index)",
"libc 0.2.71 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "ppv-lite86"
version = "0.2.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "proc-macro2"
version = "1.0.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"unicode-xid 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "quick-xml"
version = "0.18.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"memchr 2.3.3 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "quickcheck"
version = "0.9.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"env_logger 0.7.1 (registry+https://github.com/rust-lang/crates.io-index)",
"log 0.4.8 (registry+https://github.com/rust-lang/crates.io-index)",
"rand 0.7.3 (registry+https://github.com/rust-lang/crates.io-index)",
"rand_core 0.5.1 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "quickcheck_macros"
version = "0.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"proc-macro2 1.0.18 (registry+https://github.com/rust-lang/crates.io-index)",
"quote 1.0.6 (registry+https://github.com/rust-lang/crates.io-index)",
"syn 1.0.30 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "quote"
version = "1.0.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"proc-macro2 1.0.18 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "rand"
version = "0.7.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"getrandom 0.1.14 (registry+https://github.com/rust-lang/crates.io-index)",
"libc 0.2.71 (registry+https://github.com/rust-lang/crates.io-index)",
"rand_chacha 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)",
"rand_core 0.5.1 (registry+https://github.com/rust-lang/crates.io-index)",
"rand_hc 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "rand_chacha"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"ppv-lite86 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)",
"rand_core 0.5.1 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "rand_core"
version = "0.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"getrandom 0.1.14 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "rand_hc"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"rand_core 0.5.1 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "redox_syscall"
version = "0.1.56"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "regex"
version = "1.3.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"aho-corasick 0.7.10 (registry+https://github.com/rust-lang/crates.io-index)",
"memchr 2.3.3 (registry+https://github.com/rust-lang/crates.io-index)",
"regex-syntax 0.6.18 (registry+https://github.com/rust-lang/crates.io-index)",
"thread_local 1.0.1 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "regex-syntax"
version = "0.6.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "remove_dir_all"
version = "0.5.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"winapi 0.3.8 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "rustc_version"
version = "0.2.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"semver 0.9.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "ryu"
version = "1.0.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "semver"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"semver-parser 0.7.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "semver-parser"
version = "0.7.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "static_assertions"
version = "0.3.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "strsim"
version = "0.8.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "syn"
version = "1.0.30"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"proc-macro2 1.0.18 (registry+https://github.com/rust-lang/crates.io-index)",
"quote 1.0.6 (registry+https://github.com/rust-lang/crates.io-index)",
"unicode-xid 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "tempfile"
version = "3.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"cfg-if 0.1.9 (registry+https://github.com/rust-lang/crates.io-index)",
"libc 0.2.71 (registry+https://github.com/rust-lang/crates.io-index)",
"rand 0.7.3 (registry+https://github.com/rust-lang/crates.io-index)",
"redox_syscall 0.1.56 (registry+https://github.com/rust-lang/crates.io-index)",
"remove_dir_all 0.5.3 (registry+https://github.com/rust-lang/crates.io-index)",
"winapi 0.3.8 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "textwrap"
version = "0.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"unicode-width 0.1.7 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "thinp"
version = "0.1.0"
dependencies = [
"anyhow 1.0.31 (registry+https://github.com/rust-lang/crates.io-index)",
"byteorder 1.3.4 (registry+https://github.com/rust-lang/crates.io-index)",
"clap 2.33.1 (registry+https://github.com/rust-lang/crates.io-index)",
"crc32c 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)",
"fixedbitset 0.3.0 (registry+https://github.com/rust-lang/crates.io-index)",
"flate2 1.0.14 (registry+https://github.com/rust-lang/crates.io-index)",
"libc 0.2.71 (registry+https://github.com/rust-lang/crates.io-index)",
"nix 0.17.0 (registry+https://github.com/rust-lang/crates.io-index)",
"nom 5.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
"num-derive 0.3.0 (registry+https://github.com/rust-lang/crates.io-index)",
"num-traits 0.2.11 (registry+https://github.com/rust-lang/crates.io-index)",
"num_cpus 1.13.0 (registry+https://github.com/rust-lang/crates.io-index)",
"quick-xml 0.18.1 (registry+https://github.com/rust-lang/crates.io-index)",
"quickcheck 0.9.2 (registry+https://github.com/rust-lang/crates.io-index)",
"quickcheck_macros 0.9.1 (registry+https://github.com/rust-lang/crates.io-index)",
"rand 0.7.3 (registry+https://github.com/rust-lang/crates.io-index)",
"tempfile 3.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "thread_local"
version = "1.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"lazy_static 1.4.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "unicode-width"
version = "0.1.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "unicode-xid"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "vec_map"
version = "0.8.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "version_check"
version = "0.9.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "void"
version = "1.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "wasi"
version = "0.9.0+wasi-snapshot-preview1"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "winapi"
version = "0.3.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"winapi-i686-pc-windows-gnu 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)",
"winapi-x86_64-pc-windows-gnu 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "winapi-i686-pc-windows-gnu"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "winapi-x86_64-pc-windows-gnu"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
[metadata]
"checksum adler32 1.0.4 (registry+https://github.com/rust-lang/crates.io-index)" = "5d2e7343e7fc9de883d1b0341e0b13970f764c14101234857d2ddafa1cb1cac2"
"checksum aho-corasick 0.7.10 (registry+https://github.com/rust-lang/crates.io-index)" = "8716408b8bc624ed7f65d223ddb9ac2d044c0547b6fa4b0d554f3a9540496ada"
"checksum ansi_term 0.11.0 (registry+https://github.com/rust-lang/crates.io-index)" = "ee49baf6cb617b853aa8d93bf420db2383fab46d314482ca2803b40d5fde979b"
"checksum anyhow 1.0.31 (registry+https://github.com/rust-lang/crates.io-index)" = "85bb70cc08ec97ca5450e6eba421deeea5f172c0fc61f78b5357b2a8e8be195f"
"checksum arrayvec 0.4.12 (registry+https://github.com/rust-lang/crates.io-index)" = "cd9fd44efafa8690358b7408d253adf110036b88f55672a933f01d616ad9b1b9"
"checksum atty 0.2.14 (registry+https://github.com/rust-lang/crates.io-index)" = "d9b39be18770d11421cdb1b9947a45dd3f37e93092cbf377614828a319d5fee8"
"checksum autocfg 1.0.0 (registry+https://github.com/rust-lang/crates.io-index)" = "f8aac770f1885fd7e387acedd76065302551364496e46b3dd00860b2f8359b9d"
"checksum bitflags 1.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "cf1de2fe8c75bc145a2f577add951f8134889b4795d47466a54a5c846d691693"
"checksum byteorder 1.3.4 (registry+https://github.com/rust-lang/crates.io-index)" = "08c48aae112d48ed9f069b33538ea9e3e90aa263cfa3d1c24309612b1f7472de"
"checksum cc 1.0.54 (registry+https://github.com/rust-lang/crates.io-index)" = "7bbb73db36c1246e9034e307d0fba23f9a2e251faa47ade70c1bd252220c8311"
"checksum cfg-if 0.1.9 (registry+https://github.com/rust-lang/crates.io-index)" = "b486ce3ccf7ffd79fdeb678eac06a9e6c09fc88d33836340becb8fffe87c5e33"
"checksum clap 2.33.1 (registry+https://github.com/rust-lang/crates.io-index)" = "bdfa80d47f954d53a35a64987ca1422f495b8d6483c0fe9f7117b36c2a792129"
"checksum crc32c 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)" = "77ba37ef26c12988c1cee882d522d65e1d5d2ad8c3864665b88ee92767ed84c5"
"checksum crc32fast 1.2.0 (registry+https://github.com/rust-lang/crates.io-index)" = "ba125de2af0df55319f41944744ad91c71113bf74a4646efff39afe1f6842db1"
"checksum env_logger 0.7.1 (registry+https://github.com/rust-lang/crates.io-index)" = "44533bbbb3bb3c1fa17d9f2e4e38bbbaf8396ba82193c4cb1b6445d711445d36"
"checksum fixedbitset 0.3.0 (registry+https://github.com/rust-lang/crates.io-index)" = "2fc4fcacf5cd3681968f6524ea159383132937739c6c40dabab9e37ed515911b"
"checksum flate2 1.0.14 (registry+https://github.com/rust-lang/crates.io-index)" = "2cfff41391129e0a856d6d822600b8d71179d46879e310417eb9c762eb178b42"
"checksum getrandom 0.1.14 (registry+https://github.com/rust-lang/crates.io-index)" = "7abc8dd8451921606d809ba32e95b6111925cd2906060d2dcc29c070220503eb"
"checksum hermit-abi 0.1.13 (registry+https://github.com/rust-lang/crates.io-index)" = "91780f809e750b0a89f5544be56617ff6b1227ee485bcb06ebe10cdf89bd3b71"
"checksum lazy_static 1.4.0 (registry+https://github.com/rust-lang/crates.io-index)" = "e2abad23fbc42b3700f2f279844dc832adb2b2eb069b2df918f455c4e18cc646"
"checksum lexical-core 0.6.7 (registry+https://github.com/rust-lang/crates.io-index)" = "f86d66d380c9c5a685aaac7a11818bdfa1f733198dfd9ec09c70b762cd12ad6f"
"checksum libc 0.2.71 (registry+https://github.com/rust-lang/crates.io-index)" = "9457b06509d27052635f90d6466700c65095fdf75409b3fbdd903e988b886f49"
"checksum log 0.4.8 (registry+https://github.com/rust-lang/crates.io-index)" = "14b6052be84e6b71ab17edffc2eeabf5c2c3ae1fdb464aae35ac50c67a44e1f7"
"checksum memchr 2.3.3 (registry+https://github.com/rust-lang/crates.io-index)" = "3728d817d99e5ac407411fa471ff9800a778d88a24685968b36824eaf4bee400"
"checksum miniz_oxide 0.3.6 (registry+https://github.com/rust-lang/crates.io-index)" = "aa679ff6578b1cddee93d7e82e263b94a575e0bfced07284eb0c037c1d2416a5"
"checksum nix 0.17.0 (registry+https://github.com/rust-lang/crates.io-index)" = "50e4785f2c3b7589a0d0c1dd60285e1188adac4006e8abd6dd578e1567027363"
"checksum nodrop 0.1.14 (registry+https://github.com/rust-lang/crates.io-index)" = "72ef4a56884ca558e5ddb05a1d1e7e1bfd9a68d9ed024c21704cc98872dae1bb"
"checksum nom 5.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "0b471253da97532da4b61552249c521e01e736071f71c1a4f7ebbfbf0a06aad6"
"checksum num-derive 0.3.0 (registry+https://github.com/rust-lang/crates.io-index)" = "0c8b15b261814f992e33760b1fca9fe8b693d8a65299f20c9901688636cfb746"
"checksum num-traits 0.2.11 (registry+https://github.com/rust-lang/crates.io-index)" = "c62be47e61d1842b9170f0fdeec8eba98e60e90e5446449a0545e5152acd7096"
"checksum num_cpus 1.13.0 (registry+https://github.com/rust-lang/crates.io-index)" = "05499f3756671c15885fee9034446956fff3f243d6077b91e5767df161f766b3"
"checksum ppv-lite86 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)" = "237a5ed80e274dbc66f86bd59c1e25edc039660be53194b5fe0a482e0f2612ea"
"checksum proc-macro2 1.0.18 (registry+https://github.com/rust-lang/crates.io-index)" = "beae6331a816b1f65d04c45b078fd8e6c93e8071771f41b8163255bbd8d7c8fa"
"checksum quick-xml 0.18.1 (registry+https://github.com/rust-lang/crates.io-index)" = "3cc440ee4802a86e357165021e3e255a9143724da31db1e2ea540214c96a0f82"
"checksum quickcheck 0.9.2 (registry+https://github.com/rust-lang/crates.io-index)" = "a44883e74aa97ad63db83c4bf8ca490f02b2fc02f92575e720c8551e843c945f"
"checksum quickcheck_macros 0.9.1 (registry+https://github.com/rust-lang/crates.io-index)" = "608c156fd8e97febc07dc9c2e2c80bf74cfc6ef26893eae3daf8bc2bc94a4b7f"
"checksum quote 1.0.6 (registry+https://github.com/rust-lang/crates.io-index)" = "54a21852a652ad6f610c9510194f398ff6f8692e334fd1145fed931f7fbe44ea"
"checksum rand 0.7.3 (registry+https://github.com/rust-lang/crates.io-index)" = "6a6b1679d49b24bbfe0c803429aa1874472f50d9b363131f0e89fc356b544d03"
"checksum rand_chacha 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)" = "f4c8ed856279c9737206bf725bf36935d8666ead7aa69b52be55af369d193402"
"checksum rand_core 0.5.1 (registry+https://github.com/rust-lang/crates.io-index)" = "90bde5296fc891b0cef12a6d03ddccc162ce7b2aff54160af9338f8d40df6d19"
"checksum rand_hc 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)" = "ca3129af7b92a17112d59ad498c6f81eaf463253766b90396d39ea7a39d6613c"
"checksum redox_syscall 0.1.56 (registry+https://github.com/rust-lang/crates.io-index)" = "2439c63f3f6139d1b57529d16bc3b8bb855230c8efcc5d3a896c8bea7c3b1e84"
"checksum regex 1.3.9 (registry+https://github.com/rust-lang/crates.io-index)" = "9c3780fcf44b193bc4d09f36d2a3c87b251da4a046c87795a0d35f4f927ad8e6"
"checksum regex-syntax 0.6.18 (registry+https://github.com/rust-lang/crates.io-index)" = "26412eb97c6b088a6997e05f69403a802a92d520de2f8e63c2b65f9e0f47c4e8"
"checksum remove_dir_all 0.5.3 (registry+https://github.com/rust-lang/crates.io-index)" = "3acd125665422973a33ac9d3dd2df85edad0f4ae9b00dafb1a05e43a9f5ef8e7"
"checksum rustc_version 0.2.3 (registry+https://github.com/rust-lang/crates.io-index)" = "138e3e0acb6c9fb258b19b67cb8abd63c00679d2851805ea151465464fe9030a"
"checksum ryu 1.0.5 (registry+https://github.com/rust-lang/crates.io-index)" = "71d301d4193d031abdd79ff7e3dd721168a9572ef3fe51a1517aba235bd8f86e"
"checksum semver 0.9.0 (registry+https://github.com/rust-lang/crates.io-index)" = "1d7eb9ef2c18661902cc47e535f9bc51b78acd254da71d375c2f6720d9a40403"
"checksum semver-parser 0.7.0 (registry+https://github.com/rust-lang/crates.io-index)" = "388a1df253eca08550bef6c72392cfe7c30914bf41df5269b68cbd6ff8f570a3"
"checksum static_assertions 0.3.4 (registry+https://github.com/rust-lang/crates.io-index)" = "7f3eb36b47e512f8f1c9e3d10c2c1965bc992bd9cdb024fa581e2194501c83d3"
"checksum strsim 0.8.0 (registry+https://github.com/rust-lang/crates.io-index)" = "8ea5119cdb4c55b55d432abb513a0429384878c15dde60cc77b1c99de1a95a6a"
"checksum syn 1.0.30 (registry+https://github.com/rust-lang/crates.io-index)" = "93a56fabc59dce20fe48b6c832cc249c713e7ed88fa28b0ee0a3bfcaae5fe4e2"
"checksum tempfile 3.1.0 (registry+https://github.com/rust-lang/crates.io-index)" = "7a6e24d9338a0a5be79593e2fa15a648add6138caa803e2d5bc782c371732ca9"
"checksum textwrap 0.11.0 (registry+https://github.com/rust-lang/crates.io-index)" = "d326610f408c7a4eb6f51c37c330e496b08506c9457c9d34287ecc38809fb060"
"checksum thread_local 1.0.1 (registry+https://github.com/rust-lang/crates.io-index)" = "d40c6d1b69745a6ec6fb1ca717914848da4b44ae29d9b3080cbee91d72a69b14"
"checksum unicode-width 0.1.7 (registry+https://github.com/rust-lang/crates.io-index)" = "caaa9d531767d1ff2150b9332433f32a24622147e5ebb1f26409d5da67afd479"
"checksum unicode-xid 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)" = "826e7639553986605ec5979c7dd957c7895e93eabed50ab2ffa7f6128a75097c"
"checksum vec_map 0.8.2 (registry+https://github.com/rust-lang/crates.io-index)" = "f1bddf1187be692e79c5ffeab891132dfb0f236ed36a43c7ed39f1165ee20191"
"checksum version_check 0.9.2 (registry+https://github.com/rust-lang/crates.io-index)" = "b5a972e5669d67ba988ce3dc826706fb0a8b01471c088cb0b6110b805cc36aed"
"checksum void 1.0.2 (registry+https://github.com/rust-lang/crates.io-index)" = "6a02e4885ed3bc0f2de90ea6dd45ebcbb66dacffe03547fadbb0eeae2770887d"
"checksum wasi 0.9.0+wasi-snapshot-preview1 (registry+https://github.com/rust-lang/crates.io-index)" = "cccddf32554fecc6acb585f82a32a72e28b48f8c4c1883ddfeeeaa96f7d8e519"
"checksum winapi 0.3.8 (registry+https://github.com/rust-lang/crates.io-index)" = "8093091eeb260906a183e6ae1abdba2ef5ef2257a21801128899c3fc699229c6"
"checksum winapi-i686-pc-windows-gnu 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)" = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6"
"checksum winapi-x86_64-pc-windows-gnu 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)" = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"

27
Cargo.toml
View File

@ -1,27 +0,0 @@
[package]
name = "thinp"
version = "0.1.0"
authors = ["Joe Thornber <ejt@redhat.com>"]
edition = "2018"
license = "GPL3"
[dependencies]
anyhow = "1.0"
byteorder = "1.3"
clap = "2.33"
crc32c = "0.4"
flate2 = "1.0"
fixedbitset = "0.3"
libc = "0.2.71"
quick-xml = "0.18"
nix = "0.17"
nom = "5.1"
num_cpus = "1.13"
rand = "0.7"
tempfile = "3.1"
num-traits = "0.2"
num-derive = "0.3"
[dev-dependencies]
quickcheck = "0.9"
quickcheck_macros = "0.9"

160
Makefile.in
View File

@ -20,18 +20,22 @@ V=@
PROGRAMS=\
bin/pdata_tools
ifeq ("@TESTING@", "yes")
DEV_TOOLS=\
bin/pdata_tools_dev
TESTLIBS=\
lib/libft.so
endif
.PHONY: all
all: $(PROGRAMS) $(TESTLIBS)
.PHONY: all dev-tools
all: $(PROGRAMS)
dev-tools: $(DEV_TOOLS)
ifeq ("@TESTING@", "yes")
all += $(TESTLIB)
endif
include contrib/Makefile
SOURCE=\
COMMON_SOURCE=\
base/output_file_requirements.cc \
base/application.cc \
base/base64.cc \
@ -50,13 +54,6 @@ SOURCE=\
block-cache/copier.cc \
block-cache/io_engine.cc \
block-cache/mem_pool.cc \
caching/cache_check.cc \
caching/cache_dump.cc \
caching/cache_metadata_size.cc \
caching/cache_repair.cc \
caching/cache_restore.cc \
caching/cache_writeback.cc \
caching/commands.cc \
caching/hint_array.cc \
caching/mapping_array.cc \
caching/metadata.cc \
@ -64,13 +61,8 @@ SOURCE=\
caching/restore_emitter.cc \
caching/superblock.cc \
caching/xml_format.cc \
era/commands.cc \
era/era_array.cc \
era/era_check.cc \
era/era_detail.cc \
era/era_dump.cc \
era/era_invalidate.cc \
era/era_restore.cc \
era/metadata.cc \
era/metadata_dump.cc \
era/restore_emitter.cc \
@ -95,11 +87,7 @@ SOURCE=\
persistent-data/space_map.cc \
persistent-data/transaction_manager.cc \
persistent-data/validators.cc \
thin-provisioning/commands.cc \
thin-provisioning/cache_stream.cc \
thin-provisioning/chunk_stream.cc \
thin-provisioning/device_tree.cc \
thin-provisioning/fixed_chunk_stream.cc \
thin-provisioning/human_readable_format.cc \
thin-provisioning/mapping_tree.cc \
thin-provisioning/metadata.cc \
@ -107,40 +95,71 @@ SOURCE=\
thin-provisioning/metadata_counter.cc \
thin-provisioning/metadata_dumper.cc \
thin-provisioning/override_emitter.cc \
thin-provisioning/pool_stream.cc \
thin-provisioning/restore_emitter.cc \
thin-provisioning/rmap_visitor.cc \
thin-provisioning/superblock.cc \
thin-provisioning/xml_format.cc
TOOLS_SOURCE=\
caching/commands.cc \
caching/cache_check.cc \
caching/cache_dump.cc \
caching/cache_metadata_size.cc \
caching/cache_repair.cc \
caching/cache_restore.cc \
caching/cache_writeback.cc \
era/commands.cc \
era/era_check.cc \
era/era_dump.cc \
era/era_invalidate.cc \
era/era_restore.cc \
thin-provisioning/commands.cc \
thin-provisioning/thin_check.cc \
thin-provisioning/thin_delta.cc \
thin-provisioning/thin_dump.cc \
thin-provisioning/thin_ls.cc \
thin-provisioning/thin_metadata_size.cc \
thin-provisioning/thin_pool.cc \
thin-provisioning/thin_repair.cc \
thin-provisioning/thin_restore.cc \
thin-provisioning/thin_rmap.cc \
thin-provisioning/thin_trim.cc \
thin-provisioning/xml_format.cc
thin-provisioning/thin_trim.cc
DEVTOOLS_SOURCE=\
caching/cache_debug.cc \
caching/devel_commands.cc \
dbg-lib/bitset_block_dumper.cc \
dbg-lib/command_interpreter.cc \
dbg-lib/commands.cc \
dbg-lib/index_block_dumper.cc \
dbg-lib/output_formatter.cc \
dbg-lib/simple_show_traits.cc \
dbg-lib/sm_show_traits.cc \
era/devel_commands.cc \
era/era_debug.cc \
thin-provisioning/cache_stream.cc \
thin-provisioning/chunk_stream.cc \
thin-provisioning/damage_generator.cc \
thin-provisioning/devel_commands.cc \
thin-provisioning/fixed_chunk_stream.cc \
thin-provisioning/pool_stream.cc \
thin-provisioning/thin_debug.cc \
thin-provisioning/thin_generate_damage.cc \
thin-provisioning/thin_generate_mappings.cc \
thin-provisioning/thin_generate_metadata.cc \
thin-provisioning/thin_journal.cc \
thin-provisioning/thin_journal_check.cc \
thin-provisioning/thin_ll_dump.cc \
thin-provisioning/thin_ll_restore.cc \
thin-provisioning/thin_show_duplicates.cc \
thin-provisioning/thin_generate_damage.cc \
thin-provisioning/thin_generate_metadata.cc \
thin-provisioning/thin_generate_mappings.cc \
thin-provisioning/variable_chunk_stream.cc \
thin-provisioning/thin_show_metadata.cc \
thin-provisioning/thin_patch_superblock.cc \
thin-provisioning/thin_pool.cc \
thin-provisioning/thin_scan.cc \
thin-provisioning/thin_show_duplicates.cc \
thin-provisioning/thin_show_metadata.cc \
thin-provisioning/variable_chunk_stream.cc \
ui/ui.cc
ifeq ("@DEVTOOLS@", "yes")
SOURCE+=$(DEVTOOLS_SOURCE)
endif
SOURCE=$(COMMON_SOURCE) $(TOOLS_SOURCE)
DEV_SOURCE=$(COMMON_SOURCE) $(DEVTOOLS_SOURCE)
ifeq ("@STATIC@", "yes")
SOURCE += thin-provisioning/static_library_emitter.cc
@ -153,6 +172,7 @@ CXX:=@CXX@
AR:=@AR@
STRIP:=@STRIP@
OBJECTS:=$(subst .cc,.o,$(SOURCE))
DEV_OBJECTS:=$(subst .cc,.o,$(DEV_SOURCE))
ifeq ("@STATIC@", "yes")
EMITTERS += $(PLUGIN_LIBS)
@ -164,20 +184,14 @@ CFLAGS+=-g -Wall -O3 -fPIC
CFLAGS+=@LFS_FLAGS@
CXXFLAGS+=-g -Wall -fPIC -fno-strict-aliasing -std=c++11
ifeq ("@DEVTOOLS@", "yes")
CXXFLAGS+=-DDEV_TOOLS
endif
CXXFLAGS+=@CXXOPTIMISE_FLAG@
CXXFLAGS+=@CXXDEBUG_FLAG@
CXXFLAGS+=@CXX_STRERROR_FLAG@
CXXFLAGS+=@LFS_FLAGS@
INCLUDES+=-I$(TOP_BUILDDIR) -I$(TOP_DIR) -I$(TOP_DIR)/thin-provisioning
LIBS:=-laio -lexpat -lboost_iostreams -ldl
ifeq ("@DEVTOOLS@", "yes")
LIBS+=-lncurses
endif
CPPFLAGS?=@CPPFLAGS@
CPPFLAGS+=-I$(TOP_BUILDDIR) -I$(TOP_DIR)
LIBS:=-laio -lexpat -ldl
DEV_LIBS:=-lncurses
ifeq ("@STATIC_CXX@", "yes")
CXXLIB+=-Wl,-Bstatic -lstdc++ -Wl,-Bdynamic -Wl,--as-needed
@ -201,37 +215,29 @@ INSTALL_DIR = $(INSTALL) -m 755 -d
INSTALL_PROGRAM = $(INSTALL) -m 755
INSTALL_DATA = $(INSTALL) -p -m 644
ifeq ("@TESTING@", "yes")
TEST_INCLUDES=\
-I$(TOP_DIR) \
-Igoogletest/googlemock/include \
-Igoogletest/googletest/include
else
TEST_INCLUDES=
endif
.SUFFIXES: .d .txt .8
%.o: %.cc
@echo " [CXX] $<"
@mkdir -p $(dir $@)
$(V) $(CXX) -c $(INCLUDES) $(CXXFLAGS) -o $@ $<
$(V) $(CXX) -MM -MT $(subst .cc,.o,$<) $(INCLUDES) $(TEST_INCLUDES) $(CXXFLAGS) $< > $*.$$$$; \
$(V) $(CXX) -c $(CPPFLAGS) $(CXXFLAGS) -o $@ $<
$(V) $(CXX) -MM -MT $(subst .cc,.o,$<) $(CPPFLAGS) $(CXXFLAGS) $< > $*.$$$$; \
sed 's,\([^ :]*\)\.o[ :]*,\1.o \1.gmo $* : Makefile ,g' < $*.$$$$ > $*.d; \
$(RM) $*.$$$$
%.o: %.c
@echo " [CC] $<"
@mkdir -p $(dir $@)
$(V) $(CC) -c $(INCLUDES) $(CFLAGS) -o $@ $<
$(V) $(CC) -MM -MT $(subst .cc,.o,$<) $(INCLUDES) $(TEST_INCLUDES) $(CFLAGS) $< > $*.$$$$; \
$(V) $(CC) -c $(CPPFLAGS) $(CFLAGS) -o $@ $<
$(V) $(CC) -MM -MT $(subst .cc,.o,$<) $(CPPFLAGS) $(CFLAGS) $< > $*.$$$$; \
sed 's,\([^ :]*\)\.o[ :]*,\1.o \1.gmo $* : Makefile ,g' < $*.$$$$ > $*.d; \
$(RM) $*.$$$$
%.8: %.txt bin/txt2man
@echo " [txt2man] $<"
@mkdir -p $(dir $@)
$(V) bin/txt2man -p -t $(basename $(notdir $<)) $< > $@
$(V) bin/txt2man -t $(basename $(notdir $<)) \
-s 8 -v "System Manager's Manual" -r "Device Mapper Tools" $< > $@
#----------------------------------------------------------------
@ -245,10 +251,16 @@ bin/pdata_tools: $(OBJECTS) $(EMITTERS)
@mkdir -p $(dir $@)
$(V) $(CXX) $(CXXFLAGS) $(LDFLAGS) -o $@ $+ $(LIBS) $(CXXLIB)
bin/pdata_tools_dev: $(DEV_OBJECTS)
@echo " [LD] $@"
@mkdir -p $(dir $@)
$(V) $(CXX) $(CXXFLAGS) $(LDFLAGS) -o $@ $+ $(LIBS) $(DEV_LIBS) $(CXXLIB)
#----------------------------------------------------------------
DEPEND_FILES=\
$(subst .cc,.d,$(SOURCE)) \
$(subst .cc,.d,$(DEV_SOURCE)) \
$(subst .cc,.d,$(TEST_SOURCE)) \
$(subst .cc,.d,$(CXX_PROGRAM_SOURCE)) \
$(subst .c,.d,$(C_PROGRAM_SOURCE))
@ -259,7 +271,8 @@ clean:
find . -name \*.o -delete
find . -name \*.gmo -delete
find . -name \*.d -delete
$(RM) $(TEST_PROGRAMS) $(PROGRAMS) $(GMOCK_OBJECTS) lib/*.a lib/*.so
$(RM) $(PROGRAMS) $(DEV_TOOLS) lib/*.a lib/*.so
$(RM) man8/*.8
distclean: clean
$(RM) config.cache config.log config.status configure.h version.h Makefile unit-tests/Makefile
@ -330,28 +343,11 @@ install: bin/pdata_tools $(MANPAGES)
$(INSTALL_DATA) man8/era_restore.8 $(MANPATH)/man8
$(INSTALL_DATA) man8/era_invalidate.8 $(MANPATH)/man8
$(INSTALL_DATA) man8/thin_trim.8 $(MANPATH)/man8
ifeq ("@DEVTOOLS@", "yes")
ln -s -f pdata_tools $(BINDIR)/thin_show_duplicates
ln -s -f pdata_tools $(BINDIR)/thin_ll_dump
ln -s -f pdata_tools $(BINDIR)/thin_show_duplicates
ln -s -f pdata_tools $(BINDIR)/thin_generate_metadata
ln -s -f pdata_tools $(BINDIR)/thin_scan
endif
.PHONY: install install-rust-tools rust-tools
.PHONY: install
rust-tools:
cargo build --release
#----------------------------------------------------------------
install-rust-tools: man8/thin_metadata_pack.8 man8/thin_metadata_unpack.8 rust-tools
$(INSTALL_PROGRAM) target/release/thin_metadata_pack $(BINDIR)
$(INSTALL_PROGRAM) target/release/thin_metadata_unpack $(BINDIR)
$(STRIP) $(BINDIR)/thin_metadata_pack
$(STRIP) $(BINDIR)/thin_metadata_unpack
$(INSTALL_DATA) man8/thin_metadata_pack.8 $(MANPATH)/man8
$(INSTALL_DATA) man8/thin_metadata_unpack.8 $(MANPATH)/man8
ifeq ("@TESTING@", "yes")
include unit-tests/Makefile
LIBFT_SOURCE=\
@ -367,11 +363,11 @@ lib/libft.so: $(LIBFT_OBJECTS)
.PHONEY: functional-test unit-test
functional-test: bin/pdata_tools lib/libft.so
functional-test: $(PROGRAMS) $(DEV_TOOLS) $(TESTLIBS)
cd functional-tests && ./run-tests run
test: functional-test unit-test
endif
#----------------------------------------------------------------
-include $(DEPEND_FILES)

1301
autoconf/config.guess vendored
View File

File diff suppressed because it is too large Load Diff

2899
autoconf/config.sub vendored
View File

File diff suppressed because it is too large Load Diff

10
base/application.cc
View File

@ -25,8 +25,14 @@ command::die(string const &msg)
}
::uint64_t
command::parse_uint64(string const &str, string const &desc)
command::parse_uint64(char const *str, char const *desc)
{
if (!str) {
ostringstream out;
out << "Couldn't parse " << desc << ": NULL";
die(out.str());
}
try {
// FIXME: check trailing garbage is handled
return lexical_cast<::uint64_t>(str);
@ -47,7 +53,7 @@ application::run(int argc, char **argv)
{
string cmd = get_basename(argv[0]);
if (cmd == string("pdata_tools")) {
if (cmd.find("pdata_tools") == 0) {
argc--;
argv++;

2
base/application.h
View File

@ -19,7 +19,7 @@ namespace base {
virtual ~command() {}
void die(std::string const &msg);
uint64_t parse_uint64(std::string const &str, std::string const &desc);
uint64_t parse_uint64(char const *str, char const *desc);
virtual void usage(std::ostream &out) const = 0;

35
base/file_utils.cc
View File

@ -71,11 +71,18 @@ void
file_utils::check_file_exists(string const &file, bool must_be_regular_file) {
struct stat info;
int r = ::stat(file.c_str(), &info);
if (r)
throw runtime_error("Couldn't stat file");
if (r) {
ostringstream msg;
msg << file << ": " << base::error_string(errno);
throw runtime_error(msg.str());
}
if (must_be_regular_file && !S_ISREG(info.st_mode))
throw runtime_error("Not a regular file");
if (!S_ISREG(info.st_mode)) {
if (must_be_regular_file)
throw runtime_error("Not a regular file");
if (!S_ISBLK(info.st_mode))
throw runtime_error("Not a block device or regular file");
}
}
file_utils::file_descriptor
@ -116,8 +123,11 @@ file_utils::get_file_length(string const &file) {
uint64_t nr_bytes;
int r = ::stat(file.c_str(), &info);
if (r)
throw runtime_error("Couldn't stat path");
if (r) {
ostringstream msg;
msg << file << ": " << base::error_string(errno);
throw runtime_error(msg.str());
}
if (S_ISREG(info.st_mode))
// It's okay to cast st_size to a uint64_t value.
@ -136,9 +146,11 @@ file_utils::get_file_length(string const &file) {
throw runtime_error("ioctl BLKGETSIZE64 failed");
}
::close(fd);
} else
// FIXME: needs a better message
throw runtime_error("bad path");
} else {
ostringstream msg;
msg << file << ": " << "Not a block device or regular file";
throw runtime_error(msg.str());
}
return nr_bytes;
}
@ -155,8 +167,11 @@ file_utils::zero_superblock(std::string const &path)
throw runtime_error("out of memory");
memset(buffer, 0, SUPERBLOCK_SIZE);
if (::write(fd.fd_, buffer, SUPERBLOCK_SIZE) != SUPERBLOCK_SIZE)
if (::write(fd.fd_, buffer, SUPERBLOCK_SIZE) != SUPERBLOCK_SIZE) {
free(buffer);
throw runtime_error("couldn't zero superblock");
}
free(buffer);
}
//----------------------------------------------------------------

1
bin/cache_check
View File

@ -1 +0,0 @@
pdata_tools

1
bin/cache_dump
View File

@ -1 +0,0 @@
pdata_tools

1
bin/cache_metadata_size
View File

@ -1 +0,0 @@
pdata_tools

1
bin/cache_repair
View File

@ -1 +0,0 @@
pdata_tools

1
bin/cache_restore
View File

@ -1 +0,0 @@
pdata_tools

1
bin/era_check
View File

@ -1 +0,0 @@
pdata_tools

1
bin/era_dump
View File

@ -1 +0,0 @@
pdata_tools

1
bin/era_invalidate
View File

@ -1 +0,0 @@
pdata_tools

1
bin/era_restore
View File

@ -1 +0,0 @@
pdata_tools

1
bin/thin_check
View File

@ -1 +0,0 @@
pdata_tools

1
bin/thin_delta
View File

@ -1 +0,0 @@
pdata_tools

1
bin/thin_dump
View File

@ -1 +0,0 @@
pdata_tools

1
bin/thin_generate_metadata
View File

@ -1 +0,0 @@
pdata_tools

1
bin/thin_journal_check
View File

@ -1 +0,0 @@
pdata_tools

1
bin/thin_ll_dump
View File

@ -1 +0,0 @@
pdata_tools

1
bin/thin_ll_restore
View File

@ -1 +0,0 @@
pdata_tools

1
bin/thin_ls
View File

@ -1 +0,0 @@
pdata_tools

1
bin/thin_metadata_pack
View File

@ -1 +0,0 @@
../target/release/thin_metadata_pack

1
bin/thin_metadata_size
View File

@ -1 +0,0 @@
pdata_tools

1
bin/thin_metadata_unpack
View File

@ -1 +0,0 @@
../target/release/thin_metadata_unpack

1
bin/thin_repair
View File

@ -1 +0,0 @@
pdata_tools

1
bin/thin_restore
View File

@ -1 +0,0 @@
pdata_tools

1
bin/thin_rmap
View File

@ -1 +0,0 @@
pdata_tools

1
bin/thin_scan
View File

@ -1 +0,0 @@
pdata_tools

1
bin/thin_show_duplicates
View File

@ -1 +0,0 @@
pdata_tools

1
bin/thin_show_metadata
View File

@ -1 +0,0 @@
pdata_tools

168
bin/txt2man
View File

@ -1,5 +1,4 @@
#!/bin/sh
test "$HOME" = ~ || exec ksh $0 "$@" # try ksh if sh too old (not yet POSIX)
# Copyright (C) 2001, 2002, 2003 Marc Vertes
@ -18,7 +17,7 @@ test "$HOME" = ~ || exec ksh $0 "$@" # try ksh if sh too old (not yet POSIX)
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA
# 02111-1307, USA.
# txt2man-1.5
# release 1.7.1
usage()
{
@ -27,7 +26,7 @@ NAME
txt2man - convert flat ASCII text to man page format
SYNOPSIS
txt2man [-hpTX] [-t mytitle] [-P pname] [-r rel] [-s sect]
[-v vol] [-I txt] [-B txt] [ifile]
[-v vol] [-I txt] [-B txt] [-d date] [ifile]
DESCRIPTION
txt2man converts the input text into nroff/troff standard man(7)
macros used to format Unix manual pages. Nice pages can be generated
@ -45,26 +44,32 @@ DESCRIPTION
Here is how text patterns are recognized and processed:
Sections These headers are defined by a line in upper case, starting
column 1. If there is one or more leading spaces, a
sub-section will be generated instead.
sub-section will be generated instead. Optionally, the
Section name can be preceded by a blank line. This is useful
for a better visualization of the source text to be used to
generate the manpage.
Paragraphs They must be separated by a blank line, and left aligned.
Alternatively two blank spaces can be used to produce the
same result. This option will provide a better visualization
of the source text to be used to generate the manpage.
Tag list The item definition is separated from the item description
by at least 2 blank spaces, even before a new line, if
definition is too long. Definition will be emphasized
by default.
Bullet list
Bullet list items are defined by the first word being "-"
or "*" or "o".
or "*" or "o".
Enumerated list
The first word must be a number followed by a dot.
The first word must be a number followed by a dot.
Literal display blocks
This paragraph type is used to display unmodified text,
for example source code. It must be separated by a blank
line, and be indented. It is primarily used to format
unmodified source code. It will be printed using fixed font
whenever possible (troff).
This paragraph type is used to display unmodified text,
for example source code. It must be separated by a blank
line and be indented by a TAB. It is primarily used to format
unmodified source code. It will be printed using fixed font
whenever possible (troff).
Cross references
A cross reference (another man page) is defined by a word
followed by a number in parenthesis.
A cross reference (another man page) is defined by a word
followed by a number in parenthesis.
Special sections:
NAME The function or command name and short description are set in
@ -72,37 +77,45 @@ DESCRIPTION
SYNOPSIS This section receives a special treatment to identify command
name, flags and arguments, and propagate corresponding
attributes later in the text. If a C like function is recognized
(word immediately followed by an open parenthesis), txt2man will
print function name in bold font, types in normal font, and
variables in italic font. The whole section will be printed using
a fixed font family (courier) whenever possible (troff).
(word immediately followed by an open parenthesis), txt2man will
print function name in bold font, types in normal font, and
variables in italic font. The whole section will be printed using
a fixed font family (courier) whenever possible (troff).
It is a good practice to embed documentation into source code, by using
comments or constant text variables. txt2man allows to do that, keeping
comments or constant text variables. txt2man allows one to do that, keeping
the document source readable, usable even without further formatting
(i.e. for online help) and easy to write. The result is high quality
and standard complying document.
OPTIONS
-h The option -h displays help.
-d date Set date in header. Defaults to current date.
-P pname Set pname as project name in header. Default to uname -s.
-p Probe title, section name and volume.
-t mytitle Set mytitle as title of generated man page.
-r rel Set rel as project name and release.
-s sect Set sect as section in heading, ususally a value from 1 to 8.
-s sect Set sect as section in heading, usually a value from 1 to 8.
-v vol Set vol as volume name, i.e. "Unix user 's manual".
-I txt Italicize txt in output. Can be specified more than once.
-B txt Emphasize (bold) txt in output. Can be specified more than once.
-T Text result previewing using PAGER, usually more(1).
-X X11 result previewing using gxditview(1).
ENVIRONMENT
PAGER name of paging command, usually more(1), or less(1). If not set
falls back to more(1).
EXAMPLE
PAGER name of paging command, usually more(1), or less(1). If not set
falls back to more(1).
SOURCE_DATE_EPOCH Unix timestamp that is used for date in header instead
of current date.
EXAMPLES
Try this command to format this text itself:
$ txt2man -h 2>&1 | txt2man -T
$ txt2man -h 2>&1 | txt2man -T
The following command will generate a manpage level 1 to foo-1.1.0 program,
from foo.txt file, used as source code to previously mentioned manpage:
$ txt2man -d "15 May 2016" -t foo -r foo-1.1.0 -s 1 -v "show stars on screen" foo.txt > foo.1
HINTS
To obtain an overall good formating of output document, keep paragraphs
To obtain an overall good formatting of output document, keep paragraphs
indented correctly. If you have unwanted bold sections, search for
multiple spaces between words, which are used to identify a tag list
(term followed by a description). Choose also carefully the name of
@ -127,35 +140,53 @@ doprobe=
itxt=
btxt=
post=cat
while getopts :hpTXr:s:t:v:P:I:B: opt
while getopts :d:hpTXr:s:t:v:P:I:B: opt
do
case $opt in
r) rel=$OPTARG;;
t) title=$OPTARG;;
s) section=$OPTARG;;
v) volume=$OPTARG;;
P) sys=$OPTARG;;
p) doprobe=1;;
I) itxt="$OPTARG§$itxt";;
B) btxt=$OPTARG;;
T) post="groff -mandoc -Tlatin1 | ${PAGER:-more}";;
X) post="groff -mandoc -X";;
*) usage; exit;;
(d) date=$OPTARG;;
(r) rel=$OPTARG;;
(t) title=$OPTARG;;
(s) section=$OPTARG;;
(v) volume=$OPTARG;;
(P) sys=$OPTARG;;
(p) doprobe=1;;
(I) itxt="$OPTARG§$itxt";;
(B) btxt="$OPTARG§$btxt";;
(T) post="groff -mandoc -Tlatin1 | ${PAGER:-more}";;
(X) post="groff -mandoc -TX100-12 -rS12";;
(*) usage; exit;;
esac
done
shift $(($OPTIND - 1))
# Compatibility wrapper for BSD/GNU date, for parsing dates
if date -j >/dev/null 2>&1; then
pdate() { date -u -j -f '@%s' "$@"; }
else
pdate() { date -u -d "$@"; }
fi
if [ -n "$SOURCE_DATE_EPOCH" ]; then
date=$(LC_ALL=C pdate "@$SOURCE_DATE_EPOCH" +'%d %B %Y')
fi
date=${date:-$(LC_ALL=C date -u +'%d %B %Y')}
if test "$doprobe"
then
title=${1##*/}; title=${title%.txt}
section="8"
volume="System Manager's Manual"
if grep -q '#include ' $1
then
section=${section:-3}
volume=${volume:-"$sys Programmer's Manual"}
else
section=${section:-1}
volume=${volume:-"$sys Reference Manual"}
fi
# get release from path
#rel=$(pwd | sed 's:/.*[^0-9]/::g; s:/.*::g')
rel="Device Mapper Tools"
rel=${rel:-"$(pwd | sed 's:/.*[^0-9]/::g; s:/.*::g')"}
fi
head=".\\\" Text automatically generated by txt2man
head="\" Text automatically generated by txt2man
.TH $title $section \"$rel\" \"$volume\""
# All tabs converted to spaces
@ -163,11 +194,11 @@ expand $* |
# gawk is needed because use of non standard regexp
gawk --re-interval -v head="$head" -v itxt="$itxt" -v btxt="$btxt" '
BEGIN {
print head
print ".\\" head
avar[1] = btxt; avar[2] = itxt
for (k in avar) {
mark = (k == 1) ? "\\fB" : "\\fI"
split(avar[k], tt, "§")
split(avar[k], tt, "§")
for (i in tt)
if (tt[i] != "")
subwords["\\<" tt[i] "\\>"] = mark tt[i] "\\fP"
@ -179,11 +210,11 @@ BEGIN {
}
{
# to avoid some side effects in regexp
sub(/\.\.\./, "\\.\\.\\.")
gsub(/\.\.\./, "\\.\\.\\.")
# remove spaces in empty lines
sub(/^ +$/,"")
}
/^[[:upper:][:space:]]+$/ {
/^[:space:]*[[:upper:][:digit:]]+[[:upper:][:space:][:digit:][:punct:]]+$/ {
# Section header
if ((in_bd + 0) == 1) {
in_bd = 0
@ -199,8 +230,10 @@ BEGIN {
print ".SS" $0
sub(/^ +/, "")
section = $0
if (section == "SYNOPSIS")
if (section == "SYNOPSIS") {
print ".nf\n.fam C"
in_bd = 1
}
ls = 0 # line start index
pls = 0 # previous line start index
pnzls = 0 # previous non zero line start index
@ -216,7 +249,7 @@ BEGIN {
pnzls = ls
match($0, /[^ ]/)
ls = RSTART
if (pls == 0 && pnzls > 0 && ls > pnzls && $1 !~ /^[0-9\-\*\o]\.*$/) {
if (in_bd == 0 && pls == 0 && pnzls > 0 && ls > pnzls && $1 !~ /^[\-\*o]$|^[0-9]+\.$/) {
# example display block
if (prevblankline == 1) {
print ".PP"
@ -230,8 +263,10 @@ BEGIN {
ind[0] = ls
}
(in_bd + 0) == 1 {
# In example display block
if (ls != 0 && ls < eoff) {
# In block display
if (section == "SYNOPSIS")
;
else if (ls != 0 && ls < eoff) {
# End of litteral display block
in_bd = 0
print ".fam T\n.fi"
@ -244,11 +279,12 @@ section == "NAME" {
section == "SYNOPSIS" {
# Identify arguments of fcts and cmds
if (type["SYNOPSIS"] == "") {
if (index($0, "(") == 0 && index($0, ")") == 0 &&
index($0, "#include") == 0)
type["SYNOPSIS"] = "cmd"
else
if ($0 ~ /\(/)
type["SYNOPSIS"] = "fct"
else if ($1 == "struct" || $2 == "struct")
type["SYNOPSIS"] = "struct"
else if ($1 && $1 !~ /^#|typedef|struct|union|enum/)
type["SYNOPSIS"] = "cmd"
}
if (type["SYNOPSIS"] == "cmd") {
# Line is a command line
@ -263,19 +299,19 @@ section == "SYNOPSIS" {
if (a ~ /^[^\-]/)
subwords["\\<" a "\\>"] = "\\fI" a "\\fP"
}
} else {
} else if (type["SYNOPSIS"] == "fct") {
# Line is a C function definition
if ($1 == "typedef")
subwords["\\<" $2 "\\>"] = "\\fI" $2 "\\fP"
else if ($1 == "#define")
if ($1 == "typedef") {
if ($0 !~ /\(\*/)
subwords["\\<" $2 "\\>"] = "\\fI" $2 "\\fP"
} else if ($1 == "#define")
subwords["\\<" $2 "\\>"] = "\\fI" $2 "\\fP"
for (i = 1; i <= NF; i++) {
if ($i ~ /[\,\)]/) {
if ($i ~ /[,\)];*$/) {
a = $i
sub(/.*\(/, "", a)
gsub(/\W/, "", a)
if (a !~ /^void$/)
subwords["\\<" a "\\>"] = "\\fI" a "\\fP"
subwords["\\<" a "\\>"] = "\\fI" a "\\fP"
}
}
}
@ -298,13 +334,19 @@ section == "SYNOPSIS" {
}
}
# word attributes
for (i in subwords)
gsub(i, subwords[i])
n = asorti(subwords, indices)
for (i = 1; i <= n; i++)
gsub(indices[i], subwords[indices[i]])
# shell options
gsub(/\B\-+\w+(\-\w+)*/, "\\fB&\\fP")
# unprotect dots inside words
gsub(/_dOt_/, ".")
if (section == "SYNOPSIS") {
sub(/^ /, "")
print
next
}
if (match($0, /[^ ] +/) > 0) {
# tag list item
adjust_indent()
@ -347,6 +389,8 @@ section == "SYNOPSIS" {
}
if (section != "SYNOPSIS" || $0 ~ /^ {1,4}/)
sub(/ */,"")
# Protect lines starting by simple quotes
sub(/^'\''/, "\\(cq")
print
}

22
block-cache/copier.cc
View File

@ -10,12 +10,14 @@ using namespace std;
copier::copier(io_engine &engine,
string const &src, string const &dest,
sector_t block_size, size_t mem)
sector_t block_size, size_t mem,
sector_t src_offset, sector_t dest_offset)
: pool_(block_size * 512, mem, PAGE_SIZE),
block_size_(block_size),
engine_(engine),
src_handle_(engine_.open_file(src, io_engine::M_READ_ONLY)),
dest_handle_(engine_.open_file(dest, io_engine::M_READ_WRITE)),
src_offset_(src_offset), dest_offset_(dest_offset),
genkey_count_(0)
{
}
@ -45,8 +47,8 @@ copier::issue(copy_op const &op)
auto r = engine_.issue_io(src_handle_,
io_engine::D_READ,
to_sector(op.src_b),
to_sector(op.src_e),
to_src_sector(op.src_b),
to_src_sector(op.src_e),
data,
key);
@ -151,8 +153,8 @@ copier::wait_successful(io_engine::wait_result const &p)
j.op.read_complete = true;
if (!engine_.issue_io(dest_handle_,
io_engine::D_WRITE,
to_sector(j.op.dest_b),
to_sector(j.op.dest_b + (j.op.src_e - j.op.src_b)),
to_dest_sector(j.op.dest_b),
to_dest_sector(j.op.dest_b + (j.op.src_e - j.op.src_b)),
j.data,
it->first)) {
complete(j);
@ -177,9 +179,15 @@ copier::complete(copy_job const &j)
}
sector_t
copier::to_sector(block_address b) const
copier::to_src_sector(block_address b) const
{
return b * block_size_;
return src_offset_ + b * block_size_;
}
sector_t
copier::to_dest_sector(block_address b) const
{
return dest_offset_ + b * block_size_;
}
unsigned

16
block-cache/copier.h
View File

@ -61,13 +61,22 @@ namespace bcache {
public:
copier(io_engine &engine,
std::string const &src, std::string const &dest,
sector_t block_size, size_t mem);
sector_t block_size, size_t mem,
sector_t src_offset, sector_t dest_offset);
~copier();
sector_t get_block_size() const {
return block_size_;
}
sector_t get_src_offset() const {
return src_offset_;
}
sector_t get_dest_offset() const {
return dest_offset_;
}
// Blocks if out of memory.
void issue(copy_op const &op);
@ -83,7 +92,8 @@ namespace bcache {
void wait_();
void complete(copy_job const &j);
sector_t to_sector(block_address b) const;
sector_t to_src_sector(block_address b) const;
sector_t to_dest_sector(block_address b) const;
unsigned genkey();
mempool pool_;
@ -91,6 +101,8 @@ namespace bcache {
io_engine &engine_;
io_engine::handle src_handle_;
io_engine::handle dest_handle_;
sector_t src_offset_;
sector_t dest_offset_;
unsigned genkey_count_;
using job_map = std::map<unsigned, copy_job>;

3
block-cache/io_engine.cc
View File

@ -174,9 +174,6 @@ aio_engine::wait_(timespec *ts)
cbs_.free(cb);
return optional<wait_result>(make_pair(false, context));
}
// shouldn't get here
return optional<wait_result>(make_pair(false, 0));
}
struct timespec

231
caching/cache_debug.cc Normal file
View File

@ -0,0 +1,231 @@
// Copyright (C) 2012 Red Hat, Inc. All rights reserved.
//
// This file is part of the thin-provisioning-tools source.
//
// thin-provisioning-tools is free software: you can redistribute it
// and/or modify it under the terms of the GNU General Public License
// as published by the Free Software Foundation, either version 3 of
// the License, or (at your option) any later version.
//
// thin-provisioning-tools is distributed in the hope that it will be
// useful, but WITHOUT ANY WARRANTY; without even the implied warranty
// of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License along
// with thin-provisioning-tools. If not, see
// <http://www.gnu.org/licenses/>.
#include <boost/lexical_cast.hpp>
#include <getopt.h>
#include <iostream>
#include <string>
#include "dbg-lib/array_block_dumper.h"
#include "dbg-lib/btree_node_dumper.h"
#include "dbg-lib/bitset_block_dumper.h"
#include "dbg-lib/command_interpreter.h"
#include "dbg-lib/commands.h"
#include "dbg-lib/index_block_dumper.h"
#include "dbg-lib/output_formatter.h"
#include "dbg-lib/sm_show_traits.h"
#include "persistent-data/file_utils.h"
#include "persistent-data/space-maps/disk_structures.h"
#include "caching/commands.h"
#include "caching/metadata.h"
#include "version.h"
using namespace dbg;
using namespace persistent_data;
using namespace std;
using namespace caching;
//----------------------------------------------------------------
namespace {
class help : public dbg::command {
virtual void exec(strings const &args, ostream &out) {
out << "Commands:" << endl
<< " superblock [block#]" << endl
<< " block_node <block# of array block-tree node>" << endl
<< " bitset_block <block# of bitset block>" << endl
<< " index_block <block# of metadata space map root>" << endl
<< " mapping_block <block# of mappings array block>" << endl
<< " exit" << endl;
}
};
class show_superblock : public dbg::command {
public:
explicit show_superblock(block_manager::ptr bm)
: bm_(bm) {
}
virtual void exec(strings const &args, ostream &out) {
if (args.size() > 2)
throw runtime_error("incorrect number of arguments");
block_address b = caching::SUPERBLOCK_LOCATION;
if (args.size() == 2)
b = boost::lexical_cast<block_address>(args[1]);
caching::superblock sb = read_superblock(bm_, b);
formatter::ptr f = create_xml_formatter();
ostringstream version;
field(*f, "csum", sb.csum);
field(*f, "flags", sb.flags.encode());
field(*f, "blocknr", sb.blocknr);
field(*f, "uuid", sb.uuid); // FIXME: delimit, and handle non-printable chars
field(*f, "magic", sb.magic);
field(*f, "version", sb.version);
field(*f, "policy_name", reinterpret_cast<char const*>(sb.policy_name));
version << sb.policy_version[0] << "."
<< sb.policy_version[1] << "."
<< sb.policy_version[2];
field(*f, "policy_version", version.str().c_str());
field(*f, "policy_hint_size", sb.policy_hint_size);
sm_disk_detail::sm_root_disk const *d;
sm_disk_detail::sm_root v;
{
d = reinterpret_cast<sm_disk_detail::sm_root_disk const *>(sb.metadata_space_map_root);
sm_disk_detail::sm_root_traits::unpack(*d, v);
formatter::ptr f2 = create_xml_formatter();
sm_root_show_traits::show(f2, "value", v);
f->child("metadata_space_map_root", f2);
}
field(*f, "mapping_root", sb.mapping_root);
if (sb.version >= 2)
field(*f, "dirty_root", *sb.dirty_root);
field(*f, "hint_root", sb.hint_root);
field(*f, "discard_root", sb.discard_root);
field(*f, "discard_block_size", sb.discard_block_size);
field(*f, "discard_nr_blocks", sb.discard_nr_blocks);
field(*f, "data_block_size", sb.data_block_size);
field(*f, "metadata_block_size", sb.metadata_block_size);
field(*f, "cache_blocks", sb.cache_blocks);
field(*f, "compat_flags", sb.compat_flags);
field(*f, "compat_ro_flags", sb.compat_ro_flags);
field(*f, "incompat_flags", sb.incompat_flags);
field(*f, "read_hits", sb.read_hits);
field(*f, "read_misses", sb.read_misses);
field(*f, "write_hits", sb.write_hits);
field(*f, "write_misses", sb.write_misses);
f->output(out, 0);
}
private:
block_manager::ptr bm_;
};
class mapping_show_traits : public caching::mapping_traits {
public:
typedef mapping_traits value_trait;
static void show(formatter::ptr f, string const &key, caching::mapping const &value) {
field(*f, "oblock", value.oblock_);
field(*f, "flags", value.flags_);
}
};
//--------------------------------
template <typename ShowTraits>
dbg::command::ptr
create_btree_node_handler(block_manager::ptr bm) {
return create_block_handler(bm, create_btree_node_dumper<ShowTraits>());
}
template <typename ShowTraits>
dbg::command::ptr
create_array_block_handler(block_manager::ptr bm,
typename ShowTraits::value_trait::ref_counter rc) {
return create_block_handler(bm, create_array_block_dumper<ShowTraits>(rc));
}
dbg::command::ptr
create_bitset_block_handler(block_manager::ptr bm) {
return create_block_handler(bm, create_bitset_block_dumper());
}
dbg::command::ptr
create_index_block_handler(block_manager::ptr bm) {
return create_block_handler(bm, create_index_block_dumper());
}
int debug(string const &path) {
using dbg::command;
try {
block_manager::ptr bm = open_bm(path, block_manager::READ_ONLY);
command_interpreter::ptr interp = create_command_interpreter(cin, cout);
interp->register_command("hello", create_hello_handler());
interp->register_command("superblock", command::ptr(new show_superblock(bm)));
interp->register_command("block_node", create_btree_node_handler<uint64_show_traits>(bm));
interp->register_command("bitset_block", create_bitset_block_handler(bm));
interp->register_command("index_block", create_index_block_handler(bm));
interp->register_command("mapping_block", create_array_block_handler<mapping_show_traits>(bm,
mapping_traits::ref_counter()));
interp->register_command("help", command::ptr(new help));
interp->register_command("exit", create_exit_handler(interp));
interp->enter_main_loop();
} catch (std::exception &e) {
cerr << e.what();
return 1;
}
return 0;
}
}
//----------------------------------------------------------------
cache_debug_cmd::cache_debug_cmd()
: command("cache_debug")
{
}
void
cache_debug_cmd::usage(std::ostream &out) const
{
out << "Usage: " << get_name() << " {device|file}" << endl
<< "Options:" << endl
<< " {-h|--help}" << endl
<< " {-V|--version}" << endl;
}
int
cache_debug_cmd::run(int argc, char **argv)
{
int c;
const char shortopts[] = "hV";
const struct option longopts[] = {
{ "help", no_argument, NULL, 'h'},
{ "version", no_argument, NULL, 'V'},
{ NULL, no_argument, NULL, 0 }
};
while ((c = getopt_long(argc, argv, shortopts, longopts, NULL)) != -1) {
switch(c) {
case 'h':
usage(cout);
return 0;
case 'V':
cerr << THIN_PROVISIONING_TOOLS_VERSION << endl;
return 0;
}
}
if (argc == optind) {
usage(cerr);
exit(1);
}
return debug(argv[optind]);
}
//----------------------------------------------------------------

2
caching/cache_repair.cc
View File

@ -32,7 +32,7 @@ namespace {
int repair(string const &old_path, string const &new_path) {
bool metadata_touched = false;
try {
file_utils::check_file_exists(new_path, false);
file_utils::check_file_exists(old_path, false);
metadata_touched = true;
metadata_dump(open_metadata_for_read(old_path),
output_emitter(new_path),

29
caching/cache_writeback.cc
View File

@ -31,6 +31,8 @@ namespace {
flags()
: cache_size(4 * 1024 * 1024),
sort_buffers(16 * 1024),
origin_dev_offset(0),
fast_dev_offset(0),
list_failed_blocks(false),
update_metadata(true) {
}
@ -52,6 +54,8 @@ namespace {
maybe_string metadata_dev;
maybe_string origin_dev;
maybe_string fast_dev;
sector_t origin_dev_offset;
sector_t fast_dev_offset;
bool list_failed_blocks;
bool update_metadata;
};
@ -297,13 +301,15 @@ namespace {
int writeback_(flags const &f) {
block_manager::ptr bm = open_bm(*f.metadata_dev, block_manager::READ_WRITE);
metadata md(bm);
metadata md(bm, true);
// FIXME: we're going to have to copy runs to get the through put with small block sizes
unsigned max_ios = f.cache_size / (md.sb_.data_block_size << SECTOR_SHIFT);
aio_engine engine(max_ios);
copier c(engine, *f.fast_dev, *f.origin_dev,
md.sb_.data_block_size, f.cache_size);
md.sb_.data_block_size, f.cache_size,
f.fast_dev_offset >> SECTOR_SHIFT,
f.origin_dev_offset >> SECTOR_SHIFT);
auto bar = create_progress_bar("Copying data");
copy_visitor cv(c, f.sort_buffers, clean_shutdown(md), f.list_failed_blocks,
@ -364,6 +370,8 @@ cache_writeback_cmd::usage(std::ostream &out) const
<< "\t\t--buffer-size-meg <size>\n"
<< "\t\t--list-failed-blocks\n"
<< "\t\t--no-metadata-update\n"
<< "\t\t--origin-device-offset <bytes>\n"
<< "\t\t--fast-device-offset <bytes>\n"
<< "Options:\n"
<< " {-h|--help}\n"
<< " {-V|--version}" << endl;
@ -382,6 +390,8 @@ cache_writeback_cmd::run(int argc, char **argv)
{ "buffer-size-meg", required_argument, NULL, 3 },
{ "list-failed-blocks", no_argument, NULL, 4 },
{ "no-metadata-update", no_argument, NULL, 5 },
{ "origin-device-offset", required_argument, NULL, 6 },
{ "fast-device-offset", required_argument, NULL, 7 },
{ "help", no_argument, NULL, 'h'},
{ "version", no_argument, NULL, 'V'},
{ NULL, no_argument, NULL, 0 }
@ -413,6 +423,14 @@ cache_writeback_cmd::run(int argc, char **argv)
fs.update_metadata = false;
break;
case 6:
fs.origin_dev_offset = parse_uint64(optarg, "origin dev offset");
break;
case 7:
fs.fast_dev_offset = parse_uint64(optarg, "fast dev offset");
break;
case 'h':
usage(cout);
return 0;
@ -452,6 +470,13 @@ cache_writeback_cmd::run(int argc, char **argv)
return 1;
}
if (fs.origin_dev_offset & (SECTOR_SHIFT - 1) ||
fs.fast_dev_offset & (SECTOR_SHIFT - 1)) {
cerr << "Offset must be sector-aligned\n\n";
usage(cerr);
return 1;
}
return writeback(fs);
}

12
caching/commands.h
View File

@ -70,6 +70,18 @@ namespace caching {
virtual int run(int argc, char **argv);
};
//------------------------------------------------------
class cache_debug_cmd : public base::command {
public:
cache_debug_cmd();
virtual void usage(std::ostream &out) const;
virtual int run(int argc, char **argv);
};
//------------------------------------------------------
void register_cache_commands(base::application &app);
}

14
caching/devel_commands.cc Normal file
View File

@ -0,0 +1,14 @@
#include "caching/commands.h"
using namespace base;
using namespace caching;
//----------------------------------------------------------------
void
caching::register_cache_commands(application &app)
{
app.add_cmd(command::ptr(new cache_debug_cmd));
}
//----------------------------------------------------------------

6
caching/hint_array.cc
View File

@ -56,9 +56,6 @@ namespace {
default:
throw runtime_error("invalid hint width");
}
// never get here
return std::shared_ptr<array_base>();
}
//--------------------------------
@ -93,9 +90,6 @@ namespace {
default:
throw runtime_error("invalid hint width");
}
// never get here
return std::shared_ptr<array_base>();
}
//--------------------------------

11
caching/metadata.cc
View File

@ -45,9 +45,9 @@ metadata::metadata(block_manager::ptr bm, open_type ot, unsigned metadata_versio
}
}
metadata::metadata(block_manager::ptr bm)
metadata::metadata(block_manager::ptr bm, bool read_space_map)
{
open_metadata(bm);
open_metadata(bm, read_space_map);
}
void
@ -90,11 +90,16 @@ metadata::create_metadata(block_manager::ptr bm, unsigned metadata_version)
}
void
metadata::open_metadata(block_manager::ptr bm)
metadata::open_metadata(block_manager::ptr bm, bool read_space_map)
{
tm_ = open_tm(bm);
sb_ = read_superblock(tm_->get_bm());
if (read_space_map) {
metadata_sm_ = open_metadata_sm(*tm_, &sb_.metadata_space_map_root);
tm_->set_sm(metadata_sm_);
}
mappings_ = mapping_array::ptr(
new mapping_array(*tm_,
mapping_array::ref_counter(),

4
caching/metadata.h
View File

@ -27,7 +27,7 @@ namespace caching {
typedef std::shared_ptr<metadata> ptr;
metadata(block_manager::ptr bm, open_type ot, unsigned metadata_version = 2); // Create only
metadata(block_manager::ptr bm);
metadata(block_manager::ptr bm, bool read_space_map = false);
void commit(bool clean_shutdown = true);
void setup_hint_array(size_t width);
@ -46,7 +46,7 @@ namespace caching {
void init_superblock();
void create_metadata(block_manager::ptr bm, unsigned metadata_version);
void open_metadata(block_manager::ptr bm);
void open_metadata(block_manager::ptr bm, bool read_space_map);
void commit_space_map();
void commit_mappings();

9
configure.ac
View File

@ -150,14 +150,6 @@ AC_ARG_ENABLE(testing,
TESTING=$enableval, TESTING=no)
AC_MSG_RESULT($TESTING)
################################################################################
dnl -- Enable development tools
AC_MSG_CHECKING(whether to enable development tools)
AC_ARG_ENABLE(dev-tools,
AC_HELP_STRING(--enable-dev-tools, [enable development tools in the makefile]),
DEVTOOLS=$enableval, DEVTOOLS=no)
AC_MSG_RESULT($DEVTOOLS)
################################################################################
dnl -- Enable static libstdc++
AC_MSG_CHECKING(whether to statically link libstdc++)
@ -214,6 +206,5 @@ Makefile
contrib/Makefile
unit-tests/Makefile
version.h
src/version.rs
])
AC_OUTPUT

2
contrib/Makefile.in
View File

@ -18,7 +18,7 @@ contrib/%.a: contrib/%.o
$(V)echo " [AR] $@"
$(V)$(AR) rcs $@ $^
contrib/%.so: contrib/%.a
contrib/%.so: contrib/%.o
$(V)echo " [LD] $@"
$(V)$(CC) -shared -Wl,-soname,$@ -o $@ $<

53
dbg-lib/array_block_dumper.h Normal file
View File

@ -0,0 +1,53 @@
#include "dbg-lib/block_dumper.h"
#include "dbg-lib/output_formatter.h"
#include "persistent-data/data-structures/array_block.h"
//----------------------------------------------------------------
namespace dbg {
using persistent_data::block_manager;
using persistent_data::array_block;
template <typename ShowTraits>
class array_block_dumper : public block_dumper {
public:
array_block_dumper(typename ShowTraits::value_trait::ref_counter rc)
: rc_(rc) {
}
virtual void show(block_manager::read_ref &rr, std::ostream &out) {
rblock b(rr, rc_);
show_array_entries(b, out);
}
private:
typedef array_block<typename ShowTraits::value_trait, block_manager::read_ref> rblock;
void show_array_entries(rblock const& b, std::ostream &out) {
formatter::ptr f = create_xml_formatter();
uint32_t nr_entries = b.nr_entries();
field(*f, "max_entries", b.max_entries());
field(*f, "nr_entries", nr_entries);
field(*f, "value_size", b.value_size());
for (unsigned i = 0; i < nr_entries; i++) {
formatter::ptr f2 = create_xml_formatter();
ShowTraits::show(f2, "value", b.get(i));
f->child(boost::lexical_cast<std::string>(i), f2);
}
f->output(out, 0);
}
typename ShowTraits::value_trait::ref_counter rc_;
};
template <typename ShowTraits>
block_dumper::ptr
create_array_block_dumper(typename ShowTraits::value_trait::ref_counter rc) {
return block_dumper::ptr(new array_block_dumper<ShowTraits>(rc));
}
}
//----------------------------------------------------------------

137
dbg-lib/bitset_block_dumper.cc Normal file
View File

@ -0,0 +1,137 @@
#include "dbg-lib/bitset_block_dumper.h"
#include "dbg-lib/output_formatter.h"
#include "persistent-data/data-structures/array_block.h"
#include "persistent-data/data-structures/simple_traits.h"
using namespace dbg;
using namespace persistent_data;
//----------------------------------------------------------------
namespace {
class bitset_block_dumper : public dbg::block_dumper {
typedef array_block<uint64_traits, block_manager::read_ref> rblock;
public:
explicit bitset_block_dumper()
: BITS_PER_ARRAY_ENTRY(64) {
}
virtual void show(block_manager::read_ref &rr, ostream &out) {
rblock b(rr, rc_);
show_bitset_entries(b, out);
}
private:
void show_bitset_entries(rblock const& b, ostream &out) {
formatter::ptr f = create_xml_formatter();
uint32_t nr_entries = b.nr_entries();
field(*f, "max_entries", b.max_entries());
field(*f, "nr_entries", nr_entries);
field(*f, "value_size", b.value_size());
uint32_t end_pos = b.nr_entries() * BITS_PER_ARRAY_ENTRY;
std::pair<uint32_t, uint32_t> range = next_set_bits(b, 0);
for (; range.first < end_pos; range = next_set_bits(b, range.second)) {
formatter::ptr f2 = create_xml_formatter();
field(*f2, "begin", range.first);
field(*f2, "end", range.second);
f->child("set_bits", f2);
}
f->output(out, 0);
}
// Returns the range of set bits, starts from the offset.
pair<uint32_t, uint32_t> next_set_bits(rblock const &b, uint32_t offset) {
uint32_t end_pos = b.nr_entries() * BITS_PER_ARRAY_ENTRY;
uint32_t begin = find_first_set(b, offset);
if (begin == end_pos) // not found
return make_pair(end_pos, end_pos);
uint32_t end = find_first_unset(b, begin + 1);
return make_pair(begin, end);
}
// Returns the position (zero-based) of the first bit set
// in the array block, starts from the offset.
// Returns the pass-the-end position if not found.
uint32_t find_first_set(rblock const &b, uint32_t offset) {
uint32_t entry = offset / BITS_PER_ARRAY_ENTRY;
uint32_t nr_entries = b.nr_entries();
if (entry >= nr_entries)
return entry * BITS_PER_ARRAY_ENTRY;
uint32_t idx = offset % BITS_PER_ARRAY_ENTRY;
uint64_t v = b.get(entry++) >> idx;
while (!v && entry < nr_entries) {
v = b.get(entry++);
idx = 0;
}
if (!v) // not found
return entry * BITS_PER_ARRAY_ENTRY;
return (entry - 1) * BITS_PER_ARRAY_ENTRY + idx + ffsll(static_cast<long long>(v)) - 1;
}
// Returns the position (zero-based) of the first zero bit
// in the array block, starts from the offset.
// Returns the pass-the-end position if not found.
// FIXME: improve efficiency
uint32_t find_first_unset(rblock const& b, uint32_t offset) {
uint32_t entry = offset / BITS_PER_ARRAY_ENTRY;
uint32_t nr_entries = b.nr_entries();
if (entry >= nr_entries)
return entry * BITS_PER_ARRAY_ENTRY;
uint32_t idx = offset % BITS_PER_ARRAY_ENTRY;
uint64_t v = b.get(entry++);
while (all_bits_set(v, idx) && entry < nr_entries) {
v = b.get(entry++);
idx = 0;
}
if (all_bits_set(v, idx)) // not found
return entry * BITS_PER_ARRAY_ENTRY;
return (entry - 1) * BITS_PER_ARRAY_ENTRY + idx + count_leading_bits(v, idx);
}
// Returns true if all the bits beyond the position are set.
bool all_bits_set(uint64_t v, uint32_t offset) {
return (v >> offset) == (numeric_limits<uint64_t>::max() >> offset);
}
// Counts the number of leading 1's in the given value, starts from the offset
// FIXME: improve efficiency
uint32_t count_leading_bits(uint64_t v, uint32_t offset) {
uint32_t count = 0;
v >>= offset;
while (v & 0x1) {
v >>= 1;
count++;
}
return count;
}
block_manager::ptr bm_;
uint64_traits::ref_counter rc_;
const uint32_t BITS_PER_ARRAY_ENTRY;
};
}
//----------------------------------------------------------------
block_dumper::ptr
dbg::create_bitset_block_dumper() {
return block_dumper::ptr(new bitset_block_dumper());
}
//----------------------------------------------------------------

14
dbg-lib/bitset_block_dumper.h Normal file
View File

@ -0,0 +1,14 @@
#ifndef DBG_BITSET_BLOCK_DUMPER
#define DBG_BITSET_BLOCK_DUMPER
#include "dbg-lib/block_dumper.h"
//----------------------------------------------------------------
namespace dbg {
block_dumper::ptr create_bitset_block_dumper();
}
//----------------------------------------------------------------
#endif

21
dbg-lib/block_dumper.h Normal file
View File

@ -0,0 +1,21 @@
#ifndef DBG_BLOCK_DUMPER_H
#define DBG_BLOCK_DUMPER_H
#include "persistent-data/block.h"
//----------------------------------------------------------------
namespace dbg {
class block_dumper {
public:
typedef std::shared_ptr<block_dumper> ptr;
// pass the read_ref by reference since the caller already held the ref-count
virtual void show(persistent_data::block_manager::read_ref &rr,
std::ostream &out) = 0;
};
}
//----------------------------------------------------------------
#endif

57
dbg-lib/btree_node_dumper.h Normal file
View File

@ -0,0 +1,57 @@
#ifndef DBG_BTREE_NODE_DUMPER
#define DBG_BTREE_NODE_DUMPER
#include "dbg-lib/block_dumper.h"
#include "dbg-lib/simple_show_traits.h"
#include "persistent-data/data-structures/btree.h"
//----------------------------------------------------------------
namespace dbg {
using persistent_data::block_manager;
using persistent_data::btree_detail::node_ref;
template <typename ShowTraits>
class btree_node_dumper : public block_dumper {
public:
virtual void show(block_manager::read_ref &rr, std::ostream &out) {
node_ref<uint64_traits> n = btree_detail::to_node<uint64_traits>(rr);
if (n.get_type() == INTERNAL)
btree_node_dumper<uint64_show_traits>::show_node(n, out);
else {
node_ref<typename ShowTraits::value_trait> n = btree_detail::to_node<typename ShowTraits::value_trait>(rr);
show_node(n, out);
}
}
static void show_node(node_ref<typename ShowTraits::value_trait> n, std::ostream &out) {
formatter::ptr f = create_xml_formatter();
field(*f, "csum", n.get_checksum());
field(*f, "blocknr", n.get_block_nr());
field(*f, "type", n.get_type() == INTERNAL ? "internal" : "leaf");
field(*f, "nr_entries", n.get_nr_entries());
field(*f, "max_entries", n.get_max_entries());
field(*f, "value_size", n.get_value_size());
for (unsigned i = 0; i < n.get_nr_entries(); i++) {
formatter::ptr f2 = create_xml_formatter();
field(*f2, "key", n.key_at(i));
ShowTraits::show(f2, "value", n.value_at(i));
f->child(boost::lexical_cast<std::string>(i), f2);
}
f->output(out, 0);
}
};
template <typename ShowTraits>
block_dumper::ptr
create_btree_node_dumper() {
return block_dumper::ptr(new btree_node_dumper<ShowTraits>());
}
}
//----------------------------------------------------------------
#endif

88
dbg-lib/command_interpreter.cc Normal file
View File

@ -0,0 +1,88 @@
#include "dbg-lib/command_interpreter.h"
using namespace dbg;
using namespace std;
//----------------------------------------------------------------
namespace {
class command_interpreter_impl : public command_interpreter {
public:
typedef std::shared_ptr<command_interpreter> ptr;
command_interpreter_impl(std::istream &in, std::ostream &out)
: in_(in),
out_(out),
exit_(false) {
}
void register_command(std::string const &str, command::ptr cmd) {
commands_.insert(make_pair(str, cmd));
}
void enter_main_loop() {
while (!exit_)
do_once();
}
void exit_main_loop() {
exit_ = true;
}
private:
void do_once();
std::istream &in_;
std::ostream &out_;
std::map <std::string, command::ptr> commands_;
bool exit_;
};
//--------------------------------
strings read_input(std::istream &in)
{
using namespace boost::algorithm;
std::string input;
getline(in, input);
strings toks;
split(toks, input, is_any_of(" \t"), token_compress_on);
return toks;
}
}
//----------------------------------------------------------------
void command_interpreter_impl::do_once()
{
if (in_.eof())
throw runtime_error("input closed");
out_ << "> ";
strings args = read_input(in_);
std::map<std::string, command::ptr>::iterator it;
it = commands_.find(args[0]);
if (it == commands_.end())
out_ << "Unrecognised command" << endl;
else {
try {
it->second->exec(args, out_);
} catch (std::exception &e) {
cerr << e.what() << endl;
}
}
}
//----------------------------------------------------------------
command_interpreter::ptr
dbg::create_command_interpreter(std::istream &in, std::ostream &out)
{
return command_interpreter::ptr(new command_interpreter_impl(in, out));
}
//----------------------------------------------------------------

40
dbg-lib/command_interpreter.h Normal file
View File

@ -0,0 +1,40 @@
#ifndef DBG_COMMAND_INTERPRETER
#define DBG_COMMAND_INTERPRETER
#include <boost/algorithm/string/classification.hpp>
#include <boost/algorithm/string/split.hpp>
#include <iostream>
#include <map>
#include <string>
#include <vector>
//----------------------------------------------------------------
namespace dbg {
typedef std::vector<std::string> strings;
class command {
public:
typedef std::shared_ptr<command> ptr;
virtual ~command() {}
virtual void exec(strings const &args, std::ostream &out) = 0;
};
class command_interpreter {
public:
typedef std::shared_ptr<command_interpreter> ptr;
virtual void register_command(std::string const &str, command::ptr cmd) = 0;
virtual void enter_main_loop() = 0;
virtual void exit_main_loop() = 0;
};
command_interpreter::ptr
create_command_interpreter(std::istream &in, std::ostream &out);
}
//----------------------------------------------------------------
#endif

79
dbg-lib/commands.cc Normal file
View File

@ -0,0 +1,79 @@
#include "dbg-lib/commands.h"
#include "persistent-data/block.h"
#include <boost/lexical_cast.hpp>
#include <stdexcept>
using namespace dbg;
using namespace persistent_data;
using namespace std;
//----------------------------------------------------------------
namespace {
class hello : public dbg::command {
virtual void exec(dbg::strings const &args, ostream &out) {
out << "Hello, world!" << endl;
}
};
class exit_handler : public dbg::command {
public:
exit_handler(command_interpreter::ptr interpreter)
: interpreter_(interpreter) {
}
virtual void exec(dbg::strings const &args, ostream &out) {
out << "Goodbye!" << endl;
interpreter_->exit_main_loop();
}
private:
command_interpreter::ptr interpreter_;
};
class block_handler : public dbg::command {
public:
block_handler(block_manager::ptr bm, block_dumper::ptr dumper)
: bm_(bm), dumper_(dumper) {
}
virtual void exec(dbg::strings const &args, ostream &out) {
if (args.size() != 2)
throw runtime_error("incorrect number of arguments");
block_address block = boost::lexical_cast<block_address>(args[1]);
// no checksum validation for debugging purpose
block_manager::read_ref rr = bm_->read_lock(block);
dumper_->show(rr, out);
}
private:
block_manager::ptr bm_;
block_dumper::ptr dumper_;
};
}
//----------------------------------------------------------------
command::ptr
dbg::create_hello_handler()
{
return command::ptr(new hello());
}
command::ptr
dbg::create_exit_handler(command_interpreter::ptr interp)
{
return command::ptr(new exit_handler(interp));
}
command::ptr
dbg::create_block_handler(block_manager::ptr bm, block_dumper::ptr dumper)
{
return command::ptr(new block_handler(bm, dumper));
}
//----------------------------------------------------------------

24
dbg-lib/commands.h Normal file
View File

@ -0,0 +1,24 @@
#ifndef DBG_COMMANDS_H
#define DBG_COMMANDS_H
#include "dbg-lib/command_interpreter.h"
#include "dbg-lib/block_dumper.h"
#include "persistent-data/block.h"
//----------------------------------------------------------------
namespace dbg {
dbg::command::ptr
create_hello_handler();
dbg::command::ptr
create_exit_handler(dbg::command_interpreter::ptr interp);
dbg::command::ptr
create_block_handler(persistent_data::block_manager::ptr bm,
dbg::block_dumper::ptr dumper);
}
//----------------------------------------------------------------
#endif

51
dbg-lib/index_block_dumper.cc Normal file
View File

@ -0,0 +1,51 @@
#include "dbg-lib/index_block_dumper.h"
#include "dbg-lib/output_formatter.h"
#include "dbg-lib/sm_show_traits.h"
#include "persistent-data/space-maps/disk_structures.h"
using namespace dbg;
//----------------------------------------------------------------
namespace {
using persistent_data::block_manager;
class index_block_dumper : public dbg::block_dumper {
public:
virtual void show(block_manager::read_ref &rr, std::ostream &out) {
sm_disk_detail::metadata_index const *mdi =
reinterpret_cast<sm_disk_detail::metadata_index const *>(rr.data());
show_metadata_index(mdi, sm_disk_detail::MAX_METADATA_BITMAPS, out);
}
private:
void show_metadata_index(sm_disk_detail::metadata_index const *mdi, unsigned nr_indexes, std::ostream &out) {
formatter::ptr f = create_xml_formatter();
field(*f, "csum", to_cpu<uint32_t>(mdi->csum_));
field(*f, "padding", to_cpu<uint32_t>(mdi->padding_));
field(*f, "blocknr", to_cpu<uint64_t>(mdi->blocknr_));
sm_disk_detail::index_entry ie;
for (unsigned i = 0; i < nr_indexes; i++) {
sm_disk_detail::index_entry_traits::unpack(*(mdi->index + i), ie);
if (!ie.blocknr_ && !ie.nr_free_ && !ie.none_free_before_)
continue;
formatter::ptr f2 = create_xml_formatter();
index_entry_show_traits::show(f2, "value", ie);
f->child(boost::lexical_cast<string>(i), f2);
}
f->output(out, 0);
}
};
}
//----------------------------------------------------------------
block_dumper::ptr
dbg::create_index_block_dumper() {
return block_dumper::ptr(new index_block_dumper());
}
//----------------------------------------------------------------

10
dbg-lib/index_block_dumper.h Normal file
View File

@ -0,0 +1,10 @@
#include "dbg-lib/block_dumper.h"
//----------------------------------------------------------------
namespace dbg {
block_dumper::ptr
create_index_block_dumper();
}
//----------------------------------------------------------------

86
dbg-lib/output_formatter.cc Normal file
View File

@ -0,0 +1,86 @@
#include <string>
#include "dbg-lib/output_formatter.h"
using namespace dbg;
using namespace std;
//----------------------------------------------------------------
namespace {
class abstract_formatter : public formatter {
typedef boost::optional<std::string> maybe_string;
void field(std::string const &name, std::string const &value) {
fields_.push_back(field_type(name, value));
}
void child(std::string const &name, formatter::ptr t) {
children_.push_back(field_type(name, t));
}
protected:
typedef boost::variant<std::string, ptr> value;
typedef boost::tuple<std::string, value> field_type;
std::vector<field_type> fields_;
std::vector<field_type> children_;
};
class xml_formatter : public abstract_formatter {
public:
void output(std::ostream &out, int depth = 0,
boost::optional<std::string> name = boost::none);
};
void indent(int depth, std::ostream &out) {
for (int i = 0; i < depth * 2; i++)
out << ' ';
}
}
//----------------------------------------------------------------
void xml_formatter::output(std::ostream &out,
int depth,
boost::optional<std::string> name) {
indent(depth, out);
out << "<fields";
if (name && (*name).length())
out << " id=\"" << *name << "\"";
/* output non-child fields */
std::vector<field_type>::const_iterator it;
for (it = fields_.begin(); it != fields_.end(); ++it) {
if (string const *s = boost::get<string>(&it->get<1>())) {
out << " " << it->get<0>() << "=\"" << *s << "\"";
}
}
if (children_.size() == 0) {
out << " />" << endl;
return;
}
/* output child fields */
out << ">" << endl;
for (it = children_.begin(); it != children_.end(); ++it) {
if (!boost::get<string>(&it->get<1>())) {
formatter::ptr f = boost::get<formatter::ptr>(it->get<1>());
f->output(out, depth + 1, it->get<0>());
}
}
indent(depth, out);
out << "</fields>" << endl;
}
//----------------------------------------------------------------
formatter::ptr
dbg::create_xml_formatter()
{
return formatter::ptr(new xml_formatter());
}
//----------------------------------------------------------------

38
dbg-lib/output_formatter.h Normal file
View File

@ -0,0 +1,38 @@
#ifndef DBG_OUTPUT_FORMATTER_H
#define DBG_OUTPUT_FORMATTER_H
#include <boost/lexical_cast.hpp>
#include <boost/optional.hpp>
#include <boost/tuple/tuple.hpp>
#include <boost/variant.hpp>
#include <memory>
#include <vector>
#include <string>
//----------------------------------------------------------------
namespace dbg {
class formatter {
public:
typedef std::shared_ptr<formatter> ptr;
virtual ~formatter() {}
virtual void field(std::string const &name, std::string const &value) = 0;
virtual void child(std::string const &name, formatter::ptr t) = 0;
virtual void output(std::ostream &out, int depth = 0,
boost::optional<std::string> name = boost::none) = 0;
};
template <typename T>
void
field(formatter &t, std::string const &name, T const &value) {
t.field(name, boost::lexical_cast<std::string>(value));
}
formatter::ptr create_xml_formatter();
}
//----------------------------------------------------------------
#endif

19
dbg-lib/simple_show_traits.cc Normal file
View File

@ -0,0 +1,19 @@
#include "dbg-lib/simple_show_traits.h"
using namespace dbg;
//----------------------------------------------------------------
void
uint32_show_traits::show(formatter::ptr f, string const &key, uint32_t const &value)
{
field(*f, key, boost::lexical_cast<string>(value));
}
void
uint64_show_traits::show(formatter::ptr f, string const &key, uint64_t const &value)
{
field(*f, key, boost::lexical_cast<string>(value));
}
//----------------------------------------------------------------

27
dbg-lib/simple_show_traits.h Normal file
View File

@ -0,0 +1,27 @@
#ifndef DBG_SIMPLE_SHOW_TRAITS_H
#define DBG_SIMPLE_SHOW_TRAITS_H
#include "dbg-lib/output_formatter.h"
#include "persistent-data/data-structures/simple_traits.h"
//----------------------------------------------------------------
namespace dbg {
class uint32_show_traits {
public:
typedef persistent_data::uint32_traits value_trait;
static void show(dbg::formatter::ptr f, std::string const &key, uint32_t const &value);
};
class uint64_show_traits {
public:
typedef persistent_data::uint64_traits value_trait;
static void show(dbg::formatter::ptr f, std::string const &key, uint64_t const &value);
};
}
//----------------------------------------------------------------
#endif

27
dbg-lib/sm_show_traits.cc Normal file
View File

@ -0,0 +1,27 @@
#include "dbg-lib/sm_show_traits.h"
using namespace dbg;
using namespace std;
//----------------------------------------------------------------
void
index_entry_show_traits::show(formatter::ptr f, string const &key,
persistent_data::sm_disk_detail::index_entry const &value)
{
field(*f, "blocknr", value.blocknr_);
field(*f, "nr_free", value.nr_free_);
field(*f, "none_free_before", value.none_free_before_);
}
void
sm_root_show_traits::show(formatter::ptr f, string const &key,
persistent_data::sm_disk_detail::sm_root const &value)
{
field(*f, "nr_blocks", value.nr_blocks_);
field(*f, "nr_allocated", value.nr_allocated_);
field(*f, "bitmap_root", value.bitmap_root_);
field(*f, "ref_count_root", value.ref_count_root_);
}
//----------------------------------------------------------------

29
dbg-lib/sm_show_traits.h Normal file
View File

@ -0,0 +1,29 @@
#ifndef DBG_SM_SHOW_TRAITS_H
#define DBG_SM_SHOW_TRAITS_H
#include "dbg-lib/output_formatter.h"
#include "persistent-data/space-maps/disk_structures.h"
//----------------------------------------------------------------
namespace dbg {
class index_entry_show_traits {
public:
typedef persistent_data::sm_disk_detail::index_entry_traits value_trait;
static void show(dbg::formatter::ptr f, std::string const &key,
persistent_data::sm_disk_detail::index_entry const &value);
};
class sm_root_show_traits {
public:
typedef persistent_data::sm_disk_detail::sm_root_traits value_trait;
static void show(dbg::formatter::ptr f, std::string const &key,
persistent_data::sm_disk_detail::sm_root const &value);
};
}
//----------------------------------------------------------------
#endif

12
era/commands.h
View File

@ -34,6 +34,18 @@ namespace era {
virtual int run(int argc, char **argv);
};
//------------------------------------------------------
class era_debug_cmd : public base::command {
public:
era_debug_cmd();
virtual void usage(std::ostream &out) const;
virtual int run(int argc, char **argv);
};
//------------------------------------------------------
void register_era_commands(base::application &app);
}

14
era/devel_commands.cc Normal file
View File

@ -0,0 +1,14 @@
#include "era/commands.h"
using namespace base;
using namespace era;
//----------------------------------------------------------------
void
era::register_era_commands(base::application &app)
{
app.add_cmd(command::ptr(new era_debug_cmd));
}
//----------------------------------------------------------------

228
era/era_debug.cc Normal file
View File

@ -0,0 +1,228 @@
// Copyright (C) 2012 Red Hat, Inc. All rights reserved.
//
// This file is part of the thin-provisioning-tools source.
//
// thin-provisioning-tools is free software: you can redistribute it
// and/or modify it under the terms of the GNU General Public License
// as published by the Free Software Foundation, either version 3 of
// the License, or (at your option) any later version.
//
// thin-provisioning-tools is distributed in the hope that it will be
// useful, but WITHOUT ANY WARRANTY; without even the implied warranty
// of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License along
// with thin-provisioning-tools. If not, see
// <http://www.gnu.org/licenses/>.
#include <boost/lexical_cast.hpp>
#include <getopt.h>
#include <iostream>
#include <string>
#include "dbg-lib/array_block_dumper.h"
#include "dbg-lib/btree_node_dumper.h"
#include "dbg-lib/bitset_block_dumper.h"
#include "dbg-lib/command_interpreter.h"
#include "dbg-lib/commands.h"
#include "dbg-lib/index_block_dumper.h"
#include "dbg-lib/output_formatter.h"
#include "dbg-lib/sm_show_traits.h"
#include "persistent-data/file_utils.h"
#include "persistent-data/space-maps/disk_structures.h"
#include "era/commands.h"
#include "era/metadata.h"
#include "version.h"
using namespace dbg;
using namespace persistent_data;
using namespace std;
using namespace era;
//----------------------------------------------------------------
namespace {
class help : public dbg::command {
virtual void exec(strings const &args, ostream &out) {
out << "Commands:" << endl
<< " superblock [block#]" << endl
<< " block_node <block# of array block-tree node>" << endl
<< " bitset_block <block# of bitset block>" << endl
<< " era_block <block# of era array block>" << endl
<< " index_block <block# of metadata space map root>" << endl
<< " writeset_node <block# of writeset tree node>" << endl
<< " exit" << endl;
}
};
// for displaying the writeset tree
class writeset_show_traits : public era::era_detail_traits {
public:
typedef era_detail_traits value_trait;
static void show(formatter::ptr f, string const &key, era_detail const &value) {
field(*f, "nr_bits", value.nr_bits);
field(*f, "writeset_root", value.writeset_root);
}
};
class show_superblock : public dbg::command {
public:
explicit show_superblock(block_manager::ptr bm)
: bm_(bm) {
}
virtual void exec(strings const &args, ostream &out) {
if (args.size() > 2)
throw runtime_error("incorrect number of arguments");
block_address b = era::SUPERBLOCK_LOCATION;
if (args.size() == 2)
b = boost::lexical_cast<block_address>(args[1]);
era::superblock sb = read_superblock(bm_, b);
formatter::ptr f = create_xml_formatter();
ostringstream version;
field(*f, "csum", sb.csum);
field(*f, "flags", sb.flags.encode());
field(*f, "blocknr", sb.blocknr);
field(*f, "uuid", sb.uuid); // FIXME: delimit, and handle non-printable chars
field(*f, "magic", sb.magic);
field(*f, "version", sb.version);
sm_disk_detail::sm_root_disk const *d;
sm_disk_detail::sm_root v;
{
d = reinterpret_cast<sm_disk_detail::sm_root_disk const *>(sb.metadata_space_map_root);
sm_disk_detail::sm_root_traits::unpack(*d, v);
formatter::ptr f2 = create_xml_formatter();
sm_root_show_traits::show(f2, "value", v);
f->child("metadata_space_map_root", f2);
}
field(*f, "data_block_size", sb.data_block_size);
field(*f, "metadata_block_size", sb.metadata_block_size);
field(*f, "nr_blocks", sb.nr_blocks);
field(*f, "current_era", sb.current_era);
{
formatter::ptr f2 = create_xml_formatter();
writeset_show_traits::show(f2, "value", sb.current_detail);
f->child("current_writeset", f2);
}
field(*f, "writeset_tree_root", sb.writeset_tree_root);
field(*f, "era_array_root", sb.era_array_root);
if (sb.metadata_snap)
field(*f, "metadata_snap", *sb.metadata_snap);
f->output(out, 0);
}
private:
block_manager::ptr bm_;
};
//--------------------------------
template <typename ShowTraits>
dbg::command::ptr
create_btree_node_handler(block_manager::ptr bm) {
return create_block_handler(bm, create_btree_node_dumper<ShowTraits>());
}
template <typename ShowTraits>
dbg::command::ptr
create_array_block_handler(block_manager::ptr bm,
typename ShowTraits::value_trait::ref_counter rc) {
return create_block_handler(bm, create_array_block_dumper<ShowTraits>(rc));
}
dbg::command::ptr
create_bitset_block_handler(block_manager::ptr bm) {
return create_block_handler(bm, create_bitset_block_dumper());
}
dbg::command::ptr
create_index_block_handler(block_manager::ptr bm) {
return create_block_handler(bm, create_index_block_dumper());
}
int debug(string const &path) {
using dbg::command;
try {
block_manager::ptr bm = open_bm(path, block_manager::READ_ONLY);
transaction_manager::ptr null_tm = open_tm(bm, era::SUPERBLOCK_LOCATION);
command_interpreter::ptr interp = create_command_interpreter(cin, cout);
interp->register_command("hello", create_hello_handler());
interp->register_command("superblock", command::ptr(new show_superblock(bm)));
interp->register_command("block_node", create_btree_node_handler<uint64_show_traits>(bm));
interp->register_command("bitset_block", create_bitset_block_handler(bm));
interp->register_command("era_block", create_array_block_handler<uint32_show_traits>(bm,
uint32_traits::ref_counter()));
interp->register_command("index_block", create_index_block_handler(bm));
interp->register_command("writeset_node", create_btree_node_handler<writeset_show_traits>(bm));
interp->register_command("help", command::ptr(new help));
interp->register_command("exit", create_exit_handler(interp));
interp->enter_main_loop();
} catch (std::exception &e) {
cerr << e.what();
return 1;
}
return 0;
}
}
//----------------------------------------------------------------
era_debug_cmd::era_debug_cmd()
: command("era_debug")
{
}
void
era_debug_cmd::usage(std::ostream &out) const
{
out << "Usage: " << get_name() << " {device|file}" << endl
<< "Options:" << endl
<< " {-h|--help}" << endl
<< " {-V|--version}" << endl;
}
int
era_debug_cmd::run(int argc, char **argv)
{
int c;
const char shortopts[] = "hV";
const struct option longopts[] = {
{ "help", no_argument, NULL, 'h'},
{ "version", no_argument, NULL, 'V'},
{ NULL, no_argument, NULL, 0 }
};
while ((c = getopt_long(argc, argv, shortopts, longopts, NULL)) != -1) {
switch(c) {
case 'h':
usage(cout);
return 0;
case 'V':
cerr << THIN_PROVISIONING_TOOLS_VERSION << endl;
return 0;
}
}
if (argc == optind) {
usage(cerr);
exit(1);
}
return debug(argv[optind]);
}
//----------------------------------------------------------------

10
era/era_detail.h
View File

@ -22,6 +22,14 @@ namespace era {
uint64_t writeset_root;
};
inline bool operator==(era_detail const& lhs, era_detail const& rhs) {
return lhs.nr_bits == rhs.nr_bits && lhs.writeset_root == rhs.writeset_root;
}
inline bool operator!=(era_detail const& lhs, era_detail const& rhs) {
return !(lhs == rhs);
}
struct era_detail_ref_counter {
era_detail_ref_counter(persistent_data::transaction_manager::ptr tm)
: tm_(tm) {
@ -31,7 +39,7 @@ namespace era {
tm_->get_sm()->inc(d.writeset_root);
}
void dec(persistent_data::block_address b) {
void dec(era_detail const &d) {
// I don't think we ever do this in the tools
throw std::runtime_error("not implemented");
}

2
era/era_restore.cc
View File

@ -37,7 +37,7 @@ namespace {
bool metadata_touched = false;
try {
block_manager::ptr bm = open_bm(*fs.output, block_manager::READ_WRITE);
file_utils::check_file_exists(*fs.output);
file_utils::check_file_exists(*fs.input);
metadata_touched = true;
metadata::ptr md(new metadata(bm, metadata::CREATE));
emitter::ptr restorer = create_restore_emitter(*md);

2
era/metadata.cc
View File

@ -44,7 +44,7 @@ metadata::metadata(block_manager::ptr bm, open_type ot)
metadata::metadata(block_manager::ptr bm, block_address metadata_snap)
{
open_metadata(bm);
open_metadata(bm, metadata_snap);
}
void

1
era/restore_emitter.cc
View File

@ -14,6 +14,7 @@ namespace {
: md_(md),
in_superblock_(false),
in_writeset_(false),
era_(0),
in_era_array_(false) {
}

322
functional-tests/cache-functional-tests.scm
View File

@ -49,335 +49,13 @@
;; to run.
(define (register-cache-tests) #t)
;;;-----------------------------------------------------------
;;; cache_check scenarios
;;;-----------------------------------------------------------
(define-scenario (cache-check v)
"cache_check -V"
(run-ok-rcv (stdout _) (cache-check "-V")
(assert-equal tools-version stdout)))
(define-scenario (cache-check version)
"cache_check --version"
(run-ok-rcv (stdout _) (cache-check "--version")
(assert-equal tools-version stdout)))
(define-scenario (cache-check h)
"cache_check -h"
(run-ok-rcv (stdout _) (cache-check "-h")
(assert-equal cache-check-help stdout)))
(define-scenario (cache-check help)
"cache_check --help"
(run-ok-rcv (stdout _) (cache-check "--help")
(assert-equal cache-check-help stdout)))
(define-scenario (cache-check must-specify-metadata)
"Metadata file must be specified"
(run-fail-rcv (_ stderr) (cache-check)
(assert-equal
(string-append "No input file provided.\n"
cache-check-help)
stderr)))
(define-scenario (cache-check no-such-metadata)
"Metadata file doesn't exist."
(let ((bad-path "/arbitrary/filename"))
(run-fail-rcv (_ stderr) (cache-check bad-path)
(assert-starts-with
(string-append bad-path ": No such file or directory")
stderr))))
(define-scenario (cache-check metadata-file-cannot-be-a-directory)
"Metadata file must not be a directory"
(let ((bad-path "/tmp"))
(run-fail-rcv (_ stderr) (cache-check bad-path)
(assert-starts-with
(string-append bad-path ": Not a block device or regular file")
stderr))))
(define-scenario (cache-check unreadable-metadata)
"Metadata file exists, but is unreadable."
(with-valid-metadata (md)
(run-ok "chmod" "-r" md)
(run-fail-rcv (_ stderr) (cache-check md)
(assert-starts-with "syscall 'open' failed: Permission denied" stderr))))
(define-scenario (cache-check fails-with-corrupt-metadata)
"Fail with corrupt superblock"
(with-corrupt-metadata (md)
(run-fail (cache-check md))))
(define-scenario (cache-check failing-q)
"Fail quietly with -q"
(with-corrupt-metadata (md)
(run-fail-rcv (stdout stderr) (cache-check "-q" md)
(assert-eof stdout)
(assert-eof stderr))))
(define-scenario (cache-check failing-quiet)
"Fail quietly with --quiet"
(with-corrupt-metadata (md)
(run-fail-rcv (stdout stderr) (cache-check "--quiet" md)
(assert-eof stdout)
(assert-eof stderr))))
(define-scenario (cache-check valid-metadata-passes)
"A valid metadata area passes"
(with-valid-metadata (md)
(run-ok (cache-check md))))
(define-scenario (cache-check bad-metadata-version)
"Invalid metadata version fails"
(with-cache-xml (xml)
(with-empty-metadata (md)
(cache-restore "-i" xml "-o" md "--debug-override-metadata-version" "12345")
(run-fail (cache-check md)))))
(define-scenario (cache-check tiny-metadata)
"Prints helpful message in case tiny metadata given"
(with-temp-file-sized ((md "cache.bin" 1024))
(run-fail-rcv (_ stderr) (cache-check md)
(assert-starts-with "Metadata device/file too small. Is this binary metadata?" stderr))))
(define-scenario (cache-check spot-accidental-xml-data)
"Prints helpful message if XML metadata given"
(with-cache-xml (xml)
(system (fmt #f "man bash >> " xml))
(run-fail-rcv (_ stderr) (cache-check xml)
(assert-matches ".*This looks like XML. cache_check only checks the binary metadata format." stderr))))
;;;-----------------------------------------------------------
;;; cache_restore scenarios
;;;-----------------------------------------------------------
(define-scenario (cache-restore v)
"print version (-V flag)"
(run-ok-rcv (stdout _) (cache-restore "-V")
(assert-equal tools-version stdout)))
(define-scenario (cache-restore version)
"print version (--version flags)"
(run-ok-rcv (stdout _) (cache-restore "--version")
(assert-equal tools-version stdout)))
(define-scenario (cache-restore h)
"cache_restore -h"
(run-ok-rcv (stdout _) (cache-restore "-h")
(assert-equal cache-restore-help stdout)))
(define-scenario (cache-restore help)
"cache_restore --help"
(run-ok-rcv (stdout _) (cache-restore "--help")
(assert-equal cache-restore-help stdout)))
(define-scenario (cache-restore no-input-file)
"forget to specify an input file"
(with-empty-metadata (md)
(run-fail-rcv (_ stderr) (cache-restore "-o" md)
(assert-starts-with "No input file provided." stderr))))
(define-scenario (cache-restore missing-input-file)
"the input file can't be found"
(with-empty-metadata (md)
(run-fail-rcv (_ stderr) (cache-restore "-i no-such-file -o" md)
(assert-superblock-all-zeroes md)
(assert-starts-with "Couldn't stat file" stderr))))
(define-scenario (cache-restore garbage-input-file)
"the input file is just zeroes"
(with-empty-metadata (md)
(with-temp-file-sized ((xml "cache.xml" 4096))
(run-fail-rcv (_ stderr) (cache-restore "-i" xml "-o" md)
(assert-superblock-all-zeroes md)))))
(define-scenario (cache-restore missing-output-file)
"the output file can't be found"
(with-cache-xml (xml)
(run-fail-rcv (_ stderr) (cache-restore "-i" xml)
(assert-starts-with "No output file provided." stderr))))
(define-scenario (cache-restore tiny-output-file)
"Fails if the output file is too small."
(with-temp-file-sized ((md "cache.bin" (* 1024 4)))
(with-cache-xml (xml)
(run-fail-rcv (_ stderr) (cache-restore "-i" xml "-o" md)
(assert-starts-with cache-restore-outfile-too-small-text stderr)))))
(define-scenario (cache-restore successfully-restores)
"Restore succeeds."
(with-empty-metadata (md)
(with-cache-xml (xml)
(run-ok (cache-restore "-i" xml "-o" md)))))
(define-scenario (cache-restore q)
"cache_restore accepts -q"
(with-empty-metadata (md)
(with-cache-xml (xml)
(run-ok-rcv (stdout stderr) (cache-restore "-i" xml "-o" md "-q")
(assert-eof stdout)
(assert-eof stderr)))))
(define-scenario (cache-restore quiet)
"cache_restore accepts --quiet"
(with-empty-metadata (md)
(with-cache-xml (xml)
(run-ok-rcv (stdout stderr) (cache-restore "-i" xml "-o" md "--quiet")
(assert-eof stdout)
(assert-eof stderr)))))
(define-scenario (cache-restore override-metadata-version)
"we can set any metadata version"
(with-empty-metadata (md)
(with-cache-xml (xml)
(run-ok
(cache-restore "-i" xml "-o" md "--debug-override-metadata-version 10298")))))
(define-scenario (cache-restore omit-clean-shutdown)
"accepts --omit-clean-shutdown"
(with-empty-metadata (md)
(with-cache-xml (xml)
(run-ok
(cache-restore "-i" xml "-o" md "--omit-clean-shutdown")))))
;;;-----------------------------------------------------------
;;; cache_dump scenarios
;;;-----------------------------------------------------------
(define-scenario (cache-dump v)
"print version (-V flag)"
(run-ok-rcv (stdout _) (cache-dump "-V")
(assert-equal tools-version stdout)))
(define-scenario (cache-dump version)
"print version (--version flags)"
(run-ok-rcv (stdout _) (cache-dump "--version")
(assert-equal tools-version stdout)))
(define-scenario (cache-dump h)
"cache_dump -h"
(run-ok-rcv (stdout _) (cache-dump "-h")
(assert-equal cache-dump-help stdout)))
(define-scenario (cache-dump help)
"cache_dump --help"
(run-ok-rcv (stdout _) (cache-dump "--help")
(assert-equal cache-dump-help stdout)))
(define-scenario (cache-dump missing-input-file)
"Fails with missing input file."
(run-fail-rcv (stdout stderr) (cache-dump)
(assert-starts-with "No input file provided." stderr)))
(define-scenario (cache-dump small-input-file)
"Fails with small input file"
(with-temp-file-sized ((md "cache.bin" 512))
(run-fail
(cache-dump md))))
(define-scenario (cache-dump restore-is-noop)
"cache_dump followed by cache_restore is a noop."
(with-valid-metadata (md)
(run-ok-rcv (d1-stdout _) (cache-dump md)
(with-temp-file-containing ((xml "cache.xml" d1-stdout))
(run-ok (cache-restore "-i" xml "-o" md))
(run-ok-rcv (d2-stdout _) (cache-dump md)
(assert-equal d1-stdout d2-stdout))))))
;;;-----------------------------------------------------------
;;; cache_metadata_size scenarios
;;;-----------------------------------------------------------
(define-scenario (cache-metadata-size v)
"cache_metadata_size -V"
(run-ok-rcv (stdout _) (cache-metadata-size "-V")
(assert-equal tools-version stdout)))
(define-scenario (cache-metadata-size version)
"cache_metadata_size --version"
(run-ok-rcv (stdout _) (cache-metadata-size "--version")
(assert-equal tools-version stdout)))
(define-scenario (cache-metadata-size h)
"cache_metadata_size -h"
(run-ok-rcv (stdout _) (cache-metadata-size "-h")
(assert-equal cache-metadata-size-help stdout)))
(define-scenario (cache-metadata-size help)
"cache_metadata_size --help"
(run-ok-rcv (stdout _) (cache-metadata-size "--help")
(assert-equal cache-metadata-size-help stdout)))
(define-scenario (cache-metadata-size no-args)
"No arguments specified causes fail"
(run-fail-rcv (_ stderr) (cache-metadata-size)
(assert-equal "Please specify either --device-size and --block-size, or --nr-blocks."
stderr)))
(define-scenario (cache-metadata-size device-size-only)
"Just --device-size causes fail"
(run-fail-rcv (_ stderr) (cache-metadata-size "--device-size" (to-bytes (meg 100)))
(assert-equal "If you specify --device-size you must also give --block-size."
stderr)))
(define-scenario (cache-metadata-size block-size-only)
"Just --block-size causes fail"
(run-fail-rcv (_ stderr) (cache-metadata-size "--block-size" 128)
(assert-equal "If you specify --block-size you must also give --device-size."
stderr)))
(define-scenario (cache-metadata-size conradictory-info-fails)
"Contradictory info causes fail"
(run-fail-rcv (_ stderr) (cache-metadata-size "--device-size 102400 --block-size 1000 --nr-blocks 6")
(assert-equal "Contradictory arguments given, --nr-blocks doesn't match the --device-size and --block-size."
stderr)))
(define-scenario (cache-metadata-size all-args-agree)
"All args agreeing succeeds"
(run-ok-rcv (stdout stderr) (cache-metadata-size "--device-size" 102400 "--block-size" 100 "--nr-blocks" 1024)
(assert-equal "8248 sectors" stdout)
(assert-eof stderr)))
(define-scenario (cache-metadata-size nr-blocks-alone)
"Just --nr-blocks succeeds"
(run-ok-rcv (stdout stderr) (cache-metadata-size "--nr-blocks" 1024)
(assert-equal "8248 sectors" stdout)
(assert-eof stderr)))
(define-scenario (cache-metadata-size dev-size-and-block-size-succeeds)
"Specifying --device-size with --block-size succeeds"
(run-ok-rcv (stdout stderr) (cache-metadata-size "--device-size" 102400 "--block-size" 100)
(assert-equal "8248 sectors" stdout)
(assert-eof stderr)))
(define-scenario (cache-metadata-size big-config)
"A big configuration succeeds"
(run-ok-rcv (stdout stderr) (cache-metadata-size "--nr-blocks 67108864")
(assert-equal "3678208 sectors" stdout)
(assert-eof stderr)))
;;;-----------------------------------------------------------
;;; cache_repair scenarios
;;;-----------------------------------------------------------
(define-scenario (cache-repair missing-input-file)
"the input file can't be found"
(with-empty-metadata (md)
(run-fail-rcv (_ stderr) (cache-repair "-i no-such-file -o" md)
(assert-superblock-all-zeroes md)
(assert-starts-with "Couldn't stat path" stderr))))
(define-scenario (cache-repair garbage-input-file)
"the input file is just zeroes"
(with-empty-metadata (md1)
(with-corrupt-metadata (md2)
(run-fail-rcv (_ stderr) (cache-repair "-i" md1 "-o" md2)
(assert-superblock-all-zeroes md2)))))
(define-scenario (cache-repair missing-output-file)
"the output file can't be found"
(with-cache-xml (xml)
(run-fail-rcv (_ stderr) (cache-repair "-i" xml)
(assert-starts-with "No output file provided." stderr))))
)

169
functional-tests/era-functional-tests.scm
View File

@ -43,153 +43,9 @@
(define (register-era-tests) #t)
;;;-----------------------------------------------------------
;;; era_check scenarios
;;;-----------------------------------------------------------
(define-scenario (era-check v)
"era_check -V"
(run-ok-rcv (stdout _) (era-check "-V")
(assert-equal tools-version stdout)))
(define-scenario (era-check version)
"era_check --version"
(run-ok-rcv (stdout _) (era-check "--version")
(assert-equal tools-version stdout)))
(define-scenario (era-check h)
"era_check -h"
(run-ok-rcv (stdout _) (era-check "-h")
(assert-equal era-check-help stdout)))
(define-scenario (era-check help)
"era_check --help"
(run-ok-rcv (stdout _) (era-check "--help")
(assert-equal era-check-help stdout)))
(define-scenario (era-check no-device-specified)
"Fail if no device specified"
(run-fail-rcv (_ stderr) (era-check)
(assert-starts-with "No input file provided." stderr)))
(define-scenario (era-check dev-not-exist)
"Fail if specified device doesn't exist"
(run-fail-rcv (_ stderr) (era-check "/dev/unlikely")
(assert-starts-with "/dev/unlikely: No such file or directory" stderr)))
(define-scenario (era-check dev-is-a-directory)
"Fail if given a directory instead of a file or device"
(run-fail-rcv (_ stderr) (era-check "/tmp")
(assert-starts-with "/tmp: Not a block device or regular file" stderr)))
(define-scenario (era-check bad-permissions)
"Fail if given a device with inadequate access permissions"
(with-temp-file-sized ((md "era.bin" (meg 4)))
(run-ok "chmod -r" md)
(run-fail-rcv (_ stderr) (era-check md)
(assert-starts-with "syscall 'open' failed: Permission denied" stderr))))
(define-scenario (era-check empty-dev)
"Fail if given a file of zeroes"
(with-empty-metadata (md)
(run-fail (era-check md))))
(define-scenario (era-check quiet)
"Fail should give no output if --quiet"
(with-empty-metadata (md)
(run-fail-rcv (stdout stderr) (era-check "--quiet" md)
(assert-eof stdout)
(assert-eof stderr))))
(define-scenario (era-check q)
"Fail should give no output if -q"
(with-empty-metadata (md)
(run-fail-rcv (stdout stderr) (era-check "-q" md)
(assert-eof stdout)
(assert-eof stderr))))
(define-scenario (era-check tiny-metadata)
"Prints helpful message in case tiny metadata given"
(with-temp-file-sized ((md "era.bin" 1024))
(run-fail-rcv (_ stderr) (era-check md)
(assert-starts-with "Metadata device/file too small. Is this binary metadata?" stderr))))
(define-scenario (era-check spot-accidental-xml-data)
"Prints helpful message if XML metadata given"
(with-era-xml (xml)
(system (fmt #f "man bash >> " xml))
(run-fail-rcv (_ stderr) (era-check xml)
(assert-matches ".*This looks like XML. era_check only checks the binary metadata format." stderr))))
;;;-----------------------------------------------------------
;;; era_restore scenarios
;;;-----------------------------------------------------------
(define-scenario (era-restore v)
"era_restore -V"
(run-ok-rcv (stdout _) (era-restore "-V")
(assert-equal tools-version stdout)))
(define-scenario (era-restore version)
"era_restore --version"
(run-ok-rcv (stdout _) (era-restore "--version")
(assert-equal tools-version stdout)))
(define-scenario (era-restore h)
"era_restore -h"
(run-ok-rcv (stdout _) (era-restore "-h")
(assert-equal era-restore-help stdout)))
(define-scenario (era-restore help)
"era_restore --help"
(run-ok-rcv (stdout _) (era-restore "--help")
(assert-equal era-restore-help stdout)))
(define-scenario (era-restore input-unspecified)
"Fails if no xml specified"
(with-empty-metadata (md)
(run-fail-rcv (_ stderr) (era-restore "-o" md)
(assert-starts-with "No input file provided." stderr))))
(define-scenario (era-restore missing-input-file)
"the input file can't be found"
(with-empty-metadata (md)
(run-fail-rcv (_ stderr) (era-restore "-i no-such-file -o" md)
(assert-superblock-all-zeroes md)
(assert-starts-with "Couldn't stat file" stderr))))
(define-scenario (era-restore garbage-input-file)
"the input file is just zeroes"
(with-empty-metadata (md)
(with-temp-file-sized ((xml "era.xml" 4096))
(run-fail-rcv (_ stderr) (era-restore "-i " xml "-o" md)
(assert-superblock-all-zeroes md)))))
(define-scenario (era-restore output-unspecified)
"Fails if no metadata dev specified"
(with-era-xml (xml)
(run-fail-rcv (_ stderr) (era-restore "-i" xml)
(assert-starts-with "No output file provided." stderr))))
(define-scenario (era-restore success)
"Succeeds with xml and metadata"
(with-era-xml (xml)
(with-empty-metadata (md)
(run-ok (era-restore "-i" xml "-o" md)))))
(define-scenario (era-restore quiet)
"No output with --quiet (succeeding)"
(with-era-xml (xml)
(with-empty-metadata (md)
(run-ok-rcv (stdout stderr) (era-restore "--quiet" "-i" xml "-o" md)
(assert-eof stdout)
(assert-eof stderr)))))
(define-scenario (era-restore q)
"No output with -q (succeeding)"
(with-era-xml (xml)
(with-empty-metadata (md)
(run-ok-rcv (stdout stderr) (era-restore "-q" "-i" xml "-o" md)
(assert-eof stdout)
(assert-eof stderr)))))
(define-scenario (era-restore quiet-fail)
"No output with --quiet (failing)"
@ -197,7 +53,9 @@
(with-empty-metadata (md)
(run-fail-rcv (stdout stderr) (era-restore "--quiet" "-i" bad-xml "-o" md)
(assert-eof stdout)
(assert-starts-with "Couldn't stat file" stderr)))))
(assert-starts-with
(string-append bad-xml ": No such file or directory")
stderr)))))
(define-scenario (era-restore q-fail)
"No output with --q(failing)"
@ -205,22 +63,7 @@
(with-empty-metadata (md)
(run-fail-rcv (stdout stderr) (era-restore "-q" "-i" bad-xml "-o" md)
(assert-eof stdout)
(assert-starts-with "Couldn't stat file" stderr)))))
;;;-----------------------------------------------------------
;;; era_dump scenarios
;;;-----------------------------------------------------------
(define-scenario (era-dump small-input-file)
"Fails with small input file"
(with-temp-file-sized ((md "era.bin" 512))
(run-fail (era-dump md))))
(define-scenario (era-dump restore-is-noop)
"era_dump followed by era_restore is a noop."
(with-valid-metadata (md)
(run-ok-rcv (d1-stdout _) (era-dump md)
(with-temp-file-containing ((xml "era.xml" d1-stdout))
(run-ok (era-restore "-i" xml "-o" md))
(run-ok-rcv (d2-stdout _) (era-dump md)
(assert-equal d1-stdout d2-stdout))))))
(assert-starts-with
(string-append bad-xml ": No such file or directory")
stderr)))))
)

4
functional-tests/run-tests.scm
View File

@ -1,10 +1,8 @@
(import (rnrs)
(test-runner)
(cache-functional-tests)
(era-functional-tests)
(thin-functional-tests))
(era-functional-tests))
(register-thin-tests)
(register-cache-tests)
(register-era-tests)

3
functional-tests/test-runner.scm
View File

@ -14,8 +14,7 @@
(only (srfi s1 lists) break)
(regex)
(srfi s8 receive)
(temp-file)
(thin-functional-tests))
(temp-file))
;;------------------------------------------------

635
functional-tests/thin-functional-tests.scm
View File

@ -1,635 +0,0 @@
(library
(thin-functional-tests)
(export register-thin-tests)
(import
(chezscheme)
(bcache block-manager)
(disk-units)
(fmt fmt)
(functional-tests)
(process)
(scenario-string-constants)
(temp-file)
(thin xml)
(srfi s8 receive))
(define-tool thin-check)
(define-tool thin-delta)
(define-tool thin-dump)
(define-tool thin-restore)
(define-tool thin-rmap)
(define-tool thin-repair)
(define-tool thin-metadata-pack)
(define-tool thin-metadata-unpack)
(define-syntax with-thin-xml
(syntax-rules ()
((_ (v) b1 b2 ...)
(with-temp-file-containing ((v "thin.xml" (fmt #f (generate-xml 10 1000))))
b1 b2 ...))))
(define-syntax with-valid-metadata
(syntax-rules ()
((_ (md) b1 b2 ...)
(with-temp-file-sized ((md "thin.bin" (meg 4)))
(with-thin-xml (xml)
(run-ok (thin-restore "-i" xml "-o" md))
b1 b2 ...)))))
;;; It would be nice if the metadata was at least similar to valid data.
;;; Here I'm just using the start of the ls binary as 'random' data.
(define-syntax with-corrupt-metadata
(syntax-rules ()
((_ (md) b1 b2 ...)
(with-temp-file-sized ((md "thin.bin" (meg 4)))
(system (fmt #f "dd if=/usr/bin/ls of=" md " bs=4096 > /dev/null 2>&1"))
b1 b2 ...))))
(define-syntax with-empty-metadata
(syntax-rules ()
((_ (md) b1 b2 ...)
(with-temp-file-sized ((md "thin.bin" (meg 4)))
b1 b2 ...))))
(define (damage-superblock md)
(system (string-append "dd if=/dev/zero of=" md " bs=4K count=1 conv=notrunc > /dev/null 2>&1")))
(define-syntax with-damaged-superblock
(syntax-rules ()
((_ (md) b1 b2 ...)
(with-valid-metadata (md)
(damage-superblock md)
b1 b2 ...))))
;; We have to export something that forces all the initialisation expressions
;; to run.
(define (register-thin-tests) #t)
;;;-----------------------------------------------------------
;;; thin_check scenarios
;;;-----------------------------------------------------------
(define-scenario (thin-check v)
"thin_check -V"
(run-ok-rcv (stdout _) (thin-check "-V")
(assert-equal tools-version stdout)))
(define-scenario (thin-check version)
"thin_check --version"
(run-ok-rcv (stdout _) (thin-check "--version")
(assert-equal tools-version stdout)))
(define-scenario (thin-check h)
"print help (-h)"
(run-ok-rcv (stdout _) (thin-check "-h")
(assert-equal thin-check-help stdout)))
(define-scenario (thin-check help)
"print help (--help)"
(run-ok-rcv (stdout _) (thin-check "--help")
(assert-equal thin-check-help stdout)))
(define-scenario (thin-check bad-option)
"Unrecognised option should cause failure"
(run-fail (thin-check "--hedgehogs-only")))
(define-scenario (thin-check incompatible-options auto-repair)
"Incompatible options should cause failure"
(with-valid-metadata (md)
(run-fail (thin-check "--auto-repair" "-m" md))
(run-fail (thin-check "--auto-repair" "--override-mapping-root 123" md))
(run-fail (thin-check "--auto-repair" "--super-block-only" md))
(run-fail (thin-check "--auto-repair" "--skip-mappings" md))
(run-fail (thin-check "--auto-repair" "--ignore-non-fatal-errors" md))))
(define-scenario (thin-check incompatible-options clear-needs-check-flag)
"Incompatible options should cause failure"
(with-valid-metadata (md)
(run-fail (thin-check "--clear-needs-check-flag" "-m" md))
(run-fail (thin-check "--clear-needs-check-flag" "--override-mapping-root 123" md))
(run-fail (thin-check "--clear-needs-check-flag" "--super-block-only" md))
(run-fail (thin-check "--clear-needs-check-flag" "--skip-mappings" md))
(run-fail (thin-check "--clear-needs-check-flag" "--ignore-non-fatal-errors" md))))
(define-scenario (thin-check superblock-only-valid)
"--super-block-only check passes on valid metadata"
(with-valid-metadata (md)
(run-ok (thin-check "--super-block-only" md))))
(define-scenario (thin-check superblock-only-invalid)
"--super-block-only check fails with corrupt metadata"
(with-corrupt-metadata (md)
(run-fail (thin-check "--super-block-only" md))))
(define-scenario (thin-check skip-mappings-valid)
"--skip-mappings check passes on valid metadata"
(with-valid-metadata (md)
(run-ok (thin-check "--skip-mappings" md))))
(define-scenario (thin-check ignore-non-fatal-errors)
"--ignore-non-fatal-errors check passes on valid metadata"
(with-valid-metadata (md)
(run-ok (thin-check "--ignore-non-fatal-errors" md))))
(define-scenario (thin-check quiet)
"--quiet should give no output"
(with-valid-metadata (md)
(run-ok-rcv (stdout stderr) (thin-check "--quiet" md)
(assert-eof stdout)
(assert-eof stderr))))
(define-scenario (thin-check clear-needs-check-flag)
"Accepts --clear-needs-check-flag"
(with-valid-metadata (md)
(run-ok (thin-check "--clear-needs-check-flag" md))))
(define-scenario (thin-check auto-repair)
"Accepts --auto-repair"
(with-valid-metadata (md)
(run-ok (thin-check "--auto-repair" md))))
(define-scenario (thin-check tiny-metadata)
"Prints helpful message in case tiny metadata given"
(with-temp-file-sized ((md "thin.bin" 1024))
(run-fail-rcv (_ stderr) (thin-check md)
(assert-starts-with "Metadata device/file too small. Is this binary metadata?" stderr))))
(define-scenario (thin-check spot-accidental-xml-data)
"Prints helpful message if XML metadata given"
(with-thin-xml (xml)
(system (fmt #f "man bash >> " xml))
(run-fail-rcv (_ stderr) (thin-check xml)
(assert-matches ".*This looks like XML. thin_check only checks the binary metadata format." stderr))))
(define-scenario (thin-check info-fields)
"Outputs info fields"
(with-valid-metadata (md)
(run-ok-rcv (stdout stderr) (thin-check md)
(assert-matches ".*TRANSACTION_ID=[0-9]+.*" stdout)
(assert-matches ".*METADATA_FREE_BLOCKS=[0-9]+.*" stdout))))
;;;-----------------------------------------------------------
;;; thin_restore scenarios
;;;-----------------------------------------------------------
(define-scenario (thin-restore print-version-v)
"print help (-V)"
(run-ok-rcv (stdout _) (thin-restore "-V")
(assert-equal tools-version stdout)))
(define-scenario (thin-restore print-version-long)
"print help (--version)"
(run-ok-rcv (stdout _) (thin-restore "--version")
(assert-equal tools-version stdout)))
(define-scenario (thin-restore h)
"print help (-h)"
(run-ok-rcv (stdout _) (thin-restore "-h")
(assert-equal thin-restore-help stdout)))
(define-scenario (thin-restore help)
"print help (-h)"
(run-ok-rcv (stdout _) (thin-restore "--help")
(assert-equal thin-restore-help stdout)))
(define-scenario (thin-restore no-input-file)
"forget to specify an input file"
(with-empty-metadata (md)
(run-fail-rcv (_ stderr) (thin-restore "-o" md)
(assert-starts-with "No input file provided." stderr))))
(define-scenario (thin-restore missing-input-file)
"the input file can't be found"
(with-empty-metadata (md)
(run-fail-rcv (_ stderr) (thin-restore "-i no-such-file -o" md)
(assert-superblock-all-zeroes md)
(assert-starts-with "Couldn't stat file" stderr))))
(define-scenario (thin-restore garbage-input-file)
"the input file is just zeroes"
(with-empty-metadata (md)
(with-temp-file-sized ((xml "thin.xml" 4096))
(run-fail-rcv (_ stderr) (thin-restore "-i " xml "-o" md)
(assert-superblock-all-zeroes md)))))
(define-scenario (thin-restore missing-output-file)
"the output file can't be found"
(with-thin-xml (xml)
(run-fail-rcv (_ stderr) (thin-restore "-i " xml)
(assert-starts-with "No output file provided." stderr))))
(define-scenario (thin-restore tiny-output-file)
"Fails if the output file is too small."
(with-temp-file-sized ((md "thin.bin" 4096))
(with-thin-xml (xml)
(run-fail-rcv (_ stderr) (thin-restore "-i" xml "-o" md)
(assert-starts-with thin-restore-outfile-too-small-text stderr)))))
(define-scenario (thin-restore q)
"thin_restore accepts -q"
(with-empty-metadata (md)
(with-thin-xml (xml)
(run-ok-rcv (stdout _) (thin-restore "-i" xml "-o" md "-q")
(assert-eof stdout)))))
(define-scenario (thin-restore quiet)
"thin_restore accepts --quiet"
(with-empty-metadata (md)
(with-thin-xml (xml)
(run-ok-rcv (stdout _) (thin-restore "-i" xml "-o" md "--quiet")
(assert-eof stdout)))))
(define-scenario (thin-restore override transaction-id)
"thin_restore obeys the --transaction-id override"
(with-empty-metadata (md)
(with-thin-xml (xml)
(run-ok-rcv (stdout stderr) (thin-restore "--transaction-id 2345" "-i" xml "-o" md)
(assert-eof stderr))
(run-ok-rcv (stdout stderr) (thin-dump md)
(assert-matches ".*transaction=\"2345\"" stdout)))))
(define-scenario (thin-restore override data-block-size)
"thin_restore obeys the --data-block-size override"
(with-empty-metadata (md)
(with-thin-xml (xml)
(run-ok-rcv (stdout stderr) (thin-restore "--data-block-size 8192" "-i" xml "-o" md)
(assert-eof stderr))
(run-ok-rcv (stdout stderr) (thin-dump md)
(assert-matches ".*data_block_size=\"8192\"" stdout)))))
(define-scenario (thin-restore override nr-data-blocks)
"thin_restore obeys the --nr-data-blocks override"
(with-empty-metadata (md)
(with-thin-xml (xml)
(run-ok-rcv (stdout stderr) (thin-restore "--nr-data-blocks 234500" "-i" xml "-o" md)
(assert-eof stderr))
(run-ok-rcv (stdout stderr) (thin-dump md)
(assert-matches ".*nr_data_blocks=\"234500\"" stdout)))))
;;;-----------------------------------------------------------
;;; thin_dump scenarios
;;;-----------------------------------------------------------
(define-scenario (thin-dump small-input-file)
"Fails with small input file"
(with-temp-file-sized ((md "thin.bin" 512))
(run-fail (thin-dump md))))
(define-scenario (thin-dump restore-is-noop)
"thin_dump followed by thin_restore is a noop."
(with-valid-metadata (md)
(run-ok-rcv (d1-stdout _) (thin-dump md)
(with-temp-file-containing ((xml "thin.xml" d1-stdout))
(run-ok (thin-restore "-i" xml "-o" md))
(run-ok-rcv (d2-stdout _) (thin-dump md)
(assert-equal d1-stdout d2-stdout))))))
(define-scenario (thin-dump no-stderr)
"thin_dump of clean data does not output error messages to stderr"
(with-valid-metadata (md)
(run-ok-rcv (stdout stderr) (thin-dump md)
(assert-eof stderr))))
(define-scenario (thin-dump override transaction-id)
"thin_dump obeys the --transaction-id override"
(with-valid-metadata (md)
(run-ok-rcv (stdout stderr) (thin-dump "--transaction-id 2345" md)
(assert-eof stderr)
(assert-matches ".*transaction=\"2345\"" stdout))))
(define-scenario (thin-dump override data-block-size)
"thin_dump obeys the --data-block-size override"
(with-valid-metadata (md)
(run-ok-rcv (stdout stderr) (thin-dump "--data-block-size 8192" md)
(assert-eof stderr)
(assert-matches ".*data_block_size=\"8192\"" stdout))))
(define-scenario (thin-dump override nr-data-blocks)
"thin_dump obeys the --nr-data-blocks override"
(with-valid-metadata (md)
(run-ok-rcv (stdout stderr) (thin-dump "--nr-data-blocks 234500" md)
(assert-eof stderr)
(assert-matches ".*nr_data_blocks=\"234500\"" stdout))))
(define-scenario (thin-dump repair-superblock succeeds)
"thin_dump can restore a missing superblock"
(with-valid-metadata (md)
(run-ok-rcv (expected-xml stderr) (thin-dump "--transaction-id=5" "--data-block-size=128" "--nr-data-blocks=4096000" md)
(damage-superblock md)
(run-ok-rcv (repaired-xml stderr) (thin-dump "--repair" "--transaction-id=5" "--data-block-size=128" "--nr-data-blocks=4096000" md)
(assert-eof stderr)
(assert-equal expected-xml repaired-xml)))))
(define-scenario (thin-dump repair-superblock missing-transaction-id)
"--transaction-id is mandatory if the superblock is damaged"
(with-damaged-superblock (md)
(run-fail-rcv (_ stderr) (thin-dump "--repair" "--data-block-size=128" "--nr-data-blocks=4096000" md)
(assert-matches ".*transaction id.*" stderr))))
(define-scenario (thin-dump repair-superblock missing-data-block-size)
"--data-block-size is mandatory if the superblock is damaged"
(with-damaged-superblock (md)
(run-fail-rcv (_ stderr) (thin-dump "--repair" "--transaction-id=5" "--nr-data-blocks=4096000" md)
(assert-matches ".*data block size.*" stderr))))
(define-scenario (thin-dump repair-superblock missing-nr-data-blocks)
"--nr-data-blocks is mandatory if the superblock is damaged"
(with-damaged-superblock (md)
(run-fail-rcv (_ stderr) (thin-dump "--repair" "--transaction-id=5" "--data-block-size=128" md)
(assert-matches ".*nr data blocks.*" stderr))))
;;;-----------------------------------------------------------
;;; thin_rmap scenarios
;;;-----------------------------------------------------------
(define-scenario (thin-rmap v)
"thin_rmap accepts -V"
(run-ok-rcv (stdout _) (thin-rmap "-V")
(assert-equal tools-version stdout)))
(define-scenario (thin-rmap version)
"thin_rmap accepts --version"
(run-ok-rcv (stdout _) (thin-rmap "--version")
(assert-equal tools-version stdout)))
(define-scenario (thin-rmap h)
"thin_rmap accepts -h"
(run-ok-rcv (stdout _) (thin-rmap "-h")
(assert-equal thin-rmap-help stdout)))
(define-scenario (thin-rmap help)
"thin_rmap accepts --help"
(run-ok-rcv (stdout _) (thin-rmap "--help")
(assert-equal thin-rmap-help stdout)))
(define-scenario (thin-rmap unrecognised-flag)
"thin_rmap complains with bad flags."
(run-fail (thin-rmap "--unleash-the-hedgehogs")))
(define-scenario (thin-rmap valid-region-format-should-pass)
"thin_rmap with a valid region format should pass."
(with-valid-metadata (md)
(run-ok
(thin-rmap "--region 23..7890" md))))
(define-scenario (thin-rmap invalid-region-should-fail)
"thin_rmap with an invalid region format should fail."
(for-each (lambda (pattern)
(with-valid-metadata (md)
(run-fail (thin-rmap "--region" pattern md))))
'("23,7890" "23..six" "found..7890" "89..88" "89..89" "89.." "" "89...99")))
(define-scenario (thin-rmap multiple-regions-should-pass)
"thin_rmap should handle multiple regions."
(with-valid-metadata (md)
(run-ok (thin-rmap "--region 1..23 --region 45..78" md))))
(define-scenario (thin-rmap handles-junk-input)
"Fail gracefully if given nonsense"
(with-thin-xml (xml)
(run-fail-rcv (_ stderr) (thin-rmap "--region 0..-1" xml)
#t)))
;;;-----------------------------------------------------------
;;; thin_delta scenarios
;;;-----------------------------------------------------------
(define-scenario (thin-delta v)
"thin_delta accepts -V"
(run-ok-rcv (stdout _) (thin-delta "-V")
(assert-equal tools-version stdout)))
(define-scenario (thin-delta version)
"thin_delta accepts --version"
(run-ok-rcv (stdout _) (thin-delta "--version")
(assert-equal tools-version stdout)))
(define-scenario (thin-delta h)
"thin_delta accepts -h"
(run-ok-rcv (stdout _) (thin-delta "-h")
(assert-equal thin-delta-help stdout)))
(define-scenario (thin-delta help)
"thin_delta accepts --help"
(run-ok-rcv (stdout _) (thin-delta "--help")
(assert-equal thin-delta-help stdout)))
(define-scenario (thin-delta unrecognised-option)
"Unrecognised option should cause failure"
(with-valid-metadata (md)
(run-fail-rcv (stdout stderr) (thin-delta "--unleash-the-hedgehogs")
(assert-matches ".*thin_delta: unrecognized option '--unleash-the-hedgehogs" stderr))))
(define-scenario (thin-delta snap1-unspecified)
"Fails without --snap1 fails"
(run-fail-rcv (_ stderr) (thin-delta "--snap2 45 foo")
(assert-starts-with "--snap1 not specified." stderr)))
(define-scenario (thin-delta snap2-unspecified)
"Fails without --snap2 fails"
(run-fail-rcv (_ stderr) (thin-delta "--snap1 45 foo")
(assert-starts-with "--snap2 not specified." stderr)))
(define-scenario (thin-delta device-unspecified)
"Fails if no device given"
(run-fail-rcv (_ stderr) (thin-delta "--snap1 45 --snap2 46")
(assert-starts-with "No input device provided." stderr)))
;;;-----------------------------------------------------------
;;; thin_repair scenarios
;;;-----------------------------------------------------------
(define-scenario (thin-repair dont-repair-xml)
"Fails gracefully if run on XML rather than metadata"
(with-thin-xml (xml)
(with-empty-metadata (md)
(run-fail-rcv (_ stderr) (thin-repair "-i" xml "-o" md)
#t))))
(define-scenario (thin-repair missing-input-file)
"the input file can't be found"
(with-empty-metadata (md)
(run-fail-rcv (_ stderr) (thin-repair "-i no-such-file -o" md)
(assert-superblock-all-zeroes md)
(assert-starts-with "Couldn't stat file" stderr))))
(define-scenario (thin-repair garbage-input-file)
"the input file is just zeroes"
(with-empty-metadata (md1)
(with-corrupt-metadata (md2)
(run-fail-rcv (_ stderr) (thin-repair "-i " md1 "-o" md2)
(assert-superblock-all-zeroes md2)))))
(define-scenario (thin-repair missing-output-file)
"the output file can't be found"
(with-thin-xml (xml)
(run-fail-rcv (_ stderr) (thin-repair "-i " xml)
(assert-starts-with "No output file provided." stderr))))
(define-scenario (thin-repair override transaction-id)
"thin_repair obeys the --transaction-id override"
(with-valid-metadata (md1)
(with-empty-metadata (md2)
(run-ok-rcv (stdout stderr) (thin-repair "--transaction-id 2345" "-i" md1 "-o" md2)
(assert-eof stderr))
(run-ok-rcv (stdout stderr) (thin-dump md2)
(assert-matches ".*transaction=\"2345\"" stdout)))))
(define-scenario (thin-repair override data-block-size)
"thin_repair obeys the --data-block-size override"
(with-valid-metadata (md1)
(with-empty-metadata (md2)
(run-ok-rcv (stdout stderr) (thin-repair "--data-block-size 8192" "-i" md1 "-o" md2)
(assert-eof stderr))
(run-ok-rcv (stdout stderr) (thin-dump md2)
(assert-matches ".*data_block_size=\"8192\"" stdout)))))
(define-scenario (thin-repair override nr-data-blocks)
"thin_repair obeys the --nr-data-blocks override"
(with-valid-metadata (md1)
(with-empty-metadata (md2)
(run-ok-rcv (stdout stderr) (thin-repair "--nr-data-blocks 234500" "-i" md1 "-o" md2)
(assert-eof stderr))
(run-ok-rcv (stdout stderr) (thin-dump md2)
(assert-matches ".*nr_data_blocks=\"234500\"" stdout)))))
(define-scenario (thin-repair superblock succeeds)
"thin_repair can restore a missing superblock"
(with-valid-metadata (md1)
(run-ok-rcv (expected-xml stderr) (thin-dump "--transaction-id=5" "--data-block-size=128" "--nr-data-blocks=4096000" md1)
(damage-superblock md1)
(with-empty-metadata (md2)
(run-ok-rcv (_ stderr) (thin-repair "--transaction-id=5" "--data-block-size=128" "--nr-data-blocks=4096000" "-i" md1 "-o" md2)
(assert-eof stderr))
(run-ok-rcv (repaired-xml stderr) (thin-dump md2)
(assert-eof stderr)
(assert-equal expected-xml repaired-xml))))))
(define-scenario (thin-repair superblock missing-transaction-id)
"--transaction-id is mandatory if the superblock is damaged"
(with-damaged-superblock (md1)
(with-empty-metadata (md2)
(run-fail-rcv (_ stderr) (thin-repair "--data-block-size=128" "--nr-data-blocks=4096000" "-i" md1 "-o" md2)
(assert-matches ".*transaction id.*" stderr)))))
(define-scenario (thin-repair superblock missing-data-block-size)
"--data-block-size is mandatory if the superblock is damaged"
(with-damaged-superblock (md1)
(with-empty-metadata (md2)
(run-fail-rcv (_ stderr) (thin-repair "--transaction-id=5" "--nr-data-blocks=4096000" "-i" md1 "-o" md2)
(assert-matches ".*data block size.*" stderr)))))
(define-scenario (thin-repair superblock missing-nr-data-blocks)
"--nr-data-blocks is mandatory if the superblock is damaged"
(with-damaged-superblock (md1)
(with-empty-metadata (md2)
(run-fail-rcv (_ stderr) (thin-repair "--transaction-id=5" "--data-block-size=128" "-i" md1 "-o" md2)
(assert-matches ".*nr data blocks.*" stderr)))))
;;;-----------------------------------------------------------
;;; thin_metadata_pack scenarios
;;;-----------------------------------------------------------
(define-scenario (thin-metadata-pack version)
"accepts --version"
(run-ok-rcv (stdout _) (thin-metadata-pack "--version")
(assert-equal "thin_metadata_pack 0.9.0-rc2" stdout)))
(define-scenario (thin-metadata-pack h)
"accepts -h"
(run-ok-rcv (stdout _) (thin-metadata-pack "-h")
(assert-equal thin-metadata-pack-help stdout)))
(define-scenario (thin-metadata-pack help)
"accepts --help"
(run-ok-rcv (stdout _) (thin-metadata-pack "--help")
(assert-equal thin-metadata-pack-help stdout)))
(define-scenario (thin-metadata-pack unrecognised-option)
"Unrecognised option should cause failure"
(with-valid-metadata (md)
(run-fail-rcv (stdout stderr) (thin-metadata-pack "--unleash-the-hedgehogs")
(assert-starts-with "error: Found argument '--unleash-the-hedgehogs'" stderr))))
(define-scenario (thin-metadata-pack missing-input-file)
"the input file wasn't specified"
(with-empty-metadata (md)
(run-fail-rcv (_ stderr) (thin-metadata-pack "-o " md)
(assert-starts-with "error: The following required arguments were not provided:\n -i <DEV>" stderr))))
(define-scenario (thin-metadata-pack no-such-input-file)
"the input file can't be found"
(with-empty-metadata (md)
(run-fail-rcv (_ stderr) (thin-metadata-pack "-i no-such-file -o" md)
(assert-starts-with "Couldn't find input file" stderr))))
(define-scenario (thin-metadata-pack missing-output-file)
"the output file wasn't specified"
(with-empty-metadata (md)
(run-fail-rcv (_ stderr) (thin-metadata-pack "-i" md)
(assert-starts-with "error: The following required arguments were not provided:\n -o <FILE>" stderr))))
;;;-----------------------------------------------------------
;;; thin_metadata_unpack scenarios
;;;-----------------------------------------------------------
(define-scenario (thin-metadata-unpack version)
"accepts --version"
(run-ok-rcv (stdout _) (thin-metadata-unpack "--version")
(assert-equal "thin_metadata_unpack 0.9.0-rc2" stdout)))
(define-scenario (thin-metadata-unpack h)
"accepts -h"
(run-ok-rcv (stdout _) (thin-metadata-unpack "-h")
(assert-equal thin-metadata-unpack-help stdout)))
(define-scenario (thin-metadata-unpack help)
"accepts --help"
(run-ok-rcv (stdout _) (thin-metadata-unpack "--help")
(assert-equal thin-metadata-unpack-help stdout)))
(define-scenario (thin-metadata-unpack unrecognised-option)
"Unrecognised option should cause failure"
(with-valid-metadata (md)
(run-fail-rcv (stdout stderr) (thin-metadata-unpack "--unleash-the-hedgehogs")
(assert-starts-with "error: Found argument '--unleash-the-hedgehogs'" stderr))))
(define-scenario (thin-metadata-unpack missing-input-file)
"the input file wasn't specified"
(with-empty-metadata (md)
(run-fail-rcv (_ stderr) (thin-metadata-unpack "-o " md)
(assert-starts-with "error: The following required arguments were not provided:\n -i <DEV>" stderr))))
(define-scenario (thin-metadata-unpack no-such-input-file)
"the input file can't be found"
(with-empty-metadata (md)
(run-fail-rcv (_ stderr) (thin-metadata-unpack "-i no-such-file -o" md)
(assert-starts-with "Couldn't find input file" stderr))))
(define-scenario (thin-metadata-unpack missing-output-file)
"the output file wasn't specified"
(with-empty-metadata (md)
(run-fail-rcv (_ stderr) (thin-metadata-unpack "-i" md)
(assert-starts-with "error: The following required arguments were not provided:\n -o <FILE>" stderr))))
(define-scenario (thin-metadata-unpack garbage-input-file)
"the input file is just zeroes"
(with-empty-metadata (bad-pack)
(run-fail-rcv (_ stderr) (thin-metadata-unpack "-i " bad-pack "-o junk")
(assert-starts-with "Not a pack file." stderr))))
;;;-----------------------------------------------------------
;;; thin_metadata_pack/unpack end to end scenario
;;;-----------------------------------------------------------)
(define-scenario (thin-metadata-pack end-to-end)
"pack -> unpack recovers metadata"
(let ((pack-file "md.pack"))
(with-valid-metadata (md-in)
(with-empty-metadata (md-out)
(run-ok (thin-metadata-pack "-i" md-in "-o" pack-file))
(run-ok (thin-metadata-unpack "-i" pack-file "-o" md-out))
(run-ok-rcv (dump1 _) (thin-dump md-in)
(run-ok-rcv (dump2 _) (thin-dump md-out)
(assert-equal dump1 dump2)))))))
)

9
persistent-data/block_counter.h
View File

@ -62,6 +62,10 @@ namespace persistent_data {
return stop_on_error_;
}
virtual void clear() {
counts_.clear();
}
private:
count_map counts_;
bool stop_on_error_;
@ -86,6 +90,11 @@ namespace persistent_data {
base::run_set<block_address> const& get_visited() const {
return visited_;
}
virtual void clear() {
visited_.clear();
}
private:
base::run_set<block_address> visited_;
};

2
persistent-data/data-structures/array.h
View File

@ -104,7 +104,7 @@ namespace persistent_data {
template <typename ValueTraits>
class array : public array_base {
public:
class block_ref_counter : public ref_counter<uint64_t> {
class block_ref_counter : public persistent_data::ref_counter<uint64_t> {
public:
block_ref_counter(space_map::ptr sm,
array<ValueTraits> &a)

3
persistent-data/data-structures/btree-remove.tcc
View File

@ -145,11 +145,11 @@ namespace persistent_data {
{
internal_node n = spine.get_node<block_traits>();
// compact the path if there's only one child
if (n.get_nr_entries() == 1) {
block_address b = n.value_at(0);
read_ref child = tm_.read_lock(b, validator_);
// FIXME: is it safe?
::memcpy(n.raw(), child.data(), read_ref::BLOCK_SIZE);
tm_.get_sm()->dec(child.get_location());
@ -341,7 +341,6 @@ namespace persistent_data {
if (nr_left < nr_right) {
int s = nr_left - target_left;
// FIXME: signed & unsigned comparison
if (s < 0 && nr_center < static_cast<unsigned>(-s)) {
// not enough in central node
left.move_entries(center, -nr_center);

19
persistent-data/data-structures/btree.tcc
View File

@ -338,7 +338,7 @@ namespace persistent_data {
unsigned nr_right = rhs.get_nr_entries();
unsigned max_entries = get_max_entries();
if (nr_left - count > max_entries || nr_right - count > max_entries)
if (nr_left - count > max_entries || nr_right + count > max_entries)
throw runtime_error("too many entries");
if (count > 0) {
@ -693,9 +693,15 @@ namespace persistent_data {
leaf_node n = spine.template get_node<ValueTraits>();
if (need_insert)
n.insert_at(index, key[Levels - 1], value);
else
// FIXME: check if we're overwriting with the same value.
n.set_value(index, value);
else {
typename ValueTraits::value_type old_value = n.value_at(index);
if (value != old_value) {
// do decrement the old value if it already exists
rc_.dec(old_value);
n.set_value(index, value);
}
}
root_ = spine.get_root();
@ -981,11 +987,6 @@ namespace persistent_data {
if (i < 0 || leaf.key_at(i) != key)
i++;
// do decrement the old value if it already exists
// FIXME: I'm not sure about this, I don't understand the |inc| reference
if (static_cast<unsigned>(i) < leaf.get_nr_entries() && leaf.key_at(i) == key && inc) {
// dec old entry
}
*index = i;
return ((static_cast<unsigned>(i) >= leaf.get_nr_entries()) ||

22
persistent-data/data-structures/btree_counter.h
View File

@ -14,12 +14,14 @@ namespace persistent_data {
public:
typedef btree<Levels, ValueTraits> tree;
counting_visitor(block_counter &bc, ValueCounter &vc)
counting_visitor(block_counter &bc, ValueCounter &vc,
bool ignore_non_fatal = false)
: bc_(bc),
vc_(vc),
error_outcome_(bc.stop_on_error() ?
tree::visitor::RETHROW_EXCEPTION :
tree::visitor::EXCEPTION_HANDLED) {
tree::visitor::EXCEPTION_HANDLED),
ignore_non_fatal_(ignore_non_fatal) {
}
virtual bool visit_internal(node_location const &l,
@ -66,7 +68,7 @@ namespace persistent_data {
if (!checker_.check_block_nr(n) ||
!checker_.check_value_size(n) ||
!checker_.check_max_entries(n) ||
!checker_.check_nr_entries(n, l.is_sub_root()) ||
!check_nr_entries(l, n) ||
!checker_.check_ordered_keys(n) ||
!checker_.check_parent_key(n, l.is_sub_root() ? boost::optional<uint64_t>() : l.key))
return false;
@ -83,7 +85,7 @@ namespace persistent_data {
if (!checker_.check_block_nr(n) ||
!checker_.check_value_size(n) ||
!checker_.check_max_entries(n) ||
!checker_.check_nr_entries(n, l.is_sub_root()) ||
!check_nr_entries(l, n) ||
!checker_.check_ordered_keys(n) ||
!checker_.check_parent_key(n, l.is_sub_root() ? boost::optional<uint64_t>() : l.key) ||
!checker_.check_leaf_key(n, last_leaf_key_[l.level()]))
@ -109,11 +111,18 @@ namespace persistent_data {
return !seen;
}
template <typename ValueTraits2>
bool check_nr_entries(node_location const &loc,
btree_detail::node_ref<ValueTraits2> const &n) {
return ignore_non_fatal_ || checker_.check_nr_entries(n, loc.is_sub_root());
}
block_counter &bc_;
ValueCounter &vc_;
btree_node_checker checker_;
boost::optional<uint64_t> last_leaf_key_[Levels];
error_outcome error_outcome_;
bool ignore_non_fatal_;
};
}
@ -141,8 +150,9 @@ namespace persistent_data {
// walked. This walk should only be done once you're sure the tree
// is not corrupt.
template <unsigned Levels, typename ValueTraits, typename ValueCounter>
void count_btree_blocks(btree<Levels, ValueTraits> const &tree, block_counter &bc, ValueCounter &vc) {
btree_count_detail::counting_visitor<Levels, ValueTraits, ValueCounter> v(bc, vc);
void count_btree_blocks(btree<Levels, ValueTraits> const &tree, block_counter &bc, ValueCounter &vc,
bool ignore_non_fatal = false) {
btree_count_detail::counting_visitor<Levels, ValueTraits, ValueCounter> v(bc, vc, ignore_non_fatal);
tree.visit_depth_first(v);
}
}

11
persistent-data/space-maps/disk.cc
View File

@ -666,9 +666,14 @@ namespace {
if (!ie.blocknr_)
return;
block_manager::read_ref rr = tm_.read_lock(ie.blocknr_, bitmap_validator_);
if (rr.data())
bc_.inc(ie.blocknr_);
try {
block_manager::read_ref rr = tm_.read_lock(ie.blocknr_, bitmap_validator_);
if (rr.data())
bc_.inc(ie.blocknr_);
} catch (std::exception &e) {
if (bc_.stop_on_error())
throw;
}
}
private:

10
persistent-data/space-maps/disk_structures.h
View File

@ -42,6 +42,16 @@ namespace persistent_data {
uint32_t none_free_before_;
};
inline bool operator==(index_entry const& lhs, index_entry const& rhs) {
// The return value doesn't matter, since the ref-counts of bitmap blocks
// are managed by shadow operations.
return false;
}
inline bool operator!=(index_entry const& lhs, index_entry const& rhs) {
return !(lhs == rhs);
}
struct index_entry_traits {
typedef index_entry_disk disk_type;
typedef index_entry value_type;

7
persistent-data/transaction_manager.cc
View File

@ -37,6 +37,13 @@ transaction_manager::~transaction_manager()
{
}
void
transaction_manager::commit()
{
wipe_shadow_table();
bm_->flush();
}
transaction_manager::write_ref
transaction_manager::begin(block_address superblock, validator v)
{

2
persistent-data/transaction_manager.h
View File

@ -42,6 +42,8 @@ namespace persistent_data {
space_map::ptr sm);
~transaction_manager();
void commit();
// Drop the superblock reference to commit
write_ref begin(block_address superblock, validator v);
write_ref new_block(validator v);

39
src/bin/thin_metadata_pack.rs
View File

@ -1,39 +0,0 @@
extern crate clap;
extern crate thinp;
use clap::{App, Arg};
use std::path::Path;
use std::process::exit;
use thinp::file_utils;
fn main() {
let parser = App::new("thin_metadata_pack")
.version(thinp::version::TOOLS_VERSION)
.about("Produces a compressed file of thin metadata. Only packs metadata blocks that are actually used.")
.arg(Arg::with_name("INPUT")
.help("Specify thinp metadata binary device/file")
.required(true)
.short("i")
.value_name("DEV")
.takes_value(true))
.arg(Arg::with_name("OUTPUT")
.help("Specify packed output file")
.required(true)
.short("o")
.value_name("FILE")
.takes_value(true));
let matches = parser.get_matches();
let input_file = Path::new(matches.value_of("INPUT").unwrap());
let output_file = Path::new(matches.value_of("OUTPUT").unwrap());
if !file_utils::file_exists(&input_file) {
eprintln!("Couldn't find input file '{}'.", &input_file.display());
exit(1);
}
if let Err(reason) = thinp::pack::toplevel::pack(&input_file, &output_file) {
println!("Application error: {}\n", reason);
exit(1);
}
}

45
src/bin/thin_metadata_unpack.rs
View File

@ -1,45 +0,0 @@
extern crate clap;
extern crate thinp;
use clap::{App, Arg};
use std::path::Path;
use std::process;
use thinp::file_utils;
use std::process::exit;
fn main() {
let parser = App::new("thin_metadata_unpack")
.version(thinp::version::TOOLS_VERSION)
.about("Unpack a compressed file of thin metadata.")
.arg(
Arg::with_name("INPUT")
.help("Specify thinp metadata binary device/file")
.required(true)
.short("i")
.value_name("DEV")
.takes_value(true),
)
.arg(
Arg::with_name("OUTPUT")
.help("Specify packed output file")
.required(true)
.short("o")
.value_name("FILE")
.takes_value(true),
);
let matches = parser.get_matches();
let input_file = Path::new(matches.value_of("INPUT").unwrap());
let output_file = Path::new(matches.value_of("OUTPUT").unwrap());
if !file_utils::file_exists(input_file) {
eprintln!("Couldn't find input file '{}'.", &input_file.display());
exit(1);
}
if let Err(reason) = thinp::pack::toplevel::unpack(&input_file, &output_file) {
println!("Application error: {}", reason);
process::exit(1);
}
}

74
src/bin/thin_shrink.rs
View File

@ -1,74 +0,0 @@
extern crate clap;
extern crate thinp;
use clap::{App, Arg};
use std::path::Path;
use std::process::exit;
use thinp::file_utils;
fn main() {
let parser = App::new("thin_shrink")
.version(thinp::version::TOOLS_VERSION)
.about("Rewrite xml metadata and move data in an inactive pool.")
.arg(
Arg::with_name("INPUT")
.help("Specify thinp metadata xml file")
.required(true)
.long("input")
.value_name("INPUT")
.takes_value(true),
)
.arg(
Arg::with_name("OUTPUT")
.help("Specify output xml file")
.required(true)
.long("output")
.value_name("OUTPUT")
.takes_value(true),
)
.arg(
Arg::with_name("DATA")
.help("Specify pool data device where data will be moved")
.required(true)
.long("data")
.value_name("DATA")
.takes_value(true),
)
.arg(
Arg::with_name("NOCOPY")
.help("Skip the copying of data, useful for benchmarking")
.required(false)
.long("no-copy")
.value_name("NOCOPY")
.takes_value(false),
)
.arg(
Arg::with_name("SIZE")
.help("Specify new size for the pool (in data blocks)")
.required(true)
.long("nr-blocks")
.value_name("SIZE")
.takes_value(true),
);
let matches = parser.get_matches();
// FIXME: check these look like xml
let input_file = Path::new(matches.value_of("INPUT").unwrap());
let output_file = Path::new(matches.value_of("OUTPUT").unwrap());
let size = matches.value_of("SIZE").unwrap().parse::<u64>().unwrap();
let data_file = Path::new(matches.value_of("DATA").unwrap());
let do_copy = !matches.is_present("NOCOPY");
if !file_utils::file_exists(input_file) {
eprintln!("Couldn't find input file '{}'.", input_file.display());
exit(1);
}
if let Err(reason) =
thinp::shrink::toplevel::shrink(&input_file, &output_file, &data_file, size, do_copy)
{
println!("Application error: {}\n", reason);
exit(1);
}
}

51
src/block_manager.rs
View File

@ -1,51 +0,0 @@
use std::io;
use std::io::{Read, Seek};
use std::fs::OpenOptions;
use std::os::unix::fs::OpenOptionsExt;
use std::fs::File;
pub const BLOCK_SIZE: usize = 4096;
#[repr(align(4096))]
pub struct Block {
pub data: [u8; BLOCK_SIZE as usize],
}
pub struct BlockManager {
pub nr_blocks: u64,
input: File,
}
fn get_nr_blocks(path: &str) -> io::Result<u64> {
let metadata = std::fs::metadata(path)?;
Ok(metadata.len() / (BLOCK_SIZE as u64))
}
impl BlockManager {
pub fn new(path: &str, _cache_size: usize) -> io::Result<BlockManager> {
let input = OpenOptions::new()
.read(true)
.write(false)
.custom_flags(libc::O_DIRECT)
.open(path)?;
Ok(BlockManager {
nr_blocks: get_nr_blocks(path)?,
input,
})
}
pub fn get(&mut self, b: u64) -> io::Result<Block> {
self.read_block(b)
}
fn read_block(&mut self, b: u64) -> io::Result<Block>
{
let mut buf = Block {data: [0; BLOCK_SIZE]};
self.input.seek(io::SeekFrom::Start(b * (BLOCK_SIZE as u64)))?;
self.input.read_exact(&mut buf.data)?;
Ok(buf)
}
}

13
src/check.rs
View File

@ -1,13 +0,0 @@
use std::error::Error;
use crate::block_manager::BlockManager;
pub fn check(dev: &str) -> Result<(), Box<dyn Error>> {
let mut bm = BlockManager::new(dev, 1024)?;
for b in 0..100 {
let _block = bm.get(b)?;
}
Ok(())
}

100
src/file_utils.rs
View File

@ -1,100 +0,0 @@
use nix::sys::stat;
use nix::sys::stat::{FileStat, SFlag};
use std::fs::{File, OpenOptions};
use std::io;
use std::io::{Seek, Write};
use std::os::unix::io::AsRawFd;
use std::path::Path;
use tempfile::tempfile;
//---------------------------------------
fn check_bits(mode: u32, flag: &SFlag) -> bool {
(mode & flag.bits()) != 0
}
pub fn is_file_or_blk(info: FileStat) -> bool {
check_bits(info.st_mode, &stat::SFlag::S_IFBLK)
|| check_bits(info.st_mode, &stat::SFlag::S_IFREG)
}
pub fn file_exists(path: &Path) -> bool {
match stat::stat(path) {
Ok(info) => is_file_or_blk(info),
_ => {
// FIXME: assuming all errors indicate the file doesn't
// exist.
false
}
}
}
//---------------------------------------
const BLKGETSIZE64_CODE: u8 = 0x12;
const BLKGETSIZE64_SEQ: u8 = 114;
ioctl_read!(ioctl_blkgetsize64, BLKGETSIZE64_CODE, BLKGETSIZE64_SEQ, u64);
pub fn fail<T>(msg: &str) -> io::Result<T> {
let e = io::Error::new(io::ErrorKind::Other, msg);
Err(e)
}
fn get_device_size(path: &Path) -> io::Result<u64> {
let file = File::open(path)?;
let fd = file.as_raw_fd();
let mut cap = 0u64;
unsafe {
match ioctl_blkgetsize64(fd, &mut cap) {
Ok(_) => Ok(cap),
_ => fail("BLKGETSIZE64 ioctl failed"),
}
}
}
pub fn file_size(path: &Path) -> io::Result<u64> {
match stat::stat(path) {
Ok(info) => {
if check_bits(info.st_mode, &SFlag::S_IFREG) {
Ok(info.st_size as u64)
} else if check_bits(info.st_mode, &SFlag::S_IFBLK) {
get_device_size(path)
} else {
fail("not a regular file or block device")
}
}
_ => fail("stat failed"),
}
}
//---------------------------------------
fn set_size<W: Write + Seek>(w: &mut W, nr_bytes: u64) -> io::Result<()> {
let zeroes: Vec<u8> = vec![0; 1];
if nr_bytes > 0 {
w.seek(io::SeekFrom::Start(nr_bytes - 1))?;
w.write_all(&zeroes)?;
}
Ok(())
}
pub fn temp_file_sized(nr_bytes: u64) -> io::Result<std::fs::File> {
let mut file = tempfile()?;
set_size(&mut file, nr_bytes)?;
Ok(file)
}
pub fn create_sized_file(path: &Path, nr_bytes: u64) -> io::Result<std::fs::File> {
let mut file = OpenOptions::new()
.read(false)
.write(true)
.create(true)
.truncate(true)
.open(path)?;
set_size(&mut file, nr_bytes)?;
Ok(file)
}
//---------------------------------------

24
src/lib.rs
View File

@ -1,24 +0,0 @@
extern crate anyhow;
extern crate byteorder;
extern crate crc32c;
extern crate flate2;
extern crate nom;
extern crate num_cpus;
#[macro_use]
extern crate nix;
#[cfg(test)]
extern crate quickcheck;
#[cfg(test)]
#[macro_use(quickcheck)]
#[cfg(test)]
extern crate quickcheck_macros;
pub mod block_manager;
pub mod check;
pub mod file_utils;
pub mod pack;
pub mod shrink;
pub mod thin;
pub mod version;

175
src/pack/delta_list.rs
View File

@ -1,175 +0,0 @@
//-------------------------------------------------
#[derive(PartialEq, Debug, Clone)]
pub enum Delta {
Base { n: u64 },
Const { count: u64 },
Pos { delta: u64, count: u64 },
Neg { delta: u64, count: u64 },
}
use Delta::*;
pub fn to_delta(ns: &[u64]) -> Vec<Delta> {
use std::cmp::Ordering::*;
let mut ds = Vec::new();
if !ns.is_empty() {
let mut base = ns[0];
ds.push(Base { n: base });
let mut i = 1;
while i < ns.len() {
let n = ns[i];
match n.cmp(&base) {
Less => {
let delta = base - n;
let mut count = 1;
while i < ns.len() && (ns[i] + (count * delta) == base) {
i += 1;
count += 1;
}
count -= 1;
ds.push(Neg {
delta,
count,
});
base -= delta * count;
}
Equal => {
let mut count = 1;
while i < ns.len() && ns[i] == base {
i += 1;
count += 1;
}
count -= 1;
ds.push(Const { count });
}
Greater => {
let delta = n - base;
let mut count = 1;
while i < ns.len() && (ns[i] == (base + (count * delta))) {
i += 1;
count += 1;
}
count -= 1;
ds.push(Pos {
delta,
count,
});
base += delta * count;
}
}
}
}
ds
}
#[cfg(test)]
mod tests {
use super::*;
fn from_delta(ds: &[Delta]) -> Vec<u64> {
let mut ns: Vec<u64> = Vec::new();
let mut base = 0u64;
for d in ds {
match d {
Base { n } => {
ns.push(*n);
base = *n;
}
Const { count } => {
for _ in 0..*count {
ns.push(base);
}
}
Pos { delta, count } => {
for _ in 0..*count {
base += delta;
ns.push(base);
}
}
Neg { delta, count } => {
for _ in 0..*count {
assert!(base >= *delta);
base -= delta;
ns.push(base);
}
}
}
}
ns
}
#[test]
fn test_to_delta() {
struct TestCase(Vec<u64>, Vec<Delta>);
let cases = [
TestCase(vec![], vec![]),
TestCase(vec![1], vec![Base { n: 1 }]),
TestCase(vec![1, 2], vec![Base { n: 1 }, Pos { delta: 1, count: 1 }]),
TestCase(
vec![1, 2, 3, 4],
vec![Base { n: 1 }, Pos { delta: 1, count: 3 }],
),
TestCase(
vec![2, 4, 6, 8],
vec![Base { n: 2 }, Pos { delta: 2, count: 3 }],
),
TestCase(
vec![7, 14, 21, 28],
vec![Base { n: 7 }, Pos { delta: 7, count: 3 }],
),
TestCase(
vec![10, 9],
vec![Base { n: 10 }, Neg { delta: 1, count: 1 }],
),
TestCase(
vec![10, 9, 8, 7],
vec![Base { n: 10 }, Neg { delta: 1, count: 3 }],
),
TestCase(
vec![10, 8, 6, 4],
vec![Base { n: 10 }, Neg { delta: 2, count: 3 }],
),
TestCase(
vec![28, 21, 14, 7],
vec![Base { n: 28 }, Neg { delta: 7, count: 3 }],
),
TestCase(
vec![42, 42, 42, 42],
vec![Base { n: 42 }, Const { count: 3 }],
),
TestCase(
vec![1, 2, 3, 10, 20, 30, 40, 38, 36, 34, 0, 0, 0, 0],
vec![
Base { n: 1 },
Pos { delta: 1, count: 2 },
Pos { delta: 7, count: 1 },
Pos {
delta: 10,
count: 3,
},
Neg { delta: 2, count: 3 },
Neg {
delta: 34,
count: 1,
},
Const { count: 3 },
],
),
];
for t in &cases {
assert_eq!(to_delta(&t.0), t.1);
assert_eq!(from_delta(&t.1), t.0);
}
}
}
//-------------------------------------------------

5
src/pack/mod.rs
View File

@ -1,5 +0,0 @@
pub mod toplevel;
mod delta_list;
mod node_encode;
mod vm;

127
src/pack/node_encode.rs
View File

@ -1,127 +0,0 @@
use std::{io, io::Write};
use nom::{bytes::complete::*, number::complete::*, IResult};
use crate::pack::vm::*;
//-------------------------------------------
#[derive(Debug)]
pub enum PackError {
ParseError,
IOError,
}
impl std::error::Error for PackError {}
pub type PResult<T> = Result<T, PackError>;
fn nom_to_pr<T>(r: IResult<&[u8], T>) -> PResult<(&[u8], T)> {
match r {
Ok(v) => Ok(v),
Err(_) => Err(PackError::ParseError),
}
}
fn io_to_pr<T>(r: io::Result<T>) -> PResult<T> {
match r {
Ok(v) => Ok(v),
Err(_) => Err(PackError::IOError),
}
}
//-------------------------------------------
impl std::fmt::Display for PackError {
fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
match self {
PackError::ParseError => write!(f, "parse error"),
PackError::IOError => write!(f, "IO error"),
}
}
}
fn run64(i: &[u8], count: usize) -> IResult<&[u8], Vec<u64>> {
let (i, ns) = nom::multi::many_m_n(count, count, le_u64)(i)?;
Ok((i, ns))
}
struct NodeSummary {
is_leaf: bool,
max_entries: usize,
value_size: usize
}
fn summarise_node(data: &[u8]) -> IResult<&[u8], NodeSummary> {
let (i, _csum) = le_u32(data)?;
let (i, flags) = le_u32(i)?;
let (i, _blocknr) = le_u64(i)?;
let (i, _nr_entries) = le_u32(i)?;
let (i, max_entries) = le_u32(i)?;
let (i, value_size) = le_u32(i)?;
let (i, _padding) = le_u32(i)?;
Ok((i, NodeSummary {
is_leaf: flags == 2,
max_entries: max_entries as usize,
value_size: value_size as usize,
}))
}
pub fn pack_btree_node<W: Write>(w: &mut W, data: &[u8]) -> PResult<()> {
let (_, info) = nom_to_pr(summarise_node(data))?;
if info.is_leaf {
if info.value_size == std::mem::size_of::<u64>() {
let (i, hdr) = nom_to_pr(take(32usize)(data))?;
let (i, keys) = nom_to_pr(run64(i, info.max_entries))?;
let (tail, values) = nom_to_pr(run64(i, info.max_entries))?;
io_to_pr(pack_literal(w, hdr))?;
io_to_pr(pack_u64s(w, &keys))?;
io_to_pr(pack_shifted_u64s(w, &values))?;
if !tail.is_empty() {
io_to_pr(pack_literal(w, tail))?;
}
Ok(())
} else {
// We don't bother packing the values if they aren't u64
let (i, hdr) = nom_to_pr(take(32usize)(data))?;
let (tail, keys) = nom_to_pr(run64(i, info.max_entries))?;
io_to_pr(pack_literal(w, hdr))?;
io_to_pr(pack_u64s(w, &keys))?;
io_to_pr(pack_literal(w, tail))?;
Ok(())
}
} else {
// Internal node, values are also u64s
let (i, hdr) = nom_to_pr(take(32usize)(data))?;
let (i, keys) = nom_to_pr(run64(i, info.max_entries))?;
let (tail, values) = nom_to_pr(run64(i, info.max_entries))?;
io_to_pr(pack_literal(w, hdr))?;
io_to_pr(pack_u64s(w, &keys))?;
io_to_pr(pack_u64s(w, &values))?;
if !tail.is_empty() {
io_to_pr(pack_literal(w, tail))?;
}
Ok(())
}
}
pub fn pack_superblock<W: Write>(w: &mut W, bytes: &[u8]) -> PResult<()> {
io_to_pr(pack_literal(w, bytes))
}
pub fn pack_bitmap<W: Write>(w: &mut W, bytes: &[u8]) -> PResult<()> {
io_to_pr(pack_literal(w, bytes))
}
pub fn pack_index<W: Write>(w: &mut W, bytes: &[u8]) -> PResult<()> {
io_to_pr(pack_literal(w, bytes))
}
//-------------------------------------

357
src/pack/toplevel.rs
View File

@ -1,357 +0,0 @@
use anyhow::Result;
use byteorder::{LittleEndian, ReadBytesExt, WriteBytesExt};
use flate2::{read::ZlibDecoder, write::ZlibEncoder, Compression};
use std::os::unix::fs::OpenOptionsExt;
use std::{
error::Error,
fs::OpenOptions,
path::Path,
io,
io::prelude::*,
io::Cursor,
io::Write,
ops::DerefMut,
sync::{Arc, Mutex},
thread::spawn,
};
use rand::prelude::*;
use std::sync::mpsc::{sync_channel, Receiver};
use crate::file_utils;
use crate::pack::node_encode::*;
const BLOCK_SIZE: u64 = 4096;
const MAGIC: u64 = 0xa537a0aa6309ef77;
const PACK_VERSION: u64 = 3;
const SUPERBLOCK_CSUM_XOR: u32 = 160774;
const BITMAP_CSUM_XOR: u32 = 240779;
const INDEX_CSUM_XOR: u32 = 160478;
const BTREE_CSUM_XOR: u32 = 121107;
fn shuffle<T>(v: &mut Vec<T>) {
let mut rng = rand::thread_rng();
v.shuffle(&mut rng);
}
// Each thread processes multiple contiguous runs of blocks, called
// chunks. Chunks are shuffled so each thread gets chunks spread
// across the dev in case there are large regions that don't contain
// metadata.
fn mk_chunk_vecs(nr_blocks: u64, nr_jobs: u64) -> Vec<Vec<(u64, u64)>> {
use std::cmp::{max, min};
let chunk_size = min(4 * 1024u64, max(128u64, nr_blocks / (nr_jobs * 64)));
let nr_chunks = nr_blocks / chunk_size;
let mut chunks = Vec::with_capacity(nr_chunks as usize);
for i in 0..nr_chunks {
chunks.push((i * chunk_size, (i + 1) * chunk_size));
}
// there may be a smaller chunk at the back of the file.
if nr_chunks * chunk_size < nr_blocks {
chunks.push((nr_chunks * chunk_size, nr_blocks));
}
shuffle(&mut chunks);
let mut vs = Vec::with_capacity(nr_jobs as usize);
for _ in 0..nr_jobs {
vs.push(Vec::new());
}
for c in 0..nr_chunks {
vs[(c % nr_jobs) as usize].push(chunks[c as usize]);
}
vs
}
pub fn pack(input_file: &Path, output_file: &Path) -> Result<(), Box<dyn Error>> {
let nr_blocks = get_nr_blocks(&input_file)?;
let nr_jobs = std::cmp::max(1, std::cmp::min(num_cpus::get() as u64, nr_blocks / 128));
let chunk_vecs = mk_chunk_vecs(nr_blocks, nr_jobs);
let input = OpenOptions::new()
.read(true)
.write(false)
.custom_flags(libc::O_EXCL)
.open(input_file)?;
let output = OpenOptions::new()
.read(false)
.write(true)
.create(true)
.truncate(true)
.open(output_file)?;
write_header(&output, nr_blocks)?;
let sync_input = Arc::new(Mutex::new(input));
let sync_output = Arc::new(Mutex::new(output));
let mut threads = Vec::new();
for job in 0..nr_jobs {
let sync_input = Arc::clone(&sync_input);
let sync_output = Arc::clone(&sync_output);
let chunks = chunk_vecs[job as usize].clone();
threads.push(spawn(move || crunch(sync_input, sync_output, chunks)));
}
for t in threads {
t.join().unwrap()?;
}
Ok(())
}
fn crunch<R, W>(input: Arc<Mutex<R>>, output: Arc<Mutex<W>>, ranges: Vec<(u64, u64)>) -> Result<()>
where
R: Read + Seek,
W: Write,
{
let mut written = 0u64;
let mut z = ZlibEncoder::new(Vec::new(), Compression::default());
for (lo, hi) in ranges {
// We read multiple blocks at once to reduce contention
// on input.
let mut input = input.lock().unwrap();
let big_data = read_blocks(input.deref_mut(), lo, hi - lo)?;
drop(input);
for b in lo..hi {
let block_start = ((b - lo) * BLOCK_SIZE) as usize;
let data = &big_data[block_start..(block_start + BLOCK_SIZE as usize)];
let kind = metadata_block_type(data);
if kind != BT::UNKNOWN {
z.write_u64::<LittleEndian>(b)?;
pack_block(&mut z, kind, &data);
written += 1;
if written == 1024 {
let compressed = z.reset(Vec::new())?;
let mut output = output.lock().unwrap();
output.write_u64::<LittleEndian>(compressed.len() as u64)?;
output.write_all(&compressed)?;
written = 0;
}
}
}
}
if written > 0 {
let compressed = z.finish()?;
let mut output = output.lock().unwrap();
output.write_u64::<LittleEndian>(compressed.len() as u64)?;
output.write_all(&compressed)?;
}
Ok(())
}
fn write_header<W>(mut w: W, nr_blocks: u64) -> io::Result<()>
where
W: byteorder::WriteBytesExt,
{
w.write_u64::<LittleEndian>(MAGIC)?;
w.write_u64::<LittleEndian>(PACK_VERSION)?;
w.write_u64::<LittleEndian>(4096)?;
w.write_u64::<LittleEndian>(nr_blocks)?;
Ok(())
}
fn read_header<R>(mut r: R) -> io::Result<u64>
where
R: byteorder::ReadBytesExt,
{
use std::process::exit;
let magic = r.read_u64::<LittleEndian>()?;
if magic != MAGIC {
eprintln!("Not a pack file.");
exit(1);
}
let version = r.read_u64::<LittleEndian>()?;
if version != PACK_VERSION {
eprintln!("unsupported pack file version ({}).", PACK_VERSION);
exit(1);
}
let block_size = r.read_u64::<LittleEndian>()?;
if block_size != BLOCK_SIZE {
eprintln!("block size is not {}", BLOCK_SIZE);
exit(1);
}
r.read_u64::<LittleEndian>()
}
fn get_nr_blocks(path: &Path) -> io::Result<u64> {
let len = file_utils::file_size(path)?;
Ok(len / (BLOCK_SIZE as u64))
}
fn read_blocks<R>(rdr: &mut R, b: u64, count: u64) -> io::Result<Vec<u8>>
where
R: io::Read + io::Seek,
{
let mut buf: Vec<u8> = vec![0; (BLOCK_SIZE * count) as usize];
rdr.seek(io::SeekFrom::Start(b * BLOCK_SIZE))?;
rdr.read_exact(&mut buf)?;
Ok(buf)
}
fn checksum(buf: &[u8]) -> u32 {
crc32c::crc32c(&buf[4..]) ^ 0xffffffff
}
#[derive(PartialEq)]
enum BT {
SUPERBLOCK,
NODE,
INDEX,
BITMAP,
UNKNOWN,
}
fn metadata_block_type(buf: &[u8]) -> BT {
if buf.len() != BLOCK_SIZE as usize {
return BT::UNKNOWN;
}
// The checksum is always stored in the first u32 of the buffer.
let mut rdr = Cursor::new(buf);
let sum_on_disk = rdr.read_u32::<LittleEndian>().unwrap();
let csum = checksum(buf);
let btype = csum ^ sum_on_disk;
match btype {
SUPERBLOCK_CSUM_XOR => BT::SUPERBLOCK,
BTREE_CSUM_XOR => BT::NODE,
BITMAP_CSUM_XOR => BT::BITMAP,
INDEX_CSUM_XOR => BT::INDEX,
_ => BT::UNKNOWN,
}
}
fn check<T>(r: &PResult<T>) {
match r {
Ok(_) => {}
Err(PackError::ParseError) => panic!("parse error"),
Err(PackError::IOError) => panic!("io error"),
}
}
fn pack_block<W: Write>(w: &mut W, kind: BT, buf: &[u8]) {
match kind {
BT::SUPERBLOCK => check(&pack_superblock(w, buf)),
BT::NODE => check(&pack_btree_node(w, buf)),
BT::INDEX => check(&pack_index(w, buf)),
BT::BITMAP => check(&pack_bitmap(w, buf)),
BT::UNKNOWN => panic!("asked to pack an unknown block type"),
}
}
fn write_zero_block<W>(w: &mut W, b: u64) -> io::Result<()>
where
W: Write + Seek,
{
let zeroes: Vec<u8> = vec![0; BLOCK_SIZE as usize];
w.seek(io::SeekFrom::Start(b * BLOCK_SIZE))?;
w.write_all(&zeroes)?;
Ok(())
}
fn write_blocks<W>(w: &Arc<Mutex<W>>, blocks: &mut Vec<(u64, Vec<u8>)>) -> io::Result<()>
where
W: Write + Seek,
{
let mut w = w.lock().unwrap();
while let Some((b, block)) = blocks.pop() {
w.seek(io::SeekFrom::Start(b * BLOCK_SIZE))?;
w.write_all(&block[0..])?;
}
Ok(())
}
fn decode_worker<W>(rx: Receiver<Vec<u8>>, w: Arc<Mutex<W>>) -> io::Result<()>
where
W: Write + Seek,
{
let mut blocks = Vec::new();
while let Ok(bytes) = rx.recv() {
let mut z = ZlibDecoder::new(&bytes[0..]);
while let Ok(b) = z.read_u64::<LittleEndian>() {
let block = crate::pack::vm::unpack(&mut z, BLOCK_SIZE as usize).unwrap();
assert!(metadata_block_type(&block[0..]) != BT::UNKNOWN);
blocks.push((b, block));
if blocks.len() >= 32 {
write_blocks(&w, &mut blocks)?;
}
}
}
write_blocks(&w, &mut blocks)?;
Ok(())
}
pub fn unpack(input_file: &Path, output_file: &Path) -> Result<(), Box<dyn Error>> {
let mut input = OpenOptions::new()
.read(true)
.write(false)
.open(input_file)?;
let mut output = OpenOptions::new()
.read(false)
.write(true)
.create(true)
.truncate(true)
.open(output_file)?;
let nr_blocks = read_header(&input)?;
// zero the last block to size the file
write_zero_block(&mut output, nr_blocks - 1)?;
// Run until we hit the end
let output = Arc::new(Mutex::new(output));
// kick off the workers
let nr_jobs = num_cpus::get();
let mut senders = Vec::new();
let mut threads = Vec::new();
for _ in 0..nr_jobs {
let (tx, rx) = sync_channel(1);
let output = Arc::clone(&output);
senders.push(tx);
threads.push(spawn(move || decode_worker(rx, output)));
}
// Read z compressed chunk, and hand to worker thread.
let mut next_worker = 0;
while let Ok(len) = input.read_u64::<LittleEndian>() {
let mut bytes = vec![0; len as usize];
input.read_exact(&mut bytes)?;
senders[next_worker].send(bytes).unwrap();
next_worker = (next_worker + 1) % nr_jobs;
}
for s in senders {
drop(s);
}
for t in threads {
t.join().unwrap()?;
}
Ok(())
}

490
src/pack/vm.rs
View File

@ -1,490 +0,0 @@
use byteorder::{LittleEndian, ReadBytesExt, WriteBytesExt};
use std::io;
use std::io::{Cursor, Read, Write};
use num_derive::FromPrimitive;
use num_traits::FromPrimitive;
use crate::pack::delta_list::*;
//-------------------------------------------------
// Deltas are converted to instructions. A delta may not fit
// into a single instruction.
#[derive(Debug, FromPrimitive)]
enum Tag {
Set, // Operand width given in nibble
Pos, // Delta in nibble
PosW, // Delta in operand, whose width is in nibble
Neg, // Delta in nibble
NegW, // Delta in operand, whose width is in nibble
Const, // Count in nibble
Const8, // count = (nibble << 8) | byte
// Controls how many times the next instruction is applied.
// Not applicable to Const instructions which hold their own count.
Count, // count stored in nibble
Count8, // count = (nibble << 8) | byte
Lit, // len in nibble
LitW,
ShiftedRun,
}
fn pack_tag<W: Write>(w: &mut W, t: Tag, nibble: u8) -> io::Result<()> {
assert!(nibble < 16);
let mut b: u8 = t as u8;
assert!(b < 16);
b = (b << 4) | nibble;
w.write_u8(b)
}
fn pack_count<W>(w: &mut W, count: u64) -> io::Result<()>
where
W: Write,
{
if count == 1u64 {
Ok(())
} else if count < 16 {
pack_tag(w, Tag::Count, count as u8)
} else {
assert!(count < 4096);
let nibble = count >> 8;
assert!(nibble < 16);
let byte = count & 0xff;
pack_tag(w, Tag::Count8, nibble as u8)?;
w.write_u8(byte as u8)
}
}
fn pack_delta<W: Write>(w: &mut W, d: &Delta) -> io::Result<()> {
use Tag::*;
match d {
Delta::Base { n } => {
if *n <= std::u8::MAX as u64 {
pack_tag(w, Set, 1)?;
w.write_u8(*n as u8)
} else if *n <= std::u16::MAX as u64 {
pack_tag(w, Set, 2)?;
w.write_u16::<LittleEndian>(*n as u16)
} else if *n <= u32::MAX as u64 {
pack_tag(w, Set, 4)?;
w.write_u32::<LittleEndian>(*n as u32)
} else {
pack_tag(w, Set, 8)?;
w.write_u64::<LittleEndian>(*n)
}
}
Delta::Pos { delta, count } => {
pack_count(w, *count)?;
if *delta < 16 {
pack_tag(w, Tag::Pos, *delta as u8)
} else if *delta <= u8::MAX as u64 {
pack_tag(w, PosW, 1)?;
w.write_u8(*delta as u8)
} else if *delta <= u16::MAX as u64 {
pack_tag(w, PosW, 2)?;
w.write_u16::<LittleEndian>(*delta as u16)
} else if *delta <= u32::MAX as u64 {
pack_tag(w, PosW, 4)?;
w.write_u32::<LittleEndian>(*delta as u32)
} else {
pack_tag(w, PosW, 8)?;
w.write_u64::<LittleEndian>(*delta as u64)
}
}
Delta::Neg { delta, count } => {
pack_count(w, *count)?;
if *delta < 16 {
pack_tag(w, Neg, *delta as u8)
} else if *delta <= u8::MAX as u64 {
pack_tag(w, NegW, 1)?;
w.write_u8(*delta as u8)
} else if *delta <= u16::MAX as u64 {
pack_tag(w, NegW, 2)?;
w.write_u16::<LittleEndian>(*delta as u16)
} else if *delta <= u32::MAX as u64 {
pack_tag(w, NegW, 4)?;
w.write_u32::<LittleEndian>(*delta as u32)
} else {
pack_tag(w, NegW, 8)?;
w.write_u64::<LittleEndian>(*delta as u64)
}
}
Delta::Const { count } => {
if *count < 16 {
pack_tag(w, Tag::Const, *count as u8)
} else {
assert!(*count < 4096);
let nibble = *count >> 8;
assert!(nibble < 16);
pack_tag(w, Tag::Const8, nibble as u8)?;
w.write_u8((*count & 0xff) as u8)
}
}
}
}
fn pack_deltas<W: Write>(w: &mut W, ds: &[Delta]) -> io::Result<()> {
for d in ds {
pack_delta(w, d)?;
}
Ok(())
}
//-------------------------------------------------
pub fn pack_u64s<W: Write>(w: &mut W, ns: &[u64]) -> io::Result<()> {
let ds = to_delta(ns);
pack_deltas(w, &ds[0..])
}
fn unshift_nrs(shift: usize, ns: &[u64]) -> (Vec<u64>, Vec<u64>) {
let mut values = Vec::new();
let mut shifts = Vec::new();
let mask = (1 << shift) - 1;
for n in ns {
values.push(n >> shift);
shifts.push(n & mask);
}
(values, shifts)
}
pub fn pack_shifted_u64s<W: Write>(w: &mut W, ns: &[u64]) -> io::Result<()> {
let len = ns.len();
let nibble = len >> 8;
assert!(nibble < 16);
pack_tag(w, Tag::ShiftedRun, nibble as u8)?;
w.write_u8((len & 0xff) as u8)?;
let (high, low) = unshift_nrs(24, ns);
pack_u64s(w, &high[0..])?;
pack_u64s(w, &low[0..])
}
pub fn pack_literal<W: Write>(w: &mut W, bs: &[u8]) -> io::Result<()> {
use Tag::LitW;
let len = bs.len() as u64;
if len < 16 as u64 {
pack_tag(w, Tag::Lit, len as u8)?;
} else if len <= u8::MAX as u64 {
pack_tag(w, LitW, 1)?;
w.write_u8(len as u8)?;
} else if len <= u16::MAX as u64 {
pack_tag(w, LitW, 2)?;
w.write_u16::<LittleEndian>(len as u16)?;
} else if len <= u32::MAX as u64 {
pack_tag(w, LitW, 4)?;
w.write_u32::<LittleEndian>(len as u32)?;
} else {
pack_tag(w, LitW, 8)?;
w.write_u64::<LittleEndian>(len as u64)?;
}
w.write_all(bs)
}
//-------------------------------------------------
fn unpack_with_width<R: Read>(r: &mut R, nibble: u8) -> io::Result<u64> {
let v = match nibble {
1 => r.read_u8()? as u64,
2 => r.read_u16::<LittleEndian>()? as u64,
4 => r.read_u32::<LittleEndian>()? as u64,
8 => r.read_u64::<LittleEndian>()? as u64,
_ => {
panic!("SET with bad width");
}
};
Ok(v)
}
fn unpack_u64s<R: Read>(r: &mut R, count: usize) -> io::Result<Vec<u64>> {
let mut v = Vec::new();
for _ in 0..count {
let n = r.read_u64::<LittleEndian>()?;
v.push(n);
}
Ok(v)
}
struct VM {
base: u64,
bytes_written: usize,
}
impl VM {
fn new() -> VM {
VM {
base: 0,
bytes_written: 0,
}
}
fn emit_u64<W: Write>(&mut self, w: &mut W, n: u64) -> io::Result<()> {
w.write_u64::<LittleEndian>(n)?;
self.bytes_written += 8;
Ok(())
}
fn emit_base<W: Write>(&mut self, w: &mut W) -> io::Result<()> {
self.emit_u64(w, self.base)
}
fn emit_bytes<W: Write>(&mut self, w: &mut W, bytes: &[u8]) -> io::Result<()> {
let len = bytes.len();
w.write_all(bytes)?;
self.bytes_written += len;
Ok(())
}
fn unpack_instr<R: Read, W: Write>(
&mut self,
r: &mut R,
w: &mut W,
count: usize,
) -> io::Result<()> {
use Tag::*;
let b = r.read_u8()?;
let kind: Tag = match Tag::from_u8(b >> 4) {
Some(k) => k,
None => {
panic!("bad tag");
}
};
let nibble = b & 0xf;
match kind {
Set => {
self.base = unpack_with_width(r, nibble)?;
for _ in 0..count {
self.emit_base(w)?;
}
}
Pos => {
for _ in 0..count {
self.base += nibble as u64;
self.emit_base(w)?;
}
}
PosW => {
let delta = unpack_with_width(r, nibble)?;
for _ in 0..count {
self.base += delta;
self.emit_base(w)?;
}
}
Neg => {
for _ in 0..count {
self.base -= nibble as u64;
self.emit_base(w)?;
}
}
NegW => {
let delta = unpack_with_width(r, nibble)?;
for _ in 0..count {
self.base -= delta;
self.emit_base(w)?;
}
}
Const => {
assert_eq!(count, 1);
for _ in 0..nibble as usize {
self.emit_base(w)?;
}
}
Const8 => {
assert_eq!(count, 1);
let count = ((nibble as usize) << 8) | (r.read_u8()? as usize);
for _ in 0..count {
self.emit_base(w)?;
}
}
Count => {
self.unpack_instr(r, w, nibble as usize)?;
}
Count8 => {
let count = ((nibble as usize) << 8) | (r.read_u8()? as usize);
self.unpack_instr(r, w, count as usize)?;
}
Lit => {
assert_eq!(count, 1);
let len = nibble as usize;
let mut bytes = vec![0; len];
r.read_exact(&mut bytes[0..])?;
self.emit_bytes(w, &bytes)?;
}
LitW => {
assert_eq!(count, 1);
let len = unpack_with_width(r, nibble)? as usize;
let mut bytes = vec![0; len];
r.read_exact(&mut bytes[0..])?;
self.emit_bytes(w, &bytes)?;
}
ShiftedRun => {
// FIXME: repeated unpack, pack, unpack
let len = ((nibble as usize) << 8) | (r.read_u8()? as usize);
let nr_bytes = (len as usize) * std::mem::size_of::<u64>() as usize;
let mut high_bytes: Vec<u8> = Vec::with_capacity(nr_bytes);
let written = self.exec(r, &mut high_bytes, nr_bytes)?;
self.bytes_written -= written; // hack
let mut high_r = Cursor::new(high_bytes);
let high = unpack_u64s(&mut high_r, len)?;
let mut low_bytes: Vec<u8> = Vec::with_capacity(nr_bytes);
let written = self.exec(r, &mut low_bytes, nr_bytes)?;
self.bytes_written -= written; // hack
let mut low_r = Cursor::new(low_bytes);
let low = unpack_u64s(&mut low_r, len)?;
let mask = (1 << 24) - 1;
for i in 0..len {
self.emit_u64(w, (high[i] << 24) | (low[i] & mask))?;
}
}
}
Ok(())
}
// Runs until at least a number of bytes have been emitted. Returns nr emitted.
fn exec<R: Read, W: Write>(
&mut self,
r: &mut R,
w: &mut W,
emit_bytes: usize,
) -> io::Result<usize> {
let begin = self.bytes_written;
while (self.bytes_written - begin) < emit_bytes {
self.unpack_instr(r, w, 1)?;
}
Ok(self.bytes_written - begin)
}
}
pub fn unpack<R: Read>(r: &mut R, count: usize) -> io::Result<Vec<u8>> {
let mut w = Vec::with_capacity(4096);
let mut cursor = Cursor::new(&mut w);
let mut vm = VM::new();
let written = vm.exec(r, &mut cursor, count)?;
assert_eq!(w.len(), count);
assert_eq!(written, count);
Ok(w)
}
//-------------------------------------------------
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_pack_literals() {
struct TestCase(Vec<u8>);
let cases = [
// This is a bad test case, because unpack will not exec
// any instructions.
TestCase(b"".to_vec()),
TestCase(b"foo".to_vec()),
TestCase(vec![42; 15]),
TestCase(vec![42; 256]),
TestCase(vec![42; 4096]),
];
for t in &cases {
let mut bs = Vec::with_capacity(4096);
let mut w = Cursor::new(&mut bs);
pack_literal(&mut w, &t.0[0..]).unwrap();
let mut r = Cursor::new(&mut bs);
let unpacked = unpack(&mut r, t.0.len()).unwrap();
assert_eq!(&t.0[0..], &unpacked[0..]);
}
}
fn check_u64s_match(ns: &Vec<u64>, bytes: &[u8]) -> bool {
let mut packed = Vec::with_capacity(ns.len() * 8);
let mut w = Cursor::new(&mut packed);
for n in ns {
w.write_u64::<LittleEndian>(*n).unwrap();
}
packed == bytes
}
fn check_pack_u64s(ns: &Vec<u64>) -> bool {
println!("packing {:?}", &ns);
let mut bs = Vec::with_capacity(4096);
let mut w = Cursor::new(&mut bs);
pack_u64s(&mut w, &ns[0..]).unwrap();
println!("unpacked len = {}, packed len = {}", ns.len() * 8, bs.len());
let mut r = Cursor::new(&mut bs);
let unpacked = unpack(&mut r, ns.len() * 8).unwrap();
check_u64s_match(&ns, &unpacked[0..])
}
#[test]
fn test_pack_u64s() {
let cases = [
vec![0],
vec![1, 5, 9, 10],
b"the quick brown fox jumps over the lazy dog"
.iter()
.map(|b| *b as u64)
.collect(),
];
for t in &cases {
assert!(check_pack_u64s(&t));
}
}
#[quickcheck]
fn prop_pack_u64s(mut ns: Vec<u64>) -> bool {
ns.push(42); // We don't handle empty vecs
check_pack_u64s(&ns)
}
fn check_pack_shifted_u64s(ns: &Vec<(u64, u64)>) -> bool {
let shifted: Vec<u64> = ns
.iter()
.map(|(h, l)| (h << 24) | (l & ((1 << 24) - 1)))
.collect();
println!("packing {:?}", &ns);
let mut bs = Vec::with_capacity(4096);
let mut w = Cursor::new(&mut bs);
pack_shifted_u64s(&mut w, &shifted[0..]).unwrap();
println!("unpacked len = {}, packed len = {}", ns.len() * 8, bs.len());
let mut r = Cursor::new(&mut bs);
let unpacked = unpack(&mut r, ns.len() * 8).unwrap();
check_u64s_match(&shifted, &unpacked[0..])
}
#[quickcheck]
fn prop_pack_shifted_u64s(mut ns: Vec<(u64, u64)>) -> bool {
ns.push((42, 42));
check_pack_shifted_u64s(&ns)
}
}
//-------------------------------------------------

Some files were not shown because too many files have changed in this diff Show More