emo/thin-provisioning-tools
1
0 Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,828 Commits 2 Branches 0 Tags
e1628f9004f518050b790bb3ba092d80c0af0766
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Ming-Hung Tsai e1628f9004 [cache_check (rust)] Add more checks 
- Check array indices continuity
- Support ignore_non_fatal
- Report blocknr of IoErrors
- Report array block indeices
 2021-04-21 23:35:22 +08:00
Ming-Hung Tsai 239ff7dfa1 [cache_check (rust)] Add more checks 
- Report errors
- Support reading partially broken bitset
  - The output is a bitmap of 2-bit entries, indicating availability of bits
 2021-04-21 14:09:38 +08:00
Ming-Hung Tsai 3279d8381b [array_walker] Read multiple array blocks at once 2021-04-17 00:10:14 +08:00
Ming-Hung Tsai c17559791f [bitset] Rename bitset module 2021-04-17 00:10:14 +08:00
Ming-Hung Tsai 1964015d81 [array_walker] Handle the whole array block at once 
That gives the visitor more controls over the data processing and locking,
and also improves the performance by 10-15%.
 2021-04-17 00:06:09 +08:00
Ming-Hung Tsai 95dee9f66d [cache_check (rust)] Do not remap ArrayErrors to keep the error context 2021-04-16 21:46:30 +08:00
Ming-Hung Tsai 9b4a0607ea [cache_check (rust)] Detect structural errors on arrays 
- Define error types for arrays
- Propagate low-level errors to tools
- Rename array_block.rs to array.rs
 2021-04-14 16:02:50 +08:00
Ming-Hung Tsai ace9c1d1e3 [cache_check (rust)] Add more checks 
- check version-2 dirty bitset size and consistency
- check discard bitset size
 2021-04-11 22:48:10 +08:00
Ming-Hung Tsai ae630f1fd8 [btree_walker] Fix error returning 2021-04-11 20:18:44 +08:00
Ming-Hung Tsai 860b3ca7d2 [cache_check (rust)] Add more checks 
- Check the version-1 dirty flag
- Check mappings against the origin device size, if the cache is clean
- Check superblock fields
 2021-04-11 20:18:25 +08:00
Ming-Hung Tsai eb3d181f95 [cache_dump (rust)] First draft of cache_dump 2021-03-03 12:27:51 +08:00
Ming-Hung Tsai fde0e0e2b8 [cache (rust)] Add Mapping::is_dirty() 2021-03-03 12:14:46 +08:00
Ming-Hung Tsai 74fcb9d505 [cache (rust)] Fix data types 2021-02-26 23:31:12 +08:00
Ming-Hung Tsai a7ecfba2b4 [cache (rust)] Fix merge conflicts 2021-02-24 20:21:10 +08:00
Ming-Hung Tsai 2bb3bf65b7 [cache_check (rust)] Implement basic functions 2021-02-24 17:39:11 +08:00
Ming-Hung Tsai e8410dec04 [dbg] Add missing commands to cache/era_debug 2021-02-21 19:16:51 +08:00
Ming-Hung Tsai 29f9182078 [dbg] Remove nested function template to reduce code size 2021-02-21 01:07:55 +08:00
Ming-Hung Tsai a81cef4467 [dbg] Pull out common code into dbg-lib 
- Modularize common routines
- Extract the block_dumper interface for displaying blocks
- Remove inheritance from show_traits
 2021-02-21 01:04:35 +08:00
Ming-Hung Tsai c95e31bef6 [era_debug] Display bitset entries in run-length fashion 2021-02-21 00:02:57 +08:00
Ming-Hung Tsai afbd913e22 [era] Add era_debug 2021-02-18 23:40:39 +08:00
Ming-Hung Tsai 1bc88bfde8 [cache_debug] Simplify the trait types 2021-02-18 18:20:48 +08:00
Ming-Hung Tsai e3a646e4d2 [dbg] Improve error resilience of debugging tools 
Do not open the metadata. The tool interprets user specified blocks one-by-one.
Thus, there's no need to preload the metadata structures.

Also remove the unused --ignore-metadata-space-map option. It was designed
to control the loading of space maps with the old metadata constructors.
 2021-02-18 00:00:55 +08:00
Ming-Hung Tsai 08bd2f47bd [cache_debug] Fork thin_debug into cache_debug 2021-02-04 14:29:18 +08:00
Ming-Hung Tsai a05ac553b6 [dbg] Hide implementations of shared components 2021-02-04 14:28:17 +08:00
Ming-Hung Tsai b9b04dc872 [thin_debug] Factor out reusable componments 2021-02-04 14:19:28 +08:00
Ming-Hung Tsai e046bbf5c4 [thin_debug] Simplify the output format 
Turn nested fields into attributes. Also replace spaces in field names
by underscores.
 2021-02-03 15:51:52 +08:00
Ming-Hung Tsai 0816430ba0 [thin_debug] Remove the using boost directive to avoid namespace pollution 
boost::uint32_t might conflict with the toolchain defined uint32_t.
which causes template argument deduction failed.
 2021-02-03 15:51:52 +08:00
Ming-Hung Tsai ab3b2cbda2 [thin_debug] Refine the outputs 
- Show the blocknr from node header
- Display the index of node entries
 2021-02-03 15:51:52 +08:00
Ming-Hung Tsai 6dc5086643 [thin_debug] Add commands to show space maps 2021-02-03 15:51:52 +08:00
Ming-Hung Tsai 3bfa775887 [thin_debug] Show space map roots in superblock 2021-02-03 15:51:52 +08:00
Ming-Hung Tsai 127f44c66b [thin_debug] Enhance error handling 
- Handle exception thrown by commands
- Add help and exit commands
 2021-02-03 15:51:52 +08:00
Ming-Hung Tsai 62d09c6752 [thin_debug] Reduce code size by eliminating duplicated types 
This patch doesn't have a significant effect - only a few KBs of code
is reduced. However, it's still a nice have.
 2021-02-03 15:51:52 +08:00
Ming-Hung Tsai faa371c208 [cache (rust)] Implement cache primitives 2021-02-03 15:18:47 +08:00
Ming-Hung Tsai 087a4711a5 [array (rust)] Implement array basis 2021-02-03 15:18:41 +08:00
Ming-Hung Tsai 4c17076f09 [cache_writeback] Support offset within the source and destination devices 2021-01-20 12:02:13 +08:00
Ming-Hung Tsai 6a2fa73924 [build] Enable building the dev-tools for functional tests 
- Factor out the dev-tools into a stand-alone, no-installed program
- Built the dev-tools if --enable-testing is specified
- Remove the --enable-dev-tools configure option
- Allow suffix on the binary name
- Update symlinks
- Cleanup Makefile
 2021-01-12 16:08:51 +08:00
Ming-Hung Tsai 8d59a83f7c [functional-tests] Update thin_delta help text 2021-01-12 02:50:46 +08:00
Ming-Hung Tsai ca7e79a828 Merge commit 'b67b587' into main 2021-01-12 02:50:12 +08:00
Ming-Hung Tsai 565c656ed2 [thin_generate_damage] Do not open a new transaction to prevent ref-count underflow 
There's a chance that thin_generate_damage tries to change ref-counts of
space map blocks due to its random nature, which could lead to problems.
If the ref-counts of metadata space map blocks (shadow source) is changed
to zero, then the ref-counts will become underflow after a shadow operation.

In-place space map modification is a way to prevent that value underflow.
An alternative approach is to avoid changing ref-counts of space map blocks.
 2020-11-24 18:18:21 +08:00
Ming-Hung Tsai c932a76f08 [unit-tests] Add underfull nodes counting tests 2020-11-24 18:18:21 +08:00
Ming-Hung Tsai 61f07573e1 [metadata_counter] Count under populated nodes if the option is provided 2020-11-24 18:18:21 +08:00
Ming-Hung Tsai 1fe8a0dbde [thin_check] Allow using --clear-needs-check and --skip-mappings together 
Although it is not recommended to clear the flag without a full
examination, however, the usage has been documented as an approach
to reduce lvchange run time [1]. For the purpose of backward
compatibility and avoiding boot failure after upgrading thin_check [2],
the limitation is now removed.

[1] https://wiki.archlinux.org/index.php/LVM#Thinly-provisioned_root_volume_device_times_out
[2] Community feedback on previous commit:
    https://github.com/jthornber/thin-provisioning-tools/commit/b278f4f
 2020-11-24 18:17:36 +08:00
Ming-Hung Tsai f364de35bc [unit-tests] Fix unflushed trashed blocks and variable referencing 2020-11-24 17:55:17 +08:00
Ming-Hung Tsai becdbbdb49 [build] Update Rust package version 2020-11-24 16:03:58 +08:00
Ming-Hung Tsai 7ceb500fc8 [thin_delta] Support comparing two specific subtrees 2020-11-24 15:58:01 +08:00
Ming-Hung Tsai 1d5b52b0dd [thin_delta] Clean up duplicated code 2020-11-24 15:17:35 +08:00
Ming-Hung Tsai b42408ef41 [thin] Introduce thin_patch_superblock to override superblock fields 2020-11-24 14:57:05 +08:00
Ming-Hung Tsai 44d025be0c [unit-tests] Add space map counting tests 2020-08-17 22:57:46 +08:00
Ming-Hung Tsai 3c49949796 [space-maps/disk] Support ignoring broken bitmaps on counting index_store 2020-08-17 22:57:26 +08:00
Ming-Hung Tsai 27ca8cc009 [block_counter] Add block_counter::clear() 2020-08-17 15:43:09 +08:00
Ming-Hung Tsai de843991e3 [transaction_manager] Add transaction_manager::commit() 
It should be called by metadata::commit() and reserve_metadata_snap()
(issue #73)
 2020-08-17 15:43:04 +08:00
Ming-Hung Tsai 9f3823c97d [metadata_checker] Rename function to reflect command line changes 2020-08-14 18:49:41 +08:00
Ming-Hung Tsai 6c90f9483e [metadata_checker] Support in-place metadata space map modification 2020-08-12 16:57:16 +08:00
Ming-Hung Tsai 0349e9c9e2 [space-maps/disk] Show the block address in exception string 2020-08-12 16:57:16 +08:00
Ming-Hung Tsai 1cfdcd58c5 [metadata_checker] Disable checking data block ref-counts due to performance concerns 2020-08-12 16:57:16 +08:00
Ming-Hung Tsai b16ff123b7 [thin] Stop metadata counting on the first error 2020-08-12 15:56:26 +08:00
Ming-Hung Tsai 12725983db [block_counter] Add the stop-on-error option to prevent silent errors 
The stop-on-error option aims to solve the lack of damage_visitor support
in space_map::count_metadata and btree counting_visitor. To prevent
possibly silent errors during space map or btree counting, the option
stops the counting process immediately when a btree or bitmap error
was found.

The bitmap blocks are also validated to avoid potential false-alarm,
if broken bitmap is not counted, and the corresponding ref-count is
also zeroed.
 2020-08-12 15:56:26 +08:00
Ming-Hung Tsai 3609b8bee5 [unit-tests] Add test cases for space maps 
- Increase test metadata size
- Test block allocation among multiple transactions
- Test the interaction between sm_recursive and sm_careful_alloc
- Test intended in-place modification
 2020-08-12 15:53:08 +08:00
Ming-Hung Tsai c952f52c79 [metadata_checker] Fix expected ref-counts in data space map comparison 2020-08-10 20:39:22 +08:00
Ming-Hung Tsai 7c3145633b [space-maps/core] Fix the search start position 2020-08-10 20:39:22 +08:00
Ming-Hung Tsai 2d9eaa1c56 [metadata_checker] Do not update the superblock if the needs_check flag is not set 2020-08-08 01:20:15 +08:00
Ming-Hung Tsai 3f19818c56 [thin] Adopt stateful random number generators 2020-08-05 22:14:01 +08:00
Ming-Hung Tsai 90f2d43a95 [thin_generate_damage] Add help text 2020-07-30 23:00:50 +08:00
Ming-Hung Tsai 80965944c6 [thin_check] Add functional tests for --fix-metadata-leaks 2020-07-30 23:00:50 +08:00
Ming-Hung Tsai 1932c243dd [thin_check] Add functional tests for incompatible options 2020-07-30 23:00:50 +08:00
Ming-Hung Tsai 7b2a18a865 [functional-tests] Update thin_check help 2020-07-30 23:00:50 +08:00
Ming-Hung Tsai b278f4ffa4 [thin_check] Change the policy of --clear-needs-check-flag to prevent error recurrence 
- Disallow clearing the needs_check flag if there's any error,
  i.e., the metadata must be fully examined, and the result must
  be NO_ERROR.
- Disallow combining --clear-needs-check with -m, --super-blocks-only,
  --skip-mappings, --override-mapping-root, or --ignore-non-fatal-errors.
 2020-07-30 23:00:50 +08:00
Ming-Hung Tsai 49efa06ffa [thin_check] Support fixing leaked metadata blocks 
- The leaked blocks are defined as allocated but not used ones,
  i.e., expected ref-count == 0 and actual ref-count <= 2.
- Blocks with expected ref-count > 0, or actual ref-count > 2,
  are treated as unrepairable leaks.
- The fixing processing won't be executed if there's any
  unrepairable leak or errors.
- The fixing process implicitly clearing the superblock
  needs_check flag if there's no error.
 2020-07-30 23:00:50 +08:00
Ming-Hung Tsai 7be6b91420 [metadata_checker] Tidying: rename variables 2020-07-29 16:47:22 +08:00
Ming-Hung Tsai ecb5226acc [thin] Fix indent and trailing whitespaces 2020-07-29 16:47:22 +08:00
Ming-Hung Tsai b693c7d6b0 [metadata_checker] Fix missing member initialization 2020-07-29 16:47:22 +08:00
Ming-Hung Tsai 0c3c5f6c1e [sequence_generator] Fix parameters and span boundary checking 2020-07-29 16:26:14 +08:00
Ming-Hung Tsai 661b4de5de [run_set] Add lower_bound and upper_bound functions 2020-07-29 16:18:23 +08:00
Ming-Hung Tsai 4d56cef5de [unit-tests] Add test cases for sequence_generator 2020-07-29 16:18:23 +08:00
Ming-Hung Tsai 105b8ec1cf [base] Factor out sequence_generator 2020-07-29 15:33:25 +08:00
Ming-Hung Tsai e62022a200 [base] Move run_set to namespace base 2020-07-29 15:33:25 +08:00
Ming-Hung Tsai 7a332d43c3 [build] Fix the default value for the AR variable (#145) 
Initialize the AR variable during configuration, to avoid leaving
unsubstituted string if the AR shell/environment variable is not set.
 2020-07-29 10:06:39 +08:00
Ming-Hung Tsai 061d966e7a [io_generator] Support mixed sequential and random io pattern 2020-07-25 17:46:59 +08:00
Ming-Hung Tsai 7ed013fcab [io_generator] Rename offset_generator classes, and fix bugs 
- Rename the classes for general-purpose usage
- Fix duplicated sequential number generation while wrap around
- Enable unaligned random number generation
 2020-07-25 17:20:59 +08:00
Ming-Hung Tsai d1fed5f5ec [thin_generate_damage] Introduce tools to generate metadata damage 2020-07-25 16:53:44 +08:00
Ming-Hung Tsai e46fabde1a [thin_generate_metadata] Fix command line options 
- Remove unused options
- Fix potential option conflict
 2020-07-25 16:53:44 +08:00
Ming-Hung Tsai 8d8ee51599 [thin_generate_metadata] Support setting the needs_check flag 2020-07-25 16:53:44 +08:00
Ming-Hung Tsai aabc3a9dbc [io_generator] Add random map to avoid generating repeated addresses 2020-07-25 16:53:44 +08:00
Ming-Hung Tsai cf127f3471 [btree] Fix reference counts of children below a shadow 2020-06-30 17:24:55 +08:00
Ming-Hung Tsai 4f8466c489 [btree] Fix parent key index for the new shadow 2020-06-30 17:22:47 +08:00
Ming-Hung Tsai ce8945e829 [io_generator] Fix address boundary for doing IO 
Limit the addresses to be within the specified range
 2020-06-30 17:12:42 +08:00
Ming-Hung Tsai 19e482fd2f [thin_generate_mappings] Update help text 2020-06-24 12:26:15 +08:00
Ming-Hung Tsai 8006e18173 [thin_pool] Fix the number of mapped blocks 2020-06-24 12:26:14 +08:00
Ming-Hung Tsai 084ac4b950 [io_generator] Simplify the class interface 2020-06-24 12:26:09 +08:00
Ming-Hung Tsai 8fa59553c4 [thin_generate_mappings] Introduce the mapping generation tool 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai 621b466333 [all] Eliminate duplicated sector_t typedefs 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai a345b894d3 [math_utils] Add is_power_of_two() 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai 5260a87c0b [base] Move math_utils to namespace base 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai 12d7178199 [base] Introduce io_generator 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai e724a72045 [btree] Implement btree::remove() 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai ef8e94b22c [thin_generate_metadata] Support setting transaction id and metadata snapshots 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai 524de134a4 [thin_pool] Support creating metadadata snapshot 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai e33c60c7e2 [thin_pool] Add close_thin() 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai 89409927b4 [thin_generate_metadata] Support device deletion 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai a5614f7db2 [thin_pool] Complete device deletion 2020-06-19 18:07:55 +08:00