emo/thin-provisioning-tools
1
0 Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,616 Commits 2 Branches 0 Tags
75ba654b7a050db24cedcb9eb6c11683091fc511
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Joe Thornber 75ba654b7a Merge pull request #144 from mingnus/thin-generate-metadata-wip 
Enhance thin_generate_mappings, and introduce thin_generate_damage
 2020-07-27 15:55:22 +01:00
Joe Thornber e9055f5d7d Merge branch 'master' of github.com:jthornber/thin-provisioning-tools 2020-07-22 16:09:33 +01:00
Joe Thornber 67622f3c76 Merge branch '2020-06-23-thin-remap' 2020-07-22 16:08:19 +01:00
Joe Thornber 32019ac388 [thin_shrink] add a trivial snapshot test. 
Still need much more work on snap tests.
 2020-07-22 16:07:34 +01:00
Joe Thornber b7e02d0ae4 [thin_metadata_pack] FIx bug where pack would try and read too much. 2020-07-09 13:53:46 +01:00
Joe Thornber 19138dbd81 [thin_shrink] add unit tests for build_remaps 2020-07-06 11:44:49 +01:00
Joe Thornber 23f3033f61 [functional-tests] bump version nr 2020-07-03 13:28:02 +01:00
Joe Thornber 73d26393ab [functional-tests] bump version in help strings 2020-07-03 13:13:59 +01:00
Joe Thornber 9c45d830d9 bump version 2020-07-03 12:08:01 +01:00
Joe Thornber fec11289b0 [thin_check/dump] Under populated nodes are now 'non fatal errors' 
thin_dump always ignores non-fatal-errors.
 2020-07-02 16:03:23 +01:00
Joe Thornber 9adcc7911f Merge branch 'master' of github.com:jthornber/thin-provisioning-tools 2020-07-02 10:53:10 +01:00
Joe Thornber a407c831df [thin_repair] Fix segfault with metadata containing loops (bz1853241) 
'seen' bit was being set too late.
 2020-07-02 10:51:10 +01:00
Joe Thornber 470d0b7532 [thin_shrink] more test cases 2020-07-01 07:52:53 +01:00
Joe Thornber 7f4fa9c58a Merge pull request #143 from mingnus/thin-generate-metadata-wip 
btree::remove bug fixes
 2020-06-30 15:23:20 +01:00
Joe Thornber 4d7f9c7ddc [thin_shrink] add some more test cases 2020-06-30 08:32:04 +01:00
Joe Thornber 3618da3a12 [thin_shrink] Get test framework built 
This verifies data has been moved to the correct place.
 2020-06-29 10:49:40 +01:00
Joe Thornber d03dac8f75 [thin_shrink] write test harness 2020-06-26 16:44:47 +01:00
Joe Thornber 07da5704d5 [thin_shrink] add --no-copy 2020-06-26 08:57:26 +01:00
Joe Thornber abf06236b1 [thin_shrink] Tidy up some messages 2020-06-26 08:47:40 +01:00
Joe Thornber d8a0805753 [thin_shrink] fix some clippy warnings 2020-06-26 08:31:02 +01:00
Joe Thornber 7df56a5a04 [thin_shrink] Squash some warnings 2020-06-26 08:00:53 +01:00
Joe Thornber 27c6e36f5c [thin_shrink] Very simple copier implementation 2020-06-25 16:50:11 +01:00
Joe Thornber fcf44d46b7 [thin_shrink] remove some debug printlns 2020-06-25 16:12:35 +01:00
Joe Thornber 31abc468be [thin_shrink] Stub the copier 2020-06-25 15:28:45 +01:00
Joe Thornber d8957e3d86 [thin_shrink] Rewrites xml 
Just need to do copying now
 2020-06-25 14:57:37 +01:00
Joe Thornber 259eef9eee [thin_shrink] calculate remaps 2020-06-25 10:44:57 +01:00
Joe Thornber 3f1b776359 [thin_shrink] Add NoopVisitor 2020-06-24 13:55:08 +01:00
Joe Thornber 861b2f21ff [thin_shrink] Write thinp xml format reader and writer. 
Still need to tidy up the error handling.
 2020-06-24 13:32:47 +01:00
Joe Thornber 8aaea7f45b Merge pull request #142 from mingnus/thin-generate-metadata-wip 
thin_generate_metadata minor fixes
 2020-06-24 12:29:31 +01:00
Joe Thornber fe754d81a4 Merge branch 'master' of github.com:jthornber/thin-provisioning-tools 2020-06-19 14:50:51 +01:00
Joe Thornber 1fd0bfd4e8 Merge pull request #141 from mingnus/thin-generate-metadata-wip 
Complete the major features of thin_generate_metadata, and introduce thin_generate_mappings
 2020-06-19 14:46:52 +01:00
Joe Thornber d1cf87712b Bump version to 0.9.0-rc1 2020-06-15 12:50:58 +01:00
Joe Thornber b8eebf59c7 Update CHANGES file 2020-06-15 12:49:16 +01:00
Joe Thornber 7d625eb6ff [src/file_utils.rs] Don't print to stderr in file_exists. 2020-06-15 12:48:19 +01:00
Joe Thornber 0d2b49aae7 Merge branch 'master' of github.com:jthornber/thin-provisioning-tools 2020-06-15 11:29:01 +01:00
Joe Thornber bb64189d26 [thin_check] update thin_check manpage. 2020-06-15 11:27:15 +01:00
Joe Thornber dc5bb3559b [build] remove need for zlib. 
Disappeared with the C++ version of thin_metadata_pack
 2020-06-12 19:09:19 +01:00
Joe Thornber 3e24264442 Merge branch '2020-05-27-check-data-space-map' 2020-06-12 13:42:35 +01:00
Joe Thornber 0b5afc6cb0 [thin_check] fix bugs in thin_check -m 2020-06-12 13:41:47 +01:00
Joe Thornber 0ab509578f [*_check] Remove the inane "This is not the btree you are looking for." message. 2020-06-11 13:05:03 +01:00
Joe Thornber 7a85e47a20 [rust] squash clippy (linter) warnings 2020-06-10 16:46:38 +01:00
Joe Thornber ace8b39a8a Merge branch 'master' into 2020-05-27-check-data-space-map 2020-06-10 15:12:46 +01:00
Joe Thornber f90010e22a [pack/unpack] fix bug in file_utils::file_size() 2020-06-10 15:07:40 +01:00
Joe Thornber 380632565d [thin_metadata_{pack/unpack}] Fix get_nr_blocks() 
It didn't work with block devices.
 2020-06-10 14:38:42 +01:00
Joe Thornber 324c0050bf [install-rust-tools] add dep to generate man pages 2020-06-10 10:32:36 +01:00
Joe Thornber 549c0b9499 Merge branch 'master' into 2020-05-27-check-data-space-map 2020-06-10 10:15:16 +01:00
Joe Thornber 821628b004 [pack/unpack] add missing file. 2020-06-09 14:59:13 +01:00
Joe Thornber 86704deacb [thin_check] Add support for --metadata-snap. 
Not tested yet.
 2020-06-09 14:25:19 +01:00
Joe Thornber 01a7462ece Merge branch 'master' into 2020-05-27-check-data-space-map 2020-06-09 13:39:46 +01:00
Joe Thornber 409a660082 [pack/unpack] Get the functional tests working again. 
There's some hard coded version numbers in the tests, but I'm
leaving for now since I'll rewrite in Rust to avoid too much
of a proliferation of languages.
 2020-06-09 13:03:39 +01:00
Joe Thornber db5a71a53c [rust tools] Get the tools version from autoconf 2020-06-09 10:23:52 +01:00
Joe Thornber 61de3f9287 [thin_metadata_pack/unpack] Replace C++ implementation with a Rust one. 
The Rust implementation is multithreaded, performs better in general and
does custom compression of btree nodes to achieve much better compression
ratios.  unpack also checksums expanded metadata to validate it.

Format version has jumped to 3, no backwards compatibility, but I think
that's ok since we never made a release that contained the C++ version
of these tools.

Benchmarks
==========

On an 8 core, 16 hyperthread machine.

metadata 1G, full:

      Pack size    pack time     unpack time
------------------------------------------------------
C++      193M        50.3s          6.9s (no verify)
Rust      70M         1.4s          1.8s (verify)

metadata 16G, sparse:

       Pack size    pack time     unpack time
------------------------------------------------------
C++      21M          68s           1s   (no verify)
Rust      4M           8.6s         0.5s (verify)
 2020-06-09 09:15:00 +01:00
Joe Thornber c48851e747 Merge branch 'master' of github.com:jthornber/thin-provisioning-tools 2020-06-01 11:14:24 +01:00
Joe Thornber 9f2b747ffa [thin_metadata_pack] Prefetch blocks. 
Doubles performance
 2020-06-01 11:13:51 +01:00
Joe Thornber 16a10d2554 [thin_check] Remove the metadata_checker base class. 
It's really only a single method.
 2020-05-28 14:43:03 +01:00
Joe Thornber c94f560be8 [thin_check] Check the ref counts in the data space map. 
Hadn't realised this was being done.
 2020-05-28 14:35:51 +01:00
Joe Thornber 774111b34d Merge pull request #136 from mingnus/thin-generate-metadata-wip 
[thin_generate_metadata] Introduce the new transactional metadata generate tool
 2020-05-28 10:47:04 +01:00
Joe Thornber 4cdffafe88 [sm_recursive] Fix bug that allowed the same block to be allocated twice. 
See issue 70.  Soln from mingnus.
 2020-05-28 17:30:15 +08:00
Joe Thornber 128ecc27f2 [space-maps] remove count_possibly_greater_than_one 
This is only ever called with a recursive space map wrapper,
and that causes too many false positives.  False positives trigger
a copy of a metadata page, so this is a false economy.
 2020-05-28 07:38:58 +01:00
Joe Thornber 29cfdd8979 [space-maps/core] rewrite the core space map to use less memory. 2020-05-27 12:00:40 +01:00
Joe Thornber e92ae10408 [thin_metadata_pack/unpack] Add version field to header. 2020-05-26 10:59:13 +01:00
Joe Thornber 9daa726870 [thin_metadata_pack/unpack] functional tests 2020-05-26 10:55:03 +01:00
Joe Thornber a934a34d22 [thin_metadata_pack/unpack] Move the file magic nr before the compressed stream. 
Makes it easier to spot whether it's a pack file or not.
 2020-05-26 10:53:14 +01:00
Joe Thornber d9e18fecf1 Merge pull request #135 from mingnus/master 
[thin_metadata_pack] Improve block scanning performance
 2020-05-26 09:53:27 +01:00
Joe Thornber 371d1bbd6e [space-maps/recursive] fix bug in flush_ops() 
Decrements weren't being passed down properly.
 2020-05-26 09:24:15 +01:00
Joe Thornber 50f8e792d3 [space-map] Make the version of inc/dec that take a count the only one. 2020-05-26 09:12:45 +01:00
Joe Thornber cc806a0daa [space-maps/disk] add implementation for the variant of inc/dec that take a count 2020-05-26 08:57:13 +01:00
Joe Thornber 12b6157c5b [space-maps/recursive] lift a lock, and add a lock 2020-05-26 08:45:35 +01:00
Joe Thornber 0e54de95e1 [space-maps/recursive] Remove some code duplication. 2020-05-26 08:38:05 +01:00
Joe Thornber 65fcda062f [space-maps/recursive] Aggregate the BOPs 2020-05-26 08:30:52 +01:00
Joe Thornber 2ed089a83f [space-map] Add new virtual methods to inc or dec by a specific amount. 
Provide default implementation that just loops
 2020-05-26 08:29:52 +01:00
Joe Thornber 5e6eec1bb0 [recursive space map] switch to new form 'for' statements 2020-05-26 07:49:26 +01:00
Joe Thornber 0e1700fbe9 [thin_metadata_pack] First pass at pack/unpack 2020-05-22 14:11:48 +01:00
Joe Thornber b7d20bce48 [dev-tools] Update dev tools to use non-template version of block_manager. 2020-05-01 14:11:33 +01:00
Joe Thornber 0882022120 [build] remove [DEP] compilation messages. 
It just means each file appears twice.
 2020-04-30 15:50:58 +01:00
Joe Thornber 4313469475 [all] Switch from boost::shared_ptr -> std::shared_ptr. 
Shared_ptr has moved into the standard library since these tools were
first written.
 2020-04-30 15:02:43 +01:00
Joe Thornber e801cc607b [block_manager] Hard code block size to 4k. 
We're never going to use anything other than 4k, and by hard coding it we
avoid making block_manager a template.
 2020-04-30 14:30:01 +01:00
Joe Thornber 3e5de399a7 [functional tests] Remove dependency on the ThunderChez library. 
I've just moved the relevant code into the functional-tests dir.
 2020-04-30 12:07:42 +01:00
Joe Thornber ad79b627a4 Merge branch 'master' of github.com:jthornber/thin-provisioning-tools 2020-04-08 12:30:26 +01:00
Joe Thornber 0098803345 [thinp_*] Make mtree_ref_counter derive from ref_counter. 
Slight performance hit, but I'd rather everything was derived
from ref_counter.

Also implement it!  Not sure why this wasn't done before.
 2020-04-08 12:24:37 +01:00
Joe Thornber ed22bfcdda [space-map-disk] Add virtual destructor to index_store. 
Bug. Cleaning up index stores would not be correct.
 2020-04-08 12:23:28 +01:00
Joe Thornber 1481ed0f8a [ref_counter] Make ref_counter an abstract base class. 
Forces people to override correctly.

Also make no_op_ref_counter final.
 2020-04-08 12:22:28 +01:00
Joe Thornber 501299a4ee [cache_*] array ref_counter wasn't overloaded correctly. 
Major bug, explains some space map issues I've seen when
restoring cache metadata.
 2020-04-08 12:19:55 +01:00
Joe Thornber 8db3800e2c [base/application] One of the run methods wasn't overriding 2020-04-08 12:19:06 +01:00
Joe Thornber 0ccee1759e [various] Squash a couple of clang warning wrt friend declarations. 2020-04-08 12:18:24 +01:00
Joe Thornber 5e347dddbf [various] remove dead code. 
Squashes warnings from clang++
 2020-04-08 12:12:59 +01:00
Joe Thornber ed44a1128f Merge pull request #134 from Arusekk/issue-12 
Fix compilation with uClibc
 2020-03-25 15:00:28 +00:00
Joe Thornber 47da81b6b7 Merge pull request #133 from mingnus/master 
Improve performance, and remove unused code
 2020-03-02 11:22:49 +00:00
Joe Thornber 70f11e0866 Merge pull request #132 from mingnus/master 
Fix potential file descriptor leak and compile flags
 2020-02-25 16:08:36 +00:00
Joe Thornber 5f2c3bed69 Add some ftests, and fixup whitespace from Nikhil's work 2019-10-28 11:52:21 +00:00
Joe Thornber 0fc7529c01 [cache_repair, thin_repair] fix bug introduced in recent patch 
I hadn't realised that check_file_exists() also checked that it was
a regular file, which we don't want for the couple of uses I recently
added.

This patch adds an optional arg must_be_regular_file, and defaults
it to true, preserving the original behaviour.  The recent additions
have this set to false.
 2019-10-14 09:21:38 +01:00
Joe Thornber 1dd7b454bb [thin_repair, cache_repair] Check input file exists earlier 
and zero superblock if we fail part way through a repair.

bz1499781
 2019-10-08 14:34:24 +01:00
Joe Thornber a3b8e9db78 [thin_trim] remove all mention of --pool-inactive from man page. 
This option is obsolete now that we open O_EXCL.  If used a warning message
will be output.
 2019-10-07 13:32:52 +01:00
Joe Thornber 6332962ee8 [ft-lib/bcache] rename raise() -> raise_() 
Name clash with signal.h on Debian and Gentoo.
 2019-06-05 15:02:05 +01:00
Joe Thornber 5e5409f48b [space-map/noop] Be explicit about returning an optional 
Needed for older version of g++
 2019-06-05 11:07:28 +01:00
Joe Thornber f5acd0426d bump version 2019-06-04 10:39:59 +01:00
Joe Thornber 7be9be1494 [space-maps/noop] Add the noops space map 
Used when we know we're npt modifying the metadata.
 2019-06-04 10:38:58 +01:00
Joe Thornber c19a05f28f bump version 2019-06-03 14:11:37 +01:00
Joe Thornber 733c7df798 [thin_repair/thin_dump] Fix some more spurious error messages when doing repair. 
Repair was falling back to non-repair behaviour if it thought the roots
were ok.  Now if --repair is specified the same dumping code is always
executed.
 2019-06-03 14:03:24 +01:00
Joe Thornber e45a211902 [thin/metadata_dumper] Change a couple of confusing class names. 
metadata_emitter is actually a visitor that passes on it's data
to an encapsulated emitter object.

metadata_emitter -> metadata_emit_visitor
metadata_tree_emitter -> metadata_tree_emit_visitor
 2019-06-03 11:07:32 +01:00