emo/thin-provisioning-tools
1
0 Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,558 Commits 2 Branches 0 Tags
a5614f7db23e270fc1e694bf1be31cbe0640c4ee
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Joe Thornber dc5bb3559b [build] remove need for zlib. 
Disappeared with the C++ version of thin_metadata_pack
 2020-06-12 19:09:19 +01:00
Joe Thornber 3e24264442 Merge branch '2020-05-27-check-data-space-map' 2020-06-12 13:42:35 +01:00
Joe Thornber 0b5afc6cb0 [thin_check] fix bugs in thin_check -m 2020-06-12 13:41:47 +01:00
Joe Thornber 0ab509578f [*_check] Remove the inane "This is not the btree you are looking for." message. 2020-06-11 13:05:03 +01:00
Joe Thornber 7a85e47a20 [rust] squash clippy (linter) warnings 2020-06-10 16:46:38 +01:00
Joe Thornber ace8b39a8a Merge branch 'master' into 2020-05-27-check-data-space-map 2020-06-10 15:12:46 +01:00
Joe Thornber f90010e22a [pack/unpack] fix bug in file_utils::file_size() 2020-06-10 15:07:40 +01:00
Joe Thornber 380632565d [thin_metadata_{pack/unpack}] Fix get_nr_blocks() 
It didn't work with block devices.
 2020-06-10 14:38:42 +01:00
Joe Thornber 324c0050bf [install-rust-tools] add dep to generate man pages 2020-06-10 10:32:36 +01:00
Joe Thornber 549c0b9499 Merge branch 'master' into 2020-05-27-check-data-space-map 2020-06-10 10:15:16 +01:00
Joe Thornber 821628b004 [pack/unpack] add missing file. 2020-06-09 14:59:13 +01:00
Joe Thornber 86704deacb [thin_check] Add support for --metadata-snap. 
Not tested yet.
 2020-06-09 14:25:19 +01:00
Joe Thornber 01a7462ece Merge branch 'master' into 2020-05-27-check-data-space-map 2020-06-09 13:39:46 +01:00
Joe Thornber 409a660082 [pack/unpack] Get the functional tests working again. 
There's some hard coded version numbers in the tests, but I'm
leaving for now since I'll rewrite in Rust to avoid too much
of a proliferation of languages.
 2020-06-09 13:03:39 +01:00
Joe Thornber db5a71a53c [rust tools] Get the tools version from autoconf 2020-06-09 10:23:52 +01:00
Joe Thornber 61de3f9287 [thin_metadata_pack/unpack] Replace C++ implementation with a Rust one. 
The Rust implementation is multithreaded, performs better in general and
does custom compression of btree nodes to achieve much better compression
ratios.  unpack also checksums expanded metadata to validate it.

Format version has jumped to 3, no backwards compatibility, but I think
that's ok since we never made a release that contained the C++ version
of these tools.

Benchmarks
==========

On an 8 core, 16 hyperthread machine.

metadata 1G, full:

      Pack size    pack time     unpack time
------------------------------------------------------
C++      193M        50.3s          6.9s (no verify)
Rust      70M         1.4s          1.8s (verify)

metadata 16G, sparse:

       Pack size    pack time     unpack time
------------------------------------------------------
C++      21M          68s           1s   (no verify)
Rust      4M           8.6s         0.5s (verify)
 2020-06-09 09:15:00 +01:00
Joe Thornber c48851e747 Merge branch 'master' of github.com:jthornber/thin-provisioning-tools 2020-06-01 11:14:24 +01:00
Joe Thornber 9f2b747ffa [thin_metadata_pack] Prefetch blocks. 
Doubles performance
 2020-06-01 11:13:51 +01:00
Joe Thornber 16a10d2554 [thin_check] Remove the metadata_checker base class. 
It's really only a single method.
 2020-05-28 14:43:03 +01:00
Joe Thornber c94f560be8 [thin_check] Check the ref counts in the data space map. 
Hadn't realised this was being done.
 2020-05-28 14:35:51 +01:00
Joe Thornber 774111b34d Merge pull request #136 from mingnus/thin-generate-metadata-wip 
[thin_generate_metadata] Introduce the new transactional metadata generate tool
 2020-05-28 10:47:04 +01:00
Joe Thornber 4cdffafe88 [sm_recursive] Fix bug that allowed the same block to be allocated twice. 
See issue 70.  Soln from mingnus.
 2020-05-28 17:30:15 +08:00
Joe Thornber 128ecc27f2 [space-maps] remove count_possibly_greater_than_one 
This is only ever called with a recursive space map wrapper,
and that causes too many false positives.  False positives trigger
a copy of a metadata page, so this is a false economy.
 2020-05-28 07:38:58 +01:00
Joe Thornber 29cfdd8979 [space-maps/core] rewrite the core space map to use less memory. 2020-05-27 12:00:40 +01:00
Joe Thornber e92ae10408 [thin_metadata_pack/unpack] Add version field to header. 2020-05-26 10:59:13 +01:00
Joe Thornber 9daa726870 [thin_metadata_pack/unpack] functional tests 2020-05-26 10:55:03 +01:00
Joe Thornber a934a34d22 [thin_metadata_pack/unpack] Move the file magic nr before the compressed stream. 
Makes it easier to spot whether it's a pack file or not.
 2020-05-26 10:53:14 +01:00
Joe Thornber d9e18fecf1 Merge pull request #135 from mingnus/master 
[thin_metadata_pack] Improve block scanning performance
 2020-05-26 09:53:27 +01:00
Joe Thornber 371d1bbd6e [space-maps/recursive] fix bug in flush_ops() 
Decrements weren't being passed down properly.
 2020-05-26 09:24:15 +01:00
Joe Thornber 50f8e792d3 [space-map] Make the version of inc/dec that take a count the only one. 2020-05-26 09:12:45 +01:00
Joe Thornber cc806a0daa [space-maps/disk] add implementation for the variant of inc/dec that take a count 2020-05-26 08:57:13 +01:00
Joe Thornber 12b6157c5b [space-maps/recursive] lift a lock, and add a lock 2020-05-26 08:45:35 +01:00
Joe Thornber 0e54de95e1 [space-maps/recursive] Remove some code duplication. 2020-05-26 08:38:05 +01:00
Joe Thornber 65fcda062f [space-maps/recursive] Aggregate the BOPs 2020-05-26 08:30:52 +01:00
Joe Thornber 2ed089a83f [space-map] Add new virtual methods to inc or dec by a specific amount. 
Provide default implementation that just loops
 2020-05-26 08:29:52 +01:00
Joe Thornber 5e6eec1bb0 [recursive space map] switch to new form 'for' statements 2020-05-26 07:49:26 +01:00
Joe Thornber 0e1700fbe9 [thin_metadata_pack] First pass at pack/unpack 2020-05-22 14:11:48 +01:00
Joe Thornber b7d20bce48 [dev-tools] Update dev tools to use non-template version of block_manager. 2020-05-01 14:11:33 +01:00
Joe Thornber 0882022120 [build] remove [DEP] compilation messages. 
It just means each file appears twice.
 2020-04-30 15:50:58 +01:00
Joe Thornber 4313469475 [all] Switch from boost::shared_ptr -> std::shared_ptr. 
Shared_ptr has moved into the standard library since these tools were
first written.
 2020-04-30 15:02:43 +01:00
Joe Thornber e801cc607b [block_manager] Hard code block size to 4k. 
We're never going to use anything other than 4k, and by hard coding it we
avoid making block_manager a template.
 2020-04-30 14:30:01 +01:00
Joe Thornber 3e5de399a7 [functional tests] Remove dependency on the ThunderChez library. 
I've just moved the relevant code into the functional-tests dir.
 2020-04-30 12:07:42 +01:00
Joe Thornber ad79b627a4 Merge branch 'master' of github.com:jthornber/thin-provisioning-tools 2020-04-08 12:30:26 +01:00
Joe Thornber 0098803345 [thinp_*] Make mtree_ref_counter derive from ref_counter. 
Slight performance hit, but I'd rather everything was derived
from ref_counter.

Also implement it!  Not sure why this wasn't done before.
 2020-04-08 12:24:37 +01:00
Joe Thornber ed22bfcdda [space-map-disk] Add virtual destructor to index_store. 
Bug. Cleaning up index stores would not be correct.
 2020-04-08 12:23:28 +01:00
Joe Thornber 1481ed0f8a [ref_counter] Make ref_counter an abstract base class. 
Forces people to override correctly.

Also make no_op_ref_counter final.
 2020-04-08 12:22:28 +01:00
Joe Thornber 501299a4ee [cache_*] array ref_counter wasn't overloaded correctly. 
Major bug, explains some space map issues I've seen when
restoring cache metadata.
 2020-04-08 12:19:55 +01:00
Joe Thornber 8db3800e2c [base/application] One of the run methods wasn't overriding 2020-04-08 12:19:06 +01:00
Joe Thornber 0ccee1759e [various] Squash a couple of clang warning wrt friend declarations. 2020-04-08 12:18:24 +01:00
Joe Thornber 5e347dddbf [various] remove dead code. 
Squashes warnings from clang++
 2020-04-08 12:12:59 +01:00
Joe Thornber ed44a1128f Merge pull request #134 from Arusekk/issue-12 
Fix compilation with uClibc
 2020-03-25 15:00:28 +00:00
Joe Thornber 47da81b6b7 Merge pull request #133 from mingnus/master 
Improve performance, and remove unused code
 2020-03-02 11:22:49 +00:00
Joe Thornber 70f11e0866 Merge pull request #132 from mingnus/master 
Fix potential file descriptor leak and compile flags
 2020-02-25 16:08:36 +00:00
Joe Thornber 5f2c3bed69 Add some ftests, and fixup whitespace from Nikhil's work 2019-10-28 11:52:21 +00:00
Joe Thornber 0fc7529c01 [cache_repair, thin_repair] fix bug introduced in recent patch 
I hadn't realised that check_file_exists() also checked that it was
a regular file, which we don't want for the couple of uses I recently
added.

This patch adds an optional arg must_be_regular_file, and defaults
it to true, preserving the original behaviour.  The recent additions
have this set to false.
 2019-10-14 09:21:38 +01:00
Joe Thornber 1dd7b454bb [thin_repair, cache_repair] Check input file exists earlier 
and zero superblock if we fail part way through a repair.

bz1499781
 2019-10-08 14:34:24 +01:00
Joe Thornber a3b8e9db78 [thin_trim] remove all mention of --pool-inactive from man page. 
This option is obsolete now that we open O_EXCL.  If used a warning message
will be output.
 2019-10-07 13:32:52 +01:00
Joe Thornber 6332962ee8 [ft-lib/bcache] rename raise() -> raise_() 
Name clash with signal.h on Debian and Gentoo.
 2019-06-05 15:02:05 +01:00
Joe Thornber 5e5409f48b [space-map/noop] Be explicit about returning an optional 
Needed for older version of g++
 2019-06-05 11:07:28 +01:00
Joe Thornber f5acd0426d bump version 2019-06-04 10:39:59 +01:00
Joe Thornber 7be9be1494 [space-maps/noop] Add the noops space map 
Used when we know we're npt modifying the metadata.
 2019-06-04 10:38:58 +01:00
Joe Thornber c19a05f28f bump version 2019-06-03 14:11:37 +01:00
Joe Thornber 733c7df798 [thin_repair/thin_dump] Fix some more spurious error messages when doing repair. 
Repair was falling back to non-repair behaviour if it thought the roots
were ok.  Now if --repair is specified the same dumping code is always
executed.
 2019-06-03 14:03:24 +01:00
Joe Thornber e45a211902 [thin/metadata_dumper] Change a couple of confusing class names. 
metadata_emitter is actually a visitor that passes on it's data
to an encapsulated emitter object.

metadata_emitter -> metadata_emit_visitor
metadata_tree_emitter -> metadata_tree_emit_visitor
 2019-06-03 11:07:32 +01:00
Joe Thornber a13178ffd1 Bump version 2019-05-28 09:24:22 +01:00
Joe Thornber 0624ded67f [thin_repair, thin_dump] Don't look for better roots if the ones in the sb are ok. 
Calculating the most recent trees is inexact, so trees other than the ones
in the superblock, which presumably are the most recent, could be chosen.
 2019-05-27 16:43:32 +01:00
Joe Thornber 5e6ddbf2da [thin_dump, thin_repair] Fix bad metadata_damage exception 
The first pass of the repair process scans all metadata working out the
largest orphan btrees.  This scan doesn't use as much validation as
the btree_walk function which subsequently gets called.

This patch catches any exceptions thrown by the btree walk function
and removes that btree from consideration.
 2019-05-27 16:06:59 +01:00
Joe Thornber d299cc66d1 [functional-tests/thin_dump] check no error msg with clean metadata 2019-05-27 15:38:45 +01:00
Joe Thornber acf2e6e625 [unit-tests] squash some warnings 
ASSERT_THAT(), defined in google-test needs to be wrapped in braces if
it's the only statement in an if block (dangling else).
 2019-05-27 09:19:14 +01:00
Joe Thornber 65264911c1 [thin_repair, thin_dump] --repair falg wasn't being passed through properly. 
Recent regression.
 2019-05-27 09:04:55 +01:00
Joe Thornber 9e87ad8298 bump version to 0.8.1 2019-05-03 15:19:06 +01:00
Joe Thornber 595ddfc50f [thin_repair/thin_dump] repair flag wasn't being set internally. 
Introduced by recent changes
 2019-05-03 15:15:01 +01:00
Joe Thornber 4bca324c9f Bump version to 0.8.0 2019-04-17 12:55:15 +01:00
Joe Thornber 9e20465fd1 [thin-repair, thin_dump] When repairing we now hunt for the best btree roots. 
We've had a trickle of users who accidentally activate the same pool on a
VM and host at the same time.  Typically the host doesn't do any IO, but
the kernel will still rewrite the superblock on shutdown.  This leaves
the superblock pointing to very out of date btree roots and so we get
massive metadata loss.

This patch changes thin_repair, and thin_dump --repair.  They now hunt
for the most recent, undamaged and consistent roots of the device and
mapping trees, and use that as the starting point of the repair.
 2019-04-17 12:17:13 +01:00
Joe Thornber b027a1039f [thin-provisioning] Add a new constructor for the metadata class 
This one takes a superblock.  Used when repairing.
 2019-04-17 12:15:46 +01:00
Joe Thornber b777a77023 [thin-provisioning] pass the tm by reference to the metadata object 2019-04-04 15:06:41 +01:00
Joe Thornber aace49cdd6 [thin_repair] work in progress 2019-03-29 12:56:31 +00:00
Joe Thornber 95ae4fef6a [functional tests] Use dd to create zeroed files 
Previously I was using fallocate, which has issues on some filesystems.
 2018-12-12 13:14:09 +00:00
Joe Thornber e03b878df3 [functional tests] Move test output from /tmp to . 
tmpfs has trouble with O_DIRECT.
 2018-12-12 13:13:11 +00:00
Joe Thornber cc56214eb8 Merge branch 'master' of github.com:jthornber/thin-provisioning-tools 2018-12-12 10:25:52 +00:00
Joe Thornber 207ef304c3 [era/restore_emitter] remove a noop 
Pointless self assignment.  Value in suerpblock does get assigned properly.
 2018-12-12 10:23:35 +00:00
Joe Thornber d1a41d01cc [thin-check] print out some info fields. 
These are consumed by lvm.
 2018-11-09 12:43:21 +00:00
Joe Thornber 5a309dd3bd Merge branch 'master' of github.com:jthornber/thin-provisioning-tools 2018-11-09 11:18:27 +00:00
Joe Thornber de7c9a5781 [thin_journal_check] Checks journal of block manager activity. 
You need to apply doc/bm-journal.patch to create the journal.

thin_journal_check confirms that if the machine had crashed at any time
during the test run no metadata corruption would have occured.
 2018-09-24 14:51:46 +01:00
Joe Thornber 70cdfe12a2 [thin_journal_check] first code drop 2018-08-28 13:12:20 +01:00
Joe Thornber a512b8754b Revert "fix compile failed with libc musl (#103)" 
This reverts commit 9311aa6483.
 2018-08-02 07:12:22 +01:00
Joe Thornber 82ff03e5c1 Merge branch 'master' of github.com:jthornber/thin-provisioning-tools 2018-06-21 10:11:27 +01:00
Joe Thornber 68284c12a8 Merge branch '2018-04-25-thin-discard-bug' 2018-06-21 10:11:21 +01:00
Joe Thornber 498057d00e ft-lib: add discard() 2018-06-21 10:06:01 +01:00
Joe Thornber cfe421ab8b Merge commit '570e11b2da6ee55bc755f5211cf80c4f942b4644' 2018-06-13 16:58:07 +01:00
Joe Thornber 570e11b2da [dmtests] Add failing test thin/discard/recover-space 
See bz1563697
 2018-06-13 16:54:36 +01:00
Joe Thornber b6ba3a070d [restore_emitter] squash a strncpy warning 2018-06-13 12:50:48 +01:00
Joe Thornber af785461e9 [build] [CXX] -> [CC] for C files 2018-06-13 12:49:57 +01:00
Joe Thornber 80e046a7df [build] support separate build directories 2018-06-13 09:34:14 +01:00
Joe Thornber c1e6adc72c [build] support separate build directories 2018-06-13 09:33:25 +01:00
Joe Thornber 5d3197b8ab [build] quieten build of C files 2018-04-25 14:05:28 +01:00
Joe Thornber 6f936992b8 bump version 2018-04-23 10:41:48 +01:00
Joe Thornber 1172b2f450 [functional-tests] with-temp-file-sized now takes *either* bytes or a disk-size object 2018-04-18 15:12:28 +01:00
Joe Thornber e619c6edd0 Merge branch 'master' of github.com:jthornber/thin-provisioning-tools 2018-04-17 13:33:32 +01:00
Joe Thornber 1433d11f59 [thin-provisioning/metadata] Use latest metadata version (2) 2018-04-17 13:32:49 +01:00