emo/thin-provisioning-tools
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,569 Commits 2 Branches 0 Tags 3.7 MiB
1fd0bfd4e8
Commit Graph

1569 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Joe Thornber
1fd0bfd4e8
Merge pull request #141 from mingnus/thin-generate-metadata-wip 
Complete the major features of thin_generate_metadata, and introduce thin_generate_mappings
 2020-06-19 14:46:52 +01:00
Ming-Hung Tsai
8fa59553c4 [thin_generate_mappings] Introduce the mapping generation tool 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai
621b466333 [all] Eliminate duplicated sector_t typedefs 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai
a345b894d3 [math_utils] Add is_power_of_two() 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai
5260a87c0b [base] Move math_utils to namespace base 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai
12d7178199 [base] Introduce io_generator 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai
e724a72045 [btree] Implement btree::remove() 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai
ef8e94b22c [thin_generate_metadata] Support setting transaction id and metadata snapshots 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai
524de134a4 [thin_pool] Support creating metadadata snapshot 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai
e33c60c7e2 [thin_pool] Add close_thin() 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai
89409927b4 [thin_generate_metadata] Support device deletion 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai
a5614f7db2 [thin_pool] Complete device deletion 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai
f803c44e93 [btree] Implement btree::destroy() to delete the entire tree 
Also decrease the reference count of the mapped values.
(e.g., bitmap blocks of btree_index_store)
 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai
4e1ec9326b [thin_pool] Tidying 
- Remove unused headers
- Use predefined types
 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai
598696bc32 [thin_pool] Implement synchronous mapping insertion and removal 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai
383ae78871 [thin_pool] Determine shared blocks while looking up mappings 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai
a842b4ecfd [thin_generate_metadata] Make use of thin_pool.cc 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai
66b519db83 [thin_pool] Fix snapshot creation 2020-06-19 17:22:19 +08:00
Ming-Hung Tsai
2cce8a51a7 [thin_pool] Fix mapped block time 2020-06-19 17:22:19 +08:00
Ming-Hung Tsai
7f7ba950ef [thin_pool] Complete device creation 
- Set up the device details
- Cache device details of opened devices
- Update changed devices at once while committing the pool
 2020-06-19 17:22:19 +08:00
Ming-Hung Tsai
77adb08c3f [thin_pool] Replace the member pointer by references 2020-06-17 19:09:24 +08:00
Ming-Hung Tsai
0b058a2850 [thin_pool] Encapsulate metadata into the pool class 2020-06-17 19:09:24 +08:00
Ming-Hung Tsai
5810846409 [thin] Add more constructors to device_details 2020-06-17 19:09:24 +08:00
Joe Thornber
dc5bb3559b [build] remove need for zlib. 
Disappeared with the C++ version of thin_metadata_pack
 2020-06-12 19:09:19 +01:00
Joe Thornber
3e24264442 Merge branch '2020-05-27-check-data-space-map' 2020-06-12 13:42:35 +01:00
Joe Thornber
0b5afc6cb0 [thin_check] fix bugs in thin_check -m 2020-06-12 13:41:47 +01:00
Joe Thornber
0ab509578f [*_check] Remove the inane "This is not the btree you are looking for." message. 2020-06-11 13:05:03 +01:00
Joe Thornber
7a85e47a20 [rust] squash clippy (linter) warnings 2020-06-10 16:46:38 +01:00
Joe Thornber
ace8b39a8a Merge branch 'master' into 2020-05-27-check-data-space-map 2020-06-10 15:12:46 +01:00
Joe Thornber
f90010e22a [pack/unpack] fix bug in file_utils::file_size() 2020-06-10 15:07:40 +01:00
Joe Thornber
380632565d [thin_metadata_{pack/unpack}] Fix get_nr_blocks() 
It didn't work with block devices.
 2020-06-10 14:38:42 +01:00
Joe Thornber
324c0050bf [install-rust-tools] add dep to generate man pages 2020-06-10 10:32:36 +01:00
Joe Thornber
549c0b9499 Merge branch 'master' into 2020-05-27-check-data-space-map 2020-06-10 10:15:16 +01:00
Joe Thornber
821628b004 [pack/unpack] add missing file. 2020-06-09 14:59:13 +01:00
Joe Thornber
86704deacb [thin_check] Add support for --metadata-snap. 
Not tested yet.
 2020-06-09 14:25:19 +01:00
Joe Thornber
01a7462ece Merge branch 'master' into 2020-05-27-check-data-space-map 2020-06-09 13:39:46 +01:00
Joe Thornber
409a660082 [pack/unpack] Get the functional tests working again. 
There's some hard coded version numbers in the tests, but I'm
leaving for now since I'll rewrite in Rust to avoid too much
of a proliferation of languages.
 2020-06-09 13:03:39 +01:00
Joe Thornber
db5a71a53c [rust tools] Get the tools version from autoconf 2020-06-09 10:23:52 +01:00
Joe Thornber
61de3f9287 [thin_metadata_pack/unpack] Replace C++ implementation with a Rust one. 
The Rust implementation is multithreaded, performs better in general and
does custom compression of btree nodes to achieve much better compression
ratios.  unpack also checksums expanded metadata to validate it.

Format version has jumped to 3, no backwards compatibility, but I think
that's ok since we never made a release that contained the C++ version
of these tools.

Benchmarks
==========

On an 8 core, 16 hyperthread machine.

metadata 1G, full:

      Pack size    pack time     unpack time
------------------------------------------------------
C++      193M        50.3s          6.9s (no verify)
Rust      70M         1.4s          1.8s (verify)

metadata 16G, sparse:

       Pack size    pack time     unpack time
------------------------------------------------------
C++      21M          68s           1s   (no verify)
Rust      4M           8.6s         0.5s (verify)
 2020-06-09 09:15:00 +01:00
Joe Thornber
c48851e747 Merge branch 'master' of github.com:jthornber/thin-provisioning-tools 2020-06-01 11:14:24 +01:00
Joe Thornber
9f2b747ffa [thin_metadata_pack] Prefetch blocks. 
Doubles performance
 2020-06-01 11:13:51 +01:00
Joe Thornber
16a10d2554 [thin_check] Remove the metadata_checker base class. 
It's really only a single method.
 2020-05-28 14:43:03 +01:00
Joe Thornber
c94f560be8 [thin_check] Check the ref counts in the data space map. 
Hadn't realised this was being done.
 2020-05-28 14:35:51 +01:00
Joe Thornber
774111b34d
Merge pull request #136 from mingnus/thin-generate-metadata-wip 
[thin_generate_metadata] Introduce the new transactional metadata generate tool
 2020-05-28 10:47:04 +01:00
Ming-Hung Tsai
86139cf6bd [sm_recursive] Fix atomicity when applying BOPs 
Any BOP should be taken only once. Therefore, we should remove BOPs
that are being processed or had been processed from the uncommitted
list.
 2020-05-28 17:30:15 +08:00
Joe Thornber
4cdffafe88 [sm_recursive] Fix bug that allowed the same block to be allocated twice. 
See issue 70.  Soln from mingnus.
 2020-05-28 17:30:15 +08:00
Ming-Hung Tsai
0f2918e989 [thin_generate_metadata] Implement format_metadata and create_thin 2020-05-28 17:30:15 +08:00
Ming-Hung Tsai
0c6d132774 [thin_generate_metadata] Preparation for revising the features 2020-05-28 15:09:20 +08:00
Joe Thornber
128ecc27f2 [space-maps] remove count_possibly_greater_than_one 
This is only ever called with a recursive space map wrapper,
and that causes too many false positives.  False positives trigger
a copy of a metadata page, so this is a false economy.
 2020-05-28 07:38:58 +01:00
Joe Thornber
29cfdd8979 [space-maps/core] rewrite the core space map to use less memory. 2020-05-27 12:00:40 +01:00