Vitaliy Filippov
ec10a25905
Use 32-bit big write location (OK for up to 512 TB OSDs)
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
923c41b928
Do not store offset & len in big_writes
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
8f179eb9a6
Fix object crc32c calculation
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
172b106d99
Add entry_type to heap_object_t too
...
This is required to:
1) later inline the last "big_write" entry into the object to slightly reduce memory usage
2) eliminate an ugly hack where entry type is determined by its size
3) make the storage scheme extensible i.e. when adding new entry types
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
59a51363d0
Rename flags to entry_type
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
65d5d69ae3
Use robin_hood::unordered_flat_map - it has 1 byte overhead instead of 8 byte
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
5d100b4e35
Revert try_get
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
7a894efcb7
WIP dump/load heap
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
3bad0f6fa8
Add missing list_stable_limit support
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
f215c791c8
Fix checksum validation in !inmemory_journal mode
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
0a3260672b
Fix checksum padding during read
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
171b339eb3
Remove unused bs->buffer_area + buffer calc
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
ac331fa77b
Add missing memset zero_object to 0
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
915be747d6
Mark in-memory data reads as SKIP_CSUM
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
b14fbbf5cc
Add missing calc_crc32c after updating block checksums
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
e379fc6df4
Fix checksum padding during flush
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
eec2f2b799
Fix free_read_buffers
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
d85e1dc204
Fix assert(region_marker) at the end of the block in find_block_run()
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
c2edcf63e1
Fix assert(is_buffer_area_free) with size=0
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
3a8f0f53de
Add a copy of wyhash
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
9b70bfc2e4
Fix test build with isa-l
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
d1678c67d9
Remove assert !region_marker & FREE
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
ad308ed92a
Use wyhash
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
b0bf45c26f
Change emhashes
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
2a550d4d13
unordered_map mvcc
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
d4d3d34a84
Fix bad resharding due to the lack of iteration order in a hashmap
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
b8bec78879
Use emhash::try_get
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
fa77a321ab
Use emhash hashmap (2x faster)
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
cfce09be03
Use unordered_maps for object-block index
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
ed3332ea83
B-tree is slower...
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
0a94c437e2
Unordered_map for pool settings
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
3bb514321f
Fix read fio bench
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
1a4c6746c4
Fix vitastor-disk prepare and param validation
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
3040bcfbfa
Fix skipping of corrupted objects, fix use_buffer_area with zero size
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
08e16ba985
Support heap format in simple-offsets
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
9afa3f9a13
Return all unstable versions in listing
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
bbf5063b25
Actually fsync after stabilize
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
f45c88d38c
Move test & build_test to top-level cmakelists
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
8fa0638347
Add missing request_trim
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
0c2e0b8a49
Use multilist_index_t instead of multiple bitmap allocators
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
36ccf63922
Implement another multilist-style allocator for metadata blocks
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
4bc1d09fbd
Fix a bug with unstable_big over unstable_small
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
dc31650110
Limit the number of unstable versions per object
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
598e1ed1db
Integrate moving objects
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
a49c176423
Support moving objects between blocks
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
ecd26ebac3
Fix zero-length writes
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
d0690091c1
Fix op_stable slowdowns
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
9369b643a1
Fix a bs_read bug
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
ed2a886ab4
Fix test dependencies
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
98bdcfbff9
Add read_blocks() API
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
3cce10d9c7
Fix block checksum calculation in write_journal for the old blockstore version
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
8762ae07a6
Remove block_order parameter
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
b2b30201c4
Extract (flags & BS_HEAP_TYPE) into a function
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
a912420424
Make dump-journal --format data default
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
27c250e4da
Add fio options
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
0b28ec8fc9
Fix loading for out-of-order lsns
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
b9b2f8cec1
Add include for older gcc
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
4581084540
Fix vitastor-disk build (with old metadata and journal formats)
2025-11-23 19:08:24 +03:00
Vitaliy Filippov
4dfb80e6a1
Collapse intent_writes on other write types too
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
1aff129f99
Fix buffer overflow in test_heap
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
2282ed0345
Add a test for parallel reads with block checksums
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
4b95fe4919
Remove cancel_all_writes
...
Not needed because a) parallel writes to the same object are forbidden
b) subsequent writes don't depend on previous ones anyway.
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
ad6354a537
Add a test for "perfect_csum_update" mode
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
346d20ce29
Add 2 tests for intent writes
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
eb9633f21f
Add a test with fsync
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
f86bed106c
Implement buffered disk_mock_t mode, extract ringloop_mock.cpp
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
cc2975efe6
unaligned_intent does not need special handling anymore
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
27ef64cef4
Add an option for global coverage
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
7f7d79fdc9
Fix some blockstore bugs discovered by the mocked test!
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
7b94969869
Add a basic mocked blockstore test
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
92a0942cd6
Add mocks for blockstore integration tests: timerfd, ring_loop_mock_t and disk_mock_t
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
c5c332cb9a
Test calc_checksums
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
cedbedc1e5
Check 2/3 blocks in test_recheck
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
b28c67ba5f
Fix intent writes with padded checksums
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
26fb08cc02
Fix reads from intent writes
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
186f37e286
Disable punching block checksums and allow to enable it with a parameter
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
8c3b83ff74
Process big_writes as intents to avoid fsyncs
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
5fe474573e
Fix collapsing intent-over-intent checksums
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
89644692ac
Allow 1 intent_write over big_write in fsync mode
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
8581988c2c
Block lists by previous writes
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
fb83ebbfff
Allow multiple writes with the same version
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
abf741048a
Add tests for intent writes
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
cb056b4413
Call finish_load after async recheck
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
43c4510a29
More tests for incorrect data cases
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
2ba34bfd75
Do not block writes on previous writes
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
481005b062
Add atomic_write_size parameter
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
6464dae129
Use ui32 for block sizes
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
e0d60dd49f
Remove extra unneeded read_entry-s
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
abeb8b73fa
Do not recheck data location on intent-write
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
15d47a2695
Remove FIXMEs
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
661c45cbb5
Fsync after stabilizing
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
bff3d35c85
Fsync & update metadata when block checksums are enabled
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
18a2858c62
Return new_lsn from erase and rollback
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
999d0e2961
Allow to cancel compaction for unfinished writes
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
df1ab722dd
Fix space allocation & compaction on start
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
40a4952592
Use inflight_lsn iterators
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
298a5ec499
Free space on overwrites correctly
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
dd28f2f963
Use fsynced_lsn in flusher
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
41256ea460
Correctly track fsynced_lsn when fsyncs are enabled
2025-11-23 19:08:23 +03:00
Vitaliy Filippov
229c19dee4
Extract multilist_alloc_t
2025-11-23 19:08:23 +03:00
Vitaliy Filippov