thebestvibedev/tromcho.net
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,421 Commits 74 Branches 106 Tags
debb00a53549dfd97031a3f154d7f12b8167b613
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Vitaliy Filippov debb00a535 Check for "Checksum mismatch" and "BUG" messages during test_heal 2023-07-13 01:49:45 +03:00
Vitaliy Filippov c8891ab1d6 Use clean_dyn_size for space check 2023-07-13 01:49:45 +03:00
Vitaliy Filippov 7062b73d87 Log more details about checksum mismatch in big_writes 2023-07-13 01:49:45 +03:00
Vitaliy Filippov 02e24f5144 Use find_holes() in flusher for unification 2023-07-13 01:49:45 +03:00
Vitaliy Filippov 55f506f6e0 Fill journal header to know checksum type & size when dumping journal with --all 2023-07-13 01:49:45 +03:00
Vitaliy Filippov e12dd9b82c Fix journal read checksum verification with inmemory_journal=false 2023-07-13 01:49:45 +03:00
Vitaliy Filippov 82d8848b8f Call fill_partial_checksum_blocks() correctly in regard to COPY_BUF_CSUM_FILL 2023-07-13 01:49:45 +03:00
Vitaliy Filippov ef800408dc Wait for journal reads before checking them in clear_incomplete_csum_block_bits 2023-07-13 01:49:45 +03:00
Vitaliy Filippov 3b1150c478 Check for checksum mismatch absence in test_heal 2023-07-13 01:49:45 +03:00
Vitaliy Filippov ccdf87dc81 Use zero checksum size for zero-length writes 2023-07-13 01:49:45 +03:00
Vitaliy Filippov bede73d158 Fix journal data checksum mangling on corrupted block overwrite 2023-07-13 01:49:45 +03:00
Vitaliy Filippov 8e35319a34 Check journal entry size when checking block checksums 2023-07-13 01:49:45 +03:00
Vitaliy Filippov dc6e88e2ca Fix journal data checksum verification on start 2023-07-13 01:49:45 +03:00
Vitaliy Filippov 4c3370220b Add more details to "journal entry data is corrupt" messages 2023-07-13 01:49:45 +03:00
Vitaliy Filippov f02344c0a4 ...and partially remove the perversion with bitmap inlining 2023-07-13 01:49:45 +03:00
Vitaliy Filippov b369032665 Sadly we have to refcount dyn_data... 2023-07-13 01:49:45 +03:00
Vitaliy Filippov 319b0833eb Fix clean block checksum read 2023-07-13 01:49:45 +03:00
Vitaliy Filippov 0641b06fb1 Allow to forcibly set meta_format 2023-07-13 01:49:45 +03:00
Vitaliy Filippov 3b9873b9a9 Rename meta_version to meta_format 2023-07-13 01:49:45 +03:00
Vitaliy Filippov eef97a0dc4 Support old metadata format in vitastor-disk dump-meta 2023-07-13 01:49:45 +03:00
Vitaliy Filippov 5b16e5ab5b Fix journal big_write simple reads after checksum changes 2023-07-13 01:49:45 +03:00
Vitaliy Filippov bb430fccd5 Verify checksums in test_heal in different combinations 2023-07-13 01:49:45 +03:00
Vitaliy Filippov bafadd5559 Fix bitmap-granular checksums 2023-07-13 01:49:45 +03:00
Vitaliy Filippov 6ebca5fedc Fix wait_journal_count not being zeroed 2023-07-13 01:49:45 +03:00
Vitaliy Filippov e3e2325ef5 Rewrite and fix find_holes into a more obvious version 2023-07-13 01:49:45 +03:00
Vitaliy Filippov 9a908f3e66 Fix missing checksum read offset 2023-07-13 01:49:45 +03:00
Vitaliy Filippov 89f6fef920 Add a test for checksums 2023-07-13 01:49:45 +03:00
Vitaliy Filippov 63b9382067 Fix checksum verification in big_write journal reads 2023-07-13 01:49:45 +03:00
Vitaliy Filippov 8a7dea9fa2 Verify checksums during journal reads 2023-07-13 01:49:45 +03:00
Vitaliy Filippov 1adf77f8fb Add backwards compatibility with non-checksum metadata and journal formats 2023-07-13 01:49:45 +03:00
Vitaliy Filippov 7a530346a6 Implement large csum_block_size support (more than 4k) + refactor blockstore_flush 2023-07-13 01:49:45 +03:00
Vitaliy Filippov cc1f03971d Implement bitmap-granular (4k) metadata & data checksums 2023-07-13 01:49:45 +03:00
Vitaliy Filippov a0aac7eb2a Update drives 2023-07-13 01:49:45 +03:00
Vitaliy Filippov ac7b834af3 Disable journal_no_same_sector_overwrites by default for HDD-only 2023-07-10 00:34:35 +03:00
Vitaliy Filippov ee0c78fd74 Fix default HDD block size in docs (actual size is 1 MB) 2023-07-09 13:12:30 +03:00
Vitaliy Filippov e6646a5b2f Bump QEMU version to vitastor3 2023-07-09 13:01:04 +03:00
Vitaliy Filippov ae69662b17 Add "Recommended drives" 2023-07-09 12:59:18 +03:00
Vitaliy Filippov 57ad4c3636 Add a note about HDD, enable throttling only for hybrid OSDs 2023-07-09 12:45:11 +03:00
Vitaliy Filippov b7e4d0c9bf Fix journal dirty_start position tracking and some debug prints 
Fixes two bugs found during HDD testing :-)
1) OSD crashed with "BUG: Attempt to overwrite used offset of the journal" during
   `fio -bs=900k -iodepth=128` test with 16 MB journal
2) OSD stalled during `fio -bs=512k -iodepth=128` test with 64 MB journal
 2023-07-09 01:17:55 +03:00
Vitaliy Filippov 161a23c966 Support reloading state when etcd says "revisions were compacted" 
Before this change, OSDs almost always died when one of the etcds was restarted,
even though the rest of them was still in quorum and the lease was still active
 2023-07-07 01:33:48 +03:00
Vitaliy Filippov 2f999d8607 Reduce etcd memory usage 
With default --snapshot-count 100000 and GOGC=100 it easily reaches 6.6 GB
even when we only store 1-2 MB of data in it
 2023-07-06 00:46:26 +03:00
Vitaliy Filippov d007a374f2 Delete extra /pool/stats/ keys for non-existing pools 2023-07-06 00:40:13 +03:00
Vitaliy Filippov 45c0694853 Clear etcd_local addresses on reload and also skip duplicates 2023-07-06 00:39:39 +03:00
Vitaliy Filippov 57bcba2406 Add notes about VDUSE 2023-07-04 16:51:46 +03:00
Vitaliy Filippov 30ac899074 Make QEMU driver compatible with older vitastor_client and with systems without io_uring 2023-07-04 15:51:43 +03:00
Vitaliy Filippov 2348d39cf4 Avoid repeated qemu_uring_handlers, add 2.0-2.7 compatibility 2023-07-04 00:28:23 +03:00
Vitaliy Filippov 3de7929fe5 Integrate v2 - direct epoll 2023-07-04 00:28:23 +03:00
Vitaliy Filippov 07b2196bc2 Integrate QEMU driver with io_uring 2023-07-04 00:28:23 +03:00
Vitaliy Filippov b8e30608d6 Bump QEMU version to vitastor2 2023-07-01 00:55:32 +03:00
Vitaliy Filippov a612cdca47 Release 0.9.3 
- Add patch for libvirt 9.0
- Add support for Proxmox VE 8.0
- Fix compatibility of the QEMU driver with iothread (QEMU rebuilds are coming)
- Fix vitastor-cli rm-data/rm/merge hanging when some OSDs are down.
  Allow deletions in unclean cluster at the cost of some data possibly
  "reappearing" when those OSDs start back. In that case you can just repeat
  the deletion request using rm-data.
- A bunch of bug fixes for snapshots:
  - Fix snapshot reads often not working at all with snapshot chain size > 2
  - Fix optimized snapshot data merge (children to parent)
  - Fix updating of image name index key during optimized merge
  - Fix auto-selection preventing the use of optimized merge with only 1 snapshot
  - Fix incorrect CAS retries during snapshot merge
  - Fix snapshot merge progress reporting
- Fix primary_read bitmap buffers use-after-free which could lead to
  incorrect allocation map reads
- Remove /usr/local/bin path from make-etcd
- Some documentation fixes
 2023-07-01 00:25:58 +03:00
Vitaliy Filippov c8d61568b5 Fix primary_read bitmap buffers being freed too early (use-after-free) 2023-06-30 12:47:45 +03:00
Vitaliy Filippov 84ed3c6395 Fix CAS retries during snapshot merge 2023-06-30 02:30:23 +03:00
Vitaliy Filippov a7b57386c0 Do not print last subcommand result twice during "inverse" snapshot merge 2023-06-30 02:07:10 +03:00
Vitaliy Filippov 9d4ea5f764 Fix inverse parent selection which prevented the use of optimized merge in case of only 1 snapshot 2023-06-30 01:39:11 +03:00
Vitaliy Filippov 000e4944ec Remove "inverse parent" image name index key from etcd during snapshot merge 2023-06-30 01:23:30 +03:00
Vitaliy Filippov 8426616d89 Warn about unfinished deletions in rm-data 2023-06-30 01:18:25 +03:00
Vitaliy Filippov 1a841344ec Print progress of all operations during snapshot merge 2023-06-30 01:13:47 +03:00
Vitaliy Filippov 8603b5cb1d Do not hang on inactive OSDs during delete, report and skip them instead 2023-06-30 00:15:16 +03:00
Vitaliy Filippov f12b8e45a9 Remove /usr/local/bin path from make-etcd 2023-06-29 23:49:31 +03:00
Vitaliy Filippov 878ccbb6ea Fix snapshot chain "down-merge" ("up-merge" worked well...) 2023-06-29 00:47:21 +03:00
Vitaliy Filippov b14220b4d0 Add a test for snapshot chain 2023-06-29 00:47:21 +03:00
Vitaliy Filippov 181d6ba407 Move npm install to CI build and remove it from tests 2023-06-28 23:41:46 +03:00
Vitaliy Filippov 63c2b9832c Fix chained (snapshot) reads often not working at all with chain size > 2 2023-06-28 18:54:03 +03:00
Vitaliy Filippov 10e2e6a7c8 Add a patch for pve-qemu 8.0 2023-06-24 01:33:52 +03:00
Vitaliy Filippov a598428992 Add a note about PVE 8.0 2023-06-24 01:29:28 +03:00
Vitaliy Filippov 08a677b684 Notes about rebuilding pve-qemu 2023-06-24 01:00:21 +03:00
Vitaliy Filippov 7c8fbdad16 Fix typo 2023-06-22 02:07:33 +03:00
Vitaliy Filippov 2f9353df60 Markdown preprocessor to compile multiple .md files into one 2023-06-22 01:27:38 +03:00
Vitaliy Filippov 57c744f288 Fix some cross-references in docs 2023-06-22 01:16:03 +03:00
Vitaliy Filippov a11ca56fb1 Fix compatibility of the QEMU driver with iothread 2023-06-21 02:11:28 +03:00
Vitaliy Filippov b84927b340 Fix \n in nbd_proxy 2023-06-19 01:48:58 +03:00
Vitaliy Filippov 83cacba226 Fix patched-qemu build 2023-06-19 01:47:55 +03:00
Vitaliy Filippov 2c8f0bc6d5 Add a note about Debian 12 2023-06-19 01:08:41 +03:00
Vitaliy Filippov 7ae5b0e368 Add patch for libvirt 9.0 2023-06-19 01:07:08 +03:00
Vitaliy Filippov 926be372fd Release 0.9.2 
- Measure and report scrub I/O statistics in vitastor-cli status
- Make aggregated statistics in vitastor-cli status much smoother
  (first derive, then sum instead of first summing and then deriving)
- Fix an old rare bug leading to journal corruption
  (try to use scrub if you think you're affected...)
- Do not start EC PGs without at least <data chunks> OSDs in each old set
  (prevents spurious read errors with EC during reconnections/restarts)
- Fix failed assert(!scrub_list_op) on OSD restart with pending scrubs
- Fix future planned scrubs not starting because of incorrect time comparison
- Build packages for Debian 12 (Bookworm)
 2023-06-18 19:44:33 +03:00
Vitaliy Filippov 6222779b52 Support debian bookworm (12) build 2023-06-18 19:44:33 +03:00
Vitaliy Filippov a4186e20aa First derive, then sum per-OSD statistics instead of first summing and then deriving 
This makes statistics reported by vitastor-cli status much smoother
 2023-06-18 01:32:24 +03:00
Vitaliy Filippov c74a424930 Report scrub I/O in vitastor-cli status 2023-06-17 21:11:21 +03:00
Vitaliy Filippov 32f2c4dd27 Measure scrub statistics 2023-06-17 20:56:26 +03:00
Vitaliy Filippov 3ad16b9a1a Fix auto_scrubs not starting because of < vs <= =)) 2023-06-17 17:32:21 +03:00
Vitaliy Filippov 1c2df841c2 Fix failed assert(!scrub_list_op) on OSD restart with pending scrubs 2023-06-17 17:02:54 +03:00
Vitaliy Filippov aa5dacc7a9 Do not start EC PGs without at least pg_data_size connections to old OSDs from each set 2023-06-17 02:16:30 +03:00
Vitaliy Filippov affe8fc270 Raise timeout also for add_osd and rebalance_verify 2023-06-17 00:29:03 +03:00
Vitaliy Filippov 4fdc49bdc7 Add another assert-type check (it does not fire, just as a safety measure for the future) 2023-06-17 00:07:22 +03:00
Vitaliy Filippov 86b4682975 Put get_trim_pos into the "critical section". Fixes rare journal corruption issue 
The consequence of this issue was that in some very rare cases (only reproduced
under load in CI when running 4+ tests in parallel) small write data written to
journal could overwrite journal entries.

Also add an assert-type safety check to be able to catch this issue in the
future again in case of a regression.
 2023-06-17 00:06:42 +03:00
Vitaliy Filippov bdd48e4cf1 Release 0.9.1 
- Fix "Client XX command out of sync" messages sometimes happening on OSD reconnections
- Fix a bug where EC reads parallel with writes to the same object failed with -ERANGE error
- Slightly reduce the amount of metadata writes during journal flushing
- Correctly unmap NBD volumes when Proxmox forces map_volume use (with SWTPM and maybe some other cases)
 2023-06-10 11:42:49 +03:00
Vitaliy Filippov af8c3411cd Correctly unmap NBD when Proxmox forces map_volume use (with SWTPM and maybe something else) 2023-06-08 01:31:49 +03:00
Vitaliy Filippov 9c405009f3 Use randrw in test_heal 2023-06-03 00:49:53 +03:00
Vitaliy Filippov f9fbea25a4 Remove double write when old and new locations are in the same metadata block 
Also add another metadata entry fool-safety check which, ideally, will never fire %)
 2023-06-03 00:47:10 +03:00
Vitaliy Filippov 2c9a10d081 Fix an idiotic bug leading to failed reads with -ERANGE with EC :D 2023-06-03 00:44:52 +03:00
Vitaliy Filippov 150968070f Slightly improve some debug prints 2023-05-29 01:04:16 +03:00
Vitaliy Filippov cdfc74665b Close client FDs only when destroying the client, after handling all async reads/writes 
Fixes "Client XX command out of sync" sometimes happening on reconnections
 2023-05-25 00:52:43 +03:00
Vitaliy Filippov 3f60fecd7c Fix typo 2023-05-21 18:37:01 +03:00
Vitaliy Filippov 3b4cf29e65 Release 0.9.0 
New features:
- Scrubbing! Check documentation: [auto_scrub](src/branch/master/docs/config/osd.en.md#auto_scrub)
- Document online-updatable configuration parameters

Bug fixes:
- Fix NaN during PG optimisation if there are nonexisting OSDs in node_placement
- Fix monitor crash on pool deletion
- Clear journal_device and meta_device before initialising the next OSD in automatic mode
- Sync unsynced deletes before overwriting them with a lower version
  (reproducted mostly/only after scrubbing)
 2023-05-21 15:07:14 +03:00
Vitaliy Filippov eeaba11ebd Use fio 3.27-8 for alma9 2023-05-21 14:48:26 +03:00
Vitaliy Filippov aea567cfbd Slightly improve scrub docs 2023-05-21 12:52:30 +03:00
Vitaliy Filippov ce02f47de6 Allow to disable scrub_find_best 2023-05-21 12:33:38 +03:00
Vitaliy Filippov 5fd3208616 Add version archive link to docs 2023-05-21 11:47:33 +03:00
Vitaliy Filippov 5997b76535 Remove -runtime=10 from fio params in test_scrub, it was breaking the test in CI :D 2023-05-21 11:03:59 +03:00
Vitaliy Filippov f1961157f0 Fix brute-force error locator for EC n+k with k > 2 2023-05-21 00:57:14 +03:00