- QEMU virtual disk migration with enabled iothread is finally fixed correctly.
- Fixed operation of Proxmox VMs with swTPM without enabling NBD for all disks.
- Debian packages are now again built with stable Antietcd released instead of
the unstable master branch.
- Antietcd cluster mode previously broken in that master branch is fixed. The
symptom was Antietcd being unable to elect the leader in a cluster.
- Fixed monitor startup with embedded Antietcd when using IPv6.
- Fixed space statistics calculation for FS and S3 pools in the new storage ([PR #127](https://github.com/vitalif/vitastor/pull/127)).
How many (bugs!) I've knifed, how many I've slit!
General note: most bug fixes now include regression tests to verify that they don't repeat in the future.
Most bugs fixed in this release were detected by using LLM analysis (Claude Opus/Fable, GPT 5.5).
OSD:
- Fix OSD hanging with an infinite loop when setting autosync_interval to 0 at runtime
- (IMPORTANT) Fix EC PGs hanging in REPEERING when the last final commit/rollback in a
batch completes with an error
- Limit pg_size by 64 because peering doesn't handle larger values with EC — they just
lead to 'incomplete' objects
- Fix OSD crash with an "assertion failed" error on EIO retry in snapshot chain read
(i.e. when some chunks belong to a corrupted replica with checksum mismatch)
- (IMPORTANT) Disable chunked PG count resharding due to possible interference with
compaction changes (will be re-enabled after fixes)
- (IMPORTANT) Fix incorrect snapshot allocation bitmap recovery during EC chained read
- Add on-wire request size validation to prevent possible OOM/DoS/heap corruption
on receiving invalid data from the network
- (IMPORTANT) Fix parity-less EC writes destroying snapshot allocation bitmaps
(i.e. when all parity OSDs in a PG are missing)
- (IMPORTANT) Fix EC N+K, K>=2 recovery destroying snapshot allocation bitmaps
of live parity chunks
- (IMPORTANT) Fix a possible OSD crash during EC misplaced object scrubbing
- (IMPORTANT) Verify object bitmap consistency during scrub (only data was checked previously)
- (IMPORTANT) Fix corrupted object chunks incorrectly marked as non-corrupted on the second scrub
- (IMPORTANT) Fix cached EC decoding of multiple stripes with ISA-L (ISA-L is the default)
New store:
- Fix a theoretically possible OSD crash on startup when using the previously added
workaround for the "double-claim" problem
- Remove theoretically possible incorrect metadata block writes during batch EC COMMITs
restarted due to a full metadata area
- Fix incorrect compaction counter tracking after OSD restart (could probably lead to
compaction not restarted correctly after a restart)
- (IMPORTANT) Fix some of parallel big_writes possibly not waiting for data fsync, thus not providing durability
- Fix possible OSD crash on sync retry when io_uring is full
- Fix a possible crash during startup on corrupted on-disk data with too small entry sizes
Old store:
- Prevent loading extra garbage metadata entries from the last 4 MB of metadata area
- Fix read operations possibly crashing if a metadata read (with inmemory_metadata=false)
was restarted due to a full io_uring
- Fix a possible memory leak of temporary buffers and bitmaps/checksums when a read
was restarted due to a full io_uring (reproducible with either inmemory_metadata=false or block_size>256k)
- Fix a possible OSD crash during padded checksum reads if buffer count exceeded 1024
(IOV_MAX) (reproducible only with csum_block_size > 4k and block_size >= 4M)
- (IMPORTANT) Fix partial padded read journal checksum verification with csum_block_size > 4k
- Fix incorrect marking of corrupted objects as non-corrupted after flushing data
from journal (with inmemory_journal=false)
- (IMPORTANT) Fix deferred freeing of a different block when a block was used by a parallel read
- Fix per-inode statistics not being disabled for FS and S3 pools correctly, leading to etcd
overload with unneeded per-inode statistics, slower etcd operation, increased memory usage,
and too many Prometheus statistics exported by the monitor
Both stores:
- Fix possibly left garbage in the metadata area if the first OSD startup was interrupted —
metadata header is now written only after initializating metadata
- Check for short reads during initialization (just in case, doesn't happen in real life)
Clients:
- Fix write-back queue item split in case when write-back is enabled at runtime
- Implement bdrv_detach_aio_context & bdrv_attach_aio_context in the QEMU driver (should fix migration with iothread)
- Do not crash on full io_uring in ublk server
- Fix missing --readonly option handling in NBD server
- Stop gracefully on NBD_CMD_DISC instead of just exit(0) in NBD server
- Fix writeback detection in ublk server for --image mode
- Limit the amount of incoming data for NFS clients to prevent choking on memory in async mount mode
Tools (vitastor-disk/vitastor-cli):
- Prevent vitastor-cli merge possibly exiting before completing the last sync/delete operations
- Fix vitastor-disk incorrectly validating too large small_write entry length
- Fix vitastor-cli merge ignoring input option validation errors
- Fix vitastor-cli rm-data always skipping the final fsync
- Fix vitastor-disk resize not moving the last used data block
- Fix vitastor-disk write-meta incorrectly importing new store small_write entries
- Fix vitastor-disk write-journal and write-meta importing old store data incorrectly
when csum_block_size is > 4k
- Support --io option for vitastor-disk dump-journal/write-journal
- Fix vitastor-disk resize crash when converting from very old (0.5.x) metadata
- Fix vitastor-disk trim incorrectly rounding block ranges with --discard_granularity
option explicitly set to a value > 4k, possibly leading to discarding live data
- Fix vitastor-disk write-meta importing new store metadata incorrectly with > 4 GB metadata area size
- (IMPORTANT) Fix vitastor-cli modify --resize to a smaller size clearing all image data O_o
Other:
- Do not crash with an uncaught exception when an invalid /osd/state/ with a non-numeric
suffix is present in etcd (in OSD and all client services)
- Fix possible crash in vitastor-kv when handling a corrupted DB due to a uint32 overflow
- Fix NFS-RDMA memory allocator crashing in some situations
- Fix small shared file extend-write potentially reading unallocated memory (NFS)
- Add bounds checks to prevent uint32 overflows in NFS/XDR
- Re-enable accidentally disabled safety checks (asserts) in files with included cpp-btree
- Fix too small memory allocation in NFS portmap
- New store fixes:
- Fix repeated rollback logic
- Fix crash on rolled back object compaction
- Fix postpone_load possibly merging different object chains
- Fix block_csums import in vitastor-disk write-meta
- Fix header checksum after vitastor-disk write-meta
- Old store fixes:
- Fix batched fsync possibly skipped by some flush coroutines
- Improve ENOSPC test, fix possible crash on ENOSPC
- Add fsyncs to vitastor-disk prepare
- Fix possible crash on pg_lock check failure in sec_read_bmp
- Fix VitastorFS initialization when local_reads are enabled
Important fixes (except the new store):
- Fixed a possible use-after-free in the OSD during error handling of initial
commit/rollback of objects in EC pools.
- Fixed a possible free of an invalid pointer in the OSD during read errors
from snapshot/clone chains in EC pools.
- Fixed possibly incorrect handling of commit/rollback operations in EC pools
during pool PG count changes.
- Fixed the inverted fsync enable parameter in the ublk driver (fsync was not
enabled on pools without immediate_commit).
- Added the raw-ls command for debugging purposes to find object versions in
the cluster using listing operations.
New store fixes:
- Improved startup speed by using LSN-based sorting only for objects with a
large number of intermediate versions.
- Added skip_double_claim option as a temporary workaround to fix the rare OSD
startup error with the "double claimed block" message, observed by several
users. This option does not affect data integrity.
- Fixed incorrect rechecking of small writes during startup, which in theory
could lead to duplicate small write object entries on the OSD.
- Fixed fsync operation for disks with a writeback cache (without capacitors):
- Fixed incorrect semantics of consecutive fsyncs (next fsync was not blocked
by the previous one).
- Added fsync when copying small writes from the buffer to the data device
(somehow forgotten during initial development).
- Added fsync after the initial garbage collection during OSD startup.
- Fixed incorrect cast of LSN from uint64 to uint32, breaking fsync when
reaching LSN 2^32.
- Added missing verification of the metadata header checksum during startup.
- Fixed incorrect updating of object checksums in perfect_csum_update=true mode.
- Fixed a possible OSD crash with "assertion failed" when processing a malformed
EC STABILIZE operation.
- Fixed the accounting of active compactor coroutines.
- Removed broken and untested new->old store conversion support.
Minor issues fixed:
- Incorrect accounting of OSD local operation statistics in replicated pools.
- Missing non-zero exitcodes on vitastor-disk resize command errors.
- Missing reset of the list of inconsistent objects during PG restarts.
- Theoretically possible hangs of various OSD operations when working with
completely corrupted objects (without a single available copy), and possibly
in some other very rare situations.
- Incorrect fsyncs when deleting objects from pools without immediate_commit
(on disks with a writeback cache), which previously could leave garbage when
deleting misplaced objects.
- Possible crash/memory corruption of the NFS server during a targeted attack
on NFS-RDMA.
- Possibly incorrect handling of ENOSPC/EIO write errors in replicated pools,
leading to inability to retry the write later.
- Possible crash instead of a clean error exit when starting an OSD with the
old storage engine on a disk with corrupted journal data.
- Shallow copying of PG configuration in the monitor, however, not related to
actual bugs.
- Incorrect checking of allocated blocks in the QEMU driver in an unused code
branch (without the BDRV_WANT_ZERO flag).
- Possible memory leak on read errors of corrupted objects.
- Possible incorrect PG states when corrupted objects are detected.
- Possible failure to mark all "bad" copies of an object during scrubs without
checksums and with a large number of replicas (> 4).
- Incorrect checksum calculation in the old storage engine when
bitmap_granularity < 4096 (a practically unused configuration).
- Theoretically possible OSD crash in rare cases during a scrub and simultaneous
object recovery.
- Theoretically possible OSD crash when handling PING operation errors.
- Slightly suboptimal logic for reusing the RDMA send buffer.
- Possible memory leak when canceling an already running scrub via no_scrub.
- Possible memory corruption when a client (e.g., QEMU code) passes invalid
buffers and the writeback cache is enabled.
- Potentially incorrect search for corrupted parts of EC objects (inability
to find a "good" combination) during a scrub with checksums disabled.
- Possible additional memory usage on the OSD side when handling failed reads
from snapshots (not a leak however - the memory was freed upon client
disconnection).
- Potential sudden write slowdown at certain pg epoch values due to incorrect
epoch update logic in etcd.
Important bug fixes (new store):
- Fix OSDs possibly refusing to start with "write metadata failed at offset xxx: Invalid argument"
(fix buffer alignment during initial garbage collection)
- Rollback change from 3.0.4 - on-disk garbage entries are not skipped on start again. This change
doesn't have any impact normally, but OSDs originally running 3.0.0-3.0.2 and then upgraded
to 3.0.9 may hit a bug where 3.0.9 refuses to start due to entries marked as garbage too
early and flushed to disk in 3.0.0-3.0.2.
Other changes:
- Auto-select the only RDMA device/port if there is only one
- Rollback one 3.0.9 change - there was no actual use-after-free :)
(the problem was only relevant to an unstable development version)
- Fix `vitastor-nfs --trace` option
- Fix an unintended 1 second sleep in vitastor-cli rm-data
- Fix inode statistics not being cleared for a deleted pool
- Fix print to stdout in client
- Fix broken garbage collection in the new store leading to very high memory usage
- Make the new store do full garbage collection on OSD startup by default (can be turned off with a new gc_on_start parameter)
- Make startup logging with the new store more informative, allow to log amount of garbage with log_level > 1
- Fix clients reading from snapshots incorrectly on retries (for example when OSDs were restarted during read)
- Fix a possible "send-after-free" in OSD which could probably also affect snapshot reads
- Fix OSD crashing when handling a read_bitmap operation from a deleted object with the new store
- Surrogate increment-only peer IDs are now used instead of file descriptor numbers,
and client cleanup sequence is reworked in another attempt to fix a possible PG hang
in 'peering' states. The history of this bug is still unclear and it still hasn't been
reproduced in tests, but several users have reported it since 3.0.4 (only with enabled RDMA).
- Fix a minor bug where an OSD could skip reporting misplaced PG state when moving
an object in non-immediate_commit mode.
- Add Ubuntu 26.04 (Resolute Raccoon) support and patches for qemu 10.2 and libvirt 12.0
- Fix one more bug in old store checksums O_o - checksum verification wasn't
working correctly on reads quickly after small initial offsetted writes
(when a BIG_WRITE was read from the journal).
- Fix clients possibly not destroyed after being switched to RDMA. Not 100%
verified, but in theory it could rarely lead to PGs hanging in peering/stopping
states with RDMA enabled.
- Use vitastor-cli in the Cinder driver for modifications instead of direct etcd
interaction.
- Fix mon_https_ca parsing.
- Fix Docker install.sh, add instructions for Podman.
A single urgent fix for the new store: metadata was written incorrectly when
filled metadata area exceeded 4 GB due to a simple & stupid 32-bit integer overflow.
This was leading to OSDs corrupting metadata and being unable to start on the next
run.
The bug affected only the new store (meta_format=3) and OSDs with metadata area size
larger than 4 GB. You can check your OSD metadata area size by checking `data_offset`
in `vitastor-disk read-sb /dev/vitastor/osdXX-data` for single-disk OSDs or
`blockdev --getsize64 /dev/vitastor/osdXX-meta` for hybrid OSDs.
If you're affected, it's highly recommended to:
- Install updated packages without restarting OSDs
- Restart OSDs **one by one**
- If a restart of an OSD triggers "has_degraded" rebalance - wait for it to finish
before restarting another OSD
- If an OSD fails to start with "double-claimed" errors in the log - purge and recreate it,
and again wait for "has_degraded" rebalance to finish before proceeding to the next OSD
New store bug fixes:
- Fix very slow OSD startup (hours) possible in unlucky cases
- Fix incorrect garbage entry validation on start in some cases, preventing OSD startup
- Fix small_write_throttling not functioning and either hanging or crashing OSDs
- Fix "different fact_versions returned from subops" error possible with EC in some cases
- Add a recovery option for the new store: skip_corrupted_meta_entries
- Disable intent writes (even 4k) for hybrid OSDs by default for better write buffering
Other changes:
- Fix chmod/chown of a new empty VitastorFS root directory not working
- Fix vitastor-dd treating all < 32k reads as always empty
- Fix vitastor-disk always setting 32k csum_block_size for HDDs when checksums are enabled
- Fix OSD missing PG state updates and PGs hanging in "starting" in rare cases
- Fix a possible OSD crash with "map::at" message in rare cases when stopping a client
- Fix a possible OSD crash on start when RDMA is available but fails to initialize
- Fix removal and overwriting of totaled objects (objects with all corrupted copies)
- Do not allow data_block_size < 8*bitmap_granularity
- Remove RDMA ODP (On-Demand Paging) support
- Update antietcd to 1.2.4
- Update Docker build to Debian Trixie
- Add packages for AlmaLinux 10
Vitaliy Filippov