- QEMU virtual disk migration with enabled iothread is finally fixed correctly.
- Fixed operation of Proxmox VMs with swTPM without enabling NBD for all disks.
- Debian packages are now again built with stable Antietcd released instead of
the unstable master branch.
- Antietcd cluster mode previously broken in that master branch is fixed. The
symptom was Antietcd being unable to elect the leader in a cluster.
- Fixed monitor startup with embedded Antietcd when using IPv6.
- Fixed space statistics calculation for FS and S3 pools in the new storage ([PR #127](https://github.com/vitalif/vitastor/pull/127)).
How many (bugs!) I've knifed, how many I've slit!
General note: most bug fixes now include regression tests to verify that they don't repeat in the future.
Most bugs fixed in this release were detected by using LLM analysis (Claude Opus/Fable, GPT 5.5).
OSD:
- Fix OSD hanging with an infinite loop when setting autosync_interval to 0 at runtime
- (IMPORTANT) Fix EC PGs hanging in REPEERING when the last final commit/rollback in a
batch completes with an error
- Limit pg_size by 64 because peering doesn't handle larger values with EC — they just
lead to 'incomplete' objects
- Fix OSD crash with an "assertion failed" error on EIO retry in snapshot chain read
(i.e. when some chunks belong to a corrupted replica with checksum mismatch)
- (IMPORTANT) Disable chunked PG count resharding due to possible interference with
compaction changes (will be re-enabled after fixes)
- (IMPORTANT) Fix incorrect snapshot allocation bitmap recovery during EC chained read
- Add on-wire request size validation to prevent possible OOM/DoS/heap corruption
on receiving invalid data from the network
- (IMPORTANT) Fix parity-less EC writes destroying snapshot allocation bitmaps
(i.e. when all parity OSDs in a PG are missing)
- (IMPORTANT) Fix EC N+K, K>=2 recovery destroying snapshot allocation bitmaps
of live parity chunks
- (IMPORTANT) Fix a possible OSD crash during EC misplaced object scrubbing
- (IMPORTANT) Verify object bitmap consistency during scrub (only data was checked previously)
- (IMPORTANT) Fix corrupted object chunks incorrectly marked as non-corrupted on the second scrub
- (IMPORTANT) Fix cached EC decoding of multiple stripes with ISA-L (ISA-L is the default)
New store:
- Fix a theoretically possible OSD crash on startup when using the previously added
workaround for the "double-claim" problem
- Remove theoretically possible incorrect metadata block writes during batch EC COMMITs
restarted due to a full metadata area
- Fix incorrect compaction counter tracking after OSD restart (could probably lead to
compaction not restarted correctly after a restart)
- (IMPORTANT) Fix some of parallel big_writes possibly not waiting for data fsync, thus not providing durability
- Fix possible OSD crash on sync retry when io_uring is full
- Fix a possible crash during startup on corrupted on-disk data with too small entry sizes
Old store:
- Prevent loading extra garbage metadata entries from the last 4 MB of metadata area
- Fix read operations possibly crashing if a metadata read (with inmemory_metadata=false)
was restarted due to a full io_uring
- Fix a possible memory leak of temporary buffers and bitmaps/checksums when a read
was restarted due to a full io_uring (reproducible with either inmemory_metadata=false or block_size>256k)
- Fix a possible OSD crash during padded checksum reads if buffer count exceeded 1024
(IOV_MAX) (reproducible only with csum_block_size > 4k and block_size >= 4M)
- (IMPORTANT) Fix partial padded read journal checksum verification with csum_block_size > 4k
- Fix incorrect marking of corrupted objects as non-corrupted after flushing data
from journal (with inmemory_journal=false)
- (IMPORTANT) Fix deferred freeing of a different block when a block was used by a parallel read
- Fix per-inode statistics not being disabled for FS and S3 pools correctly, leading to etcd
overload with unneeded per-inode statistics, slower etcd operation, increased memory usage,
and too many Prometheus statistics exported by the monitor
Both stores:
- Fix possibly left garbage in the metadata area if the first OSD startup was interrupted —
metadata header is now written only after initializating metadata
- Check for short reads during initialization (just in case, doesn't happen in real life)
Clients:
- Fix write-back queue item split in case when write-back is enabled at runtime
- Implement bdrv_detach_aio_context & bdrv_attach_aio_context in the QEMU driver (should fix migration with iothread)
- Do not crash on full io_uring in ublk server
- Fix missing --readonly option handling in NBD server
- Stop gracefully on NBD_CMD_DISC instead of just exit(0) in NBD server
- Fix writeback detection in ublk server for --image mode
- Limit the amount of incoming data for NFS clients to prevent choking on memory in async mount mode
Tools (vitastor-disk/vitastor-cli):
- Prevent vitastor-cli merge possibly exiting before completing the last sync/delete operations
- Fix vitastor-disk incorrectly validating too large small_write entry length
- Fix vitastor-cli merge ignoring input option validation errors
- Fix vitastor-cli rm-data always skipping the final fsync
- Fix vitastor-disk resize not moving the last used data block
- Fix vitastor-disk write-meta incorrectly importing new store small_write entries
- Fix vitastor-disk write-journal and write-meta importing old store data incorrectly
when csum_block_size is > 4k
- Support --io option for vitastor-disk dump-journal/write-journal
- Fix vitastor-disk resize crash when converting from very old (0.5.x) metadata
- Fix vitastor-disk trim incorrectly rounding block ranges with --discard_granularity
option explicitly set to a value > 4k, possibly leading to discarding live data
- Fix vitastor-disk write-meta importing new store metadata incorrectly with > 4 GB metadata area size
- (IMPORTANT) Fix vitastor-cli modify --resize to a smaller size clearing all image data O_o
Other:
- Do not crash with an uncaught exception when an invalid /osd/state/ with a non-numeric
suffix is present in etcd (in OSD and all client services)
- Fix possible crash in vitastor-kv when handling a corrupted DB due to a uint32 overflow
- Fix NFS-RDMA memory allocator crashing in some situations
- Fix small shared file extend-write potentially reading unallocated memory (NFS)
- Add bounds checks to prevent uint32 overflows in NFS/XDR
- Re-enable accidentally disabled safety checks (asserts) in files with included cpp-btree
- Fix too small memory allocation in NFS portmap
- New store fixes:
- Fix repeated rollback logic
- Fix crash on rolled back object compaction
- Fix postpone_load possibly merging different object chains
- Fix block_csums import in vitastor-disk write-meta
- Fix header checksum after vitastor-disk write-meta
- Old store fixes:
- Fix batched fsync possibly skipped by some flush coroutines
- Improve ENOSPC test, fix possible crash on ENOSPC
- Add fsyncs to vitastor-disk prepare
- Fix possible crash on pg_lock check failure in sec_read_bmp
- Fix VitastorFS initialization when local_reads are enabled
Important fixes (except the new store):
- Fixed a possible use-after-free in the OSD during error handling of initial
commit/rollback of objects in EC pools.
- Fixed a possible free of an invalid pointer in the OSD during read errors
from snapshot/clone chains in EC pools.
- Fixed possibly incorrect handling of commit/rollback operations in EC pools
during pool PG count changes.
- Fixed the inverted fsync enable parameter in the ublk driver (fsync was not
enabled on pools without immediate_commit).
- Added the raw-ls command for debugging purposes to find object versions in
the cluster using listing operations.
New store fixes:
- Improved startup speed by using LSN-based sorting only for objects with a
large number of intermediate versions.
- Added skip_double_claim option as a temporary workaround to fix the rare OSD
startup error with the "double claimed block" message, observed by several
users. This option does not affect data integrity.
- Fixed incorrect rechecking of small writes during startup, which in theory
could lead to duplicate small write object entries on the OSD.
- Fixed fsync operation for disks with a writeback cache (without capacitors):
- Fixed incorrect semantics of consecutive fsyncs (next fsync was not blocked
by the previous one).
- Added fsync when copying small writes from the buffer to the data device
(somehow forgotten during initial development).
- Added fsync after the initial garbage collection during OSD startup.
- Fixed incorrect cast of LSN from uint64 to uint32, breaking fsync when
reaching LSN 2^32.
- Added missing verification of the metadata header checksum during startup.
- Fixed incorrect updating of object checksums in perfect_csum_update=true mode.
- Fixed a possible OSD crash with "assertion failed" when processing a malformed
EC STABILIZE operation.
- Fixed the accounting of active compactor coroutines.
- Removed broken and untested new->old store conversion support.
Minor issues fixed:
- Incorrect accounting of OSD local operation statistics in replicated pools.
- Missing non-zero exitcodes on vitastor-disk resize command errors.
- Missing reset of the list of inconsistent objects during PG restarts.
- Theoretically possible hangs of various OSD operations when working with
completely corrupted objects (without a single available copy), and possibly
in some other very rare situations.
- Incorrect fsyncs when deleting objects from pools without immediate_commit
(on disks with a writeback cache), which previously could leave garbage when
deleting misplaced objects.
- Possible crash/memory corruption of the NFS server during a targeted attack
on NFS-RDMA.
- Possibly incorrect handling of ENOSPC/EIO write errors in replicated pools,
leading to inability to retry the write later.
- Possible crash instead of a clean error exit when starting an OSD with the
old storage engine on a disk with corrupted journal data.
- Shallow copying of PG configuration in the monitor, however, not related to
actual bugs.
- Incorrect checking of allocated blocks in the QEMU driver in an unused code
branch (without the BDRV_WANT_ZERO flag).
- Possible memory leak on read errors of corrupted objects.
- Possible incorrect PG states when corrupted objects are detected.
- Possible failure to mark all "bad" copies of an object during scrubs without
checksums and with a large number of replicas (> 4).
- Incorrect checksum calculation in the old storage engine when
bitmap_granularity < 4096 (a practically unused configuration).
- Theoretically possible OSD crash in rare cases during a scrub and simultaneous
object recovery.
- Theoretically possible OSD crash when handling PING operation errors.
- Slightly suboptimal logic for reusing the RDMA send buffer.
- Possible memory leak when canceling an already running scrub via no_scrub.
- Possible memory corruption when a client (e.g., QEMU code) passes invalid
buffers and the writeback cache is enabled.
- Potentially incorrect search for corrupted parts of EC objects (inability
to find a "good" combination) during a scrub with checksums disabled.
- Possible additional memory usage on the OSD side when handling failed reads
from snapshots (not a leak however - the memory was freed upon client
disconnection).
- Potential sudden write slowdown at certain pg epoch values due to incorrect
epoch update logic in etcd.
Important bug fixes (new store):
- Fix OSDs possibly refusing to start with "write metadata failed at offset xxx: Invalid argument"
(fix buffer alignment during initial garbage collection)
- Rollback change from 3.0.4 - on-disk garbage entries are not skipped on start again. This change
doesn't have any impact normally, but OSDs originally running 3.0.0-3.0.2 and then upgraded
to 3.0.9 may hit a bug where 3.0.9 refuses to start due to entries marked as garbage too
early and flushed to disk in 3.0.0-3.0.2.
Other changes:
- Auto-select the only RDMA device/port if there is only one
- Rollback one 3.0.9 change - there was no actual use-after-free :)
(the problem was only relevant to an unstable development version)
- Fix `vitastor-nfs --trace` option
- Fix an unintended 1 second sleep in vitastor-cli rm-data
- Fix inode statistics not being cleared for a deleted pool
- Fix print to stdout in client
- Fix broken garbage collection in the new store leading to very high memory usage
- Make the new store do full garbage collection on OSD startup by default (can be turned off with a new gc_on_start parameter)
- Make startup logging with the new store more informative, allow to log amount of garbage with log_level > 1
- Fix clients reading from snapshots incorrectly on retries (for example when OSDs were restarted during read)
- Fix a possible "send-after-free" in OSD which could probably also affect snapshot reads
- Fix OSD crashing when handling a read_bitmap operation from a deleted object with the new store
- Surrogate increment-only peer IDs are now used instead of file descriptor numbers,
and client cleanup sequence is reworked in another attempt to fix a possible PG hang
in 'peering' states. The history of this bug is still unclear and it still hasn't been
reproduced in tests, but several users have reported it since 3.0.4 (only with enabled RDMA).
- Fix a minor bug where an OSD could skip reporting misplaced PG state when moving
an object in non-immediate_commit mode.
- Add Ubuntu 26.04 (Resolute Raccoon) support and patches for qemu 10.2 and libvirt 12.0
- Fix one more bug in old store checksums O_o - checksum verification wasn't
working correctly on reads quickly after small initial offsetted writes
(when a BIG_WRITE was read from the journal).
- Fix clients possibly not destroyed after being switched to RDMA. Not 100%
verified, but in theory it could rarely lead to PGs hanging in peering/stopping
states with RDMA enabled.
- Use vitastor-cli in the Cinder driver for modifications instead of direct etcd
interaction.
- Fix mon_https_ca parsing.
- Fix Docker install.sh, add instructions for Podman.
A single urgent fix for the new store: metadata was written incorrectly when
filled metadata area exceeded 4 GB due to a simple & stupid 32-bit integer overflow.
This was leading to OSDs corrupting metadata and being unable to start on the next
run.
The bug affected only the new store (meta_format=3) and OSDs with metadata area size
larger than 4 GB. You can check your OSD metadata area size by checking `data_offset`
in `vitastor-disk read-sb /dev/vitastor/osdXX-data` for single-disk OSDs or
`blockdev --getsize64 /dev/vitastor/osdXX-meta` for hybrid OSDs.
If you're affected, it's highly recommended to:
- Install updated packages without restarting OSDs
- Restart OSDs **one by one**
- If a restart of an OSD triggers "has_degraded" rebalance - wait for it to finish
before restarting another OSD
- If an OSD fails to start with "double-claimed" errors in the log - purge and recreate it,
and again wait for "has_degraded" rebalance to finish before proceeding to the next OSD
New store bug fixes:
- Fix very slow OSD startup (hours) possible in unlucky cases
- Fix incorrect garbage entry validation on start in some cases, preventing OSD startup
- Fix small_write_throttling not functioning and either hanging or crashing OSDs
- Fix "different fact_versions returned from subops" error possible with EC in some cases
- Add a recovery option for the new store: skip_corrupted_meta_entries
- Disable intent writes (even 4k) for hybrid OSDs by default for better write buffering
Other changes:
- Fix chmod/chown of a new empty VitastorFS root directory not working
- Fix vitastor-dd treating all < 32k reads as always empty
- Fix vitastor-disk always setting 32k csum_block_size for HDDs when checksums are enabled
- Fix OSD missing PG state updates and PGs hanging in "starting" in rare cases
- Fix a possible OSD crash with "map::at" message in rare cases when stopping a client
- Fix a possible OSD crash on start when RDMA is available but fails to initialize
- Fix removal and overwriting of totaled objects (objects with all corrupted copies)
- Do not allow data_block_size < 8*bitmap_granularity
- Remove RDMA ODP (On-Demand Paging) support
- Update antietcd to 1.2.4
- Update Docker build to Debian Trixie
- Add packages for AlmaLinux 10
- Fix csum_block_size > 0 unusable with atomic writes in the new store
(almost all atomic write requests generated invalid checksums with csum_block_size > 0)
- Fix monitor sometimes randomly failing to optimize PGs with the "problem is
infeasible or unbounded" message due to not waiting to read full stdout of lp_solve
- Remove reshard_abort optimisation to fix chunked resharding introduced in 3.0.2 possibly
corrupting in-memory OSD state when handling multiple rapid PG count change requests
- Fix removed inodes not disappearing from OSD statistics in the new store, leading to
bloating of the statistics with old inodes
- Increase test coverage for the new store and fix several minor bugs:
- Enabling/disabling of used_for_app was recalculating inode space statistics incorrectly
- Fix object metadata validation on OSD startup rejecting some correct sequences of entries,
possibly leading to OSD being unable to start
- Fix PG activation with EC failing in rare cases with EBUSY when requesting to commit an
already committed write
- Fix a theoretically possible metadata writeback issue on ENOSPC during commit
- Fix monitor failing to optimize PGs in presence of a host with name convertible to
js Number (like 04e278988710) :D
- Fix vitastor-cli dd sometimes (rarely) truncating the image when writing to stdout
- Fix a theoretically possible client connection object leak when io_uring is full
- Fix a leak of RDMA-CM connection objects
- Fix crashes with data_block_size < 32KB (useless setup, but anyway) (#113)
- Antietcd is now officially safe to use: the release includes a fixed version of antietcd
which passes Jepsen transaction serializability tests.
- Fix a huge checksum bug in the old store: incorrect checksums for small initial writes.
The bug affected writes of exactly csum_block_size (4k by default) into new (unallocated)
objects and generated invalid checksums in the store for the written 4k block. Moreover,
generation of these invalid checksums was very slow because it was calculating CRC32
for 4 GB of zeroes. If the block wasn't then overwritten as a part of a larger write
request it became unreadable even though the stored data was correct. The bug affected
all versions since 1.0.0, or since 2.3.0 because vitastor-disk didn't allow to enable
checksums prior to 2.3.0 because of another bug. O:-)
- Prevent OSD disconnections due to long blocking of event-loop caused by PG resharding
(moving objects between old and new PGs in memory) when changing pool PG count or
just on restart of an OSD with a large database (for example, with a filled 8 TB SSD).
The issue should now be fixed because OSD now performs resharding in chunks with pauses
between chunks.
- Prevent pools stuck in paused state on an aborted PG count change.
- Fix a possible OSD crash with "division by zero" when trying to handle an operation
before pool PG count is applied to the in-memory store.
Important fixes:
- Disable RWF_ATOMIC by default because Linux incorrectly requires all atomic
writes to be power-of-2-sized and length-aligned. Details: [use_atomic_flag](https://vitastor.io/en/docs/config/osd.html#use_atomic_flag)
- Fix cross-pool snapshots not working at all - always reading old data after taking the snapshot
- Fix level_placement (broken in 2.2.0)
- Fix CAS write return values in the client library (broken in 2.4.4), also breaking unaligned writes in VitastorFS
- Fix VitastorFS possibly losing some of intersecting parallel unaligned writes
- Prevent possible reads of the old data during unfinished intent writes in the new store
- Tests added for all of above problems to prevent future regressions
Other fixes:
- Allow to specify OSD tags and weights during prepare
- Only clear the first block instead of whole OSD metadata and journal areas during vitastor-disk
prepare - OSD anyway clears them on the first run
- Fix vitastor-cli dd non-seekable detection and error status
- Do not stop OSD on zero-copy tcp short send errors - these aren't a bug, but just caused
by TCP client disconnections
- Fix modify-osd not working after deleting the OSD configuration key from etcd
- Fix regular antietcd websocket disconnections in OSD due to lack of WS_PONG responses to WS_PING
- Reduce the number of allocated RDMA memory regions
- Allow routed RoCE
A single new feature: the new log-structured metadata store implementation, described
in the presentation from Moscow Highload'2025 (check it out [here](https://vitastor.io/en/docs/#talks-and-presentations)).
It's now the default store for new OSDs. The support for the old store is also left in place,
you can still choose it for new OSDs with `vitastor-disk prepare --meta_format 2`.
OSDs from previous versions with the old store format will also continue to operate just like before.
- Implement `vitastor-kv rescue` command which dumps all readable blocks from the
database regardless of the tree structure and allows to recover a corrupted database.
- Simplify primary OSD selection - use just PG number instead of the hash. This allows
to distribute primary OSDs more evenly and reach better read speeds.
- Fix vitastor-kv-stress hang on parallel list & reopen.
- Fix CAS writes without immediate_commit. Fix is sufficient for vitastor-kv to
operate correctly in non-immediate_commit pools.
- Fix NFS ACCESS calls for broken inodes (allow root access to them).
- Fix broken vitastor-kv-stress --runtime parameter.
- Fix bad vitastor-kv dump output buffering leading to incorrect JSON in dumps.
- Fix writing garbage from uninitialized memory to the empty space in VitastorKV DB.
- Refactor some code in preparation to the release of the new log-structured metadata store.
- Replace assert(done != expected) error message, probably caused by kernel issues with
zero-copy TCP send, with a more concise error description.
- Daemonize before forking in NFS proxy to fix OSD RDMA support (#107)
- Fix possible PG_INCOMPLETE on node outage when allow_net_split is false
- Fix build under Ubuntu 25.10
- Fix journal_no_same_sector_overwrites mode after optimizing WA (#109)
- Optimize WA - reduce it from 4-5 to 3.x by batching journal writes
- Fix ls-pool showing wrong used % when osds are reweighted (#105)
- Fix vitastor-cli rm sometimes hanging when PGs are down (#99)
- Fix vitastor-cli ls displaying raw used space for all inodes when -p is specified (#101)
- Fix "Cannot set property of undefined" in monitor when deleting pool with live inode op stats (#103)
- Revert vitastor-kv change from 2.2.0 leading to crashes in vitastor-kv-stress (#100)
- Add pg_size change check for XOR pools
- Fix root access to VitastorFS files
- Add IPv6 support to make-etcd
- Fix IPv6 etcd address support in OSD & client library
New features:
- Support UBLK in CSI and make it the default
- Add image tree output: `vitastor-cli ls --tree`
Bug fixes:
- Fix OSDs crashing on localized reads from snapshotted images
- Several NFS fixes from github PR #95
- Fix snapshotted bitmap reads not working in EC pools (fix#92)
- Fix UBLK reporting incorrect device size (1/8 of actual size), report logical_block_size=4096
- Do not repeer when stopping PEER_CONNECTING osd_clients (increase PG peering stability on failover)
- Fix marking PGs as incomplete when peers are unavailable
- Fix FS formatter (mkfs and fsck) error handling in CSI (it didn't report mounting errors)
- Fix removal of block PVC devices not working in CSI
- Do not attempt to free outbound ops in clear_immediate_ops (fix#88)
- Add vitastor-disk prepare --dry-run option
- Add librdmacm-dev to build-deps
- Log has_invalid objects
- Do not warn on incomplete+has_invalid PG states as unexpected
- Fix OSD reweight values between 0 and 1 not working in monitor
- Fix OSD assertion failed: n_subops != sent when all object copies are corrupted
- Install ibverbs-providers in vitastor Docker builds
- Fix#86 - base64_decode on arm64 O_o
- Fix Proxmox 9.0 support (remove some kind of a whitelist added in upstream)
New features:
- Add a new kernel device mounting method: [ublk](https://vitastor.io/docs/usage/ublk.html).
It's the fastest method for random IOPS, and it's on par with VDUSE for linear MB/s.
- Disable io_uring waits being reported as iowait on kernels which support it (6.15+)
- Allow to enforce permissions at the VitastorFS NFS server side
- Add qemu_file_mirror_path option to the config to allow to trick Veeam and make it work
- Speed up CRC32C calculation in OSD by fixing a bug and enabling AVX512 version
- Support QEMU 10
- Support Debian 13 Trixie and Proxmox 9.0
- Remove the dependency on system liburing in package builds (build it statically)
Bug fixes:
- Fix checksums NEVER BEING ENABLED in vitastor-disk prepare, even when explicitly requested :-D
- Use default uid and gid from NFS AUTH_SYS when creating files
- Fix object bitmaps supposedly & possibly being corrupted in some rare cases with EC N+2+
- Avoid multiple inflight overwrites to meta blocks - fixes possible data corruption with
one specific SSD model: Memblaze PBlaze5 910 (github #79)
- Fix a bug in antietcd which was leading to leases sometimes not expiring correctly
- Fix a bug in NFS where ".." entry had its `cookie` equal to 0 instead of 1 (github #78)
- Fix snapshots not being deleted during VM deletion in Proxmox plugin (github #85)
- Fix monitor not filtering OSDs by block size correctly
- Support Ubuntu 24.04 Noble
- Fix clients hanging on online PG count change with in-flight operations
- Fix volume_size_info in PVE VitastorPlugin
- Implement bdrv_refresh_filename for the QEMU driver
- Fix RDMA-CM broken in 2.2.0
- Fix possible "invalid %N$ use detected" OSD crashes
- Fix vitastor-nfs build without RDMA
- Fix docker build by adding the forgotten apt/preferences
- Fix libfio_blockstore.so
- Fix "trigger event loop automatically" API version check in the QEMU driver
- Add Content-Type header for prometheus metrics
- Add a patch for libvirt 11.5
- Fix a bug introduced in 2.2.0 - pg_locks weren't disabled for pools without local_reads
correctly which could lead to inactive pools during various operations
- Fix an old bug where OSDs could send sub-operations to incorrect peer OSDs when their
connections were stopped and reestablished quickly, in 2.2.0 it was usually leading
to "sequencing broken" messages in OSD logs
- Fix debug use_sync_send_recv mode
- Fix vitastor-disk purge broken after adding the "OSD is still running" check
- Fix iothreads hanging after adding zero-copy send support
- Fix enabling localized reads online (without restarting OSDs) in the default PG lock mode
New features:
- [Localized read support](https://vitastor.io/docs/config/pool.html#local_reads) for multi-datacenter setups.
- io_uring-based [zero-copy send support](https://vitastor.io/docs/config/network.html#min_zerocopy_send_size) -
read the instruction carefully for optimal performance!
- Improve and speedup data distribution, especially in cases of very large hosts (100 OSD+).
Previously, PG optimization speed depended on the number of OSDs, now it only depends on
the number of failure domains. Distribution over specific OSDs is now also more even and
becomes strictly more even when you increase the number of PGs.
- Add a [very interesting instruction](https://vitastor.io/en/docs/usage/nfs.html#linux-nfs-write-size) to change NFS_MAX_FILE_IO_SIZE
- Check operation sequencing and stop connections when it breaks - should help catch some
very rare RDMA packet loss problems.
- `vitastor-cli rm-osd` now refuses to remove OSDs which are still up and suggests to use `vitastor-disk purge`.
- Allow removal of direntries referring non-existent in VitastorFS.
- Change default vitastor-etcd data dir to /var/lib/etcd/vitastor.
Bug fixes:
- Fix compatibility with ISA-L 2.31+. ⚠️Very important: please upgrade Vitastor before upgrading ISA-L to 2.31+.
- Fix in-memory state cleanup for incomplete PGs.
- Fix monitor crash with non-existent node_placement nodes.
- Slightly speedup `vitastor-kv dump` command by adding output buffering.
- Fix theoretically possible slowdowns in OSD sub-operation failure handling code.
- Fix very rare stack overflows in vitastor-kv.
- Fix a possible crash in VitastorFS during handling of file creation race condition.
- Fix modify-pool -s PG_SIZE which didn't work without --pg_minsize.
- Fix marking peer OSDs as alive on receiving data from them via RDMA - in theory,
the bug could result in instability with RDMA under high load with slow disks.
- Fix a rare OSD crash due to double handle_primary_subop() call.
- Fix latency aggregation in global stats (/vitastor/stats in etcd) - do not sum it.
- Hide "Ran out of journal space" log messages by default.
- Wait for RDMA-CM EVENT_ESTABLISHED after rdma_accept(), handle rdma_accept() before acking the event.
- Fix VitastorFS total & free numbers multiplied by extra 2.
- Fix systemd unit name in make-etcd.
- Do not allow reweight > 1 in vitastor-cli modify-osd.
- Fix docker build.
New features:
- Support separate OSD cluster network - [osd_cluster_network](https://vitastor.io/docs/config/network.html#osd_cluster_network)
and, in general, multiple OSD networks, including RDMA
- Add an alternative RDMA implementation via RDMA-CM - [use_rdmacm](https://vitastor.io/docs/config/network.html#use_rdmacm),
required for iWARP and, maybe, for some IB setups (but not for RoCE)
- Change default PG behaviour to wait for all "up" OSDs to be connected before starting it.
The old behaviour may be returned by enabling a new [allow_net_split](https://vitastor.io/docs/config/osd.html#allow_net_split)
option.
- Add a patch for QEMU 9.2
Bug fixes:
- Fix incorrect "has_xxx" PG state names in ls-pgs
- Fix possible QEMU crashes after detaching of Vitastor disks (and update all QEMU builds in Vitastor repos)
- Fix clients sometimes spamming OSDs with infinite reconnections when some PGs are offline
- Fall back to TCP on RDMA connection failures
- Add missing logging of RDMA ibv_modify_qp() errors
- Add a minimum interval for etcd_state_client to reload state
No breaking features, it's 2.0.0 just because it includes S3 and because
there are already too many 1.x releases :).
New features:
- S3 is finally available: https://vitastor.io/docs/installation/s3.html
- node.js addon is now packaged as a Debian package
- Support listing PGs by OSDs in `vitastor-cli ls-pgs`
- Implement offline TRIM support: [vitastor-disk trim](https://vitastor.io/docs/usage/disk.html#trim),
[discard_on_start](https://vitastor.io/docs/config/osd.html#discard_on_start)
- Change used_for_fs pool option to used_for_app
Bug fixes:
- Fix several bugs in the node.js addon (a memory leak, an incorrectly triggered event loop)
- Fix a client crash (vitastor-cli rm) during deletion when writeback is enabled
- Fix PG object count statistics on deletion of non-existing objects
- Fix vitastor-nbd crash when mapping by ID instead of inode name
- Fix a client memory leak with enabled immediate_commit and write-back cache
- Add seccomp=unconfined for vitastor docker OSDs to not break io_uring
- Add udev and systemd to vitastor docker image
- Fix upgrading from pre-0.7.1 (very old) systemd units O_o
- Fix total object count calculation in rm_data
New features:
- Support containerized Vitastor installations: http://vitastor.io/docs/installation/docker.html
- Add new functions to the node.js binding: delete(), get_immediate_commit(), on_ready(),
get_min_io_size(), get_max_atomic_write_size()
- S3 (Zenko Cloudserver with Vitastor support) is coming shortly and will be released separately
Bug fixes:
- Use IP-derived etcd node names in make-etcd
- Set short name of the OSD process to display in `top`
- Fix snap-create without pool_id failing when there are multiple pools
- Several bugs are fixed in the write-back cache, it should now be stable:
- Fix incorrect snapshot reads from dirty write-back cache
- Do not try to repeat pending writebacks on OSD reconnections
- Fix client hangs with multiple SYNCs in the writeback queue
- Fix client hangs do to incorrect calculation of the writeback queue size
- Several improvements for NBD mapping/unmapping:
- Add a workaround for race condition in the Linux kernel NBD driver leading
to vitastor-nbd sometimes breaking a previously mapped device instead of
setting up a new one
- Check if the device is actually mapped in vitastor-nbd unmap
- Fix device name/number validation in vitastor-nbd
- Fix OSD crashes after starting with corrupted metadata - from now it will skip
corrupted metadata entries and heal itself
- Fix scrubbing of misplaced objects and object state recalculation after
vitastor-cli fix - previously, an OSD restart could be required to fix object states
- Make primary OSD distribution more stable by using murmur3 hash instead of the old pseudo-rng
- Fix monitor sometimes racing with itself - do not touch /pool/stats from stats
aggregation if PG recheck is active
- Sort vitastor-cli ls output by name by default
- Update antietcd to 1.1.2
New features:
- Add "deleted" image flag which is set when vitastor-cli rm starts to delete an image,
but can't delete it fully due to inactive PGs or stopped OSDs
- Support JSON output in vitastor-disk prepare and purge
- Show backfillfull pools in vitastor-cli status
- Make object listings consistent (used in vitastor-cli rm/rm-data/merge/etc).
This means that there is now a guarantee that if a data block is present when you invoke rm,
rm will attempt to delete it, even if rm is invoked when the PG switches state. Previously in
such cases rm could skip and leave some objects behind as garbage, and merge probably could
incorrectly move data between snapshots.
- Make deletions (rm/rm-data) consistent. This means that rm/rm-data will either complete
successfully and delete all requested image data or complete with an error if some objects
could not be deleted or if there is a possibility that some data is left on stopped OSDs.
Previously, when some PGs or OSDs were inactive at the moment of deletion, rm-data was
behaving incorrectly: it wasn't retrying deletions failed due to dropped OSD connections,
it could hang waiting for PGs to activate, and it could return with a successful error
code while some garbage was still possibly left on some OSDs. Deletions are not fully atomic
cluster-wide yet, which means that you still have to repeat the deletion request after you
return stopped OSDs back, but now you always know for sure if you have to repeat it.
Bug fixes:
- Fix vitastor-cli rm --exact / --matching command not working
- Finally fix "Unexpected status" in the Proxmox plugin
- Fix vitastor-cli create-snap incorrectly linking multiple snapshots in a different pool
- Fix incomplete image parent_id loop check in OSD
- Fix reads from snapshots in a different pool not working if there are more than 2 snapshots
- Fix append of VITASTOR_CONF to cmdline in the opennebula prebackup script
- Fix OSDs crashing again when the cluster is full with EC (was meant to work since 1.6.0 but didn't)
- Improve logging of subop failures
New features:
- Implement basic VitastorFS support in [CSI](https://vitastor.io/docs/installation/kubernetes.html)
- Implement [NFS RDMA](https://vitastor.io/docs/usage/nfs.html#rdma) support
- Pause pool rebalance when monitor detects that it can lead to any OSD becoming full ([osd_backfillfull_ratio](https://vitastor.io/docs/config/monitor.html#osd_backfillfull_ratio))
- Auto-select correct [RDMA device and GID](https://vitastor.io/docs/config/network.html#rdma_device) based on osd_network and RoCEv2 priority
- Report slow ops in OSD stats in etcd and show them in vitastor-cli status
Bug fixes:
- Fix possibly incorrect linked list deserialization in NFS
- Fix possible crash in vitastor-nfs --block READDIR operation
- Map netlink after forking to show correct PID in vitastor-nbd ls
- Simplify and fix create-pool OSD count checks for the case of hosts split into sub-nodes
- Make monitor print "Waiting to become master" just once, not every 5s
- Take out_size from oimg if not specified in vitastor-cli dd
- Do not report OSDs with empty statistics as "full" in status
- Trigger double autosync when switching PG state to prevent leaving garbage in non-immediate_commit clusters
- Fix a lack of connection timeout for etcd websockets in OSD leading to slower etcd failover (~70s instead of ~10s)
- Fix a rare OSD crash during client disconnect
- Fix PGs sometimes sticking until OSD restart in the "has_unclean" state with EC pools
- Fix metadata partition zeroing in vitastor-disk prepare
- Add patches for qemu 9.1 and pve-qemu 9.0 and 9.1
- Fix libvirt 8 patch
- Support custom hybrid OSD creation (`vitastor-disk prepare --hybrid --fast-devices /dev/xxx,/dev/yyy`)
- Auto-change partition paths to /dev/disk/by-partuuid/ in `vitastor-disk prepare`
- Allow to select cached I/O in vitastor-disk commands
- Fix multiple bugs in vitastor-disk resize & add tests for them
- Fix vitastor-disk write-meta/write-journal in superblock-based mode writing it to an incorrect device
- Fix vitastor-disk prepare sometimes again not seeing new partitions
- Cleanup PG history and stats of deleted pools
- Fix "is already mounted" checks in CSI
New features:
- Support resizing normal vitastor-disk partitions and moving journal/metadata: [vitastor-disk resize](https://vitastor.io/docs/usage/disk.html#resize)
- Support simple forms of vitastor-disk {dump,write}-{meta,journal} for OSD partitions
Bug fixes:
- Fix block RWX volumes broken after introducing stage/unstage support
- Do not allow to create non-block RWX volumes in CSI
- Fix vitastor-disk prepare not seeing the newly created partition in rare cases
- Fix non-array tags not showing up in ls-osd/osd-tree
- Make OpenNebula oned.conf patching during installation smarter
- Fix iseek option in vitastor-cli dd not working
- Validate conv=, iflag=, oflag= options in vitastor-cli dd
- Fix vitastor-disk write-meta not writing header checksum to the disk
- Fix JSON format in vitastor-disk dump-meta
- Fix read_chain_bitmap not working for snapshot in another pool
- Fix a possible OSD crash during parallel read & write to an image with snapshots
- Several followups to the READ_CHAIN_BITMAP fix: avoid data reads, fix possible overflow in is_zero(), fix bitmap size
Hotfixes for OpenNebula and upgrade hotfix for 1.7
- Fix deploy.vitastor, save.vitastor, restore.vitastor scripts not working for nodes other than master oned
- Fix deploy.vitastor not working for VMs without Vitastor disks
- Disable clearing old PG configuration when upgrading from 1.7 or older versions (it was breaking old clients)
Bugfix release, would be 1.7.2, but etcd layout changes mandate it to be 1.8.0. :-)
- Change etcd layout: /config/pgs is now /pg/config, /pg/stats/* is now /pgstats/*.
This is required to fix a rare PG history tracking issue caused by non-atomic
delivery of etcd events sometimes resulting in `incomplete` objects in EC pools
after mass OSD restarts. Upgrading can be performed freely, downgrade requires
additional action: [1.8.0 to 1.7.1](https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/docs/usage/admin.en.md#1-8-0-to-1-7-1)
- Fix a rare client hang on PG primary OSD switch
- Fix vitastor-nfs started using mount command sometimes not stopping automatically after unmount
- Fix vitastor-nfs mounts started using mount command sometimes hanging after daemonizing
- Fix merge/flatten into a pool with different object size (image migration between pools case)
- Do not print extra "PG disappeared after reload" verbose log messages for non-existing PGs
- Fix clustered Antietcd support and persistence filter
- Do not try to purge the same OSD multiple times if its multiple devices are passed to purge
- Various node.js binding fixes
A bunch of monitor fixes
- Add noout flag for OSDs (/vitastor/config/osd/xx)
- Fix "effective" size of degraded PGs (and thus "used space") calculation in monitor
- Fix monitor not clearing PGs of deleted pools
- Fix incorrect PG generation with hosts with 0 OSDs
- Fix monitor crashing during primary OSD recheck when pool has no PGs
- Fix monitor crashing when node_placement included non-existing OSDs
- Fix possible data movement after removing OSDs reweighted to 0
- Remove extra empty keys from pool configurations created by vitastor-cli create-pool
- Fix 32-bit build
New features:
- Implement "hierarchical failure domains" and other complex distribution rules, for example
EC 4+2 over 3 DC, with 2 chunks per each DC ([documentation](docs/config/pool.en.md#level_placement))
- Make OSDs handle ENOSPC - now cluster stays online even if some OSDs fill up
to 100 %, only writes requiring free space hang
- Implement Stage/Unstage & volume locking for CSI to prevent parallel mounting
and/or modifications of the same volume
- Warn about full and almost full OSDs in vitastor-cli status
- Add an experimental NBD netlink map mode as an option ([documentation](docs/usage/nbd.en.md))
- Add --pg parameter to vitastor-cli describe, print objects with 0x in human-readable format too
- Add [administration docs](docs/usage/admin.en.md)
Bug fixes:
- Fix client operation retry timeout - previously the timeout wasn't applied and writes were
retries almost instantly
- Fix monitors crashing on invalid pool configurations
- Fix journaling - make each journal write wait for all previous journal writes
- Fix monitor thinking that OSD weight is 0 after deleting /osd/config/ key online
- Fix a write stall caused by flusher possibly not trimming journal on rollback
- Set 32k csum_block_size for HDD by default in vitastor-disk
After half a year of hard work, VitastorFS is finally here ! :-)
New features:
- VitastorFS, a full-featured clustered (read-write-many) file system.
Documentation: [VitastorFS](docs/usage/nfs.en.md)
- Embedded key-value database implementation based on Parallel Optimistic B-Tree
algorithm and used for the metadata of VitastorFS
- Pool management commands in vitastor-cli (create-pool, list-pools, rm-pool, modify-pool).
Thanks MIND Software (https://mindsw.io) for their contribution!
[Documentation](docs/usage/cli.en.md#create-pool)
Bug fixes:
- Fix a very rare "infinite loop" in the client library
- Fix a rare OSD hang on during start when zeroing out bad metadata entries left from the previous run
- Do not use \r if output is not a terminal (should fix unexpected job output in proxmox)
- Fix rm/rm-data error return code, add --down-ok option to bypass the error
- Add EIO retry timeout and allow to disable these retries, rename up_wait_retry_interval to client_retry_interval
- Add ubuntu jammy build
- Wait for blockstore initialisation before starting OSD (prevent timeouts when init takes time)
- Fix a rare use-after-free in automatic sync after delete in blockstore
- Fix another old "BUG: Attempt to overwrite used offset" in a very simple
case: bs=4k rw=write iodepth=16 from OSD start; add this case to tests
- Fix a rare crash with "unexpected state during flush: 0x51" possible with
EC since 1.4.2 during rebalance and OSD outages
- Fix a rare write stall with EC & immediate_commit=none caused by sync
operations reserving unneeded space in the journal
- Fix 32-bit build warnings, most in printf/scanf format strings
Unwavering stabilization of 1.4.x, continued :-)
- Include the accidentally lost part of 1.4.5 journal trimming fix
- Fix a possible OSD crash with "BUG: Attempt to overwrite used offset"
which was probably present for long time, but became apparent after
fixing flapping tests in CI
- Fix remaining flapping tests in CI. It was the first time when tests
actually passed without retries :-)
- Fix a write stall caused by incorrect journal trimming introduced in 1.4.4 :)
- Fix PGs sometimes hanging in "starting" state on mass OSD restarts
- Fix a rare crash with "map::at" during OSD pings
- Use new defaults for non-capacitor (desktop) SSDs - improves T1Q256 random write from ~6k iops to ~45k iops
- Make journal_trim_interval configurable
A couple of fixes for EC pools
- Fix a segfault possible on partial EC overwrite in 1234 -> 5030 rebalance scenario
- Fix two problems leading to EC pools stalling on rebalance & parallel sudden stops
of OSDs, for example during a sudden poweroff of a host:
- Recovery auto-tuning (1.4.0 feature) could apply too large delays and stall
the EC journal - fixed by limiting delays with a new recovery_tune_sleep_cutoff_us
parameter (10 seconds by default) and applying recovery pauses before write
operations, not after them, to not occupy space in the journal for long time
- Dynamic journal space reservation (1.3.0 feature) wasn't accounting new writes
when checking the limit so OSDs could still fill the journal fully and stall -
fixed by including new writes into the limit
- Print etcd dbSize instead of dbSizeInUse in status
Vitaliy Filippov