- Antietcd is now officially safe to use: the release includes a fixed version of antietcd
which passes Jepsen transaction serializability tests.
- Fix a huge checksum bug in the old store: incorrect checksums for small initial writes.
The bug affected writes of exactly csum_block_size (4k by default) into new (unallocated)
objects and generated invalid checksums in the store for the written 4k block. Moreover,
generation of these invalid checksums was very slow because it was calculating CRC32
for 4 GB of zeroes. If the block wasn't then overwritten as a part of a larger write
request it became unreadable even though the stored data was correct. The bug affected
all versions since 1.0.0, or since 2.3.0 because vitastor-disk didn't allow to enable
checksums prior to 2.3.0 because of another bug. O:-)
- Prevent OSD disconnections due to long blocking of event-loop caused by PG resharding
(moving objects between old and new PGs in memory) when changing pool PG count or
just on restart of an OSD with a large database (for example, with a filled 8 TB SSD).
The issue should now be fixed because OSD now performs resharding in chunks with pauses
between chunks.
- Prevent pools stuck in paused state on an aborted PG count change.
- Fix a possible OSD crash with "division by zero" when trying to handle an operation
before pool PG count is applied to the in-memory store.
Important fixes:
- Disable RWF_ATOMIC by default because Linux incorrectly requires all atomic
writes to be power-of-2-sized and length-aligned. Details: [use_atomic_flag](https://vitastor.io/en/docs/config/osd.html#use_atomic_flag)
- Fix cross-pool snapshots not working at all - always reading old data after taking the snapshot
- Fix level_placement (broken in 2.2.0)
- Fix CAS write return values in the client library (broken in 2.4.4), also breaking unaligned writes in VitastorFS
- Fix VitastorFS possibly losing some of intersecting parallel unaligned writes
- Prevent possible reads of the old data during unfinished intent writes in the new store
- Tests added for all of above problems to prevent future regressions
Other fixes:
- Allow to specify OSD tags and weights during prepare
- Only clear the first block instead of whole OSD metadata and journal areas during vitastor-disk
prepare - OSD anyway clears them on the first run
- Fix vitastor-cli dd non-seekable detection and error status
- Do not stop OSD on zero-copy tcp short send errors - these aren't a bug, but just caused
by TCP client disconnections
- Fix modify-osd not working after deleting the OSD configuration key from etcd
- Fix regular antietcd websocket disconnections in OSD due to lack of WS_PONG responses to WS_PING
- Reduce the number of allocated RDMA memory regions
- Allow routed RoCE
A single new feature: the new log-structured metadata store implementation, described
in the presentation from Moscow Highload'2025 (check it out [here](https://vitastor.io/en/docs/#talks-and-presentations)).
It's now the default store for new OSDs. The support for the old store is also left in place,
you can still choose it for new OSDs with `vitastor-disk prepare --meta_format 2`.
OSDs from previous versions with the old store format will also continue to operate just like before.
- Implement `vitastor-kv rescue` command which dumps all readable blocks from the
database regardless of the tree structure and allows to recover a corrupted database.
- Simplify primary OSD selection - use just PG number instead of the hash. This allows
to distribute primary OSDs more evenly and reach better read speeds.
- Fix vitastor-kv-stress hang on parallel list & reopen.
- Fix CAS writes without immediate_commit. Fix is sufficient for vitastor-kv to
operate correctly in non-immediate_commit pools.
- Fix NFS ACCESS calls for broken inodes (allow root access to them).
- Fix broken vitastor-kv-stress --runtime parameter.
- Fix bad vitastor-kv dump output buffering leading to incorrect JSON in dumps.
- Fix writing garbage from uninitialized memory to the empty space in VitastorKV DB.
- Refactor some code in preparation to the release of the new log-structured metadata store.
- Replace assert(done != expected) error message, probably caused by kernel issues with
zero-copy TCP send, with a more concise error description.
- Daemonize before forking in NFS proxy to fix OSD RDMA support (#107)
- Fix possible PG_INCOMPLETE on node outage when allow_net_split is false
- Fix build under Ubuntu 25.10
- Fix journal_no_same_sector_overwrites mode after optimizing WA (#109)
- Optimize WA - reduce it from 4-5 to 3.x by batching journal writes
- Fix ls-pool showing wrong used % when osds are reweighted (#105)
- Fix vitastor-cli rm sometimes hanging when PGs are down (#99)
- Fix vitastor-cli ls displaying raw used space for all inodes when -p is specified (#101)
- Fix "Cannot set property of undefined" in monitor when deleting pool with live inode op stats (#103)
- Revert vitastor-kv change from 2.2.0 leading to crashes in vitastor-kv-stress (#100)
- Add pg_size change check for XOR pools
- Fix root access to VitastorFS files
- Add IPv6 support to make-etcd
- Fix IPv6 etcd address support in OSD & client library
New features:
- Support UBLK in CSI and make it the default
- Add image tree output: `vitastor-cli ls --tree`
Bug fixes:
- Fix OSDs crashing on localized reads from snapshotted images
- Several NFS fixes from github PR #95
- Fix snapshotted bitmap reads not working in EC pools (fix#92)
- Fix UBLK reporting incorrect device size (1/8 of actual size), report logical_block_size=4096
- Do not repeer when stopping PEER_CONNECTING osd_clients (increase PG peering stability on failover)
- Fix marking PGs as incomplete when peers are unavailable
- Fix FS formatter (mkfs and fsck) error handling in CSI (it didn't report mounting errors)
- Fix removal of block PVC devices not working in CSI
- Do not attempt to free outbound ops in clear_immediate_ops (fix#88)
- Add vitastor-disk prepare --dry-run option
- Add librdmacm-dev to build-deps
- Log has_invalid objects
- Do not warn on incomplete+has_invalid PG states as unexpected
- Fix OSD reweight values between 0 and 1 not working in monitor
- Fix OSD assertion failed: n_subops != sent when all object copies are corrupted
- Install ibverbs-providers in vitastor Docker builds
- Fix#86 - base64_decode on arm64 O_o
- Fix Proxmox 9.0 support (remove some kind of a whitelist added in upstream)
New features:
- Add a new kernel device mounting method: [ublk](https://vitastor.io/docs/usage/ublk.html).
It's the fastest method for random IOPS, and it's on par with VDUSE for linear MB/s.
- Disable io_uring waits being reported as iowait on kernels which support it (6.15+)
- Allow to enforce permissions at the VitastorFS NFS server side
- Add qemu_file_mirror_path option to the config to allow to trick Veeam and make it work
- Speed up CRC32C calculation in OSD by fixing a bug and enabling AVX512 version
- Support QEMU 10
- Support Debian 13 Trixie and Proxmox 9.0
- Remove the dependency on system liburing in package builds (build it statically)
Bug fixes:
- Fix checksums NEVER BEING ENABLED in vitastor-disk prepare, even when explicitly requested :-D
- Use default uid and gid from NFS AUTH_SYS when creating files
- Fix object bitmaps supposedly & possibly being corrupted in some rare cases with EC N+2+
- Avoid multiple inflight overwrites to meta blocks - fixes possible data corruption with
one specific SSD model: Memblaze PBlaze5 910 (github #79)
- Fix a bug in antietcd which was leading to leases sometimes not expiring correctly
- Fix a bug in NFS where ".." entry had its `cookie` equal to 0 instead of 1 (github #78)
- Fix snapshots not being deleted during VM deletion in Proxmox plugin (github #85)
- Fix monitor not filtering OSDs by block size correctly
- Support Ubuntu 24.04 Noble
- Fix clients hanging on online PG count change with in-flight operations
- Fix volume_size_info in PVE VitastorPlugin
- Implement bdrv_refresh_filename for the QEMU driver
- Fix RDMA-CM broken in 2.2.0
- Fix possible "invalid %N$ use detected" OSD crashes
- Fix vitastor-nfs build without RDMA
- Fix docker build by adding the forgotten apt/preferences
- Fix libfio_blockstore.so
- Fix "trigger event loop automatically" API version check in the QEMU driver
- Add Content-Type header for prometheus metrics
- Add a patch for libvirt 11.5
- Fix a bug introduced in 2.2.0 - pg_locks weren't disabled for pools without local_reads
correctly which could lead to inactive pools during various operations
- Fix an old bug where OSDs could send sub-operations to incorrect peer OSDs when their
connections were stopped and reestablished quickly, in 2.2.0 it was usually leading
to "sequencing broken" messages in OSD logs
- Fix debug use_sync_send_recv mode
- Fix vitastor-disk purge broken after adding the "OSD is still running" check
- Fix iothreads hanging after adding zero-copy send support
- Fix enabling localized reads online (without restarting OSDs) in the default PG lock mode
New features:
- [Localized read support](https://vitastor.io/docs/config/pool.html#local_reads) for multi-datacenter setups.
- io_uring-based [zero-copy send support](https://vitastor.io/docs/config/network.html#min_zerocopy_send_size) -
read the instruction carefully for optimal performance!
- Improve and speedup data distribution, especially in cases of very large hosts (100 OSD+).
Previously, PG optimization speed depended on the number of OSDs, now it only depends on
the number of failure domains. Distribution over specific OSDs is now also more even and
becomes strictly more even when you increase the number of PGs.
- Add a [very interesting instruction](https://vitastor.io/en/docs/usage/nfs.html#linux-nfs-write-size) to change NFS_MAX_FILE_IO_SIZE
- Check operation sequencing and stop connections when it breaks - should help catch some
very rare RDMA packet loss problems.
- `vitastor-cli rm-osd` now refuses to remove OSDs which are still up and suggests to use `vitastor-disk purge`.
- Allow removal of direntries referring non-existent in VitastorFS.
- Change default vitastor-etcd data dir to /var/lib/etcd/vitastor.
Bug fixes:
- Fix compatibility with ISA-L 2.31+. ⚠️Very important: please upgrade Vitastor before upgrading ISA-L to 2.31+.
- Fix in-memory state cleanup for incomplete PGs.
- Fix monitor crash with non-existent node_placement nodes.
- Slightly speedup `vitastor-kv dump` command by adding output buffering.
- Fix theoretically possible slowdowns in OSD sub-operation failure handling code.
- Fix very rare stack overflows in vitastor-kv.
- Fix a possible crash in VitastorFS during handling of file creation race condition.
- Fix modify-pool -s PG_SIZE which didn't work without --pg_minsize.
- Fix marking peer OSDs as alive on receiving data from them via RDMA - in theory,
the bug could result in instability with RDMA under high load with slow disks.
- Fix a rare OSD crash due to double handle_primary_subop() call.
- Fix latency aggregation in global stats (/vitastor/stats in etcd) - do not sum it.
- Hide "Ran out of journal space" log messages by default.
- Wait for RDMA-CM EVENT_ESTABLISHED after rdma_accept(), handle rdma_accept() before acking the event.
- Fix VitastorFS total & free numbers multiplied by extra 2.
- Fix systemd unit name in make-etcd.
- Do not allow reweight > 1 in vitastor-cli modify-osd.
- Fix docker build.
New features:
- Support separate OSD cluster network - [osd_cluster_network](https://vitastor.io/docs/config/network.html#osd_cluster_network)
and, in general, multiple OSD networks, including RDMA
- Add an alternative RDMA implementation via RDMA-CM - [use_rdmacm](https://vitastor.io/docs/config/network.html#use_rdmacm),
required for iWARP and, maybe, for some IB setups (but not for RoCE)
- Change default PG behaviour to wait for all "up" OSDs to be connected before starting it.
The old behaviour may be returned by enabling a new [allow_net_split](https://vitastor.io/docs/config/osd.html#allow_net_split)
option.
- Add a patch for QEMU 9.2
Bug fixes:
- Fix incorrect "has_xxx" PG state names in ls-pgs
- Fix possible QEMU crashes after detaching of Vitastor disks (and update all QEMU builds in Vitastor repos)
- Fix clients sometimes spamming OSDs with infinite reconnections when some PGs are offline
- Fall back to TCP on RDMA connection failures
- Add missing logging of RDMA ibv_modify_qp() errors
- Add a minimum interval for etcd_state_client to reload state
Vitaliy Filippov
Vitaliy Filippov