- Fix broken garbage collection in the new store leading to very high memory usage
- Make the new store do full garbage collection on OSD startup by default (can be turned off with a new gc_on_start parameter)
- Make startup logging with the new store more informative, allow to log amount of garbage with log_level > 1
- Fix clients reading from snapshots incorrectly on retries (for example when OSDs were restarted during read)
- Fix a possible "send-after-free" in OSD which could probably also affect snapshot reads
- Fix OSD crashing when handling a read_bitmap operation from a deleted object with the new store
- Surrogate increment-only peer IDs are now used instead of file descriptor numbers,
and client cleanup sequence is reworked in another attempt to fix a possible PG hang
in 'peering' states. The history of this bug is still unclear and it still hasn't been
reproduced in tests, but several users have reported it since 3.0.4 (only with enabled RDMA).
- Fix a minor bug where an OSD could skip reporting misplaced PG state when moving
an object in non-immediate_commit mode.
- Add Ubuntu 26.04 (Resolute Raccoon) support and patches for qemu 10.2 and libvirt 12.0
- Fix one more bug in old store checksums O_o - checksum verification wasn't
working correctly on reads quickly after small initial offsetted writes
(when a BIG_WRITE was read from the journal).
- Fix clients possibly not destroyed after being switched to RDMA. Not 100%
verified, but in theory it could rarely lead to PGs hanging in peering/stopping
states with RDMA enabled.
- Use vitastor-cli in the Cinder driver for modifications instead of direct etcd
interaction.
- Fix mon_https_ca parsing.
- Fix Docker install.sh, add instructions for Podman.
Checksum verification wasn't working correctly on reads of small initial offsetted writes
from the journal (until the write was flushed to "clean" metadata).
It could in theory lead to hung operations on RDMA client disconnections.
Workflow (probable):
* OSD switches an outbound connection to PEER_RDMA and removes the FD from epoll
* But a receive request in io_uring is still active
* Receive response is never handled, refs remains > 0
* The connection is dropped, but osd_client_t is not destroyed because refs > 0
* Outbound ops are not canceled, suboperations hang, primary operations hang too
A single urgent fix for the new store: metadata was written incorrectly when
filled metadata area exceeded 4 GB due to a simple & stupid 32-bit integer overflow.
This was leading to OSDs corrupting metadata and being unable to start on the next
run.
The bug affected only the new store (meta_format=3) and OSDs with metadata area size
larger than 4 GB. You can check your OSD metadata area size by checking `data_offset`
in `vitastor-disk read-sb /dev/vitastor/osdXX-data` for single-disk OSDs or
`blockdev --getsize64 /dev/vitastor/osdXX-meta` for hybrid OSDs.
If you're affected, it's highly recommended to:
- Install updated packages without restarting OSDs
- Restart OSDs **one by one**
- If a restart of an OSD triggers "has_degraded" rebalance - wait for it to finish
before restarting another OSD
- If an OSD fails to start with "double-claimed" errors in the log - purge and recreate it,
and again wait for "has_degraded" rebalance to finish before proceeding to the next OSD
Vitaliy Filippov