New features:
- [Localized read support](https://vitastor.io/docs/config/pool.html#local_reads) for multi-datacenter setups.
- io_uring-based [zero-copy send support](https://vitastor.io/docs/config/network.html#min_zerocopy_send_size) -
read the instruction carefully for optimal performance!
- Improve and speedup data distribution, especially in cases of very large hosts (100 OSD+).
Previously, PG optimization speed depended on the number of OSDs, now it only depends on
the number of failure domains. Distribution over specific OSDs is now also more even and
becomes strictly more even when you increase the number of PGs.
- Add a [very interesting instruction](https://vitastor.io/en/docs/usage/nfs.html#linux-nfs-write-size) to change NFS_MAX_FILE_IO_SIZE
- Check operation sequencing and stop connections when it breaks - should help catch some
very rare RDMA packet loss problems.
- `vitastor-cli rm-osd` now refuses to remove OSDs which are still up and suggests to use `vitastor-disk purge`.
- Allow removal of direntries referring non-existent in VitastorFS.
- Change default vitastor-etcd data dir to /var/lib/etcd/vitastor.
Bug fixes:
- Fix compatibility with ISA-L 2.31+. ⚠️Very important: please upgrade Vitastor before upgrading ISA-L to 2.31+.
- Fix in-memory state cleanup for incomplete PGs.
- Fix monitor crash with non-existent node_placement nodes.
- Slightly speedup `vitastor-kv dump` command by adding output buffering.
- Fix theoretically possible slowdowns in OSD sub-operation failure handling code.
- Fix very rare stack overflows in vitastor-kv.
- Fix a possible crash in VitastorFS during handling of file creation race condition.
- Fix modify-pool -s PG_SIZE which didn't work without --pg_minsize.
- Fix marking peer OSDs as alive on receiving data from them via RDMA - in theory,
the bug could result in instability with RDMA under high load with slow disks.
- Fix a rare OSD crash due to double handle_primary_subop() call.
- Fix latency aggregation in global stats (/vitastor/stats in etcd) - do not sum it.
- Hide "Ran out of journal space" log messages by default.
- Wait for RDMA-CM EVENT_ESTABLISHED after rdma_accept(), handle rdma_accept() before acking the event.
- Fix VitastorFS total & free numbers multiplied by extra 2.
- Fix systemd unit name in make-etcd.
- Do not allow reweight > 1 in vitastor-cli modify-osd.
- Fix docker build.
New features:
- Support separate OSD cluster network - [osd_cluster_network](https://vitastor.io/docs/config/network.html#osd_cluster_network)
and, in general, multiple OSD networks, including RDMA
- Add an alternative RDMA implementation via RDMA-CM - [use_rdmacm](https://vitastor.io/docs/config/network.html#use_rdmacm),
required for iWARP and, maybe, for some IB setups (but not for RoCE)
- Change default PG behaviour to wait for all "up" OSDs to be connected before starting it.
The old behaviour may be returned by enabling a new [allow_net_split](https://vitastor.io/docs/config/osd.html#allow_net_split)
option.
- Add a patch for QEMU 9.2
Bug fixes:
- Fix incorrect "has_xxx" PG state names in ls-pgs
- Fix possible QEMU crashes after detaching of Vitastor disks (and update all QEMU builds in Vitastor repos)
- Fix clients sometimes spamming OSDs with infinite reconnections when some PGs are offline
- Fall back to TCP on RDMA connection failures
- Add missing logging of RDMA ibv_modify_qp() errors
- Add a minimum interval for etcd_state_client to reload state
No breaking features, it's 2.0.0 just because it includes S3 and because
there are already too many 1.x releases :).
New features:
- S3 is finally available: https://vitastor.io/docs/installation/s3.html
- node.js addon is now packaged as a Debian package
- Support listing PGs by OSDs in `vitastor-cli ls-pgs`
- Implement offline TRIM support: [vitastor-disk trim](https://vitastor.io/docs/usage/disk.html#trim),
[discard_on_start](https://vitastor.io/docs/config/osd.html#discard_on_start)
- Change used_for_fs pool option to used_for_app
Bug fixes:
- Fix several bugs in the node.js addon (a memory leak, an incorrectly triggered event loop)
- Fix a client crash (vitastor-cli rm) during deletion when writeback is enabled
- Fix PG object count statistics on deletion of non-existing objects
- Fix vitastor-nbd crash when mapping by ID instead of inode name
- Fix a client memory leak with enabled immediate_commit and write-back cache
- Add seccomp=unconfined for vitastor docker OSDs to not break io_uring
- Add udev and systemd to vitastor docker image
- Fix upgrading from pre-0.7.1 (very old) systemd units O_o
- Fix total object count calculation in rm_data
Vitaliy Filippov