New features:
- Support separate OSD cluster network - [osd_cluster_network](https://vitastor.io/docs/config/network.html#osd_cluster_network)
and, in general, multiple OSD networks, including RDMA
- Add an alternative RDMA implementation via RDMA-CM - [use_rdmacm](https://vitastor.io/docs/config/network.html#use_rdmacm),
required for iWARP and, maybe, for some IB setups (but not for RoCE)
- Change default PG behaviour to wait for all "up" OSDs to be connected before starting it.
The old behaviour may be returned by enabling a new [allow_net_split](https://vitastor.io/docs/config/osd.html#allow_net_split)
option.
- Add a patch for QEMU 9.2
Bug fixes:
- Fix incorrect "has_xxx" PG state names in ls-pgs
- Fix possible QEMU crashes after detaching of Vitastor disks (and update all QEMU builds in Vitastor repos)
- Fix clients sometimes spamming OSDs with infinite reconnections when some PGs are offline
- Fall back to TCP on RDMA connection failures
- Add missing logging of RDMA ibv_modify_qp() errors
- Add a minimum interval for etcd_state_client to reload state
No breaking features, it's 2.0.0 just because it includes S3 and because
there are already too many 1.x releases :).
New features:
- S3 is finally available: https://vitastor.io/docs/installation/s3.html
- node.js addon is now packaged as a Debian package
- Support listing PGs by OSDs in `vitastor-cli ls-pgs`
- Implement offline TRIM support: [vitastor-disk trim](https://vitastor.io/docs/usage/disk.html#trim),
[discard_on_start](https://vitastor.io/docs/config/osd.html#discard_on_start)
- Change used_for_fs pool option to used_for_app
Bug fixes:
- Fix several bugs in the node.js addon (a memory leak, an incorrectly triggered event loop)
- Fix a client crash (vitastor-cli rm) during deletion when writeback is enabled
- Fix PG object count statistics on deletion of non-existing objects
- Fix vitastor-nbd crash when mapping by ID instead of inode name
- Fix a client memory leak with enabled immediate_commit and write-back cache
- Add seccomp=unconfined for vitastor docker OSDs to not break io_uring
- Add udev and systemd to vitastor docker image
- Fix upgrading from pre-0.7.1 (very old) systemd units O_o
- Fix total object count calculation in rm_data
New features:
- Support containerized Vitastor installations: http://vitastor.io/docs/installation/docker.html
- Add new functions to the node.js binding: delete(), get_immediate_commit(), on_ready(),
get_min_io_size(), get_max_atomic_write_size()
- S3 (Zenko Cloudserver with Vitastor support) is coming shortly and will be released separately
Bug fixes:
- Use IP-derived etcd node names in make-etcd
- Set short name of the OSD process to display in `top`
- Fix snap-create without pool_id failing when there are multiple pools
- Several bugs are fixed in the write-back cache, it should now be stable:
- Fix incorrect snapshot reads from dirty write-back cache
- Do not try to repeat pending writebacks on OSD reconnections
- Fix client hangs with multiple SYNCs in the writeback queue
- Fix client hangs do to incorrect calculation of the writeback queue size
- Several improvements for NBD mapping/unmapping:
- Add a workaround for race condition in the Linux kernel NBD driver leading
to vitastor-nbd sometimes breaking a previously mapped device instead of
setting up a new one
- Check if the device is actually mapped in vitastor-nbd unmap
- Fix device name/number validation in vitastor-nbd
- Fix OSD crashes after starting with corrupted metadata - from now it will skip
corrupted metadata entries and heal itself
- Fix scrubbing of misplaced objects and object state recalculation after
vitastor-cli fix - previously, an OSD restart could be required to fix object states
- Make primary OSD distribution more stable by using murmur3 hash instead of the old pseudo-rng
- Fix monitor sometimes racing with itself - do not touch /pool/stats from stats
aggregation if PG recheck is active
- Sort vitastor-cli ls output by name by default
- Update antietcd to 1.1.2
Should probably have been an obvious side effect :-)
Child process gets open file descriptors to parent's epoll/timerfd,
and it's totally OK to just close() all of them, but it's absolutely NOT
OK to run destructors - they modify the kernel state of epoll/timerfd
before destroying. So, basically, when we destroy the client in the child
process, we break it in the parent too. This also means that cluster_client_t
doesn't support fork(). :-)
Do all NBD configuration in the child process, after the last fork.
Why? It's needed because there is a race condition in the Linux kernel nbd driver
in nbd_add_socket() - it saves `current` task pointer as `nbd->task_setup` and
then rechecks if the new `current` is the same. Problem is that if that process
is already dead, `current` may be freed and then replaced by another process
with the same pointer value. So the check passes and NBD allows a different process
to set up a device which is already set up. Proper fix would have to be done in the
kernel code, but the workaround is obviously to perform NBD setup from the process
which will then actually call NBD_DO_IT. That process stays alive during the whole
time of NBD device execution and the (nbd->task_setup != current) check always
works correctly, and we don't accidentally break previous NBD devices while setting
up a new device. Forking to check every device is of course rather slow, so we also
do an additional check by calling list_mapped() before searching for a free NBD device.
Vitaliy Filippov