Should probably have been an obvious side effect :-)
Child process gets open file descriptors to parent's epoll/timerfd,
and it's totally OK to just close() all of them, but it's absolutely NOT
OK to run destructors - they modify the kernel state of epoll/timerfd
before destroying. So, basically, when we destroy the client in the child
process, we break it in the parent too. This also means that cluster_client_t
doesn't support fork(). :-)
Do all NBD configuration in the child process, after the last fork.
Why? It's needed because there is a race condition in the Linux kernel nbd driver
in nbd_add_socket() - it saves `current` task pointer as `nbd->task_setup` and
then rechecks if the new `current` is the same. Problem is that if that process
is already dead, `current` may be freed and then replaced by another process
with the same pointer value. So the check passes and NBD allows a different process
to set up a device which is already set up. Proper fix would have to be done in the
kernel code, but the workaround is obviously to perform NBD setup from the process
which will then actually call NBD_DO_IT. That process stays alive during the whole
time of NBD device execution and the (nbd->task_setup != current) check always
works correctly, and we don't accidentally break previous NBD devices while setting
up a new device. Forking to check every device is of course rather slow, so we also
do an additional check by calling list_mapped() before searching for a free NBD device.
New features:
- Add "deleted" image flag which is set when vitastor-cli rm starts to delete an image,
but can't delete it fully due to inactive PGs or stopped OSDs
- Support JSON output in vitastor-disk prepare and purge
- Show backfillfull pools in vitastor-cli status
- Make object listings consistent (used in vitastor-cli rm/rm-data/merge/etc).
This means that there is now a guarantee that if a data block is present when you invoke rm,
rm will attempt to delete it, even if rm is invoked when the PG switches state. Previously in
such cases rm could skip and leave some objects behind as garbage, and merge probably could
incorrectly move data between snapshots.
- Make deletions (rm/rm-data) consistent. This means that rm/rm-data will either complete
successfully and delete all requested image data or complete with an error if some objects
could not be deleted or if there is a possibility that some data is left on stopped OSDs.
Previously, when some PGs or OSDs were inactive at the moment of deletion, rm-data was
behaving incorrectly: it wasn't retrying deletions failed due to dropped OSD connections,
it could hang waiting for PGs to activate, and it could return with a successful error
code while some garbage was still possibly left on some OSDs. Deletions are not fully atomic
cluster-wide yet, which means that you still have to repeat the deletion request after you
return stopped OSDs back, but now you always know for sure if you have to repeat it.
Bug fixes:
- Fix vitastor-cli rm --exact / --matching command not working
- Finally fix "Unexpected status" in the Proxmox plugin
- Fix vitastor-cli create-snap incorrectly linking multiple snapshots in a different pool
- Fix incomplete image parent_id loop check in OSD
- Fix reads from snapshots in a different pool not working if there are more than 2 snapshots
- Fix append of VITASTOR_CONF to cmdline in the opennebula prebackup script
- Fix OSDs crashing again when the cluster is full with EC (was meant to work since 1.6.0 but didn't)
- Improve logging of subop failures
This still doesn't make listings 100% consistent yet; for 100% consistent
listings we have to receive listings only from the primary OSD, not from all
peer OSDs, but this issue will be fixed separately.
Vitaliy Filippov