It could in theory lead to hung operations on RDMA client disconnections.
Workflow (probable):
* OSD switches an outbound connection to PEER_RDMA and removes the FD from epoll
* But a receive request in io_uring is still active
* Receive response is never handled, refs remains > 0
* The connection is dropped, but osd_client_t is not destroyed because refs > 0
* Outbound ops are not canceled, suboperations hang, primary operations hang too
Required mainly for clients, allows to scale parallel client I/O with TCP
from 100-150k iops to ~400k iops and from 2-3 GB/s to at least 7-8 GB/s
with 4 I/O threads, at the same time increasing Q=1 latency by 2x thread
switching delay, which is ~10 us when CPU powersaving is disabled and may
be as high as 200 us when it's enabled.
Vitaliy Filippov