thebestvibedev/tromcho.net
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
432 Commits 74 Branches 106 Tags
9d20839a0207f09a679e43c26140ce0d96f091d1
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Vitaliy Filippov 9d20839a02 Add license texts 2020-09-17 23:07:06 +03:00
Vitaliy Filippov 67a2e5640c Fix a GIANT memory leak on read :D 2020-09-17 00:45:59 +03:00
Vitaliy Filippov 28a0f08ce7 Add a very simple tool for calculating device offsets 2020-09-17 00:45:59 +03:00
Vitaliy Filippov 9b4e5b64ae Move monitor to mon/ 2020-09-16 02:15:26 +03:00
Vitaliy Filippov 4ca2eeafff Prefer data OSDs for EC/XOR because they can actually read something locally 2020-09-16 01:34:18 +03:00
Vitaliy Filippov 79156e0ee1 Add test systemd unit generation script 2020-09-13 12:50:15 +03:00
Vitaliy Filippov ed26c33f85 React to down OSDs instantly, set timer to recheck PGs after <osd_out_time> 2020-09-13 12:23:18 +03:00
Vitaliy Filippov 18692517be Increase receive_buffer_size 2020-09-13 00:39:20 +03:00
Vitaliy Filippov de6919b02b Add option to disable multiple overwrites of the same journal sector 
This makes sense for some SSDs like Intel D3-4510 because they don't
like overwrites of the same sector:

$ fio -direct=1 -rw=write -bs=4k -size=4k -loops=100000 -iodepth=1
  write: IOPS=3142, BW=12.3MiB/s (12.9MB/s)(97.9MiB/7977msec)

$ fio -direct=1 -rw=write -bs=4k -size=128k -loops=100000 -iodepth=1
  write: IOPS=20.8k, BW=81.4MiB/s (85.3MB/s)(543MiB/6675msec)
 2020-09-13 00:37:39 +03:00
Vitaliy Filippov 8f9f438e25 Allow zero reweights, fix changing pgs 2020-09-11 23:59:30 +03:00
Vitaliy Filippov db4b82089e connecting=true was also forgotten 2020-09-11 19:51:17 +03:00
Vitaliy Filippov faa871090f Do not die in mon on bad JSON in etcd 2020-09-11 19:25:52 +03:00
Vitaliy Filippov 49ec8c7c63 Add --verbose 1 flag for mon 2020-09-11 18:49:07 +03:00
Vitaliy Filippov eadd454992 Fix etcd key regexps 2020-09-11 18:33:49 +03:00
Vitaliy Filippov a15bd23ebd Missed a bad PG key 2020-09-11 17:56:18 +03:00
Vitaliy Filippov e3f502b466 Oops, I forgot a file 2020-09-11 17:49:44 +03:00
Vitaliy Filippov 6e72cf2732 Disable stdout/stderr buffering 2020-09-11 16:52:51 +03:00
Vitaliy Filippov 53832d184a Allow to use lazy sync with replicated pools 2020-09-06 12:08:44 +03:00
Vitaliy Filippov 352caeba14 Fix one more bug with replicated reads 2020-09-06 02:27:44 +03:00
Vitaliy Filippov fb533991b7 "Lock" retried objects from other flushers when accounting for overruns 
Fixes a rare 100% CPU consuming hang
 2020-09-06 02:19:36 +03:00
Vitaliy Filippov 73e26dbbea Add up_wait_retry_interval to config and fix it so it actually works 2020-09-05 22:05:21 +03:00
Vitaliy Filippov 44973e7f27 Fix replicated pool bugs 2020-09-05 21:45:04 +03:00
Vitaliy Filippov 242d9a42a2 Change object format in prints to %lx:%lx v%lu 2020-09-05 17:44:05 +03:00
Vitaliy Filippov 68c3e96e46 Add pool setting to fio and qemu drivers 2020-09-05 17:44:00 +03:00
Vitaliy Filippov cc4714a3a7 Basic fixes for the Monitor 2020-09-05 02:14:43 +03:00
Vitaliy Filippov e051db5a73 Check for unsuccessful memory allocations 2020-09-05 01:42:11 +03:00
Vitaliy Filippov 4f9b5286a0 Add replicated pool support to OSD logic 
...in theory :-D now it needs some testing
 2020-09-05 01:42:11 +03:00
Vitaliy Filippov 168cc2c803 Add pool support to OSD, part 1 
This just fixes all the code so it builds and works like before,
but doesn't yet bring the support for replicated pools.
 2020-09-04 17:04:17 +03:00
Vitaliy Filippov 4cdad634b5 Add pool support to the cluster client 2020-09-03 00:52:41 +03:00
Vitaliy Filippov 293cb5bd1d Parse pool configuration in etcd_state_client 2020-09-02 21:54:32 +03:00
Vitaliy Filippov 0918ea08fa Implement min/max inode filters in LIST operation 2020-09-02 14:42:40 +03:00
Vitaliy Filippov a8b3cbd6af Implement per-pool PG calculation, fix some lint warnings 2020-09-01 18:53:54 +03:00
Vitaliy Filippov fe0d78bf8e Filter configuration keys with regexps instead of "osd_tree" value checks 2020-09-01 17:07:06 +03:00
Vitaliy Filippov 085c145a18 Document etcd data (to-be state with pools) at least in some form 2020-09-01 16:29:45 +03:00
Vitaliy Filippov 30da4bddbe Extract scale_pg_count into a separate file 2020-09-01 16:18:58 +03:00
Vitaliy Filippov 14b4a4617e (re)move placement_tree 2020-09-01 16:18:58 +03:00
Vitaliy Filippov 3932c9b2e2 Add WRITE_STABLE to the secondary OSD for the upcoming replication support 2020-09-01 16:18:58 +03:00
Vitaliy Filippov 2e8c69fc5b Rename OSD_OP_SECONDARY_* to OSD_OP_SEC_* 2020-08-31 23:57:50 +03:00
Vitaliy Filippov a86788fe3b Support optimizing for the case when parity chunks occupy more space than data chunks 
Mostly as an experiment because the problem solved by this commit
comes from Ceph's EC+compression implementation details and I'm not sure
if my implementation will be the same
 2020-08-17 01:44:19 +03:00
Vitaliy Filippov 95ebfad283 Final name is Vitastor 2020-08-03 23:50:59 +03:00
Vitaliy Filippov 6022f28dc9 Add pseudo-random PG generation 2020-07-07 23:13:07 +03:00
Vitaliy Filippov 9d10a4d057 Support arbitrary pg_size in LPOptimizer 2020-07-05 20:28:05 +03:00
Vitaliy Filippov ec7acc8f3a Add WRITE_STABLE operation for future replication support 2020-07-05 01:48:02 +03:00
Vitaliy Filippov 416a80b099 Make blockstore object state a combination of type and workflow 2020-07-04 22:20:32 +03:00
Vitaliy Filippov a7929931eb Implement PG epochs to prevent the "version split" 
The "version split" is when:
- A block is written to 1 OSD out of 3, all of them die
- OSDs 2 and 3 come up, the same block is written to both of them
- The remaining OSD comes up. Now all 3 OSDs have the same version of the same object,
  but with different data.
 2020-07-04 00:55:27 +03:00
Vitaliy Filippov e680d6c1c3 Rename reconstruct_stripe and calc_rmw_parity to indicate that they are only for XOR N+1 2020-06-30 10:40:43 +03:00
Vitaliy Filippov 9b33f598d3 Fix two more cluster client bugs 
1) Sync could delete an unfinished write due to the lack of ordering (fixed by introducing syncing_writes)
2) Writes could be postponed indefinitely due to bad resuming of operations after a sync
 2020-06-27 02:13:35 +03:00
Vitaliy Filippov 592bcd3699 Fix QEMU driver bugs (QEMU and qemu-img now work! hooray!) 2020-06-26 18:25:43 +03:00
Vitaliy Filippov 5e1e39633d Implement QEMU block driver 2020-06-25 11:59:43 +03:00
Vitaliy Filippov 41c2655edd Disconnect sockets when read returns zero 2020-06-24 01:32:19 +03:00
Vitaliy Filippov d68370304e Support iovecs in cluster_client_t 2020-06-24 01:31:48 +03:00
Vitaliy Filippov a22d9f38aa Only use EPOLLOUT while connecting 2020-06-23 20:18:31 +03:00
Vitaliy Filippov 8736b3ad32 Add destructors, make ringloop optional in cluster_client_t 2020-06-23 20:10:33 +03:00
Vitaliy Filippov 62343c8022 Allow to turn synchronous recvmsg/sendmsg on with a config option 2020-06-23 01:15:07 +03:00
Vitaliy Filippov 9abaf5b735 Use epoll_manager in osd 2020-06-20 01:28:18 +03:00
Vitaliy Filippov badf68c039 Support iovecs for read operations 2020-06-19 19:47:05 +03:00
Vitaliy Filippov 0f6d193d73 Postpone op callbacks to the end of handle_read(), fix a bug where primary OSD could reply -EPIPE with data to a read operation 2020-06-16 01:36:38 +03:00
Vitaliy Filippov 27ee14a4e6 Fix bugs in cluster_client 2020-06-16 00:08:45 +03:00
Vitaliy Filippov 64afec03ec In theory, implement syncs and replay for the non-immediate commit mode 2020-06-15 00:04:16 +03:00
Vitaliy Filippov 4dde8b8a42 Oops, fix fio_sec_osd block_order parsing 2020-06-09 00:52:00 +03:00
Vitaliy Filippov f5ccb154af Benchmark reads in stub_bench, too 2020-06-08 01:54:44 +03:00
Vitaliy Filippov 73c80e2c39 Move accept_connections() to osd_messenger_t, add a simple uring OSD stub 2020-06-08 01:32:16 +03:00
Vitaliy Filippov 437dc5b630 Implement a FIO engine for testing cluster I/O 2020-06-07 00:30:15 +03:00
Vitaliy Filippov 226f5a2945 Allow to override block_size in fio_sec_osd 2020-06-07 00:10:13 +03:00
Vitaliy Filippov 2187d06eac Add a parameter to pass the initial config to client 2020-06-07 00:10:12 +03:00
Vitaliy Filippov c573bc6bb3 (Probably almost) implement cluster client 2020-06-07 00:09:36 +03:00
Vitaliy Filippov 2f6cf605a1 Rename cluster_client to osd_messenger 2020-06-04 12:57:54 +03:00
Vitaliy Filippov 05ea97119f Fix BS_OP_LIST to account for deleted objects: only list the newest stable entry of each object 
This allows list responses to be unaffected by journal flushes, which, in turn,
fixes PG peering when a peer OSD is replaying journal and journal contains deletions
 2020-06-02 23:52:48 +03:00
Vitaliy Filippov 571be0f380 Make deletions instantly stable 
"2-phase" (write->stabilize) process is pointless for deletions because it
doesn't protect us from incomplete objects. This happens because it removes
the version information from metadata after stabilization. Deletions require
"3-phase" process with a potentially very long 3rd phase.

So, deletions will be allowed to generate degraded and incomplete objects,
and for it to not affect users' ability to delete something, the cluster
will allow to delete whole inodes while storing a list of them in etcd.
Proper TRIM will be impossible until the implementation of the aforementioned
"3-phase" process, though.

By the way, this change also fixes a possible write stall after rebalancing
which was caused by the lack of "stabilize delete" operations.
 2020-06-02 23:45:22 +03:00
Vitaliy Filippov 985c309d7f Remove duplicate code between blockstore_{rollback,stable} and blockstore_init 2020-06-02 20:37:00 +03:00
Vitaliy Filippov a56f8cd14e Simplify handle_primary_subop() arguments 2020-06-02 18:44:23 +03:00
Vitaliy Filippov 46e111272f Replace assert(this_it == cur_op) with if() for the case of PG repeering 2020-06-02 14:30:57 +03:00
Vitaliy Filippov 165c204555 Fix BS_OP_DELETE (the implementation was untested up to this point) 2020-06-02 14:26:01 +03:00
Vitaliy Filippov af5cd45071 Oh crap, got SIGPIPE. Add MSG_NOSIGNAL 2020-06-02 11:41:08 +03:00
Vitaliy Filippov c3fe9ad0d1 Fix rebalancing writes (add a forgotten state resume) 2020-06-02 01:26:14 +03:00
Vitaliy Filippov 0fcdeae18b Do not die if a peer is already stopped on flush error 2020-06-01 23:07:08 +03:00
Vitaliy Filippov e6a4b634f8 Fix possible write stall 
The stall occurred during fio Q=128 random write tests with low flusher_count (4).
It was caused by flushers being unable to flush the beginning of the journal
because it contained older writes to an object that also had writes in the very end
of the journal, after dirty_start.
 2020-06-01 16:18:23 +03:00
Vitaliy Filippov c22e096943 Output journal offsets in debug trace in hex, add detailed "still waiting" messages 2020-06-01 16:18:19 +03:00
Vitaliy Filippov 45b1c2fbf1 Fix canceling of write operations on PG re-peer (which led to use-after-free, too...) 2020-06-01 16:18:14 +03:00
Vitaliy Filippov 3469bead67 Protect "delete this" with a stack refcounter 
(to fix use-after-free, too, but "delete this" was a time bomb anyway)
 2020-06-01 16:18:09 +03:00
Vitaliy Filippov 3a5d488f19 Fix use-after-free in osd_flush.cpp 2020-06-01 01:56:24 +03:00
Vitaliy Filippov 73e4e30b1f Auto-generate C++ header dependencies 2020-06-01 00:25:25 +03:00
Vitaliy Filippov 5feff1ffb9 Slightly cleanup socket send/receive code 2020-05-31 15:03:27 +03:00
Vitaliy Filippov b466e215f0 Fix queued OP_SYNC execution 2020-05-27 13:55:25 +03:00
Vitaliy Filippov 36f995367f Fix bind_address reporting 2020-05-27 10:58:40 +03:00
Vitaliy Filippov 0aca6e9ca8 Extract peer connect and read-write loop into a separate file (to be shared with the client library) 2020-05-26 22:11:30 +03:00
Vitaliy Filippov fa98be6bc0 Allow to specify multiple etcd addresses 2020-05-25 16:30:05 +03:00
Vitaliy Filippov 256a7f2667 Free op->bs_op manually 2020-05-25 15:31:22 +03:00
Vitaliy Filippov 79bf57b6e2 Allow to override pg_stripe_size 2020-05-25 15:31:22 +03:00
Vitaliy Filippov 53f6aba3e6 Die when journal_sector_buffer_count is too small 2020-05-24 17:26:47 +03:00
Vitaliy Filippov 36595eb669 Print "Ran out of journal sector buffers" warning 2020-05-24 16:48:50 +03:00
Vitaliy Filippov e09d0e0678 Several bug fixes 
- Do not block flock() requests
- Fix stop_client(0) attempts leading to std::bad_function_call
- Fix degraded writes crashing due to an unset stripes[i].missing (at least with a missing parity device)
- Fix recovery B/W reporting
 2020-05-24 01:51:35 +03:00
Vitaliy Filippov d1602b50b3 Fix BS_OP_ROLLBACK removing an incorrect version 
Instead of only removing versions with oid == X and version > Y it was
also removing the previous version in list (with the previous oid or
with version == Y)
 2020-05-24 01:51:28 +03:00
Vitaliy Filippov 7df384031a Re-peer PGs after stopping the peer 
Fixes the bug where two peers killed at once have lead to PG state PG_DEGRADED|PG_HAS_INCOMPLETE instead of PG_INCOMPLETE
 2020-05-23 18:45:12 +03:00
Vitaliy Filippov e614a98543 Add a sad FIXME :-) 2020-05-23 15:43:37 +03:00
Vitaliy Filippov 01dd3ef89e Fix timerfd_manager triggering of multiple times at the same time 2020-05-23 15:43:37 +03:00
Vitaliy Filippov cdccc23aff Print [OSD $osd_num] in stats, print B/W only for ops that log bytes 2020-05-23 15:43:37 +03:00
Vitaliy Filippov 700428829a Fix autosync_interval default not setting when autosync_interval is skipped in config 2020-05-23 15:43:37 +03:00
Vitaliy Filippov 6488d0044a Ignore EPOLL_CTL_DEL ENOENT, fix detection of the rollback version 2020-05-23 15:43:37 +03:00
Vitaliy Filippov 393fe75900 Fix creepy (osd_op_t*)(long) casts 2020-05-23 15:43:37 +03:00