thebestvibedev/tromcho.net
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
372 Commits 74 Branches 106 Tags
f5ccb154af16f0f2fdda3c4ded71d87f11e08496
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Vitaliy Filippov f5ccb154af Benchmark reads in stub_bench, too 2020-06-08 01:54:44 +03:00
Vitaliy Filippov 73c80e2c39 Move accept_connections() to osd_messenger_t, add a simple uring OSD stub 2020-06-08 01:32:16 +03:00
Vitaliy Filippov 437dc5b630 Implement a FIO engine for testing cluster I/O 2020-06-07 00:30:15 +03:00
Vitaliy Filippov 226f5a2945 Allow to override block_size in fio_sec_osd 2020-06-07 00:10:13 +03:00
Vitaliy Filippov 2187d06eac Add a parameter to pass the initial config to client 2020-06-07 00:10:12 +03:00
Vitaliy Filippov c573bc6bb3 (Probably almost) implement cluster client 2020-06-07 00:09:36 +03:00
Vitaliy Filippov 2f6cf605a1 Rename cluster_client to osd_messenger 2020-06-04 12:57:54 +03:00
Vitaliy Filippov 05ea97119f Fix BS_OP_LIST to account for deleted objects: only list the newest stable entry of each object 
This allows list responses to be unaffected by journal flushes, which, in turn,
fixes PG peering when a peer OSD is replaying journal and journal contains deletions
 2020-06-02 23:52:48 +03:00
Vitaliy Filippov 571be0f380 Make deletions instantly stable 
"2-phase" (write->stabilize) process is pointless for deletions because it
doesn't protect us from incomplete objects. This happens because it removes
the version information from metadata after stabilization. Deletions require
"3-phase" process with a potentially very long 3rd phase.

So, deletions will be allowed to generate degraded and incomplete objects,
and for it to not affect users' ability to delete something, the cluster
will allow to delete whole inodes while storing a list of them in etcd.
Proper TRIM will be impossible until the implementation of the aforementioned
"3-phase" process, though.

By the way, this change also fixes a possible write stall after rebalancing
which was caused by the lack of "stabilize delete" operations.
 2020-06-02 23:45:22 +03:00
Vitaliy Filippov 985c309d7f Remove duplicate code between blockstore_{rollback,stable} and blockstore_init 2020-06-02 20:37:00 +03:00
Vitaliy Filippov a56f8cd14e Simplify handle_primary_subop() arguments 2020-06-02 18:44:23 +03:00
Vitaliy Filippov 46e111272f Replace assert(this_it == cur_op) with if() for the case of PG repeering 2020-06-02 14:30:57 +03:00
Vitaliy Filippov 165c204555 Fix BS_OP_DELETE (the implementation was untested up to this point) 2020-06-02 14:26:01 +03:00
Vitaliy Filippov af5cd45071 Oh crap, got SIGPIPE. Add MSG_NOSIGNAL 2020-06-02 11:41:08 +03:00
Vitaliy Filippov c3fe9ad0d1 Fix rebalancing writes (add a forgotten state resume) 2020-06-02 01:26:14 +03:00
Vitaliy Filippov 0fcdeae18b Do not die if a peer is already stopped on flush error 2020-06-01 23:07:08 +03:00
Vitaliy Filippov e6a4b634f8 Fix possible write stall 
The stall occurred during fio Q=128 random write tests with low flusher_count (4).
It was caused by flushers being unable to flush the beginning of the journal
because it contained older writes to an object that also had writes in the very end
of the journal, after dirty_start.
 2020-06-01 16:18:23 +03:00
Vitaliy Filippov c22e096943 Output journal offsets in debug trace in hex, add detailed "still waiting" messages 2020-06-01 16:18:19 +03:00
Vitaliy Filippov 45b1c2fbf1 Fix canceling of write operations on PG re-peer (which led to use-after-free, too...) 2020-06-01 16:18:14 +03:00
Vitaliy Filippov 3469bead67 Protect "delete this" with a stack refcounter 
(to fix use-after-free, too, but "delete this" was a time bomb anyway)
 2020-06-01 16:18:09 +03:00
Vitaliy Filippov 3a5d488f19 Fix use-after-free in osd_flush.cpp 2020-06-01 01:56:24 +03:00
Vitaliy Filippov 73e4e30b1f Auto-generate C++ header dependencies 2020-06-01 00:25:25 +03:00
Vitaliy Filippov 5feff1ffb9 Slightly cleanup socket send/receive code 2020-05-31 15:03:27 +03:00
Vitaliy Filippov b466e215f0 Fix queued OP_SYNC execution 2020-05-27 13:55:25 +03:00
Vitaliy Filippov 36f995367f Fix bind_address reporting 2020-05-27 10:58:40 +03:00
Vitaliy Filippov 0aca6e9ca8 Extract peer connect and read-write loop into a separate file (to be shared with the client library) 2020-05-26 22:11:30 +03:00
Vitaliy Filippov fa98be6bc0 Allow to specify multiple etcd addresses 2020-05-25 16:30:05 +03:00
Vitaliy Filippov 256a7f2667 Free op->bs_op manually 2020-05-25 15:31:22 +03:00
Vitaliy Filippov 79bf57b6e2 Allow to override pg_stripe_size 2020-05-25 15:31:22 +03:00
Vitaliy Filippov 53f6aba3e6 Die when journal_sector_buffer_count is too small 2020-05-24 17:26:47 +03:00
Vitaliy Filippov 36595eb669 Print "Ran out of journal sector buffers" warning 2020-05-24 16:48:50 +03:00
Vitaliy Filippov e09d0e0678 Several bug fixes 
- Do not block flock() requests
- Fix stop_client(0) attempts leading to std::bad_function_call
- Fix degraded writes crashing due to an unset stripes[i].missing (at least with a missing parity device)
- Fix recovery B/W reporting
 2020-05-24 01:51:35 +03:00
Vitaliy Filippov d1602b50b3 Fix BS_OP_ROLLBACK removing an incorrect version 
Instead of only removing versions with oid == X and version > Y it was
also removing the previous version in list (with the previous oid or
with version == Y)
 2020-05-24 01:51:28 +03:00
Vitaliy Filippov 7df384031a Re-peer PGs after stopping the peer 
Fixes the bug where two peers killed at once have lead to PG state PG_DEGRADED|PG_HAS_INCOMPLETE instead of PG_INCOMPLETE
 2020-05-23 18:45:12 +03:00
Vitaliy Filippov e614a98543 Add a sad FIXME :-) 2020-05-23 15:43:37 +03:00
Vitaliy Filippov 01dd3ef89e Fix timerfd_manager triggering of multiple times at the same time 2020-05-23 15:43:37 +03:00
Vitaliy Filippov cdccc23aff Print [OSD $osd_num] in stats, print B/W only for ops that log bytes 2020-05-23 15:43:37 +03:00
Vitaliy Filippov 700428829a Fix autosync_interval default not setting when autosync_interval is skipped in config 2020-05-23 15:43:37 +03:00
Vitaliy Filippov 6488d0044a Ignore EPOLL_CTL_DEL ENOENT, fix detection of the rollback version 2020-05-23 15:43:37 +03:00
Vitaliy Filippov 393fe75900 Fix creepy (osd_op_t*)(long) casts 2020-05-23 15:43:37 +03:00
Vitaliy Filippov f036eecf1c Fix osd_rmw object recovery case (len==0) 2020-05-23 15:43:37 +03:00
Vitaliy Filippov e56909fb45 Remove tv_send (unused) and timerfd_interval from blockstore 2020-05-22 15:57:08 +03:00
Vitaliy Filippov fac75b0b57 Handle reweights in mon 2020-05-22 12:52:27 +03:00
Vitaliy Filippov 9f842ec9a5 Remove connect callback because it is always the same 2020-05-22 12:45:12 +03:00
Vitaliy Filippov f6a01a4819 Extract "state-watching" etcd client into a separate file 2020-05-22 12:38:40 +03:00
Vitaliy Filippov 6202260018 Extract HTTP client functions from osd_t 2020-05-21 11:39:01 +03:00
Vitaliy Filippov a61ede9951 Remove io_uring usage from osd_http and timerfd_manager 
For better future interoperability with external event loops such as QEMU's one
 2020-05-21 01:25:38 +03:00
Vitaliy Filippov f57731f8ca Calculate total stats in the monitor 2020-05-15 01:37:17 +03:00
Vitaliy Filippov 19f25c7cd5 Handle integer overflow of the op_stat_count 2020-05-15 01:37:17 +03:00
Vitaliy Filippov 2c3e84cc41 Implement stop_all_pgs() 2020-05-15 01:37:17 +03:00
Vitaliy Filippov 7bda66b866 Do not crash when optimising PGs in an undersized cluster 2020-05-15 01:29:15 +03:00
Vitaliy Filippov b467d0559f Begin node.js storage monitor service 2020-05-15 01:29:15 +03:00
Vitaliy Filippov c2c2eefea4 Duplicate host in osd/state and osd/stats, take PGs from /config/pgs.items 2020-05-15 01:29:15 +03:00
Vitaliy Filippov 5084ff7c6c Measure & report recovery op count and bandwidth 2020-05-15 01:29:15 +03:00
Vitaliy Filippov 47b6f64106 Support level names 2020-05-11 15:57:21 +03:00
Vitaliy Filippov f71d0c117b Measure & report op bandwidth, include local blockstore ops in stats 2020-05-11 02:58:13 +03:00
Vitaliy Filippov 2b854948f9 Remove dead code 2020-05-09 16:15:02 +03:00
Vitaliy Filippov e7f897ed65 Report hostname to etcd 2020-05-09 02:33:43 +03:00
Vitaliy Filippov c26b6e1fc3 Support CRUSH-like multi-level placement trees 2020-05-09 00:55:24 +03:00
Vitaliy Filippov aaa054e644 Fix optimize_change generating infeasible problems 
Mainly happened when removing PG combinations (removing OSDs)

Also randomize OSD combinations when there's a lot of them

Also remove Perl version
 2020-05-08 16:42:40 +03:00
Vitaliy Filippov 706a44d4d4 Fix optimize_initial in both perl and js versions 2020-05-06 23:12:03 +03:00
Vitaliy Filippov 842f88f94f Rewrite LPOptimizer.pm to nodejs 2020-05-06 02:08:15 +03:00
Vitaliy Filippov e8149e5848 Implement OSD_OP_DELETE 2020-05-05 00:39:51 +03:00
Vitaliy Filippov 6355b968f4 Track osd_set history and all_peers separately 2020-05-04 15:28:07 +03:00
Vitaliy Filippov 00cf24fbd7 Split osd_primary.cpp 2020-05-03 11:04:20 +03:00
Vitaliy Filippov 1bc08174f9 Sync before listing objects so flushes do not fail thereafter 2020-05-01 12:56:49 +03:00
Vitaliy Filippov cd87333091 Fix PG state comparison leading to unclean PGs not flushing 
(a & b == b) -> ((a & b) == b) !
 2020-05-01 12:56:46 +03:00
Vitaliy Filippov bd0fe6e4cc Fix PGs not stopping during sync, fix state reporting autovivification of erased PGs 2020-05-01 01:33:14 +03:00
Vitaliy Filippov ce78454215 Reply with -EROFS to write commands in readonly mode 2020-05-01 00:54:34 +03:00
Vitaliy Filippov 762bd42096 Fix use-after-free caused by "delete this" in handle_read 2020-04-30 02:15:53 +03:00
Vitaliy Filippov 7b57eeeeb3 Implement PG state locking and PG moving in response to etcd events 2020-04-29 22:23:38 +03:00
Vitaliy Filippov ec4a52af48 Fix websocket (and timer!) bugs 2020-04-26 01:59:56 +03:00
Vitaliy Filippov 268b497c0b Implement simple websocket client 2020-04-25 23:11:50 +03:00
Vitaliy Filippov 35481925b1 Implement very simple HTTP streaming to handle etcd watches 2020-04-25 01:35:52 +03:00
Vitaliy Filippov 895a80dfc4 Fix etcd 3.2 compatibility (no compare.target == LEASE, /kv/lease/revoke), fix small bugs 2020-04-25 01:35:52 +03:00
Vitaliy Filippov caa01c6aaf Acquire etcd leases, prevent starting two OSDs with the same number 2020-04-25 01:35:52 +03:00
Vitaliy Filippov d398ddfd3b Use snake_case for etcd requests 2020-04-25 01:35:52 +03:00
Vitaliy Filippov 0f2b8dbf6f Use a single timerfd_manager for all timers 2020-04-25 01:35:49 +03:00
Vitaliy Filippov 4f42e9659e Use etcd instead of Consul 2020-04-24 01:03:55 +03:00
Vitaliy Filippov 7cf71a8031 Fix timerfd_manager: remove timer, then call callback 2020-04-21 12:45:18 +03:00
Vitaliy Filippov 9d22559bcf Start peering immediately when loading PGs 2020-04-21 02:27:13 +03:00
Vitaliy Filippov 8c03e3ebab Lock Blockstore devices exclusively by default 2020-04-21 01:59:11 +03:00
Vitaliy Filippov 2a640ba2e8 Remove range port selection (leads to races) 2020-04-21 00:10:59 +03:00
Vitaliy Filippov 6a21ea207e Check peer config (at least, number) after connecting 2020-04-21 00:08:54 +03:00
Vitaliy Filippov 642802b595 Auto-select port numbers 2020-04-20 17:45:27 +03:00
Vitaliy Filippov ff38b464a5 Add consul & connect timeouts, report state before loading PGs, move init_primary to osd_cluster 2020-04-20 15:43:07 +03:00
Vitaliy Filippov 663153713b Reconnect to peers after connecting drops 2020-04-19 01:01:26 +03:00
Vitaliy Filippov dc57c5c362 Report PG states again, clear PG history on reaching active+clean 2020-04-19 00:48:23 +03:00
Vitaliy Filippov f95299b769 Take PG history into account when starting PGs 2020-04-19 00:20:18 +03:00
Vitaliy Filippov 9126ffb0f9 Fix PG loading - now it works, at least once 2020-04-17 02:33:44 +03:00
Vitaliy Filippov 2a8e40835e Fix reporting to Consul, report even if we are purely secondary 2020-04-17 01:59:06 +03:00
Vitaliy Filippov 309486d746 Implement loading PGs from Consul (in theory) 2020-04-16 23:22:32 +03:00
Vitaliy Filippov 582f485578 Extract http & getifaddr_list into a separate file 2020-04-15 15:47:06 +03:00
Vitaliy Filippov 089b4eb208 Retry consul connection attempts and then die 2020-04-15 15:33:18 +03:00
Vitaliy Filippov d78ce509c6 Add simple timer manager 2020-04-15 13:41:44 +03:00
Vitaliy Filippov f3a7ccff50 Use 4K blockstore block by default, use MEM_ALIGNMENT in osd code 2020-04-14 19:19:56 +03:00
Vitaliy Filippov 37b27c3025 Implement basic OSD status reporting to Consul 2020-04-14 14:52:06 +03:00
Vitaliy Filippov edf6d6f897 Fix http_request 2020-04-12 02:08:00 +03:00
Vitaliy Filippov d11e8dcb5e Do not flush or recover in readonly mode 2020-04-11 12:06:18 +03:00
Vitaliy Filippov dd02bc1c44 Add base64 implementation 2020-04-11 12:06:18 +03:00