thebestvibedev/tromcho.net
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
331 Commits 74 Branches 106 Tags
e56909fb450ff90ac7a4ee8a4eb3485783cf2fec
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Vitaliy Filippov e56909fb45 Remove tv_send (unused) and timerfd_interval from blockstore 2020-05-22 15:57:08 +03:00
Vitaliy Filippov fac75b0b57 Handle reweights in mon 2020-05-22 12:52:27 +03:00
Vitaliy Filippov 9f842ec9a5 Remove connect callback because it is always the same 2020-05-22 12:45:12 +03:00
Vitaliy Filippov f6a01a4819 Extract "state-watching" etcd client into a separate file 2020-05-22 12:38:40 +03:00
Vitaliy Filippov 6202260018 Extract HTTP client functions from osd_t 2020-05-21 11:39:01 +03:00
Vitaliy Filippov a61ede9951 Remove io_uring usage from osd_http and timerfd_manager 
For better future interoperability with external event loops such as QEMU's one
 2020-05-21 01:25:38 +03:00
Vitaliy Filippov f57731f8ca Calculate total stats in the monitor 2020-05-15 01:37:17 +03:00
Vitaliy Filippov 19f25c7cd5 Handle integer overflow of the op_stat_count 2020-05-15 01:37:17 +03:00
Vitaliy Filippov 2c3e84cc41 Implement stop_all_pgs() 2020-05-15 01:37:17 +03:00
Vitaliy Filippov 7bda66b866 Do not crash when optimising PGs in an undersized cluster 2020-05-15 01:29:15 +03:00
Vitaliy Filippov b467d0559f Begin node.js storage monitor service 2020-05-15 01:29:15 +03:00
Vitaliy Filippov c2c2eefea4 Duplicate host in osd/state and osd/stats, take PGs from /config/pgs.items 2020-05-15 01:29:15 +03:00
Vitaliy Filippov 5084ff7c6c Measure & report recovery op count and bandwidth 2020-05-15 01:29:15 +03:00
Vitaliy Filippov 47b6f64106 Support level names 2020-05-11 15:57:21 +03:00
Vitaliy Filippov f71d0c117b Measure & report op bandwidth, include local blockstore ops in stats 2020-05-11 02:58:13 +03:00
Vitaliy Filippov 2b854948f9 Remove dead code 2020-05-09 16:15:02 +03:00
Vitaliy Filippov e7f897ed65 Report hostname to etcd 2020-05-09 02:33:43 +03:00
Vitaliy Filippov c26b6e1fc3 Support CRUSH-like multi-level placement trees 2020-05-09 00:55:24 +03:00
Vitaliy Filippov aaa054e644 Fix optimize_change generating infeasible problems 
Mainly happened when removing PG combinations (removing OSDs)

Also randomize OSD combinations when there's a lot of them

Also remove Perl version
 2020-05-08 16:42:40 +03:00
Vitaliy Filippov 706a44d4d4 Fix optimize_initial in both perl and js versions 2020-05-06 23:12:03 +03:00
Vitaliy Filippov 842f88f94f Rewrite LPOptimizer.pm to nodejs 2020-05-06 02:08:15 +03:00
Vitaliy Filippov e8149e5848 Implement OSD_OP_DELETE 2020-05-05 00:39:51 +03:00
Vitaliy Filippov 6355b968f4 Track osd_set history and all_peers separately 2020-05-04 15:28:07 +03:00
Vitaliy Filippov 00cf24fbd7 Split osd_primary.cpp 2020-05-03 11:04:20 +03:00
Vitaliy Filippov 1bc08174f9 Sync before listing objects so flushes do not fail thereafter 2020-05-01 12:56:49 +03:00
Vitaliy Filippov cd87333091 Fix PG state comparison leading to unclean PGs not flushing 
(a & b == b) -> ((a & b) == b) !
 2020-05-01 12:56:46 +03:00
Vitaliy Filippov bd0fe6e4cc Fix PGs not stopping during sync, fix state reporting autovivification of erased PGs 2020-05-01 01:33:14 +03:00
Vitaliy Filippov ce78454215 Reply with -EROFS to write commands in readonly mode 2020-05-01 00:54:34 +03:00
Vitaliy Filippov 762bd42096 Fix use-after-free caused by "delete this" in handle_read 2020-04-30 02:15:53 +03:00
Vitaliy Filippov 7b57eeeeb3 Implement PG state locking and PG moving in response to etcd events 2020-04-29 22:23:38 +03:00
Vitaliy Filippov ec4a52af48 Fix websocket (and timer!) bugs 2020-04-26 01:59:56 +03:00
Vitaliy Filippov 268b497c0b Implement simple websocket client 2020-04-25 23:11:50 +03:00
Vitaliy Filippov 35481925b1 Implement very simple HTTP streaming to handle etcd watches 2020-04-25 01:35:52 +03:00
Vitaliy Filippov 895a80dfc4 Fix etcd 3.2 compatibility (no compare.target == LEASE, /kv/lease/revoke), fix small bugs 2020-04-25 01:35:52 +03:00
Vitaliy Filippov caa01c6aaf Acquire etcd leases, prevent starting two OSDs with the same number 2020-04-25 01:35:52 +03:00
Vitaliy Filippov d398ddfd3b Use snake_case for etcd requests 2020-04-25 01:35:52 +03:00
Vitaliy Filippov 0f2b8dbf6f Use a single timerfd_manager for all timers 2020-04-25 01:35:49 +03:00
Vitaliy Filippov 4f42e9659e Use etcd instead of Consul 2020-04-24 01:03:55 +03:00
Vitaliy Filippov 7cf71a8031 Fix timerfd_manager: remove timer, then call callback 2020-04-21 12:45:18 +03:00
Vitaliy Filippov 9d22559bcf Start peering immediately when loading PGs 2020-04-21 02:27:13 +03:00
Vitaliy Filippov 8c03e3ebab Lock Blockstore devices exclusively by default 2020-04-21 01:59:11 +03:00
Vitaliy Filippov 2a640ba2e8 Remove range port selection (leads to races) 2020-04-21 00:10:59 +03:00
Vitaliy Filippov 6a21ea207e Check peer config (at least, number) after connecting 2020-04-21 00:08:54 +03:00
Vitaliy Filippov 642802b595 Auto-select port numbers 2020-04-20 17:45:27 +03:00
Vitaliy Filippov ff38b464a5 Add consul & connect timeouts, report state before loading PGs, move init_primary to osd_cluster 2020-04-20 15:43:07 +03:00
Vitaliy Filippov 663153713b Reconnect to peers after connecting drops 2020-04-19 01:01:26 +03:00
Vitaliy Filippov dc57c5c362 Report PG states again, clear PG history on reaching active+clean 2020-04-19 00:48:23 +03:00
Vitaliy Filippov f95299b769 Take PG history into account when starting PGs 2020-04-19 00:20:18 +03:00
Vitaliy Filippov 9126ffb0f9 Fix PG loading - now it works, at least once 2020-04-17 02:33:44 +03:00
Vitaliy Filippov 2a8e40835e Fix reporting to Consul, report even if we are purely secondary 2020-04-17 01:59:06 +03:00
Vitaliy Filippov 309486d746 Implement loading PGs from Consul (in theory) 2020-04-16 23:22:32 +03:00
Vitaliy Filippov 582f485578 Extract http & getifaddr_list into a separate file 2020-04-15 15:47:06 +03:00
Vitaliy Filippov 089b4eb208 Retry consul connection attempts and then die 2020-04-15 15:33:18 +03:00
Vitaliy Filippov d78ce509c6 Add simple timer manager 2020-04-15 13:41:44 +03:00
Vitaliy Filippov f3a7ccff50 Use 4K blockstore block by default, use MEM_ALIGNMENT in osd code 2020-04-14 19:19:56 +03:00
Vitaliy Filippov 37b27c3025 Implement basic OSD status reporting to Consul 2020-04-14 14:52:06 +03:00
Vitaliy Filippov edf6d6f897 Fix http_request 2020-04-12 02:08:00 +03:00
Vitaliy Filippov d11e8dcb5e Do not flush or recover in readonly mode 2020-04-11 12:06:18 +03:00
Vitaliy Filippov dd02bc1c44 Add base64 implementation 2020-04-11 12:06:18 +03:00
Vitaliy Filippov 298b013eae Add simple http request function 2020-04-11 12:05:58 +03:00
Vitaliy Filippov 0880a77c1a 2 FIXME for the future 2020-04-06 00:55:47 +03:00
Vitaliy Filippov aa849ea07b Add a test for missing chunk overwrite 2020-04-05 16:14:03 +03:00
Vitaliy Filippov d59be0e8b4 Delete misplaced chunks after moving the object, reset object state in primary_write 2020-04-05 15:51:22 +03:00
Vitaliy Filippov cf7de0f181 (Almost) Implement misplaced recovery, integrating it into calc_rmw() 2020-04-05 15:50:53 +03:00
Vitaliy Filippov 6212195440 Implement parallel recovery 2020-04-04 19:23:12 +03:00
Vitaliy Filippov dfb6e15eaa Implement graceful stopping of PGs 2020-04-03 13:03:42 +03:00
Vitaliy Filippov afe2e76c87 Implement regular automatic syncs, split osd_t constructor into some methods 2020-04-02 22:16:46 +03:00
Vitaliy Filippov 0f43f6d3f6 Fix crashes, print some stats 
Notably:
- fix the `delete op` inside lambda callback crash (it frees the lambda itself
  which results in use-after-free with g++)
- fix stop_client() reenterability
- fix a bug in the blockstore layer which resulted in always returning version=0
  for zero-length reads
- change error codes for blockstore_stabilize
 2020-03-31 17:55:31 +03:00
Vitaliy Filippov 92c800bb64 Forget unstable writes when re-peering, rename parity_block_size -> pg_stripe_size, pg_parity_size -> pg_block_size 2020-03-31 02:09:25 +03:00
Vitaliy Filippov 8a8b619875 Handle secondary OSD connection errors [in theory] 2020-03-30 19:51:34 +03:00
Vitaliy Filippov 43fe1d88e7 Fix memory leaks with subops, fix recovery crashes 2020-03-28 19:09:20 +03:00
Vitaliy Filippov 1b30120918 Fix stripe reconstruction in recovery, only write modified object parts 2020-03-28 13:58:42 +03:00
Vitaliy Filippov c0a22d825d Fix degraded object recovery (it seems to work now) 2020-03-25 02:17:41 +03:00
Vitaliy Filippov 7acfc95f75 CONFIG_HAVE_GETTID 2020-03-25 01:20:20 +03:00
Vitaliy Filippov 250f22c0b6 Implement basic degraded object recovery (integrated into primary_write) 2020-03-25 01:17:50 +03:00
Vitaliy Filippov dbd8418798 Reply using a single finish_op() method, allow to call OSD ops from inside the OSD 2020-03-24 00:18:52 +03:00
Vitaliy Filippov 036f4c5bf3 Fix unstable flushing, include extra OSDs with old object versions in osd_set 2020-03-23 20:28:47 +03:00
Vitaliy Filippov fd8e1a8418 Slightly reorganize object state check code 2020-03-23 00:42:17 +03:00
Vitaliy Filippov a08e0bfacd Treat misplaced and degraded as separate state parts 2020-03-23 00:40:31 +03:00
Vitaliy Filippov ddc3e927d3 Solve it in integers 2020-03-20 13:58:54 +03:00
Vitaliy Filippov 2aa605f2bb Do not check 2020-03-20 13:38:35 +03:00
Vitaliy Filippov 18915b264a Extract to .pm + fix all_combinations 2020-03-19 21:35:47 +03:00
Vitaliy Filippov 60f795e7eb Add lp_solve based data distribution optimizer 2020-03-19 17:23:24 +03:00
Vitaliy Filippov 3a4279adbf Hash-based PG distribution experiments 2020-03-17 18:52:39 +03:00
Vitaliy Filippov 1ec9794376 Extract flushing into a separate file 2020-03-15 18:39:31 +03:00
Vitaliy Filippov d8164e9d84 Print PG states on every change 2020-03-14 22:19:45 +03:00
Vitaliy Filippov 21d0b06959 Implement flushing (stabilize/rollback) of unstable entries on start of the PG 2020-03-14 02:49:34 +03:00
Vitaliy Filippov 46f9bd2a69 Make blockstore list operation return consistent snapshots 2020-03-14 02:10:25 +03:00
Vitaliy Filippov 6982fe1255 Do not block reads by previous unfinished writes 2020-03-13 21:28:49 +03:00
Vitaliy Filippov eba053febe Do not start small writes before finishing the last big write to the same object 2020-03-12 02:15:01 +03:00
Vitaliy Filippov 899946ff96 Add osd_test function to unblock an OSD blocked by the lack of journal space 2020-03-10 17:19:24 +03:00
Vitaliy Filippov 3dd1b22d55 Fix segfault with concurrent OP_SYNCs 2020-03-10 17:00:23 +03:00
Vitaliy Filippov 31f9445030 Use immediate_commit to benefit the primary OSD 2020-03-10 02:20:16 +03:00
Vitaliy Filippov 3f522c66e6 Implement immediate commit mode 2020-03-10 01:59:15 +03:00
Vitaliy Filippov c3737ae3ff Add journal fsync to stabilize/rollback 2020-03-09 00:35:58 +03:00
Vitaliy Filippov c863543bfe Fix possible journal corruption caused by concurrent flushing and writing of the same journal sector 2020-03-08 01:21:19 +03:00
Vitaliy Filippov 1696446545 Rename min/max _used to _flushed 2020-03-07 16:41:58 +03:00
Vitaliy Filippov 41dddddbf2 Fix some logging 2020-03-07 16:41:53 +03:00
Vitaliy Filippov 2d4e24c9ce Add journal dumper debugging tool 2020-03-06 02:29:43 +03:00
Vitaliy Filippov 844cacd357 Allow incorrectly forbidden BS_OP_LIST in readonly mode 2020-03-06 02:29:39 +03:00