thebestvibedev/tromcho.net
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
246 Commits 74 Branches 106 Tags
d8164e9d844052f3e1ad33ea5dd778eabc2fd3d2
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Vitaliy Filippov d8164e9d84 Print PG states on every change 2020-03-14 22:19:45 +03:00
Vitaliy Filippov 21d0b06959 Implement flushing (stabilize/rollback) of unstable entries on start of the PG 2020-03-14 02:49:34 +03:00
Vitaliy Filippov 46f9bd2a69 Make blockstore list operation return consistent snapshots 2020-03-14 02:10:25 +03:00
Vitaliy Filippov 6982fe1255 Do not block reads by previous unfinished writes 2020-03-13 21:28:49 +03:00
Vitaliy Filippov eba053febe Do not start small writes before finishing the last big write to the same object 2020-03-12 02:15:01 +03:00
Vitaliy Filippov 899946ff96 Add osd_test function to unblock an OSD blocked by the lack of journal space 2020-03-10 17:19:24 +03:00
Vitaliy Filippov 3dd1b22d55 Fix segfault with concurrent OP_SYNCs 2020-03-10 17:00:23 +03:00
Vitaliy Filippov 31f9445030 Use immediate_commit to benefit the primary OSD 2020-03-10 02:20:16 +03:00
Vitaliy Filippov 3f522c66e6 Implement immediate commit mode 2020-03-10 01:59:15 +03:00
Vitaliy Filippov c3737ae3ff Add journal fsync to stabilize/rollback 2020-03-09 00:35:58 +03:00
Vitaliy Filippov c863543bfe Fix possible journal corruption caused by concurrent flushing and writing of the same journal sector 2020-03-08 01:21:19 +03:00
Vitaliy Filippov 1696446545 Rename min/max _used to _flushed 2020-03-07 16:41:58 +03:00
Vitaliy Filippov 41dddddbf2 Fix some logging 2020-03-07 16:41:53 +03:00
Vitaliy Filippov 2d4e24c9ce Add journal dumper debugging tool 2020-03-06 02:29:43 +03:00
Vitaliy Filippov 844cacd357 Allow incorrectly forbidden BS_OP_LIST in readonly mode 2020-03-06 02:29:39 +03:00
Vitaliy Filippov e19d9fde5f Fix peering_pg, begin tests 2020-03-06 02:02:49 +03:00
Vitaliy Filippov 9cb07d844b Make [un]register_consumer operate on pointers, rename get_loop_again() to has_work() 2020-03-04 21:00:20 +03:00
Vitaliy Filippov 1e21555343 Add FIXME with Oops 2020-03-04 20:34:45 +03:00
Vitaliy Filippov 94cdbcd085 Stop reading when less than <buffer> data is available 2020-03-04 18:03:16 +03:00
Vitaliy Filippov 8315407558 Incoming data pre-buffering 2020-03-04 17:34:45 +03:00
Vitaliy Filippov b27ad550cf Use btree_map instead of sparsepp 2020-03-04 17:12:27 +03:00
Vitaliy Filippov 8e63995306 Allow to specify data area size 2020-03-04 02:32:49 +03:00
Vitaliy Filippov c9f3654905 Add ringloop.h to dependencies 2020-03-03 17:23:33 +03:00
Vitaliy Filippov 2a5ca4ff6f Measure OP_STAB in both ways 2020-03-03 02:18:59 +03:00
Vitaliy Filippov 2575431176 Fix EPOLLET race on Linux 5.5+ 2020-03-03 01:47:56 +03:00
Vitaliy Filippov 20125db181 Use clock_gettime() 2020-03-03 00:54:42 +03:00
Vitaliy Filippov 7eac7b6d55 Use writev in libfio_sec_osd 2020-03-02 21:17:21 +03:00
Vitaliy Filippov 79839ec31d Start sending immediately instead of waiting for another loop 2020-03-02 00:20:28 +03:00
Vitaliy Filippov 9d96e4bf0b Add simple "stub bench" using sync I/O 2020-03-01 23:00:46 +03:00
Vitaliy Filippov 9e18293497 Allow to specify multiple peers 2020-03-01 00:46:07 +03:00
Vitaliy Filippov 56765ab750 Send all iovecs at once 2020-02-29 02:27:19 +03:00
Vitaliy Filippov fd05e13bc4 Use EPOLLET 
Its latency is slightly better, too
 2020-02-29 01:56:59 +03:00
Vitaliy Filippov c41fd7ea18 Measure sending subops with data 2020-02-29 01:46:03 +03:00
Vitaliy Filippov c6334afc94 Measure OSD op/subop latency 
Something is wrong: loopback RTT between OSDs is sometimes as high as 70us (should be 20us or less probably)
 2020-02-28 12:26:49 +03:00
Vitaliy Filippov 2be4824a7a Fix a small memory leak and BS_OP_SYNC mishandling, now fio does not hang during primary-osd test 2020-02-28 01:46:39 +03:00
Vitaliy Filippov 1733de2db6 Test & fix single-PG primary OSD 
- Add support for benchmarking single primary OSD in fio_sec_osd
- Do not wait for the next event in flushers (return resume_0 back)
- Fix flushing of zero-length writes
- Print PG object count when peering
- Print journal free space when starting and when congested
 2020-02-26 19:05:29 +03:00
Vitaliy Filippov df66a76ce2 ...and make it work :) 2020-02-25 22:52:03 +03:00
Vitaliy Filippov a406c62a71 Implement basic primary-sync-stabilize 2020-02-25 20:10:21 +03:00
Vitaliy Filippov 74673c761f Make basic primary-write work 2020-02-25 02:55:58 +03:00
Vitaliy Filippov 09588a349f Transform primary_r/w into "coroutines" 2020-02-24 02:40:52 +03:00
Vitaliy Filippov 4c0178f180 Fix some memory freeing 2020-02-24 01:04:23 +03:00
Vitaliy Filippov 5dd04abbac Make bs_op pointer 2020-02-23 23:46:00 +03:00
Vitaliy Filippov fe3c47812c Rename osd_exec_secondary to osd_secondary 2020-02-23 23:43:57 +03:00
Vitaliy Filippov 88e56a564f Rename osd_read_stripe_t to osd_rmw_stripe_t 2020-02-23 23:43:57 +03:00
Vitaliy Filippov c71b67f2f7 Move SYNC_STAB_ALL into blockstore implementation 2020-02-23 23:43:57 +03:00
Vitaliy Filippov 4a52a15564 Rename osd_op_t.op to req 2020-02-23 23:21:17 +03:00
Vitaliy Filippov 72a89be912 Move uint8_t[] buffers into any_op_t/any_reply_t 2020-02-23 23:21:17 +03:00
Vitaliy Filippov d4fd9d982a Implement read-modify-write calculation and extract it into a separate file 2020-02-23 02:11:43 +03:00
Vitaliy Filippov 0e177c7a2a Fix possible leak in blockstore_read 2020-02-21 19:38:43 +03:00
Vitaliy Filippov ffe073473a Remove hardcode of the EC(2+1) scheme, now it supports EC(k+1), fix some bugs 2020-02-13 19:13:17 +03:00
Vitaliy Filippov b7ccd63104 Return version number from reads 2020-02-12 12:30:50 +03:00
Vitaliy Filippov 1100ad44bd Cancel outbound operations when disconnecting a peer 2020-02-11 21:17:10 +03:00
Vitaliy Filippov a66b34e04d Implement event-driven PG peering 2020-02-11 13:41:34 +03:00
Vitaliy Filippov 327f310868 Remove copy-pasted read/write _blocking 2020-02-09 19:23:03 +03:00
Vitaliy Filippov 1513d0490a Test and fix degraded-read 2020-02-09 19:17:35 +03:00
Vitaliy Filippov 97d3fc593c Test and fix primary-read 2020-02-09 19:17:32 +03:00
Vitaliy Filippov 235d15422c Mostly finish primary-OSD-read 2020-02-03 14:18:21 +03:00
Vitaliy Filippov 9fb2d3f840 Fill out the rest of the degraded read logic; now we need to make it a "coroutine" 2020-02-02 00:05:56 +03:00
Vitaliy Filippov 206c4eb655 WIP submit subops 2020-01-31 14:07:16 +03:00
Vitaliy Filippov 74d0196ba5 Stripe read reconstruction 2020-01-31 13:46:23 +03:00
Vitaliy Filippov ec50957c41 WIP primary OSD read 2020-01-30 22:06:46 +03:00
Vitaliy Filippov f4707709c6 Move ack_sync() near sync_state = SYNC_DONE so it does not hang with disable_journal_fsync 2020-01-29 16:55:06 +03:00
Vitaliy Filippov 1e286eed08 Wait for writes to complete before issuing an fsync in blockstore_sync 
Also fix a dormant bug (OP_SYNC could clear unsynced_*_writes and not be added into syncs_in_progress)
 2020-01-29 16:42:40 +03:00
Vitaliy Filippov dcc9e75c63 Wait for write completion before fsync in blockstore_init 2020-01-29 16:40:21 +03:00
Vitaliy Filippov 47663bd1dc Add (empty) osd_primary.cpp, rename osd_read to osd_receive, add FIXMEs for fsync 2020-01-28 22:40:50 +03:00
Vitaliy Filippov 1447c44b68 Calculate required stabilize/rollback actions, add more map perf tests 2020-01-27 01:34:30 +03:00
Vitaliy Filippov 079f129390 Add rollback op to the OSD code 2020-01-25 02:04:58 +03:00
Vitaliy Filippov 2b09710d6f Implement blockstore rollback operation 
Rollback operation is required for the primary OSD to kill unstable
object versions in OSD peers so they don't occupy journal space
 2020-01-24 20:18:14 +03:00
Vitaliy Filippov 98efdb78bd Extract object state calculation to a separate file and slightly test it 2020-01-24 12:56:51 +03:00
Vitaliy Filippov d2a3f0c6dd Begin object state calculation 2020-01-23 22:05:27 +03:00
Vitaliy Filippov a8bc44064d Read object lists from peers and own blockstore 2020-01-22 02:36:14 +03:00
Vitaliy Filippov 8c05ee252c Test object list sorting, begin peering code 2020-01-21 22:33:02 +03:00
Vitaliy Filippov d0ab2a20b2 Make fsync flags separate for data, metadata and journal 2020-01-17 13:41:37 +03:00
Vitaliy Filippov d5386aa958 Check presence in sync_to_repeat 2020-01-16 02:16:35 +03:00
Vitaliy Filippov 0d90966739 Add last_sync checking to fio_engine, too 2020-01-16 01:03:18 +03:00
Vitaliy Filippov 43f6cfeb73 Extract alignments to options 2020-01-16 00:54:25 +03:00
Vitaliy Filippov 57ecbb2cda Parse OSD commandline options 2020-01-15 23:20:11 +03:00
Vitaliy Filippov 8ea1ccc192 Add an OSD stub to compare sync socket I/O with io_uring + skip multiple fsyncs that fio issues 2020-01-15 22:04:58 +03:00
Vitaliy Filippov a3d3949dce Do not overwrite same journal sector multiple times 
It doesn't reduce actual WA, but it reduces tail latency (Q=32, 10% / 50% / 90% / 99% / 99.95%):
- write: 766us/979us/1090us/1303us/1729us vs 1074us/1450us/2212us/3261us/4113us
- sync: 701us/881us/1188us/1762us/2540us vs 269us/955us/1663us/2638us/4146us
 2020-01-15 02:53:01 +03:00
Vitaliy Filippov 111516381f Add FIXME 2020-01-14 18:41:56 +03:00
Vitaliy Filippov 36d8c8724f Fix sparse reads using bitmap, fix journal replay (we could sometimes lose its end) 2020-01-12 23:38:33 +03:00
Vitaliy Filippov 5739b02de8 Add sparse read to ./blockstore_test, link blockstore to libblockstore.so 2020-01-12 19:59:19 +03:00
Vitaliy Filippov cf819eb442 Implement sparse block bitmap to avoid zero-fill 2020-01-12 02:55:32 +03:00
Vitaliy Filippov 4b05bde3a2 Block writes earlier than sync/stabilize would be blocked, too 2020-01-10 20:05:17 +03:00
Vitaliy Filippov b3f2102f33 Add queue stall tracking 2020-01-10 01:23:46 +03:00
Vitaliy Filippov 522a9db0e2 Enable TCP_NODELAY 2020-01-09 20:32:58 +03:00
Vitaliy Filippov a1550bdfac EPOLL_CTL_DEL does not require &ev 2020-01-09 18:39:58 +03:00
Vitaliy Filippov 7dba43b3c6 Allow to exclude blockstore from OSD op processing for tests 2020-01-08 12:06:05 +03:00
Vitaliy Filippov ba23824561 Allow to disable zero fill 2020-01-06 21:02:15 +03:00
Vitaliy Filippov bf3eecc159 Extract 512 to constants 2020-01-06 14:11:47 +03:00
Vitaliy Filippov f3e3f8f005 Fix colocated offsets 2020-01-06 02:57:08 +03:00
Vitaliy Filippov 2709574c93 Add offset options 2020-01-06 02:12:56 +03:00
Vitaliy Filippov d35ce8ab3d Fix compile errors for newer gcc 2020-01-06 02:09:59 +03:00
Vitaliy Filippov b6f0c1cde5 Async connection to peers 2020-01-04 01:23:25 +03:00
Vitaliy Filippov b87092fcfe Split osd.cpp into multiple files 2019-12-28 01:25:55 +03:00
Vitaliy Filippov 3134b7729a Add PG structures, begin peer connection handling code 2019-12-27 12:05:44 +03:00
Vitaliy Filippov 8a386270bd Use json11, add show_config operation, begin primary operations 2019-12-23 21:56:03 +03:00
Vitaliy Filippov 4677ace4cc Allow zero-length overwrites 2019-12-21 19:04:36 +03:00
Vitaliy Filippov 4de5290619 Implement safe(?) stop 2019-12-19 22:16:04 +03:00
Vitaliy Filippov 5ebc110d35 Implement LIST operation in the OSD 2019-12-19 22:04:18 +03:00