thebestvibedev/tromcho.net
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
174 Commits 74 Branches 106 Tags
d0ab2a20b2fef3fcaeb23dfdafb6a07444513705
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Vitaliy Filippov d0ab2a20b2 Make fsync flags separate for data, metadata and journal 2020-01-17 13:41:37 +03:00
Vitaliy Filippov d5386aa958 Check presence in sync_to_repeat 2020-01-16 02:16:35 +03:00
Vitaliy Filippov 0d90966739 Add last_sync checking to fio_engine, too 2020-01-16 01:03:18 +03:00
Vitaliy Filippov 43f6cfeb73 Extract alignments to options 2020-01-16 00:54:25 +03:00
Vitaliy Filippov 57ecbb2cda Parse OSD commandline options 2020-01-15 23:20:11 +03:00
Vitaliy Filippov 8ea1ccc192 Add an OSD stub to compare sync socket I/O with io_uring + skip multiple fsyncs that fio issues 2020-01-15 22:04:58 +03:00
Vitaliy Filippov a3d3949dce Do not overwrite same journal sector multiple times 
It doesn't reduce actual WA, but it reduces tail latency (Q=32, 10% / 50% / 90% / 99% / 99.95%):
- write: 766us/979us/1090us/1303us/1729us vs 1074us/1450us/2212us/3261us/4113us
- sync: 701us/881us/1188us/1762us/2540us vs 269us/955us/1663us/2638us/4146us
 2020-01-15 02:53:01 +03:00
Vitaliy Filippov 111516381f Add FIXME 2020-01-14 18:41:56 +03:00
Vitaliy Filippov 36d8c8724f Fix sparse reads using bitmap, fix journal replay (we could sometimes lose its end) 2020-01-12 23:38:33 +03:00
Vitaliy Filippov 5739b02de8 Add sparse read to ./blockstore_test, link blockstore to libblockstore.so 2020-01-12 19:59:19 +03:00
Vitaliy Filippov cf819eb442 Implement sparse block bitmap to avoid zero-fill 2020-01-12 02:55:32 +03:00
Vitaliy Filippov 4b05bde3a2 Block writes earlier than sync/stabilize would be blocked, too 2020-01-10 20:05:17 +03:00
Vitaliy Filippov b3f2102f33 Add queue stall tracking 2020-01-10 01:23:46 +03:00
Vitaliy Filippov 522a9db0e2 Enable TCP_NODELAY 2020-01-09 20:32:58 +03:00
Vitaliy Filippov a1550bdfac EPOLL_CTL_DEL does not require &ev 2020-01-09 18:39:58 +03:00
Vitaliy Filippov 7dba43b3c6 Allow to exclude blockstore from OSD op processing for tests 2020-01-08 12:06:05 +03:00
Vitaliy Filippov ba23824561 Allow to disable zero fill 2020-01-06 21:02:15 +03:00
Vitaliy Filippov bf3eecc159 Extract 512 to constants 2020-01-06 14:11:47 +03:00
Vitaliy Filippov f3e3f8f005 Fix colocated offsets 2020-01-06 02:57:08 +03:00
Vitaliy Filippov 2709574c93 Add offset options 2020-01-06 02:12:56 +03:00
Vitaliy Filippov d35ce8ab3d Fix compile errors for newer gcc 2020-01-06 02:09:59 +03:00
Vitaliy Filippov b6f0c1cde5 Async connection to peers 2020-01-04 01:23:25 +03:00
Vitaliy Filippov b87092fcfe Split osd.cpp into multiple files 2019-12-28 01:25:55 +03:00
Vitaliy Filippov 3134b7729a Add PG structures, begin peer connection handling code 2019-12-27 12:05:44 +03:00
Vitaliy Filippov 8a386270bd Use json11, add show_config operation, begin primary operations 2019-12-23 21:56:03 +03:00
Vitaliy Filippov 4677ace4cc Allow zero-length overwrites 2019-12-21 19:04:36 +03:00
Vitaliy Filippov 4de5290619 Implement safe(?) stop 2019-12-19 22:16:04 +03:00
Vitaliy Filippov 5ebc110d35 Implement LIST operation in the OSD 2019-12-19 22:04:18 +03:00
Vitaliy Filippov e88ad3f2ff Implement object list operation in blockstore 2019-12-19 20:50:20 +03:00
Vitaliy Filippov e8f7905e08 Allow to set write/delete version explicitly 2019-12-19 19:17:54 +03:00
Vitaliy Filippov d3d21e6e0f Rename OP_ to BS_OP_ 2019-12-19 13:56:26 +03:00
Vitaliy Filippov 19abe6227e Fix submission ring overflow & ring_data_t reuse conflicts 2019-12-17 11:26:17 +03:00
Vitaliy Filippov 4fb0579b1b Change epoll polling 
1) it is incorrect to call level-triggered epoll in a loop without reading everything
2) it fixes the FIXME
 2019-12-17 01:56:46 +03:00
Vitaliy Filippov c7ae3c001c Add FIO engine for testing Blockstore through the Secondary OSD interface 2019-12-16 14:03:36 +03:00
Vitaliy Filippov e60e6978ac Extract object_id and obj_ver_id to separate header 2019-12-16 12:39:15 +03:00
Vitaliy Filippov 3061b8cf51 Add sync&stabilize test operation 2019-12-15 15:30:51 +03:00
Vitaliy Filippov 9af000e9de Init storage note 2019-12-15 15:11:26 +03:00
Vitaliy Filippov 71635f2327 Use std::hash<object_id> instead of oid_hash 2019-12-15 14:57:18 +03:00
Vitaliy Filippov a7e74670a5 Split blockstore implementation and interface header 2019-12-15 14:57:18 +03:00
Vitaliy Filippov 749ab6e2c6 Rename blockstore_operation to blockstore_op_t 2019-12-15 14:57:18 +03:00
Vitaliy Filippov f2e9749c93 Fix disconnection detection 2019-12-15 14:57:18 +03:00
Vitaliy Filippov 49f8011917 Add osd header and osd executable 2019-12-15 01:11:51 +03:00
Vitaliy Filippov d3c6314d01 Finish reply code, make it compile 2019-12-14 20:51:59 +03:00
Vitaliy Filippov 02a0eb49c2 Begin reply code 2019-12-14 20:51:49 +03:00
Vitaliy Filippov e052959d7b Remove "packets" 2019-12-13 20:16:08 +03:00
Vitaliy Filippov f6eb4326b3 Begin op enqueueing into the blockstore 2019-12-13 20:12:31 +03:00
Vitaliy Filippov 283d03ef18 Fix "address already in use" in test example, begin client read loop 2019-12-13 14:05:15 +03:00
Vitaliy Filippov 6239f6748a io_uring recvmsg test 2019-12-13 00:02:48 +03:00
Vitaliy Filippov 00ee6b7961 Accept connections 2019-12-12 11:32:20 +03:00
Vitaliy Filippov a8deb7d88c Begin OSD 2019-12-11 14:18:19 +03:00
Vitaliy Filippov a68bf129c4 Begin OSD ops header 2019-12-10 12:07:24 +03:00
Vitaliy Filippov a180b4961f Hide block_* fields 2019-12-10 00:13:59 +03:00
Vitaliy Filippov aaea3e1f99 Fix read_fulfill, use vector 2019-12-03 12:09:30 +03:00
Vitaliy Filippov a2ed38fb6b Fix segfault 2019-12-03 02:27:25 +03:00
Vitaliy Filippov 1f7381e220 Do not use goto resume_0 due to some strange -O3 bug which leads to hangs (it seems it optimizes flush_queue.size() away) 2019-12-03 02:23:29 +03:00
Vitaliy Filippov fd13965b90 Force start when journal is full 2019-12-03 02:23:29 +03:00
Vitaliy Filippov 2963ad98ec Start to flush only when there are <sync_threshold> journal entries to flush 2019-12-03 00:20:13 +03:00
Vitaliy Filippov aa3b252327 Fsync data before writing metadata 2019-12-03 00:20:13 +03:00
Vitaliy Filippov a165909745 FIXME 2019-12-02 18:16:23 +03:00
Vitaliy Filippov d24860b51a Allow to specify blockstore block size in fio options 2019-12-02 15:50:55 +03:00
Vitaliy Filippov 76caecf7c7 Inmemory metadata mode 2019-12-02 15:42:42 +03:00
Vitaliy Filippov f4d06ba102 OP_DELETE flushing 2019-12-02 02:41:14 +03:00
Vitaliy Filippov a7a0946ba8 WIP OP_DELETE 2019-12-01 17:25:59 +03:00
Vitaliy Filippov 14b2f49f4d FIXME 2019-12-01 16:43:21 +03:00
Vitaliy Filippov 00eeedae90 Add "fsync disabled" mode 2019-12-01 16:41:07 +03:00
Vitaliy Filippov 76655929c4 Add readonly flag 2019-12-01 16:41:07 +03:00
Vitaliy Filippov 9260cd263a Verify data crc32 when reading journal 2019-11-30 23:32:10 +03:00
Vitaliy Filippov 2039df76a5 Fix journal reading and make it more similar to writing :) 2019-11-30 02:27:31 +03:00
Vitaliy Filippov 40781c67b2 Trim journal on start 2019-11-29 02:13:32 +03:00
Vitaliy Filippov 45f34fb3b2 Fix linear overwrite, make metadata writes ordered, ignore older entries when recovering journal 2019-11-28 22:36:38 +03:00
Vitaliy Filippov b6fff5a77e Fix metadata area size calculation, print free space, wait for free space 
FIXME: Now it crashes with -ENOSPC on linear overwrite
 2019-11-28 20:23:27 +03:00
Vitaliy Filippov 9fa0d3325f Support inmemory journal 2019-11-28 18:06:50 +03:00
Vitaliy Filippov e1ac4dba23 Fix safe stop procedure 2019-11-28 02:27:17 +03:00
Vitaliy Filippov d56cb290ee Two FIXMEs 2019-11-28 01:00:22 +03:00
Vitaliy Filippov cbf5173d1b Allow to specify device paths in fio options 2019-11-28 00:50:52 +03:00
Vitaliy Filippov 95396e2c83 Note read errors 2019-11-28 00:37:20 +03:00
Vitaliy Filippov 35a6ed728d Fix another stall due to bad unstable_writes tracking, do not try to write beyond the end of the journal 2019-11-28 00:28:08 +03:00
Vitaliy Filippov 2630e2e3b9 Fix metadata partition length, fix journal allocation at the end 2019-11-27 19:39:18 +03:00
Vitaliy Filippov 876231d26b no new 2019-11-27 18:14:01 +03:00
Vitaliy Filippov 9ba243b3ee Add debug prints 2019-11-27 18:07:51 +03:00
Vitaliy Filippov e2b91968c5 Fix sync confirmations and some pipeline-stall bugs 2019-11-27 18:07:38 +03:00
Vitaliy Filippov 78807eb244 Fix journal space check (do not overwrite the beginning of the journal) 2019-11-27 11:35:11 +03:00
Vitaliy Filippov ffff742078 Use tcmalloc 2019-11-27 11:26:47 +03:00
Vitaliy Filippov 95b0395493 Remove trim_interval=1 2019-11-27 02:47:01 +03:00
Vitaliy Filippov ce5cd13bc8 Use fdatasync (just for testing over an FS) 2019-11-27 02:41:30 +03:00
Vitaliy Filippov 06634054c9 Note to add no-fsync mode in the future 2019-11-27 02:27:38 +03:00
Vitaliy Filippov 74d8ea2f01 Calculate data crc32c 2019-11-27 02:20:38 +03:00
Vitaliy Filippov 9568354d78 Fix old_clean_loc treatment (cannot persist sparse_hash_map iterator) 2019-11-27 02:16:43 +03:00
Vitaliy Filippov 2831d40edb Fix repeated syncs 2019-11-27 02:04:46 +03:00
Vitaliy Filippov 1080cbdf12 Fix used journal sector tracking again 2019-11-27 01:20:09 +03:00
Vitaliy Filippov 6ac1d5db08 Fix allocator bug 2019-11-27 01:12:25 +03:00
Vitaliy Filippov ff7469ee91 Make allocator a class 2019-11-27 00:50:57 +03:00
Vitaliy Filippov b8e53f8c67 Fix op validation, fix journal sector usage tracking 
Now it runs for a while with fio but then crashes with ENOSPC because
of an apparent bug in the allocator
 2019-11-26 20:38:25 +03:00
Vitaliy Filippov a6770f619a Fix crash while reading metadata 2019-11-26 12:06:42 +03:00
Vitaliy Filippov a8f250c68f fio engine starts now 2019-11-26 02:18:42 +03:00
Vitaliy Filippov a7dc759f74 Compile fio engine 2019-11-26 01:50:26 +03:00
Vitaliy Filippov 3e46728321 Continue fio engine 2019-11-26 00:03:09 +03:00
Vitaliy Filippov b67406e764 Start fio engine 2019-11-26 00:03:04 +03:00
Vitaliy Filippov d0fdcbd7ff Add optimized crc32c 2019-11-25 02:30:06 +03:00
Vitaliy Filippov be3015169f Safe stop procedure 2019-11-25 01:29:07 +03:00