thebestvibedev/tromcho.net
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
542 Commits 74 Branches 106 Tags
b9e7d31aa1dbfdc3a8c79e2dccad8229fbda2fae
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Vitaliy Filippov b9e7d31aa1 Release v0.5.4 
- Fix a rare hang, more or less reproducible with very slow drives
- Fix a hang with the no_same_sector_overwrites mode
 2021-02-24 01:40:30 +03:00
Vitaliy Filippov 2d9f09dcb6 Attempt forced trim when stopping an overrun flusher 
Fixes a rare hang happening in the event of journal space running out without
new work to do for flushers except the current sector.
The hang could be reproduced more or less consistently with very slow drives.
 2021-02-24 01:33:01 +03:00
Vitaliy Filippov 7cc59260c5 Fix no_same_sector_overwrites related bug 2021-02-23 18:50:51 +03:00
Vitaliy Filippov ca0a11ec85 Release 0.5.3 2021-02-03 00:38:57 +03:00
Vitaliy Filippov 51c0b5afee Whitelist more leaks 2021-02-02 02:05:41 +03:00
Vitaliy Filippov e1e01d042e Rename sector_info.usage_count to flush_count 2021-02-02 01:32:23 +03:00
Vitaliy Filippov 534a4a657e Rename space_check.sectors_required to sectors_to_write 2021-02-02 01:30:23 +03:00
Vitaliy Filippov 9b5d8b9ad4 Fix multiple-sector journal writes, add assertions to not miss any SQEs 2021-02-02 01:29:11 +03:00
Vitaliy Filippov e66ed47515 Clear SQEs before returning them to the caller to prevent erroneous double submissions 2021-02-02 01:26:54 +03:00
Vitaliy Filippov 036c6d4c42 Add a simple test case 2021-02-01 19:43:10 +03:00
Vitaliy Filippov 4cb79a3bf8 Allow to calculate simple-offsets for files 2021-02-01 19:43:10 +03:00
Vitaliy Filippov 3bf53754c2 Fix several I/O bugs 2021-02-01 19:43:10 +03:00
Vitaliy Filippov 6023cac361 Do not stop clients before they are connected 2021-02-01 19:31:10 +03:00
Vitaliy Filippov 915d04c446 Allow empty global configuration, report OSD statistics faster 2021-02-01 19:31:10 +03:00
Vitaliy Filippov 21e06ea40d Fix memory leaks in fio engines 2021-02-01 19:31:10 +03:00
Vitaliy Filippov 9ef7f865b0 Fix incorrect calls to prepare_journal_sector_write() when flushing multiple sectors 2021-02-01 19:31:10 +03:00
Vitaliy Filippov 9dd20a31aa Do not use pg_minsize in the client code! 2021-02-01 19:31:10 +03:00
Vitaliy Filippov 28be049909 Dump only actual part of the journal by default 2021-01-01 23:04:30 +03:00
Vitaliy Filippov 78fbaacf1f External jerasure's w into defines 
In fact, w=8 looks better than w=32, so it may be changed in the future
 2020-12-31 19:15:22 +03:00
Vitaliy Filippov 1526c5a213 Add lp_solve into dependencies 2020-12-31 01:32:31 +03:00
Vitaliy Filippov c7cc414c90 Skip removed descriptors in epoll (this is possible in real clusters) 2020-12-30 17:04:18 +03:00
Vitaliy Filippov f4ea313707 Fix cl->read_op being freed without calling the completion callback 2020-12-30 16:55:54 +03:00
Vitaliy Filippov b88b76f316 Parallel usage of multiple network interfaces was a sick fantasy 2020-12-30 00:05:17 +03:00
Vitaliy Filippov 4a17a61d1f Make rm_inode work with incomplete and degraded objects, allow to wait before deleting objects 2020-12-28 16:38:08 +03:00
Vitaliy Filippov ccabbbfbcb For reference: include a spec patch for building QEMU 4.2 or CentOS 7 2020-12-06 15:43:38 +03:00
Vitaliy Filippov 26dac57083 State that jerasure is now supported 2020-12-06 15:25:48 +03:00
Vitaliy Filippov 44a53d8352 Huh. Fix rpath for packages 2020-12-05 20:16:39 +03:00
Vitaliy Filippov 9d80bd2d98 Build with jerasure, split some build scripts 2020-12-05 19:02:23 +03:00
Vitaliy Filippov 322a38a144 Fix non-preserved real_pg_count leading to inability to change pools online 2020-12-04 23:46:48 +03:00
Vitaliy Filippov 1018764c91 Fix write->delete->write bugs, add & fix some debugging output 2020-12-04 23:21:58 +03:00
Vitaliy Filippov a45e0e5e67 Use custom decoding instead of just jerasure_matrix_decode() 
- Cache the decoding matrix
- Don't do unnecessary erasures->erased conversion during decoding
- Avoid extra memory allocations during decoding
- Don't always reconstruct coding chunks
- Reconstruct chunks one-by-one, without overlapping ranges
 2020-12-04 17:43:48 +03:00
Vitaliy Filippov 44656fbf67 Allow writes with low version numbers after a delete 2020-12-04 11:54:41 +03:00
Vitaliy Filippov 089f138e0c Allow situations where the journal contains a big_write(v1) after delete(v2) and v1 < v2 
Fixes a crash in the following scenario:
- client issues a delete request (object version is at least 2)
- OSD has time to flush it to the metadata, but doesn't have time to move the journal start pointer on disk
- client overwrites the same object and it gets the version number 1 again
- OSD is restarted and sees delete(v=2), big_write(v=1) in the journal
- dirty_db sequence gets broken and OSD crashes with assert("Writes and deletes shouldn't happen at the same time")
 2020-12-04 11:47:27 +03:00
Vitaliy Filippov bcc8e697f9 Delete PGs when deleting pools 
(All OSD crash with "Online PG count change not allowed" if you try to delete an active pool though)
 2020-12-04 11:47:27 +03:00
Vitaliy Filippov a4c46ba745 Add jerasure EC support (reed_sol_van, others are slower) (not tested yet) 2020-12-04 11:47:27 +03:00
Vitaliy Filippov 5596ad8997 Use custom QEMU build for CentOS 7 2020-12-04 11:47:05 +03:00
Vitaliy Filippov 59c29b0cee Fix RPATH for CentOS builds, add additional repos into the CentOS installation instructions 2020-12-04 11:47:04 +03:00
Vitaliy Filippov 959089b919 Enable progress_notify=true for etcd watches 2020-11-17 16:29:42 +03:00
Vitaliy Filippov d3e7749616 Final fixes for packaging 2020-11-10 23:33:07 +03:00
Vitaliy Filippov b56f8820ec Container packaging for Debian 11 Bullseye, CentOS 7 and CentOS 8 2020-11-10 00:02:53 +03:00
Vitaliy Filippov 4bd2bd48eb Build Vitastor packages, too 2020-11-09 14:41:39 +03:00
Vitaliy Filippov a3fc9f8d7d Add a Dockerfile to build patched QEMU for Debian (Buster) 2020-11-09 02:30:41 +03:00
Vitaliy Filippov 530975aed7 Make it also build with GCC 8 and on Debian Buster 2020-11-09 00:07:07 +03:00
Vitaliy Filippov 1446aad107 Simple patch for qemu-kvm .spec 2020-11-08 02:14:53 +03:00
Vitaliy Filippov 46479e2456 Add RPM build scripts for CentOS 8 2020-11-08 01:55:17 +03:00
Vitaliy Filippov e41bee72a5 Lower node.js requirement to 10.x 2020-11-08 01:54:12 +03:00
Vitaliy Filippov 2e0f223ddb Add RPM build scripts for CentOS 7 2020-11-07 01:52:10 +03:00
Vitaliy Filippov 3be7bc29d8 Make it build with QEMU 2.0, too 
Also begin to work on rpms
 2020-11-06 20:05:00 +03:00
Vitaliy Filippov 0c43ff9daf Add scripts to copy fio and qemu includes to the source package 2020-11-06 18:40:42 +03:00
Vitaliy Filippov 64d471cf53 Add simple Debian packaging 2020-11-06 18:40:42 +03:00
Vitaliy Filippov 809b2ad8cd Add install target 2020-11-06 01:12:22 +03:00
Vitaliy Filippov 550d4af151 Rename test.cpp to test_shit.cpp (random shit) 2020-11-06 01:12:22 +03:00
Vitaliy Filippov cf0f23ab8e Add patches for QEMU QAPI IDL 2020-11-04 23:30:51 +03:00
Vitaliy Filippov a516fefa8c Add qemu_module_dummy and qemu_stamp_xxx to qemu_driver.c 2020-11-04 23:10:29 +03:00
Vitaliy Filippov 3b7279b376 Add Ceph EC 2+1 test results 2020-11-01 14:13:35 +03:00
Vitaliy Filippov 824ea507d0 Do not try to push more segments than IOV_MAX at once as it leads to EMSGSIZE 2020-10-30 01:25:43 +03:00
Vitaliy Filippov 23ea409081 Fix "can't get SQE, will fall out of sync with EPOLLET" when overflowing the ring 
OSDs shouldn't crash or hang with long iodepths anymore
 2020-10-30 01:06:36 +03:00
Vitaliy Filippov 2ccb75974b Fix a rare crash caused by a stopped client still being in write_ready_clients 2020-10-30 01:04:58 +03:00
Vitaliy Filippov 6561d4e040 Validate pool ID before executing the operation 
Reply -EPIPE for non-existing pools because we assume that it means
that pool config isn't loaded yet. Previously OSD crashed on such operations
 2020-10-30 01:02:46 +03:00
Vitaliy Filippov 1eda7f529d Note about Linux 5.8+ 2020-10-28 19:17:22 +03:00
Vitaliy Filippov 0a174bb313 Return success for already finished rollback operations 
There was a FIXME and I actually hit it during tests :)
 2020-10-24 18:46:19 +03:00
Vitaliy Filippov 720985e4c7 Fix NULL rmw buffer after the latest changes and add a testcase for it 2020-10-24 18:29:19 +03:00
Vitaliy Filippov 4872f617a4 Clear connect timeout in stop_client() to stop races during disconnections 2020-10-24 10:37:16 +03:00
Vitaliy Filippov e8ac08be14 Allow to overwrite incomplete objects or parts of objects to recover them 2020-10-24 02:14:41 +03:00
Vitaliy Filippov 660c2412fb Improve debugging output for incomplete/degraded 2020-10-24 01:28:47 +03:00
Vitaliy Filippov faa5e1436f Attempt journal trim even without new flushes 
This is the new guaranteed unblocking method which replaces old trims
in init and rollback, and also fixes a possible stall when just several
writes in the beginning of the journal are flushed without triggering
a subsequent trim.
 2020-10-24 01:28:47 +03:00
Vitaliy Filippov 5fbe36198a Fix journal trimming 
1) Update journal's used_start in memory only after updating journal superblock.
Doing the opposite is incorrect because part of the journal will be lost if writers
overwrite its old beginning.

2) Sync journal device after updating the superblock.

3) Do not trim in rollback and init because trimming there would also require
updating the superblock. And the only reason to trim in both those places was
to unblock writers. And a guaranteed unblocking method will follow in the next
commit :)
 2020-10-24 01:08:33 +03:00
Vitaliy Filippov 99c45bb5ed Fix debugging output during journal loading 2020-10-24 01:08:33 +03:00
Vitaliy Filippov 701eb79422 Stabilize writes before deleting extra chunks to not stall peer journals 2020-10-23 22:45:05 +03:00
Vitaliy Filippov 220bda0667 Fix possible buffer over(under)flow when handling LIST 2020-10-23 02:17:44 +03:00
Vitaliy Filippov 1e8f0328e0 Cancel outbound operations after re-peering PGs 2020-10-22 22:54:38 +00:00
Vitaliy Filippov f011e0c675 Do not block stabilize by list and list by write 2020-10-22 22:13:40 +00:00
Vitaliy Filippov 1a694c387e Print slow ops in log 2020-10-20 23:41:23 +00:00
Vitaliy Filippov 738ad5af79 Fix infinite looping in continue_recovery_op() when pg_cancel_write_queue() is called 2020-10-20 22:23:15 +00:00
Vitaliy Filippov 9abf3c17c9 Correct fix for "Pool %u PG %u configuration is invalid" during startup 
Establish watcher connection after loading PGs
 2020-10-20 21:09:14 +00:00
Vitaliy Filippov d2b901aa09 Fix default auto-created failure domains 2020-10-20 21:07:40 +00:00
Vitaliy Filippov befff09370 Fix possible crash due to uninitialized ring_data_t in ringloop 2020-10-20 10:44:38 +03:00
Vitaliy Filippov d1645551d4 Implement write batching 
Also fix possible race condition which could in theory lead to "command out of sync"
and a buffer overflow that could happen on incorrect server response.
 2020-10-20 03:29:17 +03:00
Vitaliy Filippov 7cb561f95a Add etcd to the example service generator 2020-10-20 01:50:56 +03:00
Vitaliy Filippov ae480196e2 Add a note about etcd bug, fix simple-offsets.js cmdline 2020-10-19 17:05:45 +03:00
Vitaliy Filippov 398c86f943 Improve PG-related log messages 2020-10-18 12:17:22 +00:00
Vitaliy Filippov bec5f921a6 Fix buffer overflows in the no_same_sector_overwrites mode 2020-10-17 23:30:16 +00:00
Vitaliy Filippov 5335c8de8e Do not use unordered_map for list_ops/list_results 2020-10-17 23:30:16 +00:00
Vitaliy Filippov c696a82083 Replace assert with if + error message (may happen on metadata corruption) 2020-10-17 23:30:16 +00:00
Vitaliy Filippov 900171586b XOR 2+1 test results 2020-10-17 14:58:08 +03:00
Vitaliy Filippov 70612e5df0 Do not handle change events before loading config 2020-10-17 11:18:39 +00:00
Vitaliy Filippov d952c24979 Use timeout in rw callback 2020-10-17 11:00:55 +00:00
Vitaliy Filippov 776fe954a5 Fix crashes on multiple OSD reconnects 
Identify clients by pointers instead of peer_fd as peer may be dropped
and reconnected between callbacks

Yeah maybe I need some Rust, but ... maybe in the future :)
 2020-10-17 10:53:04 +00:00
Vitaliy Filippov 9350656af6 Fix osd tags 2020-10-16 23:28:48 +00:00
Vitaliy Filippov ece14a7d65 Hide "Connected with..." client messages by default 2020-10-11 02:22:46 +03:00
Vitaliy Filippov be5f314c32 Change notes about gcc requirement to 9+, fio to 3.16+ 2020-10-11 02:00:39 +03:00
Vitaliy Filippov 15dba96375 Implement inode removal tool. Removes multiple objects from multiple OSDs in parallel 2020-10-10 01:08:19 +03:00
Vitaliy Filippov 3d05aa9362 Make it build with GCC 10, fio 3.20+ (atomics...) and QEMU 5.1 2020-10-06 02:35:11 +03:00
Vitaliy Filippov 94efb54feb Implement OSD tags (device classes), fix pool failure_domain configuration 2020-10-04 17:31:50 +03:00
Vitaliy Filippov aa2a0ee00f Do not group adjacent stripes by default as it's pointless on SSDs 2020-10-02 10:17:54 +03:00
Vitaliy Filippov 9f2a948712 Make pg_stripe_size a per-pool config 2020-10-01 18:51:49 +03:00
Vitaliy Filippov ba74eece4a More fixes to the failure model (why am I doing this?..) 2020-10-01 18:38:30 +03:00
Vitaliy Filippov 2fdd8a1b38 More correct failure model (I hope so) 2020-10-01 02:33:48 +03:00
Vitaliy Filippov 526983f7a9 Add usable CLI commands for NBD proxy (map/unmap/list) 2020-09-29 02:06:19 +03:00
Vitaliy Filippov 8e36f04482 One more experiment with cluster AFR% 2020-09-27 19:42:42 +03:00