thebestvibedev/tromcho.net
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,786 Commits 74 Branches 106 Tags
c79b38bd26835bcecfee0ff2c786dbe8a37ccdc9
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Vitaliy Filippov c79b38bd26 Move all sources to subdirs 2024-05-15 11:06:01 +03:00
Vitaliy Filippov 44692d148a Make vitastor_kv.h header public 2024-05-15 01:49:38 +03:00
Vitaliy Filippov ba52359611 Fix last master commit 2024-05-15 01:49:31 +03:00
Vitaliy Filippov 23a9aa93b5 Fix pool create/modify --block_size validation 2024-05-04 16:33:22 +03:00
Vitaliy Filippov 2412d9e239 Fix TTL comparison for lease/keepalive 2024-04-30 01:53:05 +03:00
Vitaliy Filippov 9301c857b1 Release 1.6.1 
A bunch of monitor fixes

- Add noout flag for OSDs (/vitastor/config/osd/xx)
- Fix "effective" size of degraded PGs (and thus "used space") calculation in monitor
- Fix monitor not clearing PGs of deleted pools
- Fix incorrect PG generation with hosts with 0 OSDs
- Fix monitor crashing during primary OSD recheck when pool has no PGs
- Fix monitor crashing when node_placement included non-existing OSDs
- Fix possible data movement after removing OSDs reweighted to 0
- Remove extra empty keys from pool configurations created by vitastor-cli create-pool
- Fix 32-bit build
 2024-04-22 02:01:29 +03:00
Vitaliy Filippov 3094358ec2 Fix autovivification leading to extra empty keys in pool-create 2024-04-20 02:04:09 +03:00
Vitaliy Filippov 87f666d2a2 Filter out OSDs reweighted to 0 2024-04-20 02:03:53 +03:00
Vitaliy Filippov bd7fe4ef8f Filter out non-existing OSDs added in node_placement 2024-04-20 02:03:36 +03:00
Vitaliy Filippov 1b3f9a1416 Do not set non-existing OSD weight to 0, we'll remove them instead 2024-04-20 02:03:11 +03:00
Vitaliy Filippov a7b7354f38 Do not recheck primary distribution when pool has no PGs 2024-04-20 02:02:47 +03:00
Vitaliy Filippov 765befa22f Remove empty nodes from tree because PG DSL expects that all leaf nodes are OSDs 2024-04-20 02:02:28 +03:00
Vitaliy Filippov 87b3ab94fe Do not disable require-atomic-updates and no-unused-vars 2024-04-20 02:02:13 +03:00
Vitaliy Filippov 2c0801f6e4 Configure ESLint and add it to CI 2024-04-16 02:39:31 +03:00
Vitaliy Filippov fd83fef1d9 Fix pool deletion 2024-04-16 02:20:26 +03:00
Vitaliy Filippov 8d1067971b Fix pg_effsize (and thus "used space") calculation in monitor 2024-04-16 02:20:18 +03:00
Vitaliy Filippov ae5af04fde Add noout flag for OSDs 2024-04-16 02:19:55 +03:00
Vitaliy Filippov 266d038b11 Fix 32-bit build warnings and one error again :-) 2024-04-11 22:49:33 +03:00
Vitaliy Filippov ff4414d37e Release 1.6.0 
New features:

- Implement "hierarchical failure domains" and other complex distribution rules, for example
  EC 4+2 over 3 DC, with 2 chunks per each DC ([documentation](docs/config/pool.en.md#level_placement))
- Make OSDs handle ENOSPC - now cluster stays online even if some OSDs fill up
  to 100 %, only writes requiring free space hang
- Implement Stage/Unstage & volume locking for CSI to prevent parallel mounting
  and/or modifications of the same volume
- Warn about full and almost full OSDs in vitastor-cli status
- Add an experimental NBD netlink map mode as an option ([documentation](docs/usage/nbd.en.md))
- Add --pg parameter to vitastor-cli describe, print objects with 0x in human-readable format too
- Add [administration docs](docs/usage/admin.en.md)

Bug fixes:

- Fix client operation retry timeout - previously the timeout wasn't applied and writes were
  retries almost instantly
- Fix monitors crashing on invalid pool configurations
- Fix journaling - make each journal write wait for all previous journal writes
- Fix monitor thinking that OSD weight is 0 after deleting /osd/config/ key online
- Fix a write stall caused by flusher possibly not trimming journal on rollback
- Set 32k csum_block_size for HDD by default in vitastor-disk
 2024-04-09 16:57:59 +03:00
Vitaliy Filippov 0fa7ecc03f Add also a test for OSD tags 2024-04-09 16:57:59 +03:00
Vitaliy Filippov c29bfe12eb Oops - fix filter_by_root_node, add a test for it 2024-04-09 15:48:44 +03:00
Vitaliy Filippov 57bf84ddb2 Fix filtering in mon 2024-04-09 14:51:05 +03:00
Vitaliy Filippov dff4879c8c Check if NBD_ATTR_BACKEND_IDENTIFIER is defined 2024-04-09 13:16:58 +03:00
Vitaliy Filippov af9a853db6 Move NBD netlink map&unmap to separate commands, add "netlink-revive" command 2024-04-08 16:34:41 +03:00
Vitaliy Filippov b7a3275af3 Make netlink optional 2024-04-08 01:51:28 +03:00
Vitaliy Filippov 64c5c4ca26 Fix code style 2024-04-08 01:35:03 +03:00
Vitaliy Filippov 6366972fe8 Warn about full and almost full OSDs in status 2024-04-07 19:39:51 +03:00
Vitaliy Filippov 2b863fb715 Add ENOSPC handling tests 2024-04-07 19:39:33 +03:00
Vitaliy Filippov 3bf4dd5abd Fix client op retry timeout - do not retry immediately 2024-04-07 19:08:36 +03:00
Vitaliy Filippov 3b84dcaedd Handle ENOSPC during write - rollback partial EC writes, remember partial replica writes 2024-04-07 18:02:05 +03:00
Vitaliy Filippov 20fbc4a745 Add --pg parameter to vitastor-cli describe, print objects with 0x in human-readable format too 2024-04-07 12:39:46 +03:00
Vitaliy Filippov 02993ee1dd Implement Stage/Unstage & volume locking for CSI to prevent parallel modifications of the same volume 2024-04-07 11:48:19 +03:00
Vitaliy Filippov 3629dbc54d Plug the new PG combinator into monitor 2024-04-07 02:44:17 +03:00
Vitaliy Filippov 29284bef40 Implement new DSL/rule-based PG generation algorithm 2024-04-07 00:36:20 +03:00
Vitaliy Filippov 6a924d6066 Extract PG combinator into a separate module 2024-04-07 00:36:20 +03:00
Vitaliy Filippov 9fe779a691 Do not die on invalid pool configurations 2024-04-07 00:36:20 +03:00
Vitaliy Filippov 31c2751b9b Move NBD/VDUSE map/unmap functions to a separate file 2024-04-07 00:36:09 +03:00
Vitaliy Filippov c5195666cd Fix journal sequencing: make each journal write wait for all previous journal writes 2024-04-06 23:53:12 +03:00
Vitaliy Filippov f36d7eb76c Fix monitor thinking that OSD weight is 0 after deleting /osd/config/ key 2024-04-05 23:14:46 +03:00
Vitaliy Filippov dd7f651de1 Add --max-request-bytes=104857600 to etcd params in tests 2024-04-05 23:14:46 +03:00
Vitaliy Filippov a2994ecd0d Fix flusher possibly not trimming journal on rollback 2024-04-05 23:14:39 +03:00
Vitaliy Filippov 5d3aaf016b Add administration docs 2024-03-31 01:54:52 +03:00
Vitaliy Filippov 0b097ca3f2 Set 32k csum_block_size for HDD by default 2024-03-30 16:16:49 +03:00
Vitaliy Filippov 989675a780 s/etcd_ws_keepalive_timeout/etcd_ws_keepalive_interval/ in docs 2024-03-26 01:56:08 +03:00
Vitaliy Filippov f8c403ec9e Add newer benchmark results 2024-03-23 18:28:48 +03:00
Vitaliy Filippov bfbb85e653 Replace -Oanything with -O3, not just -O/-O1/-O2 2024-03-18 02:03:44 +03:00
Vitaliy Filippov 9ad6822353 Release 1.5.0 
After half a year of hard work, VitastorFS is finally here ! :-)

New features:
- VitastorFS, a full-featured clustered (read-write-many) file system.
  Documentation: [VitastorFS](docs/usage/nfs.en.md)
- Embedded key-value database implementation based on Parallel Optimistic B-Tree
  algorithm and used for the metadata of VitastorFS
- Pool management commands in vitastor-cli (create-pool, list-pools, rm-pool, modify-pool).
  Thanks MIND Software (https://mindsw.io) for their contribution!
  [Documentation](docs/usage/cli.en.md#create-pool)

Bug fixes:
- Fix a very rare "infinite loop" in the client library
- Fix a rare OSD hang on during start when zeroing out bad metadata entries left from the previous run
 2024-03-16 15:35:10 +03:00
Vitaliy Filippov 2043b4e374 Fix build errors for gcc 8 2024-03-16 15:35:10 +03:00
Vitaliy Filippov de840e6fe3 Reduce kv-cli loadjson load parallelism to 16 2024-03-16 15:35:10 +03:00
Vitaliy Filippov b5e04bf809 Fix build warning 2024-03-16 15:35:10 +03:00
Vitaliy Filippov 8807a1623b Fix markdown tables 2024-03-16 15:35:10 +03:00
Vitaliy Filippov f12855c31b Add vitastor-kv to packages 2024-03-16 15:35:10 +03:00
Vitaliy Filippov e75dcc9a71 Add documentation for VitastorFS 2024-03-16 15:16:43 +03:00
Vitaliy Filippov 88516ab4bd Remove extra log 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 6221126b4f Allow to print simple-offsets just given the device size 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 6783d4a13c Implement fool protection for FS pools 2024-03-16 13:24:36 +03:00
Vitaliy Filippov dcbe1afac3 Store pool ID in inode metadata 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 0bde28c24a Make nfs_do_rmw a library function 2024-03-16 13:24:36 +03:00
Vitaliy Filippov bb8ca6184e Support setattr guard 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 87310ef7bb Support ctime 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 4f4b2dab80 Log NFS liveness checks 2024-03-16 13:24:36 +03:00
Vitaliy Filippov f70da82317 Add loadjson command to vitastor-kv 2024-03-16 13:24:36 +03:00
Vitaliy Filippov e42148f347 Allow to specify KV commands on command line 2024-03-16 13:24:36 +03:00
Vitaliy Filippov c289584469 Add JSON dump format 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 018e89f867 Erase verf key left from creation from ientries on every modification 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 603dc68f11 Implement async mtime change 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 7b12342933 Allow to specify additional NFS mount options 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 44bf0f16ee Fix malloc/free in nfs_kv_read/write 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 8840c84572 Fix "bad key in etcd" in mon for FS pools 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 5b747c12ec Check if already mounted before mounting 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 05f5f46162 Fix zero used space, update mtime when moving/changing inode 2024-03-16 13:24:36 +03:00
Vitaliy Filippov b5604191c8 Ignore ECANCELED in nfs-proxy (happens in io_uring on fork) 2024-03-16 13:24:36 +03:00
Vitaliy Filippov e871de27de Support unaligned shared_offsets, align shared file data instead of header 2024-03-16 13:24:36 +03:00
Vitaliy Filippov f600ce98e2 Implement auto-unmount local NFS server mode for vitastor-nfs 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 57605a5c13 Return error on failed shrink 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 29bd4561bb Implement rename over an existing file/directory 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 7142460ec8 Support --logfile in nfs-proxy 2024-03-16 13:24:36 +03:00
Vitaliy Filippov d03f19ebe5 Fix shared file overlap, add FIXMEs 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 88f9d18be3 Create inode, then direntry, not direntry, then inode; retry ID collisions 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 6213fbd8c6 Fix NFS shared/aligned write FIXMEs 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 3aee37eadd Allow to disable per-inode stats for VitastorFS pools 2024-03-16 13:24:36 +03:00
Vitaliy Filippov ecfc753e93 Add basic NFS tests, fix bugs 2024-03-16 13:24:36 +03:00
Vitaliy Filippov a574f9ad71 Return block NFS implementation back as an option too 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 7c235c9103 Move KV FS header into a separate file 2024-03-16 13:24:36 +03:00
Vitaliy Filippov e5bb986164 Implement packing small files into shared inodes 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 181795d748 Split new NFS proxy implementation into multiple files 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 8cdc38805b WIP VitastorFS with metadata storage in VitastorKV 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 0cd455d17f First just recheck version without actually re-reading block in vitastor-kv 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 32ba653ba6 Fix vitastor-kv hang on reopen & unfinished closed listing 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 231d4b15fc Add loadable dump format to vitastor-kv (dump) 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 9dc4d5fd7b Fix freeing r/w buffers on errors in kv_db 2024-03-16 13:24:36 +03:00
Vitaliy Filippov e58538fa47 Fix eviction when random_pos selects the end 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 11ac9e7024 Implement min/max list_count to make listings during performance test reasonable 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 511bc3df1c Fix and improve parallel allocation 
- Do not try to allocate more DB blocks in an inode block until it's "confirmed" and "locked" by the first write
- Do not recheck for new zero DB blocks on first write into an inode block - a CAS failure means someone else is already writing into it
- Throw new allocation blocks away regardless of whether the known_version is 0 on a CAS failure
 2024-03-16 13:24:36 +03:00
Vitaliy Filippov a64f0d1f73 Implement key_prefix for K/V stress test 2024-03-16 13:24:36 +03:00
Vitaliy Filippov ec5f7c6b87 More fixes 
- do not overwrite a block with older version if known version is newer
  (read may start before update and end after update)
- invalidated block versions can't be remembered and trusted
- right boundary for split blocks is right_half when diving down, not key_lt
- restart update also when block is "invalidated", not just on version mismatch
- copy callback in listings to avoid closure destruction bugs too
 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 3ebed9a749 Add logging and one more assert 2024-03-16 13:24:36 +03:00
Vitaliy Filippov eab67a6e8f Make get_block() wait for updating when unrelated block is found along the path 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 20993d9b7a Fix a race condition where changed blocks were parsed over existing cached blocks and getting a mix of data 2024-03-16 13:24:36 +03:00
Vitaliy Filippov 5cf9b343c0 Simplify code by removing an unneeded "optimisation" 2024-03-16 13:24:36 +03:00