Commit Graph
2865 Commits
Author SHA1 Message Date
Vitaliy Filippov 985257f2d7 Parse standard TLS record headers 2026-07-05 14:58:09 +03:00
Vitaliy Filippov a4f3383f21 Omit msgr_tls_record_hdr_t for non-tls data 2026-07-05 14:58:09 +03:00
Vitaliy Filippov b2d828ec84 Allow 2 and 4 byte per block chain_info (allow more than 255 snapshots with encryption) 2026-07-05 14:58:09 +03:00
Vitaliy Filippov b2a74de715 Implement OSD TLS support 2026-07-05 14:58:09 +03:00
Vitaliy Filippov 430761ad2e Allow to skip checksums for headers 2026-07-05 14:58:09 +03:00
Vitaliy Filippov 38a1f6937d Implement protocol-level checksums (xxhash3) 2026-07-05 14:58:09 +03:00
Vitaliy Filippov 8b46914999 Include xxhash3 x86dispatch 2026-07-05 14:58:09 +03:00
Vitaliy Filippov 1b81e60187 Do not use scrap_buffer in the client (it would block protocol checksum support) 2026-07-05 14:58:09 +03:00
Vitaliy Filippov 526c597cd0 Support TLS CN authentication and per-image permissions in vitastor-cli serve 2026-07-05 14:58:09 +03:00
Vitaliy Filippov 7a97783ead Add VitastorAuthFilter 2026-07-05 14:58:09 +03:00
Vitaliy Filippov b4425d8e65 Implement vitastor-cli ls-user, modify-user, remove-user commands 2026-07-05 14:58:09 +03:00
Vitaliy Filippov 1afda35bd3 Add image owner/owner_group/reader_group support (for antietcd VitastorAuthFilter) 2026-07-05 14:58:09 +03:00
Vitaliy Filippov 0953f5ebdd Support inline (string PEM) certificates and pkeys 2026-07-05 14:10:11 +03:00
Vitaliy Filippov 1b03615e50 Show encryption keys (only IDs) in the listing 2026-07-05 14:10:11 +03:00
Vitaliy Filippov 8f208c53df Support storing image encryption keys in Vault 2026-07-05 14:10:11 +03:00
Vitaliy Filippov c0d2dabe66 Prefer local etcd addresses and correctly cycle over them even when they need resolving
Seems slightly overcomplicated...
2026-07-05 14:10:10 +03:00
Vitaliy Filippov 6da4aaf176 Support DNS resolving via libc-ares 2026-07-05 14:10:10 +03:00
Vitaliy Filippov 5b6a6fce9a Batch handle_immediate_ops more 2026-07-05 14:10:10 +03:00
Vitaliy Filippov 79fb2def57 Add vitastor-cli create & modify --enc-key parameter 2026-07-05 14:10:10 +03:00
Vitaliy Filippov 7f5c24144b Support reading from snapshots encrypted with different keys 2026-07-05 14:10:10 +03:00
Vitaliy Filippov 3cf876fafa Support decryption with multiple keys 2026-07-05 14:10:10 +03:00
Vitaliy Filippov 771aa83282 Allow to return chain_info in response to reads 2026-07-05 14:10:10 +03:00
Vitaliy Filippov 03404ac95d Add basic AES-XTS client-side encryption support 2026-07-05 14:10:10 +03:00
Vitaliy Filippov decf314238 Rework msgr send/receive to allow encryption support 2026-07-05 14:10:10 +03:00
Vitaliy Filippov ab1849ff07 Move fromhexstr() to str_util 2026-07-05 14:10:10 +03:00
Vitaliy Filippov 33b18c7229 Add openapi description 2026-07-05 14:10:10 +03:00
Vitaliy Filippov 4a2efb29ec Slightly fix API return and input types 2026-07-05 14:10:10 +03:00
Vitaliy Filippov 00a432193a Implement vitastor-cli serve command to serve simple HTTP API 2026-07-05 14:10:10 +03:00
Vitaliy Filippov d83eb599a0 Implement HTTP server support O_o 2026-07-05 14:10:10 +03:00
Vitaliy Filippov c1c9b1975d Rename http_response_t to http_message_t 2026-07-05 14:10:10 +03:00
Vitaliy Filippov ba0d9ad9f2 Extract common HTTP context 2026-07-05 14:10:10 +03:00
Vitaliy Filippov 4feabc4ab6 Support xxhash 32-bit checksums (data_csum_type=xxh3_32) 2026-07-05 14:10:10 +03:00
Vitaliy Filippov 1a14301c50 Detect block checksums using csum_block_size, not data_csum_type 2026-07-05 14:09:57 +03:00
Vitaliy Filippov 5dc52a7c06 Add client certificate support 2026-07-05 14:09:57 +03:00
Vitaliy Filippov 4d45b27696 Do not re-initialize TLS context every connection 2026-07-05 14:09:57 +03:00
Vitaliy Filippov 5995c4a00f Add https support to antietcd 2026-07-05 14:09:57 +03:00
Vitaliy Filippov 906388adaa Implement etcd SSL support via OpenSSL
Maybe I should remove all of this and use libwebsockets :)
2026-07-05 14:09:57 +03:00
Vitaliy Filippov 5dbc679e16 Fix padded block checksums - v2 2026-07-05 14:09:57 +03:00
Vitaliy Filippov d48864a6be Fix unaligned pointer warnings 2026-07-04 21:40:29 +03:00
Vitaliy Filippov 462482d319 Fix zero-padded big_write checksum verification in the new store 2026-07-04 21:37:22 +03:00
Vitaliy Filippov 5ef9d78461 Release 3.0.15
- QEMU virtual disk migration with enabled iothread is finally fixed correctly.
- Fixed operation of Proxmox VMs with swTPM without enabling NBD for all disks.
- Debian packages are now again built with stable Antietcd released instead of
  the unstable master branch.
- Antietcd cluster mode previously broken in that master branch is fixed. The
  symptom was Antietcd being unable to elect the leader in a cluster.
- Fixed monitor startup with embedded Antietcd when using IPv6.
- Fixed space statistics calculation for FS and S3 pools in the new storage ([PR #127](https://github.com/vitalif/vitastor/pull/127)).
v3.0.15
2026-06-28 11:29:38 +03:00
Vitaliy Filippov 1b20010a69 Install npm packages in debian/rules 2026-06-28 11:29:27 +03:00
Vitaliy Filippov d750d00c6f Fix SWTPM in Proxmox 2026-06-27 16:04:51 +03:00
Vitaliy Filippov 66cb564e2c Fix background jobs in etcd_fail test 2026-06-27 15:42:53 +03:00
Vitaliy Filippov f782decbd6 Update antietcd to 1.3.1 2026-06-27 15:42:42 +03:00
Vitaliy Filippov 717582c4ba Use release builds of antietcd&tinyraft in debian packages (not master branch) 2026-06-27 10:27:15 +03:00
Vitaliy Filippov cd62c0755f Fix IPv6 antietcd configuration 2026-06-27 00:16:03 +03:00
c6f5ab7d79 fix pool stats with no_inode_stats (#127)
Co-authored-by: zhu.chengzhen <zhu.chengzhen@jingjiamicro.com>
2026-06-24 12:41:56 +03:00
Vitaliy Filippov 09607ddcbe Actual fix for live migration with iothread 2026-06-23 00:07:34 +03:00
Vitaliy Filippov 278852b4d5 Release 3.0.14
How many (bugs!) I've knifed, how many I've slit!

General note: most bug fixes now include regression tests to verify that they don't repeat in the future.
Most bugs fixed in this release were detected by using LLM analysis (Claude Opus/Fable, GPT 5.5).

OSD:
- Fix OSD hanging with an infinite loop when setting autosync_interval to 0 at runtime
- (IMPORTANT) Fix EC PGs hanging in REPEERING when the last final commit/rollback in a
  batch completes with an error
- Limit pg_size by 64 because peering doesn't handle larger values with EC — they just
  lead to 'incomplete' objects
- Fix OSD crash with an "assertion failed" error on EIO retry in snapshot chain read
  (i.e. when some chunks belong to a corrupted replica with checksum mismatch)
- (IMPORTANT) Disable chunked PG count resharding due to possible interference with
  compaction changes (will be re-enabled after fixes)
- (IMPORTANT) Fix incorrect snapshot allocation bitmap recovery during EC chained read
- Add on-wire request size validation to prevent possible OOM/DoS/heap corruption
  on receiving invalid data from the network
- (IMPORTANT) Fix parity-less EC writes destroying snapshot allocation bitmaps
  (i.e. when all parity OSDs in a PG are missing)
- (IMPORTANT) Fix EC N+K, K>=2 recovery destroying snapshot allocation bitmaps
  of live parity chunks
- (IMPORTANT) Fix a possible OSD crash during EC misplaced object scrubbing
- (IMPORTANT) Verify object bitmap consistency during scrub (only data was checked previously)
- (IMPORTANT) Fix corrupted object chunks incorrectly marked as non-corrupted on the second scrub
- (IMPORTANT) Fix cached EC decoding of multiple stripes with ISA-L (ISA-L is the default)

New store:
- Fix a theoretically possible OSD crash on startup when using the previously added
  workaround for the "double-claim" problem
- Remove theoretically possible incorrect metadata block writes during batch EC COMMITs
  restarted due to a full metadata area
- Fix incorrect compaction counter tracking after OSD restart (could probably lead to
  compaction not restarted correctly after a restart)
- (IMPORTANT) Fix some of parallel big_writes possibly not waiting for data fsync, thus not providing durability
- Fix possible OSD crash on sync retry when io_uring is full
- Fix a possible crash during startup on corrupted on-disk data with too small entry sizes

Old store:
- Prevent loading extra garbage metadata entries from the last 4 MB of metadata area
- Fix read operations possibly crashing if a metadata read (with inmemory_metadata=false)
  was restarted due to a full io_uring
- Fix a possible memory leak of temporary buffers and bitmaps/checksums when a read
  was restarted due to a full io_uring (reproducible with either inmemory_metadata=false or block_size>256k)
- Fix a possible OSD crash during padded checksum reads if buffer count exceeded 1024
  (IOV_MAX) (reproducible only with csum_block_size > 4k and block_size >= 4M)
- (IMPORTANT) Fix partial padded read journal checksum verification with csum_block_size > 4k
- Fix incorrect marking of corrupted objects as non-corrupted after flushing data
  from journal (with inmemory_journal=false)
- (IMPORTANT) Fix deferred freeing of a different block when a block was used by a parallel read
- Fix per-inode statistics not being disabled for FS and S3 pools correctly, leading to etcd
  overload with unneeded per-inode statistics, slower etcd operation, increased memory usage,
  and too many Prometheus statistics exported by the monitor

Both stores:
- Fix possibly left garbage in the metadata area if the first OSD startup was interrupted —
  metadata header is now written only after initializating metadata
- Check for short reads during initialization (just in case, doesn't happen in real life)

Clients:
- Fix write-back queue item split in case when write-back is enabled at runtime
- Implement bdrv_detach_aio_context & bdrv_attach_aio_context in the QEMU driver (should fix migration with iothread)
- Do not crash on full io_uring in ublk server
- Fix missing --readonly option handling in NBD server
- Stop gracefully on NBD_CMD_DISC instead of just exit(0) in NBD server
- Fix writeback detection in ublk server for --image mode
- Limit the amount of incoming data for NFS clients to prevent choking on memory in async mount mode

Tools (vitastor-disk/vitastor-cli):
- Prevent vitastor-cli merge possibly exiting before completing the last sync/delete operations
- Fix vitastor-disk incorrectly validating too large small_write entry length
- Fix vitastor-cli merge ignoring input option validation errors
- Fix vitastor-cli rm-data always skipping the final fsync
- Fix vitastor-disk resize not moving the last used data block
- Fix vitastor-disk write-meta incorrectly importing new store small_write entries
- Fix vitastor-disk write-journal and write-meta importing old store data incorrectly
  when csum_block_size is > 4k
- Support --io option for vitastor-disk dump-journal/write-journal
- Fix vitastor-disk resize crash when converting from very old (0.5.x) metadata
- Fix vitastor-disk trim incorrectly rounding block ranges with --discard_granularity
  option explicitly set to a value > 4k, possibly leading to discarding live data
- Fix vitastor-disk write-meta importing new store metadata incorrectly with > 4 GB metadata area size
- (IMPORTANT) Fix vitastor-cli modify --resize to a smaller size clearing all image data O_o

Other:
- Do not crash with an uncaught exception when an invalid /osd/state/ with a non-numeric
  suffix is present in etcd (in OSD and all client services)
- Fix possible crash in vitastor-kv when handling a corrupted DB due to a uint32 overflow
- Fix NFS-RDMA memory allocator crashing in some situations
- Fix small shared file extend-write potentially reading unallocated memory (NFS)
- Add bounds checks to prevent uint32 overflows in NFS/XDR
- Re-enable accidentally disabled safety checks (asserts) in files with included cpp-btree
- Fix too small memory allocation in NFS portmap
v3.0.14
2026-06-21 18:17:02 +03:00