Vitaliy Filippov
95ec750b8c
New features: - [Localized read support](https://vitastor.io/docs/config/pool.html#local_reads) for multi-datacenter setups. - io_uring-based [zero-copy send support](https://vitastor.io/docs/config/network.html#min_zerocopy_send_size) - read the instruction carefully for optimal performance! - Improve and speedup data distribution, especially in cases of very large hosts (100 OSD+). Previously, PG optimization speed depended on the number of OSDs, now it only depends on the number of failure domains. Distribution over specific OSDs is now also more even and becomes strictly more even when you increase the number of PGs. - Add a [very interesting instruction](https://vitastor.io/en/docs/usage/nfs.html#linux-nfs-write-size) to change NFS_MAX_FILE_IO_SIZE - Check operation sequencing and stop connections when it breaks - should help catch some very rare RDMA packet loss problems. - `vitastor-cli rm-osd` now refuses to remove OSDs which are still up and suggests to use `vitastor-disk purge`. - Allow removal of direntries referring non-existent in VitastorFS. - Change default vitastor-etcd data dir to /var/lib/etcd/vitastor. Bug fixes: - Fix compatibility with ISA-L 2.31+. ⚠️Very important: please upgrade Vitastor before upgrading ISA-L to 2.31+. - Fix in-memory state cleanup for incomplete PGs. - Fix monitor crash with non-existent node_placement nodes. - Slightly speedup `vitastor-kv dump` command by adding output buffering. - Fix theoretically possible slowdowns in OSD sub-operation failure handling code. - Fix very rare stack overflows in vitastor-kv. - Fix a possible crash in VitastorFS during handling of file creation race condition. - Fix modify-pool -s PG_SIZE which didn't work without --pg_minsize. - Fix marking peer OSDs as alive on receiving data from them via RDMA - in theory, the bug could result in instability with RDMA under high load with slow disks. - Fix a rare OSD crash due to double handle_primary_subop() call. - Fix latency aggregation in global stats (/vitastor/stats in etcd) - do not sum it. - Hide "Ran out of journal space" log messages by default. - Wait for RDMA-CM EVENT_ESTABLISHED after rdma_accept(), handle rdma_accept() before acking the event. - Fix VitastorFS total & free numbers multiplied by extra 2. - Fix systemd unit name in make-etcd. - Do not allow reweight > 1 in vitastor-cli modify-osd. - Fix docker build.
Vitastor
The Idea
Make Clustered Block Storage Fast Again.
Vitastor is a distributed block, file and object SDS, direct replacement of Ceph RBD, CephFS and RGW, and also internal SDS's of public clouds. However, in contrast to them, Vitastor is fast and simple at the same time. The only thing is it's slightly young :-).
Vitastor is architecturally similar to Ceph which means strong consistency, primary-replication, symmetric clustering and automatic data distribution over any number of drives of any size with configurable redundancy (replication or erasure codes/XOR).
Vitastor targets primarily SSD and SSD+HDD clusters with at least 10 Gbit/s network, supports TCP and RDMA and may achieve 4 KB read and write latency as low as ~0.1 ms with proper hardware which is ~10 times faster than other popular SDS's like Ceph or internal systems of public clouds.
Vitastor supports QEMU, NBD, NFS protocols, OpenStack, OpenNebula, Proxmox, Kubernetes drivers. More drivers may be created easily.
Read more details in the documentation. You can start from here: Quick Start.
Talks and presentations
- DevOpsConf'2021: presentation (in Russian, in English), video
- Highload'2022: presentation (in Russian), video
Documentation
- Introduction
- Installation
- Configuration
- Usage
- vitastor-cli (command-line interface)
- vitastor-disk (disk management tool)
- fio for benchmarks
- NBD for kernel mounts
- QEMU and qemu-img
- NFS clustered file system and pseudo-FS proxy
- Administration
- Performance
Author and License
Copyright (c) Vitaliy Filippov (vitalif [at] yourcmc.ru), 2019+
Join Vitastor Telegram Chat: https://t.me/vitastor
All server-side code (OSD, Monitor and so on) is licensed under the terms of Vitastor Network Public License 1.1 (VNPL 1.1), a copyleft license based on GNU GPLv3.0 with the additional "Network Interaction" clause which requires opensourcing all programs directly or indirectly interacting with Vitastor through a computer network and expressly designed to be used in conjunction with it ("Proxy Programs"). Proxy Programs may be made public not only under the terms of the same license, but also under the terms of any GPL-Compatible Free Software License, as listed by the Free Software Foundation. This is a stricter copyleft license than the Affero GPL.
Please note that VNPL doesn't require you to open the code of proprietary software running inside a VM if it's not specially designed to be used with Vitastor.
Basically, you can't use the software in a proprietary environment to provide its functionality to users without opensourcing all intermediary components standing between the user and Vitastor or purchasing a commercial license from the author 😀.
Client libraries (cluster_client and so on) are dual-licensed under the same VNPL 1.1 and also GNU GPL 2.0 or later to allow for compatibility with GPLed software like QEMU and fio.
You can find the full text of VNPL-1.1 in the file VNPL-1.1.txt. GPL 2.0 is also included in this repository as GPL-2.0.txt.