Make immediate_commit=all the default

docs/config/layout-cluster.en.md

@@ -56,14 +56,24 @@ Can't be smaller than the OSD data device sector.
 ## immediate_commit
 
 - Type: string
-- Default: false
+- Default: all
 
-Another parameter which is really important for performance.
+One of "none", "all" or "small". Global value, may be overriden [at pool level](pool.en.md#immediate_commit).
+
+This parameter is also really important for performance.
+
+TLDR: default "all" is optimal for server-grade SSDs with supercapacitor-based
+power loss protection (nonvolatile write-through cache) and also for most HDDs.
+"none" or "small" should be only selected if you use desktop SSDs without
+capacitors or drives with slow write-back cache that can't be disabled. Check
+immediate_commit of your OSDs in [ls-osd](../usage/cli.en.md#ls-osd).
+
+Detailed explanation:
 
 Desktop SSDs are very fast (100000+ iops) for simple random writes
 without cache flush. However, they are really slow (only around 1000 iops)
-if you try to fsync() each write, that is, when you want to guarantee that
-each change gets immediately persisted to the physical media.
+if you try to fsync() each write, that is, if you want to guarantee that
+each change gets actually persisted to the physical media.
 
 Server-grade SSDs with "Advanced/Enhanced Power Loss Protection" or with
 "Supercapacitor-based Power Loss Protection", on the other hand, are equally
@@ -75,8 +85,8 @@ really slow when used with desktop SSDs. Vitastor, however, can also
 efficiently utilize desktop SSDs by postponing fsync until the client calls
 it explicitly.
 
-This is what this parameter regulates. When it's set to "all" the whole
-Vitastor cluster commits each change to disks immediately and clients just
+This is what this parameter regulates. When it's set to "all" Vitastor
+cluster commits each change to disks immediately and clients just
 ignore fsyncs because they know for sure that they're unneeded. This reduces
 the amount of network roundtrips performed by clients and improves
 performance. So it's always better to use server grade SSDs with
@@ -99,9 +109,5 @@ Setting this parameter to "all" or "small" in OSD parameters requires enabling
 [disable_journal_fsync](layout-osd.en.yml#disable_journal_fsync) and
 [disable_meta_fsync](layout-osd.en.yml#disable_meta_fsync), setting it to
 "all" also requires enabling [disable_data_fsync](layout-osd.en.yml#disable_data_fsync).
-
-TLDR: For optimal performance, set immediate_commit to "all" if you only use
-SSDs with supercapacitor-based power loss protection (nonvolatile
-write-through cache) for both data and journals in the whole Vitastor
-cluster. Set it to "small" if you only use such SSDs for journals. Leave
-empty if your drives have write-back cache.
+vitastor-disk tried to do that by default, first checking/disabling drive cache.
+If it can't disable drive cache, OSD get initialized with "none".

docs/config/layout-cluster.ru.md

@@ -57,9 +57,18 @@ amplification) и эффективность распределения нагр
 ## immediate_commit
 
 - Тип: строка
-- Значение по умолчанию: false
+- Значение по умолчанию: all
 
-Ещё один важный для производительности параметр.
+Одно из значений "none", "small" или "all". Глобальное значение, может быть
+переопределено [на уровне пула](pool.ru.md#immediate_commit).
+
+Данный параметр тоже важен для производительности.
+
+Вкратце: значение по умолчанию "all" оптимально для всех серверных SSD с
+суперконденсаторами и также для большинства HDD. "none" и "small" имеет смысл
+устанавливать только при использовании SSD настольного класса без
+суперконденсаторов или дисков с медленным неотключаемым кэшем записи.
+Проверьте настройку immediate_commit своих OSD в выводе команды [ls-osd](../usage/cli.ru.md#ls-osd).
 
 Модели SSD для настольных компьютеров очень быстрые (100000+ операций в
 секунду) при простой случайной записи без сбросов кэша. Однако они очень
@@ -80,7 +89,7 @@ Power Loss Protection" - одинаково быстрые и со сбросо
 эффективно утилизировать настольные SSD.
 
 Данный параметр влияет как раз на это. Когда он установлен в значение "all",
-весь кластер Vitastor мгновенно фиксирует каждое изменение на физические
+кластер Vitastor мгновенно фиксирует каждое изменение на физические
 носители и клиенты могут просто игнорировать запросы fsync, т.к. они точно
 знают, что fsync-и не нужны. Это уменьшает число необходимых обращений к OSD
 по сети и улучшает производительность. Поэтому даже с Vitastor лучше всегда
@@ -106,10 +115,3 @@ HDD-дисках с внутренним SSD или "медиа" кэшем - н
 включения [disable_journal_fsync](layout-osd.ru.yml#disable_journal_fsync) и
 [disable_meta_fsync](layout-osd.ru.yml#disable_meta_fsync), значение "all"
 также требует включения [disable_data_fsync](layout-osd.ru.yml#disable_data_fsync).
-
-Итого, вкратце: для оптимальной производительности установите
-immediate_commit в значение "all", если вы используете в кластере только SSD
-с суперконденсаторами и для данных, и для журналов. Если вы используете
-такие SSD для всех журналов, но не для данных - можете установить параметр
-в "small". Если и какие-то из дисков журналов имеют волатильный кэш записи -
-оставьте параметр пустым.

docs/config/src/layout-cluster.yml

@@ -47,14 +47,24 @@
     Не может быть меньше размера сектора дисков данных OSD.
 - name: immediate_commit
   type: string
-  default: false
+  default: all
   info: |
-    Another parameter which is really important for performance.
+    One of "none", "all" or "small". Global value, may be overriden [at pool level](pool.en.md#immediate_commit).
+
+    This parameter is also really important for performance.
+
+    TLDR: default "all" is optimal for server-grade SSDs with supercapacitor-based
+    power loss protection (nonvolatile write-through cache) and also for most HDDs.
+    "none" or "small" should be only selected if you use desktop SSDs without
+    capacitors or drives with slow write-back cache that can't be disabled. Check
+    immediate_commit of your OSDs in [ls-osd](../usage/cli.en.md#ls-osd).
+
+    Detailed explanation:
 
     Desktop SSDs are very fast (100000+ iops) for simple random writes
     without cache flush. However, they are really slow (only around 1000 iops)
-    if you try to fsync() each write, that is, when you want to guarantee that
-    each change gets immediately persisted to the physical media.
+    if you try to fsync() each write, that is, if you want to guarantee that
+    each change gets actually persisted to the physical media.
 
     Server-grade SSDs with "Advanced/Enhanced Power Loss Protection" or with
     "Supercapacitor-based Power Loss Protection", on the other hand, are equally
@@ -66,8 +76,8 @@
     efficiently utilize desktop SSDs by postponing fsync until the client calls
     it explicitly.
 
-    This is what this parameter regulates. When it's set to "all" the whole
-    Vitastor cluster commits each change to disks immediately and clients just
+    This is what this parameter regulates. When it's set to "all" Vitastor
+    cluster commits each change to disks immediately and clients just
     ignore fsyncs because they know for sure that they're unneeded. This reduces
     the amount of network roundtrips performed by clients and improves
     performance. So it's always better to use server grade SSDs with
@@ -90,14 +100,19 @@
     [disable_journal_fsync](layout-osd.en.yml#disable_journal_fsync) and
     [disable_meta_fsync](layout-osd.en.yml#disable_meta_fsync), setting it to
     "all" also requires enabling [disable_data_fsync](layout-osd.en.yml#disable_data_fsync).
-
-    TLDR: For optimal performance, set immediate_commit to "all" if you only use
-    SSDs with supercapacitor-based power loss protection (nonvolatile
-    write-through cache) for both data and journals in the whole Vitastor
-    cluster. Set it to "small" if you only use such SSDs for journals. Leave
-    empty if your drives have write-back cache.
+    vitastor-disk tried to do that by default, first checking/disabling drive cache.
+    If it can't disable drive cache, OSD get initialized with "none".
   info_ru: |
-    Ещё один важный для производительности параметр.
+    Одно из значений "none", "small" или "all". Глобальное значение, может быть
+    переопределено [на уровне пула](pool.ru.md#immediate_commit).
+
+    Данный параметр тоже важен для производительности.
+
+    Вкратце: значение по умолчанию "all" оптимально для всех серверных SSD с
+    суперконденсаторами и также для большинства HDD. "none" и "small" имеет смысл
+    устанавливать только при использовании SSD настольного класса без
+    суперконденсаторов или дисков с медленным неотключаемым кэшем записи.
+    Проверьте настройку immediate_commit своих OSD в выводе команды [ls-osd](../usage/cli.ru.md#ls-osd).
 
     Модели SSD для настольных компьютеров очень быстрые (100000+ операций в
     секунду) при простой случайной записи без сбросов кэша. Однако они очень
@@ -118,7 +133,7 @@
     эффективно утилизировать настольные SSD.
 
     Данный параметр влияет как раз на это. Когда он установлен в значение "all",
-    весь кластер Vitastor мгновенно фиксирует каждое изменение на физические
+    кластер Vitastor мгновенно фиксирует каждое изменение на физические
     носители и клиенты могут просто игнорировать запросы fsync, т.к. они точно
     знают, что fsync-и не нужны. Это уменьшает число необходимых обращений к OSD
     по сети и улучшает производительность. Поэтому даже с Vitastor лучше всегда
@@ -144,10 +159,3 @@
     включения [disable_journal_fsync](layout-osd.ru.yml#disable_journal_fsync) и
     [disable_meta_fsync](layout-osd.ru.yml#disable_meta_fsync), значение "all"
     также требует включения [disable_data_fsync](layout-osd.ru.yml#disable_data_fsync).
-
-    Итого, вкратце: для оптимальной производительности установите
-    immediate_commit в значение "all", если вы используете в кластере только SSD
-    с суперконденсаторами и для данных, и для журналов. Если вы используете
-    такие SSD для всех журналов, но не для данных - можете установить параметр
-    в "small". Если и какие-то из дисков журналов имеют волатильный кэш записи -
-    оставьте параметр пустым.

docs/intro/quickstart.en.md

@@ -68,10 +68,6 @@ On the monitor hosts:
     but some free unpartitioned space must be available because the script creates new partitions for journals.
 - You can change OSD configuration in units or in `vitastor.conf`.
   Check [Configuration Reference](../config.en.md) for parameter descriptions.
-- If all your drives have capacitors, and even if not, but if you ran `vitastor-disk`
-  without `--disable_data_fsync off` at the first step, then put the following
-  setting into etcd: \
-  `etcdctl --endpoints=... put /vitastor/config/global '{"immediate_commit":"all"}'`
 - Start all OSDs: `systemctl start vitastor.target`
 
 ## Create a pool
@@ -88,6 +84,10 @@ For EC pools the configuration should look like the following:
 vitastor-cli create-pool testpool --ec 2+2 --pg_count 256
 ```
 
+Add `--immediate_commit none` if you added `--disable_data_fsync off` at the OSD
+initialization step, or if `vitastor-disk` complained about impossibility to
+disable drive cache.
+
 After you do this, one of the monitors will configure PGs and OSDs will start them.
 
 If you use HDDs you should also add `"block_size": 1048576` to pool configuration.

docs/intro/quickstart.ru.md

@@ -69,11 +69,6 @@
     для журналов, на SSD должно быть доступно свободное нераспределённое место.
 - Вы можете менять параметры OSD в юнитах systemd или в `vitastor.conf`. Описания параметров
   смотрите в [справке по конфигурации](../config.ru.md).
-- Если все ваши диски - серверные с конденсаторами, и даже если нет, но при этом
-  вы не добавляли опцию `--disable_data_fsync off` на первом шаге, а `vitastor-disk`
-  не ругался на невозможность отключения кэша дисков, пропишите следующую настройку
-  в глобальную конфигурацию в etcd: \
-  `etcdctl --endpoints=... put /vitastor/config/global '{"immediate_commit":"all"}'`.
 - Запустите все OSD: `systemctl start vitastor.target`
 
 ## Создайте пул
@@ -90,6 +85,10 @@ vitastor-cli create-pool testpool --pg_size 2 --pg_count 256
 vitastor-cli create-pool testpool --ec 2+2 --pg_count 256
 ```
 
+Добавьте также опцию `--immediate_commit none`, если вы добавляли `--disable_data_fsync off`
+на этапе инициализации OSD, либо если `vitastor-disk` ругался на невозможность отключения
+кэша дисков.
+
 После этого один из мониторов должен сконфигурировать PG, а OSD должны запустить их.
 
 Если вы используете HDD-диски, то добавьте в конфигурацию пулов опцию `"block_size": 1048576`.

mon/etcd_schema.js

@@ -69,7 +69,7 @@ const etcd_tree = {
             block_size: 131072,
             disk_alignment: 4096,
             bitmap_granularity: 4096,
-            immediate_commit: false, // 'all' or 'small'
+            immediate_commit: 'all', // 'none', 'all' or 'small'
             // client - configurable online
             client_max_dirty_bytes: 33554432,
             client_max_dirty_ops: 1024,
@@ -189,7 +189,7 @@ const etcd_tree = {
                 block_size: 131072,
                 bitmap_granularity: 4096,
                 // 'all'/'small'/'none', same as in OSD options
-                immediate_commit: 'none',
+                immediate_commit: 'all',
                 pg_stripe_size: 0,
                 root_node?: 'rack1',
                 // restrict pool to OSDs having all of these tags

mon/pg_gen.js

@@ -174,7 +174,7 @@ async function generate_pool_pgs(state, global_config, pool_id, osd_tree, levels
         state.osd.stats,
         pool_cfg.block_size || global_config.block_size || 131072,
         pool_cfg.bitmap_granularity || global_config.bitmap_granularity || 4096,
-        pool_cfg.immediate_commit || global_config.immediate_commit || 'none'
+        pool_cfg.immediate_commit || global_config.immediate_commit || 'all'
     );
     pool_tree = make_hier_tree(global_config, pool_tree);
     // First try last_clean_pgs to minimize data movement

src/client/etcd_state_client.cpp

@@ -573,7 +573,7 @@ void etcd_state_client_t::load_global_config()
         {
             global_bitmap_granularity = DEFAULT_BITMAP_GRANULARITY;
         }
-        global_immediate_commit = parse_immediate_commit(global_config["immediate_commit"].string_value());
+        global_immediate_commit = parse_immediate_commit(global_config["immediate_commit"].string_value(), IMMEDIATE_ALL);
         on_load_config_hook(global_config);
     });
 }
@@ -867,7 +867,7 @@ void etcd_state_client_t::parse_state(const etcd_kv_t & kv)
             pc.used_for_fs = pool_item.second["used_for_fs"].as_string();
             // Immediate Commit Mode
             pc.immediate_commit = pool_item.second["immediate_commit"].is_string()
-                ? parse_immediate_commit(pool_item.second["immediate_commit"].string_value())
+                ? parse_immediate_commit(pool_item.second["immediate_commit"].string_value(), IMMEDIATE_ALL)
                 : global_immediate_commit;
             // PG Stripe Size
             pc.pg_stripe_size = pool_item.second["pg_stripe_size"].uint64_value();
@@ -1175,10 +1175,11 @@ void etcd_state_client_t::parse_state(const etcd_kv_t & kv)
     }
 }
 
-uint32_t etcd_state_client_t::parse_immediate_commit(const std::string & immediate_commit_str)
+uint32_t etcd_state_client_t::parse_immediate_commit(const std::string & immediate_commit_str, uint32_t default_value)
 {
-    return immediate_commit_str == "all" ? IMMEDIATE_ALL :
-        (immediate_commit_str == "small" ? IMMEDIATE_SMALL : IMMEDIATE_NONE);
+    return (immediate_commit_str == "all" ? IMMEDIATE_ALL :
+        (immediate_commit_str == "small" ? IMMEDIATE_SMALL :
+        (immediate_commit_str == "none" ? IMMEDIATE_NONE : default_value)));
 }
 
 uint32_t etcd_state_client_t::parse_scheme(const std::string & scheme)

src/client/etcd_state_client.h

@@ -157,6 +157,6 @@ public:
     int address_count();
     ~etcd_state_client_t();
 
-    static uint32_t parse_immediate_commit(const std::string & immediate_commit_str);
+    static uint32_t parse_immediate_commit(const std::string & immediate_commit_str, uint32_t default_value);
     static uint32_t parse_scheme(const std::string & scheme_str);
 };

src/cmd/cli.cpp

@@ -136,7 +136,7 @@ static const char* help_text =
     "    --osd_tags <tag>[,<tag>]...   Put pool only on OSDs tagged with all specified tags\n"
     "    --block_size 128k             Put pool only on OSDs with this data block size\n"
     "    --bitmap_granularity 4k       Put pool only on OSDs with this logical sector size\n"
-    "    --immediate_commit none       Put pool only on OSDs with this or larger immediate_commit (none < small < all)\n"
+    "    --immediate_commit all        Put pool only on OSDs with this or larger immediate_commit (none < small < all)\n"
     "    --level_placement <rules>     Use additional failure domain rules (example: \"dc=112233\")\n"
     "    --raw_placement <rules>       Specify raw PG generation rules (see documentation for details)\n"
     "    --primary_affinity_tags tags  Prefer to put primary copies on OSDs with all specified tags\n"

src/cmd/cli_osd_tree.cpp

@@ -134,7 +134,7 @@ resume_1:
                 .reweight = 1,
                 .block_size = (uint32_t)kv.second["data_block_size"].uint64_value(),
                 .bitmap_granularity = (uint32_t)kv.second["bitmap_granularity"].uint64_value(),
-                .immediate_commit = etcd_state_client_t::parse_immediate_commit(kv.second["immediate_commit"].string_value()),
+                .immediate_commit = etcd_state_client_t::parse_immediate_commit(kv.second["immediate_commit"].string_value(), IMMEDIATE_NONE),
             };
             if (tree->nodes.find(osd.parent) == tree->nodes.end())
             {

src/cmd/cli_pool_cfg.cpp

@@ -319,7 +319,7 @@ std::string validate_pool_config(json11::Json::object & new_cfg, json11::Json ol
     }
 
     // immediate_commit
-    if (!cfg["immediate_commit"].is_null() && !etcd_state_client_t::parse_immediate_commit(cfg["immediate_commit"].string_value()))
+    if (!cfg["immediate_commit"].is_null() && etcd_state_client_t::parse_immediate_commit(cfg["immediate_commit"].string_value(), UINT32_MAX) == UINT32_MAX)
     {
         return "immediate_commit must be one of \"all\", \"small\", or \"none\", but it is "+cfg["immediate_commit"].as_string();
     }

src/cmd/cli_pool_create.cpp

@@ -448,7 +448,7 @@ resume_8:
             ? cfg["bitmap_granularity"].uint64_value()
             : parent->cli->st_cli.global_bitmap_granularity;
         uint32_t p_immediate_commit = cfg["immediate_commit"].is_string()
-            ? etcd_state_client_t::parse_immediate_commit(cfg["immediate_commit"].string_value())
+            ? etcd_state_client_t::parse_immediate_commit(cfg["immediate_commit"].string_value(), IMMEDIATE_ALL)
             : parent->cli->st_cli.global_immediate_commit;
 
         for (size_t i = 0; i < osd_stats.size(); i++)
@@ -459,7 +459,7 @@ resume_8:
             if (!os["data_block_size"].is_null() && os["data_block_size"] != p_block_size ||
                 !os["bitmap_granularity"].is_null() && os["bitmap_granularity"] != p_bitmap_granularity ||
                 !os["immediate_commit"].is_null() &&
-                etcd_state_client_t::parse_immediate_commit(os["immediate_commit"].string_value()) < p_immediate_commit)
+                etcd_state_client_t::parse_immediate_commit(os["immediate_commit"].string_value(), IMMEDIATE_NONE) < p_immediate_commit)
             {
                 accepted_nodes.erase(osd_num);
             }

tests/run_3osds.sh

@@ -23,7 +23,7 @@ if [ "$IMMEDIATE_COMMIT" != "" ]; then
     $ETCDCTL put /vitastor/config/global '{"recovery_queue_depth":1,"recovery_tune_util_low":1,"immediate_commit":"all","client_enable_writeback":true,"client_max_writeback_iodepth":32'$GLOBAL_CONFIG'}'
 else
     NO_SAME="--journal_sector_buffer_count 1024 --log_level 10 --etcd_stats_interval 5 --min_flusher_count 16"
-    $ETCDCTL put /vitastor/config/global '{"recovery_queue_depth":1,"recovery_tune_util_low":1,"client_enable_writeback":true,"client_max_writeback_iodepth":32'$GLOBAL_CONFIG'}'
+    $ETCDCTL put /vitastor/config/global '{"recovery_queue_depth":1,"recovery_tune_util_low":1,"immediate_commit":"none","client_enable_writeback":true,"client_max_writeback_iodepth":32'$GLOBAL_CONFIG'}'
 fi
 
 start_osd_on()

tests/test_failure_domain.sh

@@ -3,7 +3,7 @@
 . `dirname $0`/common.sh
 
 TIME=$(date '+%s')
-$ETCDCTL put /vitastor/config/global '{"placement_levels":{"rack":1,"host":2,"osd":3}}'
+$ETCDCTL put /vitastor/config/global '{"placement_levels":{"rack":1,"host":2,"osd":3},"immediate_commit":"none"}'
 $ETCDCTL put /vitastor/config/node_placement '{"rack1":{"level":"rack"},"rack2":{"level":"rack"},"host1":{"level":"host","parent":"rack1"},"host2":{"level":"host","parent":"rack1"},"host3":{"level":"host","parent":"rack2"},"host4":{"level":"host","parent":"rack2"}}'
 $ETCDCTL put /vitastor/osd/stats/1 '{"host":"host1","size":1073741824,"time":"'$TIME'"}'
 $ETCDCTL put /vitastor/osd/stats/2 '{"host":"host1","size":1073741824,"time":"'$TIME'"}'

tests/test_move_reappear.sh

@@ -11,7 +11,7 @@ for i in $(seq 1 $OSD_COUNT); do
     eval OSD${i}_PID=$!
 done
 
-$ETCDCTL put /vitastor/config/pools '{"1":{"name":"testpool","scheme":"replicated","pg_size":2,"pg_minsize":1,"pg_count":1,"failure_domain":"osd"}}'
+$ETCDCTL put /vitastor/config/pools '{"1":{"name":"testpool","scheme":"replicated","pg_size":2,"pg_minsize":1,"pg_count":1,"failure_domain":"osd","immediate_commit":"none"}}'
 
 $ETCDCTL put /vitastor/config/pgs '{"items":{"1":{"1":{"osd_set":[1,0],"primary":1}}}}'
 

tests/test_osd_tags.sh

@@ -19,7 +19,7 @@ $ETCDCTL put /vitastor/osd/stats/5 '{"host":"stor3","size":1073741824,"time":"'$
 $ETCDCTL put /vitastor/osd/stats/6 '{"host":"stor3","size":1073741824,"time":"'$TIME'"}'
 $ETCDCTL put /vitastor/osd/stats/7 '{"host":"stor4","size":1073741824,"time":"'$TIME'"}'
 $ETCDCTL put /vitastor/osd/stats/8 '{"host":"stor4","size":1073741824,"time":"'$TIME'"}'
-$ETCDCTL put /vitastor/config/pools '{"1":{"name":"testpool","scheme":"replicated","pg_size":2,"pg_minsize":2,"pg_count":16,"failure_domain":"host","osd_tags":["a"]}}'
+$ETCDCTL put /vitastor/config/pools '{"1":{"name":"testpool","scheme":"replicated","pg_size":2,"pg_minsize":2,"pg_count":16,"failure_domain":"host","osd_tags":["a"],"immediate_commit":"none"}}'
 
 node mon/mon-main.js --etcd_address $ETCD_URL --etcd_prefix "/vitastor" >>./testdata/mon.log 2>&1 &
 MON_PID=$!

tests/test_root_node.sh

@@ -3,7 +3,7 @@
 . `dirname $0`/common.sh
 
 TIME=$(date '+%s')
-$ETCDCTL put /vitastor/config/global '{"placement_levels":{"rack":100,"host":101,"osd":102}}'
+$ETCDCTL put /vitastor/config/global '{"placement_levels":{"rack":100,"host":101,"osd":102},"immediate_commit":"none"}'
 $ETCDCTL put /vitastor/config/node_placement '{"rack1":{"level":"rack"},"rack2":{"level":"rack"},"stor1":{"level":"host","parent":"rack1"},"stor2":{"level":"host","parent":"rack1"},"stor3":{"level":"host","parent":"rack2"},"stor4":{"level":"host","parent":"rack2"}}'
 $ETCDCTL put /vitastor/osd/stats/1 '{"host":"stor1","size":1073741824,"time":"'$TIME'"}'
 $ETCDCTL put /vitastor/osd/stats/2 '{"host":"stor1","size":1073741824,"time":"'$TIME'"}'