diff --git a/docs/config/layout-cluster.en.md b/docs/config/layout-cluster.en.md index b87ee9fe..1848646c 100644 --- a/docs/config/layout-cluster.en.md +++ b/docs/config/layout-cluster.en.md @@ -56,14 +56,24 @@ Can't be smaller than the OSD data device sector. ## immediate_commit - Type: string -- Default: false +- Default: all -Another parameter which is really important for performance. +One of "none", "all" or "small". Global value, may be overriden [at pool level](pool.en.md#immediate_commit). + +This parameter is also really important for performance. + +TLDR: default "all" is optimal for server-grade SSDs with supercapacitor-based +power loss protection (nonvolatile write-through cache) and also for most HDDs. +"none" or "small" should be only selected if you use desktop SSDs without +capacitors or drives with slow write-back cache that can't be disabled. Check +immediate_commit of your OSDs in [ls-osd](../usage/cli.en.md#ls-osd). + +Detailed explanation: Desktop SSDs are very fast (100000+ iops) for simple random writes without cache flush. However, they are really slow (only around 1000 iops) -if you try to fsync() each write, that is, when you want to guarantee that -each change gets immediately persisted to the physical media. +if you try to fsync() each write, that is, if you want to guarantee that +each change gets actually persisted to the physical media. Server-grade SSDs with "Advanced/Enhanced Power Loss Protection" or with "Supercapacitor-based Power Loss Protection", on the other hand, are equally @@ -75,8 +85,8 @@ really slow when used with desktop SSDs. Vitastor, however, can also efficiently utilize desktop SSDs by postponing fsync until the client calls it explicitly. -This is what this parameter regulates. When it's set to "all" the whole -Vitastor cluster commits each change to disks immediately and clients just +This is what this parameter regulates. When it's set to "all" Vitastor +cluster commits each change to disks immediately and clients just ignore fsyncs because they know for sure that they're unneeded. This reduces the amount of network roundtrips performed by clients and improves performance. So it's always better to use server grade SSDs with @@ -99,9 +109,5 @@ Setting this parameter to "all" or "small" in OSD parameters requires enabling [disable_journal_fsync](layout-osd.en.yml#disable_journal_fsync) and [disable_meta_fsync](layout-osd.en.yml#disable_meta_fsync), setting it to "all" also requires enabling [disable_data_fsync](layout-osd.en.yml#disable_data_fsync). - -TLDR: For optimal performance, set immediate_commit to "all" if you only use -SSDs with supercapacitor-based power loss protection (nonvolatile -write-through cache) for both data and journals in the whole Vitastor -cluster. Set it to "small" if you only use such SSDs for journals. Leave -empty if your drives have write-back cache. +vitastor-disk tried to do that by default, first checking/disabling drive cache. +If it can't disable drive cache, OSD get initialized with "none". diff --git a/docs/config/layout-cluster.ru.md b/docs/config/layout-cluster.ru.md index 5a703359..b7d6f6fd 100644 --- a/docs/config/layout-cluster.ru.md +++ b/docs/config/layout-cluster.ru.md @@ -57,9 +57,18 @@ amplification) и эффективность распределения нагр ## immediate_commit - Тип: строка -- Значение по умолчанию: false +- Значение по умолчанию: all -Ещё один важный для производительности параметр. +Одно из значений "none", "small" или "all". Глобальное значение, может быть +переопределено [на уровне пула](pool.ru.md#immediate_commit). + +Данный параметр тоже важен для производительности. + +Вкратце: значение по умолчанию "all" оптимально для всех серверных SSD с +суперконденсаторами и также для большинства HDD. "none" и "small" имеет смысл +устанавливать только при использовании SSD настольного класса без +суперконденсаторов или дисков с медленным неотключаемым кэшем записи. +Проверьте настройку immediate_commit своих OSD в выводе команды [ls-osd](../usage/cli.ru.md#ls-osd). Модели SSD для настольных компьютеров очень быстрые (100000+ операций в секунду) при простой случайной записи без сбросов кэша. Однако они очень @@ -80,7 +89,7 @@ Power Loss Protection" - одинаково быстрые и со сбросо эффективно утилизировать настольные SSD. Данный параметр влияет как раз на это. Когда он установлен в значение "all", -весь кластер Vitastor мгновенно фиксирует каждое изменение на физические +кластер Vitastor мгновенно фиксирует каждое изменение на физические носители и клиенты могут просто игнорировать запросы fsync, т.к. они точно знают, что fsync-и не нужны. Это уменьшает число необходимых обращений к OSD по сети и улучшает производительность. Поэтому даже с Vitastor лучше всегда @@ -106,10 +115,3 @@ HDD-дисках с внутренним SSD или "медиа" кэшем - н включения [disable_journal_fsync](layout-osd.ru.yml#disable_journal_fsync) и [disable_meta_fsync](layout-osd.ru.yml#disable_meta_fsync), значение "all" также требует включения [disable_data_fsync](layout-osd.ru.yml#disable_data_fsync). - -Итого, вкратце: для оптимальной производительности установите -immediate_commit в значение "all", если вы используете в кластере только SSD -с суперконденсаторами и для данных, и для журналов. Если вы используете -такие SSD для всех журналов, но не для данных - можете установить параметр -в "small". Если и какие-то из дисков журналов имеют волатильный кэш записи - -оставьте параметр пустым. diff --git a/docs/config/src/layout-cluster.yml b/docs/config/src/layout-cluster.yml index fe12ecfd..36115407 100644 --- a/docs/config/src/layout-cluster.yml +++ b/docs/config/src/layout-cluster.yml @@ -47,14 +47,24 @@ Не может быть меньше размера сектора дисков данных OSD. - name: immediate_commit type: string - default: false + default: all info: | - Another parameter which is really important for performance. + One of "none", "all" or "small". Global value, may be overriden [at pool level](pool.en.md#immediate_commit). + + This parameter is also really important for performance. + + TLDR: default "all" is optimal for server-grade SSDs with supercapacitor-based + power loss protection (nonvolatile write-through cache) and also for most HDDs. + "none" or "small" should be only selected if you use desktop SSDs without + capacitors or drives with slow write-back cache that can't be disabled. Check + immediate_commit of your OSDs in [ls-osd](../usage/cli.en.md#ls-osd). + + Detailed explanation: Desktop SSDs are very fast (100000+ iops) for simple random writes without cache flush. However, they are really slow (only around 1000 iops) - if you try to fsync() each write, that is, when you want to guarantee that - each change gets immediately persisted to the physical media. + if you try to fsync() each write, that is, if you want to guarantee that + each change gets actually persisted to the physical media. Server-grade SSDs with "Advanced/Enhanced Power Loss Protection" or with "Supercapacitor-based Power Loss Protection", on the other hand, are equally @@ -66,8 +76,8 @@ efficiently utilize desktop SSDs by postponing fsync until the client calls it explicitly. - This is what this parameter regulates. When it's set to "all" the whole - Vitastor cluster commits each change to disks immediately and clients just + This is what this parameter regulates. When it's set to "all" Vitastor + cluster commits each change to disks immediately and clients just ignore fsyncs because they know for sure that they're unneeded. This reduces the amount of network roundtrips performed by clients and improves performance. So it's always better to use server grade SSDs with @@ -90,14 +100,19 @@ [disable_journal_fsync](layout-osd.en.yml#disable_journal_fsync) and [disable_meta_fsync](layout-osd.en.yml#disable_meta_fsync), setting it to "all" also requires enabling [disable_data_fsync](layout-osd.en.yml#disable_data_fsync). - - TLDR: For optimal performance, set immediate_commit to "all" if you only use - SSDs with supercapacitor-based power loss protection (nonvolatile - write-through cache) for both data and journals in the whole Vitastor - cluster. Set it to "small" if you only use such SSDs for journals. Leave - empty if your drives have write-back cache. + vitastor-disk tried to do that by default, first checking/disabling drive cache. + If it can't disable drive cache, OSD get initialized with "none". info_ru: | - Ещё один важный для производительности параметр. + Одно из значений "none", "small" или "all". Глобальное значение, может быть + переопределено [на уровне пула](pool.ru.md#immediate_commit). + + Данный параметр тоже важен для производительности. + + Вкратце: значение по умолчанию "all" оптимально для всех серверных SSD с + суперконденсаторами и также для большинства HDD. "none" и "small" имеет смысл + устанавливать только при использовании SSD настольного класса без + суперконденсаторов или дисков с медленным неотключаемым кэшем записи. + Проверьте настройку immediate_commit своих OSD в выводе команды [ls-osd](../usage/cli.ru.md#ls-osd). Модели SSD для настольных компьютеров очень быстрые (100000+ операций в секунду) при простой случайной записи без сбросов кэша. Однако они очень @@ -118,7 +133,7 @@ эффективно утилизировать настольные SSD. Данный параметр влияет как раз на это. Когда он установлен в значение "all", - весь кластер Vitastor мгновенно фиксирует каждое изменение на физические + кластер Vitastor мгновенно фиксирует каждое изменение на физические носители и клиенты могут просто игнорировать запросы fsync, т.к. они точно знают, что fsync-и не нужны. Это уменьшает число необходимых обращений к OSD по сети и улучшает производительность. Поэтому даже с Vitastor лучше всегда @@ -144,10 +159,3 @@ включения [disable_journal_fsync](layout-osd.ru.yml#disable_journal_fsync) и [disable_meta_fsync](layout-osd.ru.yml#disable_meta_fsync), значение "all" также требует включения [disable_data_fsync](layout-osd.ru.yml#disable_data_fsync). - - Итого, вкратце: для оптимальной производительности установите - immediate_commit в значение "all", если вы используете в кластере только SSD - с суперконденсаторами и для данных, и для журналов. Если вы используете - такие SSD для всех журналов, но не для данных - можете установить параметр - в "small". Если и какие-то из дисков журналов имеют волатильный кэш записи - - оставьте параметр пустым. diff --git a/docs/intro/quickstart.en.md b/docs/intro/quickstart.en.md index ac5119b3..0d7ed3fa 100644 --- a/docs/intro/quickstart.en.md +++ b/docs/intro/quickstart.en.md @@ -68,10 +68,6 @@ On the monitor hosts: but some free unpartitioned space must be available because the script creates new partitions for journals. - You can change OSD configuration in units or in `vitastor.conf`. Check [Configuration Reference](../config.en.md) for parameter descriptions. -- If all your drives have capacitors, and even if not, but if you ran `vitastor-disk` - without `--disable_data_fsync off` at the first step, then put the following - setting into etcd: \ - `etcdctl --endpoints=... put /vitastor/config/global '{"immediate_commit":"all"}'` - Start all OSDs: `systemctl start vitastor.target` ## Create a pool @@ -88,6 +84,10 @@ For EC pools the configuration should look like the following: vitastor-cli create-pool testpool --ec 2+2 --pg_count 256 ``` +Add `--immediate_commit none` if you added `--disable_data_fsync off` at the OSD +initialization step, or if `vitastor-disk` complained about impossibility to +disable drive cache. + After you do this, one of the monitors will configure PGs and OSDs will start them. If you use HDDs you should also add `"block_size": 1048576` to pool configuration. diff --git a/docs/intro/quickstart.ru.md b/docs/intro/quickstart.ru.md index 71b156aa..e9d37177 100644 --- a/docs/intro/quickstart.ru.md +++ b/docs/intro/quickstart.ru.md @@ -69,11 +69,6 @@ для журналов, на SSD должно быть доступно свободное нераспределённое место. - Вы можете менять параметры OSD в юнитах systemd или в `vitastor.conf`. Описания параметров смотрите в [справке по конфигурации](../config.ru.md). -- Если все ваши диски - серверные с конденсаторами, и даже если нет, но при этом - вы не добавляли опцию `--disable_data_fsync off` на первом шаге, а `vitastor-disk` - не ругался на невозможность отключения кэша дисков, пропишите следующую настройку - в глобальную конфигурацию в etcd: \ - `etcdctl --endpoints=... put /vitastor/config/global '{"immediate_commit":"all"}'`. - Запустите все OSD: `systemctl start vitastor.target` ## Создайте пул @@ -90,6 +85,10 @@ vitastor-cli create-pool testpool --pg_size 2 --pg_count 256 vitastor-cli create-pool testpool --ec 2+2 --pg_count 256 ``` +Добавьте также опцию `--immediate_commit none`, если вы добавляли `--disable_data_fsync off` +на этапе инициализации OSD, либо если `vitastor-disk` ругался на невозможность отключения +кэша дисков. + После этого один из мониторов должен сконфигурировать PG, а OSD должны запустить их. Если вы используете HDD-диски, то добавьте в конфигурацию пулов опцию `"block_size": 1048576`. diff --git a/mon/etcd_schema.js b/mon/etcd_schema.js index 1034cc14..1e1174d6 100644 --- a/mon/etcd_schema.js +++ b/mon/etcd_schema.js @@ -69,7 +69,7 @@ const etcd_tree = { block_size: 131072, disk_alignment: 4096, bitmap_granularity: 4096, - immediate_commit: false, // 'all' or 'small' + immediate_commit: 'all', // 'none', 'all' or 'small' // client - configurable online client_max_dirty_bytes: 33554432, client_max_dirty_ops: 1024, @@ -189,7 +189,7 @@ const etcd_tree = { block_size: 131072, bitmap_granularity: 4096, // 'all'/'small'/'none', same as in OSD options - immediate_commit: 'none', + immediate_commit: 'all', pg_stripe_size: 0, root_node?: 'rack1', // restrict pool to OSDs having all of these tags diff --git a/mon/pg_gen.js b/mon/pg_gen.js index ce9e326c..bf09654c 100644 --- a/mon/pg_gen.js +++ b/mon/pg_gen.js @@ -174,7 +174,7 @@ async function generate_pool_pgs(state, global_config, pool_id, osd_tree, levels state.osd.stats, pool_cfg.block_size || global_config.block_size || 131072, pool_cfg.bitmap_granularity || global_config.bitmap_granularity || 4096, - pool_cfg.immediate_commit || global_config.immediate_commit || 'none' + pool_cfg.immediate_commit || global_config.immediate_commit || 'all' ); pool_tree = make_hier_tree(global_config, pool_tree); // First try last_clean_pgs to minimize data movement diff --git a/src/client/etcd_state_client.cpp b/src/client/etcd_state_client.cpp index 36445683..d954532b 100644 --- a/src/client/etcd_state_client.cpp +++ b/src/client/etcd_state_client.cpp @@ -573,7 +573,7 @@ void etcd_state_client_t::load_global_config() { global_bitmap_granularity = DEFAULT_BITMAP_GRANULARITY; } - global_immediate_commit = parse_immediate_commit(global_config["immediate_commit"].string_value()); + global_immediate_commit = parse_immediate_commit(global_config["immediate_commit"].string_value(), IMMEDIATE_ALL); on_load_config_hook(global_config); }); } @@ -867,7 +867,7 @@ void etcd_state_client_t::parse_state(const etcd_kv_t & kv) pc.used_for_fs = pool_item.second["used_for_fs"].as_string(); // Immediate Commit Mode pc.immediate_commit = pool_item.second["immediate_commit"].is_string() - ? parse_immediate_commit(pool_item.second["immediate_commit"].string_value()) + ? parse_immediate_commit(pool_item.second["immediate_commit"].string_value(), IMMEDIATE_ALL) : global_immediate_commit; // PG Stripe Size pc.pg_stripe_size = pool_item.second["pg_stripe_size"].uint64_value(); @@ -1175,10 +1175,11 @@ void etcd_state_client_t::parse_state(const etcd_kv_t & kv) } } -uint32_t etcd_state_client_t::parse_immediate_commit(const std::string & immediate_commit_str) +uint32_t etcd_state_client_t::parse_immediate_commit(const std::string & immediate_commit_str, uint32_t default_value) { - return immediate_commit_str == "all" ? IMMEDIATE_ALL : - (immediate_commit_str == "small" ? IMMEDIATE_SMALL : IMMEDIATE_NONE); + return (immediate_commit_str == "all" ? IMMEDIATE_ALL : + (immediate_commit_str == "small" ? IMMEDIATE_SMALL : + (immediate_commit_str == "none" ? IMMEDIATE_NONE : default_value))); } uint32_t etcd_state_client_t::parse_scheme(const std::string & scheme) diff --git a/src/client/etcd_state_client.h b/src/client/etcd_state_client.h index 0282934a..4d66cb00 100644 --- a/src/client/etcd_state_client.h +++ b/src/client/etcd_state_client.h @@ -157,6 +157,6 @@ public: int address_count(); ~etcd_state_client_t(); - static uint32_t parse_immediate_commit(const std::string & immediate_commit_str); + static uint32_t parse_immediate_commit(const std::string & immediate_commit_str, uint32_t default_value); static uint32_t parse_scheme(const std::string & scheme_str); }; diff --git a/src/cmd/cli.cpp b/src/cmd/cli.cpp index 0a04e2f0..733c775c 100644 --- a/src/cmd/cli.cpp +++ b/src/cmd/cli.cpp @@ -136,7 +136,7 @@ static const char* help_text = " --osd_tags [,]... Put pool only on OSDs tagged with all specified tags\n" " --block_size 128k Put pool only on OSDs with this data block size\n" " --bitmap_granularity 4k Put pool only on OSDs with this logical sector size\n" - " --immediate_commit none Put pool only on OSDs with this or larger immediate_commit (none < small < all)\n" + " --immediate_commit all Put pool only on OSDs with this or larger immediate_commit (none < small < all)\n" " --level_placement Use additional failure domain rules (example: \"dc=112233\")\n" " --raw_placement Specify raw PG generation rules (see documentation for details)\n" " --primary_affinity_tags tags Prefer to put primary copies on OSDs with all specified tags\n" diff --git a/src/cmd/cli_osd_tree.cpp b/src/cmd/cli_osd_tree.cpp index ede2e372..30996848 100644 --- a/src/cmd/cli_osd_tree.cpp +++ b/src/cmd/cli_osd_tree.cpp @@ -134,7 +134,7 @@ resume_1: .reweight = 1, .block_size = (uint32_t)kv.second["data_block_size"].uint64_value(), .bitmap_granularity = (uint32_t)kv.second["bitmap_granularity"].uint64_value(), - .immediate_commit = etcd_state_client_t::parse_immediate_commit(kv.second["immediate_commit"].string_value()), + .immediate_commit = etcd_state_client_t::parse_immediate_commit(kv.second["immediate_commit"].string_value(), IMMEDIATE_NONE), }; if (tree->nodes.find(osd.parent) == tree->nodes.end()) { diff --git a/src/cmd/cli_pool_cfg.cpp b/src/cmd/cli_pool_cfg.cpp index f2379c41..81d987bf 100644 --- a/src/cmd/cli_pool_cfg.cpp +++ b/src/cmd/cli_pool_cfg.cpp @@ -319,7 +319,7 @@ std::string validate_pool_config(json11::Json::object & new_cfg, json11::Json ol } // immediate_commit - if (!cfg["immediate_commit"].is_null() && !etcd_state_client_t::parse_immediate_commit(cfg["immediate_commit"].string_value())) + if (!cfg["immediate_commit"].is_null() && etcd_state_client_t::parse_immediate_commit(cfg["immediate_commit"].string_value(), UINT32_MAX) == UINT32_MAX) { return "immediate_commit must be one of \"all\", \"small\", or \"none\", but it is "+cfg["immediate_commit"].as_string(); } diff --git a/src/cmd/cli_pool_create.cpp b/src/cmd/cli_pool_create.cpp index 53534f36..0403e2bc 100644 --- a/src/cmd/cli_pool_create.cpp +++ b/src/cmd/cli_pool_create.cpp @@ -448,7 +448,7 @@ resume_8: ? cfg["bitmap_granularity"].uint64_value() : parent->cli->st_cli.global_bitmap_granularity; uint32_t p_immediate_commit = cfg["immediate_commit"].is_string() - ? etcd_state_client_t::parse_immediate_commit(cfg["immediate_commit"].string_value()) + ? etcd_state_client_t::parse_immediate_commit(cfg["immediate_commit"].string_value(), IMMEDIATE_ALL) : parent->cli->st_cli.global_immediate_commit; for (size_t i = 0; i < osd_stats.size(); i++) @@ -459,7 +459,7 @@ resume_8: if (!os["data_block_size"].is_null() && os["data_block_size"] != p_block_size || !os["bitmap_granularity"].is_null() && os["bitmap_granularity"] != p_bitmap_granularity || !os["immediate_commit"].is_null() && - etcd_state_client_t::parse_immediate_commit(os["immediate_commit"].string_value()) < p_immediate_commit) + etcd_state_client_t::parse_immediate_commit(os["immediate_commit"].string_value(), IMMEDIATE_NONE) < p_immediate_commit) { accepted_nodes.erase(osd_num); } diff --git a/tests/run_3osds.sh b/tests/run_3osds.sh index 40df4b9c..d0d65422 100644 --- a/tests/run_3osds.sh +++ b/tests/run_3osds.sh @@ -23,7 +23,7 @@ if [ "$IMMEDIATE_COMMIT" != "" ]; then $ETCDCTL put /vitastor/config/global '{"recovery_queue_depth":1,"recovery_tune_util_low":1,"immediate_commit":"all","client_enable_writeback":true,"client_max_writeback_iodepth":32'$GLOBAL_CONFIG'}' else NO_SAME="--journal_sector_buffer_count 1024 --log_level 10 --etcd_stats_interval 5 --min_flusher_count 16" - $ETCDCTL put /vitastor/config/global '{"recovery_queue_depth":1,"recovery_tune_util_low":1,"client_enable_writeback":true,"client_max_writeback_iodepth":32'$GLOBAL_CONFIG'}' + $ETCDCTL put /vitastor/config/global '{"recovery_queue_depth":1,"recovery_tune_util_low":1,"immediate_commit":"none","client_enable_writeback":true,"client_max_writeback_iodepth":32'$GLOBAL_CONFIG'}' fi start_osd_on() diff --git a/tests/test_failure_domain.sh b/tests/test_failure_domain.sh index cc5d2289..d028b8fe 100755 --- a/tests/test_failure_domain.sh +++ b/tests/test_failure_domain.sh @@ -3,7 +3,7 @@ . `dirname $0`/common.sh TIME=$(date '+%s') -$ETCDCTL put /vitastor/config/global '{"placement_levels":{"rack":1,"host":2,"osd":3}}' +$ETCDCTL put /vitastor/config/global '{"placement_levels":{"rack":1,"host":2,"osd":3},"immediate_commit":"none"}' $ETCDCTL put /vitastor/config/node_placement '{"rack1":{"level":"rack"},"rack2":{"level":"rack"},"host1":{"level":"host","parent":"rack1"},"host2":{"level":"host","parent":"rack1"},"host3":{"level":"host","parent":"rack2"},"host4":{"level":"host","parent":"rack2"}}' $ETCDCTL put /vitastor/osd/stats/1 '{"host":"host1","size":1073741824,"time":"'$TIME'"}' $ETCDCTL put /vitastor/osd/stats/2 '{"host":"host1","size":1073741824,"time":"'$TIME'"}' diff --git a/tests/test_move_reappear.sh b/tests/test_move_reappear.sh index 2dd0817c..e647aa9f 100755 --- a/tests/test_move_reappear.sh +++ b/tests/test_move_reappear.sh @@ -11,7 +11,7 @@ for i in $(seq 1 $OSD_COUNT); do eval OSD${i}_PID=$! done -$ETCDCTL put /vitastor/config/pools '{"1":{"name":"testpool","scheme":"replicated","pg_size":2,"pg_minsize":1,"pg_count":1,"failure_domain":"osd"}}' +$ETCDCTL put /vitastor/config/pools '{"1":{"name":"testpool","scheme":"replicated","pg_size":2,"pg_minsize":1,"pg_count":1,"failure_domain":"osd","immediate_commit":"none"}}' $ETCDCTL put /vitastor/config/pgs '{"items":{"1":{"1":{"osd_set":[1,0],"primary":1}}}}' diff --git a/tests/test_osd_tags.sh b/tests/test_osd_tags.sh index 405088a7..7327f4e8 100755 --- a/tests/test_osd_tags.sh +++ b/tests/test_osd_tags.sh @@ -19,7 +19,7 @@ $ETCDCTL put /vitastor/osd/stats/5 '{"host":"stor3","size":1073741824,"time":"'$ $ETCDCTL put /vitastor/osd/stats/6 '{"host":"stor3","size":1073741824,"time":"'$TIME'"}' $ETCDCTL put /vitastor/osd/stats/7 '{"host":"stor4","size":1073741824,"time":"'$TIME'"}' $ETCDCTL put /vitastor/osd/stats/8 '{"host":"stor4","size":1073741824,"time":"'$TIME'"}' -$ETCDCTL put /vitastor/config/pools '{"1":{"name":"testpool","scheme":"replicated","pg_size":2,"pg_minsize":2,"pg_count":16,"failure_domain":"host","osd_tags":["a"]}}' +$ETCDCTL put /vitastor/config/pools '{"1":{"name":"testpool","scheme":"replicated","pg_size":2,"pg_minsize":2,"pg_count":16,"failure_domain":"host","osd_tags":["a"],"immediate_commit":"none"}}' node mon/mon-main.js --etcd_address $ETCD_URL --etcd_prefix "/vitastor" >>./testdata/mon.log 2>&1 & MON_PID=$! diff --git a/tests/test_root_node.sh b/tests/test_root_node.sh index 6542202c..c8118336 100755 --- a/tests/test_root_node.sh +++ b/tests/test_root_node.sh @@ -3,7 +3,7 @@ . `dirname $0`/common.sh TIME=$(date '+%s') -$ETCDCTL put /vitastor/config/global '{"placement_levels":{"rack":100,"host":101,"osd":102}}' +$ETCDCTL put /vitastor/config/global '{"placement_levels":{"rack":100,"host":101,"osd":102},"immediate_commit":"none"}' $ETCDCTL put /vitastor/config/node_placement '{"rack1":{"level":"rack"},"rack2":{"level":"rack"},"stor1":{"level":"host","parent":"rack1"},"stor2":{"level":"host","parent":"rack1"},"stor3":{"level":"host","parent":"rack2"},"stor4":{"level":"host","parent":"rack2"}}' $ETCDCTL put /vitastor/osd/stats/1 '{"host":"stor1","size":1073741824,"time":"'$TIME'"}' $ETCDCTL put /vitastor/osd/stats/2 '{"host":"stor1","size":1073741824,"time":"'$TIME'"}'