Support local reads in client

2025-05-10 16:42:02 +03:00
parent 7e6a95c678
commit 90b1de307b
21 changed files with 295 additions and 19 deletions
@@ -720,6 +720,24 @@ jobs:
          echo ""
        done

+  test_heal_local_read:
+    runs-on: ubuntu-latest
+    needs: build
+    container: ${{env.TEST_IMAGE}}:${{github.sha}}
+    steps:
+    - name: Run test
+      id: test
+      timeout-minutes: 10
+      run: TEST_NAME=local_read POOLCFG='"local_reads":"random",' /root/vitastor/tests/test_heal.sh
+    - name: Print logs
+      if: always() && steps.test.outcome == 'failure'
+      run: |
+        for i in /root/vitastor/testdata/*.log /root/vitastor/testdata/*.txt; do
+          echo "-------- $i --------"
+          cat $i
+          echo ""
+        done
+
  test_heal_ec:
    runs-on: ubuntu-latest
    needs: build
@@ -24,6 +24,7 @@ affect their interaction with the cluster.
 - [nbd_max_devices](#nbd_max_devices)
 - [nbd_max_part](#nbd_max_part)
 - [osd_nearfull_ratio](#osd_nearfull_ratio)
+- [hostname](#hostname)

 ## client_iothread_count

@@ -215,3 +216,12 @@ just one OSD becomes 100 % full!
 However, unlike in Ceph, 100 % full Vitastor OSDs don't crash (in Ceph they're
 unable to start at all), so you'll be able to recover from "out of space" errors
 without destroying and recreating OSDs.
+
+## hostname
+
+- Type: string
+- Can be changed online: yes
+
+Clients use host name to find their distance to OSDs when [localized reads](pool.en.md#local_reads)
+are enabled. By default, standard [gethostname](https://man7.org/linux/man-pages/man2/gethostname.2.html)
+function is used to determine host name, but you can also override it with this parameter.
@@ -24,6 +24,7 @@
 - [nbd_max_devices](#nbd_max_devices)
 - [nbd_max_part](#nbd_max_part)
 - [osd_nearfull_ratio](#osd_nearfull_ratio)
+- [hostname](#hostname)

 ## client_iothread_count

@@ -219,3 +220,13 @@ RDMA и хотите повысить пиковую производитель
 заполненные на 100% OSD вообще не могут стартовать), так что вы сможете
 восстановить работу кластера после ошибок отсутствия свободного места
 без уничтожения и пересоздания OSD.
+
+## hostname
+
+- Тип: строка
+- Можно менять на лету: да
+
+Клиенты используют имя хоста для определения расстояния до OSD, когда включены
+[локальные чтения](pool.ru.md#local_reads). По умолчанию для определения имени
+хоста используется стандартная функция [gethostname](https://man7.org/linux/man-pages/man2/gethostname.2.html),
+но вы также можете задать имя хоста вручную данным параметром.
@@ -34,6 +34,7 @@ Parameters:
 - [failure_domain](#failure_domain)
 - [level_placement](#level_placement)
 - [raw_placement](#raw_placement)
+- [local_reads](#local_reads)
 - [max_osd_combinations](#max_osd_combinations)
 - [block_size](#block_size)
 - [bitmap_granularity](#bitmap_granularity)
@@ -133,8 +134,8 @@ Pool name.
 ## scheme

 - Type: string
- Required
 - One of: "replicated", "xor", "ec" or "jerasure"
+- Required

 Redundancy scheme used for data in this pool. "jerasure" is an alias for "ec",
 both use Reed-Solomon-Vandermonde codes based on ISA-L or jerasure libraries.
@@ -289,6 +290,26 @@ Examples:
 - EC 4+2 in 3 DC: `any, dc=1 host!=1, dc!=1, dc=3 host!=3, dc!=(1,3), dc=5 host!=5`
 - 1 replica in fixed DC + 2 in random DCs: `dc?=meow, dc!=1, dc!=(1,2)`

+## local_reads
+
+- Type: string
+- One of: "primary", "nearest" or "random"
+- Default: primary
+
+By default, Vitastor serves all read and write requests from the primary OSD of each PG.
+But it can also serve read requests for replicated pools from secondary OSDs in clean PGs
+(active or active+left_on_dead) which may be useful if you have OSDs with different network
+latency to the client - for example, if you have a cross-datacenter setup.
+
+If you set this parameter to "nearest", clients will try to read from the nearest OSD
+in the [Placement Tree](#placement-tree), i.e. from an OSD from the same host or datacenter.
+Distance to different OSDs will be calculated based on client hostname, determined
+automatically or set manually in the [hostname](client.en.md#hostname) parameter.
+
+If you set this parameter to "random", clients will try to distribute read requests over
+all available secondary OSDs. This mode is mainly useful for tests, but, probably, not
+really required in production setups.
+
 ## max_osd_combinations

 - Type: integer
@@ -324,7 +345,8 @@ Read more about this parameter in [Cluster-Wide Disk Layout Parameters](layout-c

 ## immediate_commit

- Type: string, one of "all", "small" and "none"
+- Type: string
+- One of: "all", "small" or "none"
 - Default: none

 Immediate commit setting for this pool. The value from /vitastor/config/global
@@ -33,6 +33,7 @@
 - [failure_domain](#failure_domain)
 - [level_placement](#level_placement)
 - [raw_placement](#raw_placement)
+- [local_reads](#local_reads)
 - [max_osd_combinations](#max_osd_combinations)
 - [block_size](#block_size)
 - [bitmap_granularity](#bitmap_granularity)
@@ -133,8 +134,8 @@ OSD игнорируется и OSD не удаляется из распред
 ## scheme

 - Тип: строка
- Обязательный
 - Возможные значения: "replicated", "xor", "ec" или "jerasure"
+- Обязательный

 Схема избыточности, используемая в данном пуле. "jerasure" - синоним для "ec",
 в обеих схемах используются коды Рида-Соломона-Вандермонда, реализованные на
@@ -287,6 +288,27 @@ meow недоступен".
 - EC 4+2 в 3 датацентрах: `any, dc=1 host!=1, dc!=1, dc=3 host!=3, dc!=(1,3), dc=5 host!=5`
 - 1 копия в фиксированном ДЦ + 2 в других ДЦ: `dc?=meow, dc!=1, dc!=(1,2)`

+## local_reads
+
+- Тип: строка
+- Возможные значения: "primary", "nearest" или "random"
+- По умолчанию: primary
+
+По умолчанию Vitastor обслуживает все запросы чтения и записи с первичного OSD каждой PG.
+Однако, в чистых PG (active или active+left_on_dead) реплицированных пулов также есть
+возможность обслуживать запросы чтения с вторичных OSD, что может быть полезно, если
+у вас сильно отличается время сетевого обращения от клиента к разным OSD - например,
+если у вас несколько дата-центров.
+
+Если данный параметр установлен в значение "nearest", клиенты будут стараться читать с
+ближайших по [Дереву размещения](#дерево-размещения) OSD, то есть, с OSD с того же хоста
+или датацентра. Расстояние до разных OSD будет рассчитываться с помощью имени хоста клиента,
+определяемого автоматически или заданного вручную параметром [hostname](client.ru.md#hostname).
+
+Если данный параметр установлен в значение "random", клиенты будут стараться распределять
+запросы чтения по всем доступным вторичным OSD. Этот режим в основном полезен для тестов,
+но, скорее всего, редко нужен в реальных инсталляциях.
+
 ## max_osd_combinations

 - Тип: целое число
@@ -324,7 +346,8 @@ meow недоступен".

 ## immediate_commit

- Тип: строка "all", "small" или "none"
+- Тип: строка
+- Возможные значения: "all", "small" или "none"
 - По умолчанию: none

 Настройка мгновенного коммита для данного пула. Если не задана, используется
@@ -271,3 +271,15 @@
    заполненные на 100% OSD вообще не могут стартовать), так что вы сможете
    восстановить работу кластера после ошибок отсутствия свободного места
    без уничтожения и пересоздания OSD.
+- name: hostname
+  type: string
+  online: true
+  info: |
+    Clients use host name to find their distance to OSDs when [localized reads](pool.en.md#local_reads)
+    are enabled. By default, standard [gethostname](https://man7.org/linux/man-pages/man2/gethostname.2.html)
+    function is used to determine host name, but you can also override it with this parameter.
+  info_ru: |
+    Клиенты используют имя хоста для определения расстояния до OSD, когда включены
+    [локальные чтения](pool.ru.md#local_reads). По умолчанию для определения имени
+    хоста используется стандартная функция [gethostname](https://man7.org/linux/man-pages/man2/gethostname.2.html),
+    но вы также можете задать имя хоста вручную данным параметром.
@@ -125,6 +125,13 @@ All other client-side components are based on the client library:
  all current read/write operations to it fail with EPIPE error and are retried by clients.
 - After completing all secondary read/write requests, primary OSD sends the response to
  the client.
+- When [localized reads](../config/pool.en.md#local_reads) are enabled for a PG in a
+  replicated pool, and the PG is in an active and clean state (active or
+  active+left_on_dead), the client can send the request to one of secondary OSDs instead
+  of the primary. Secondary OSD checks the [PG lock](../config/osd.en.md#enable_pg_locks)
+  and handles the request locally without communicating to the primary. PG lock is required
+  for the secondary OSD to know for sure that the PG is in clean state and not switching
+  primary at the moment.

 ### Nuances of request handling

@@ -125,6 +125,12 @@
  и если любое из этих соединений отключается, PG перезапускается, а все текущие запросы чтения
  и записи в неё завершаются с ошибкой EPIPE, после чего повторяются клиентами.
 - После завершения всех вторичных операций чтения/записи первичный OSD отправляет ответ клиенту.
+- Если в реплицированном пуле включены [локализованные чтения](../config/pool.ru.md#local_reads),
+  а PG находится в чистом активном состоянии (active или active+left_on_dead), клиент может
+  послать запрос к одному из вторичных OSD вместо первичного. Вторичный OSD проверяет
+  [блокировку PG](../config/osd.ru.md#enable_pg_locks) и обрабатывает запрос локально, не
+  обращаясь к первичному. Блокировка PG здесь нужна, чтобы вторичный OSD мог точно знать,
+  что PG находится в чистом состоянии и не переключается на другой первичный OSD.

 ### Особенности обработки запросов

@@ -25,6 +25,7 @@
 - Recovery of degraded blocks
 - Rebalancing (data movement between OSDs)
 - [Lazy fsync support](../config/layout-cluster.en.md#immediate_commit)
+- [Localized read support](../config/pool.en.md#local_reads) for cross-datacenter setup optimization
 - Per-OSD and per-image I/O and space usage statistics in etcd
 - Snapshots and copy-on-write image clones
 - [Write throttling to smooth random write workloads in SSD+HDD configurations](../config/osd.en.md#throttle_small_writes)
@@ -25,6 +25,7 @@
 - Восстановление деградированных блоков
 - Ребаланс, то есть перемещение данных между OSD (дисками)
 - [Поддержка "ленивого" fsync (fsync не на каждую операцию)](../config/layout-cluster.ru.md#immediate_commit)
+- [Локальные чтения](../config/pool.ru.md#local_reads) для оптимизации при нескольких датацентрах
 - Сбор статистики ввода/вывода в etcd
 - Статистика операций ввода/вывода и занятого места в разрезе инодов
 - Именование инодов через хранение их метаданных в etcd
@@ -397,6 +397,7 @@ Optional parameters:
 | `--immediate_commit none`      | Put pool only on OSDs with this or larger immediate_commit (none < small < all) |
 | `--level_placement <rules>`    | Use additional failure domain rules (example: "dc=112233")                 |
 | `--raw_placement <rules>`      | Specify raw PG generation rules ([details](../config/pool.en.md#raw_placement)) |
+| `--local_reads primary`        | Local read policy for replicated pools: primary, nearest or random         |
 | `--primary_affinity_tags tags` | Prefer to put primary copies on OSDs with all specified tags               |
 | `--scrub_interval <time>`      | Enable regular scrubbing for this pool. Format: number + unit s/m/h/d/M/y  |
 | `--used_for_app fs:<name>`     | Mark pool as used for VitastorFS with metadata in image `<name>`           |
@@ -414,6 +414,7 @@ OSD  PARENT            UP    SIZE  USED%    TAGS          WEIGHT  BLOCK  BITMAP
 | `--immediate_commit none`      | ...только OSD с этим или большим immediate_commit (none < small < all)     |
 | `--level_placement <rules>`    | Задать правила дополнительных доменов отказа (пример: "dc=112233")         |
 | `--raw_placement <rules>`      | Задать низкоуровневые правила генерации PG ([детали](../config/pool.ru.md#raw_placement)) |
+| `--local_reads primary`        | Политика локальных чтений для реплик: primary, nearest или random          |
 | `--primary_affinity_tags tags` | Предпочитать OSD со всеми данными тегами для роли первичных                |
 | `--scrub_interval <time>`      | Включить скрабы с заданным интервалом времени (число + единица s/m/h/d/M/y) |
 | `--pg_stripe_size <number>`    | Увеличить блок группировки объектов по PG                                  |
@@ -3,6 +3,7 @@

 #include <stdexcept>
 #include <assert.h>
+#include "pg_states.h"
 #include "cluster_client_impl.h"
 #include "json_util.h"

@@ -57,6 +58,7 @@ cluster_client_t::cluster_client_t(ring_loop_t *ringloop, timerfd_manager_t *tfd
    st_cli.on_change_osd_state_hook = [this](uint64_t peer_osd) { on_change_osd_state_hook(peer_osd); };
    st_cli.on_change_pool_config_hook = [this]() { on_change_pool_config_hook(); };
    st_cli.on_change_pg_state_hook = [this](pool_id_t pool_id, pg_num_t pg_num, osd_num_t prev_primary) { on_change_pg_state_hook(pool_id, pg_num, prev_primary); };
+    st_cli.on_change_node_placement_hook = [this]() { on_change_node_placement_hook(); };
    st_cli.on_load_pgs_hook = [this](bool success) { on_load_pgs_hook(success); };
    st_cli.on_reload_hook = [this]() { st_cli.load_global_config(); };

@@ -470,11 +472,95 @@ void cluster_client_t::on_load_config_hook(json11::Json::object & etcd_global_co
    }
    // log_level
    log_level = config["log_level"].uint64_value();
+    // hostname
+    conf_hostname = config["hostname"].string_value();
+    auto new_hostname = conf_hostname != "" ? conf_hostname : gethostname_str();
+    if (new_hostname != client_hostname)
+    {
+        self_tree_metrics.clear();
+        client_hostname = new_hostname;
+    }
    msgr.parse_config(config);
    st_cli.parse_config(config);
    st_cli.load_pgs();
 }

+osd_num_t cluster_client_t::select_random_osd(const std::vector<osd_num_t> & osds)
+{
+    osd_num_t alive_set[osds.size()];
+    int alive_count = 0;
+    for (auto & osd_num: osds)
+    {
+        if (!st_cli.peer_states[osd_num].is_null())
+            alive_set[alive_count++] = osd_num;
+    }
+    if (!alive_count)
+        return 0;
+    return alive_set[lrand48() % alive_count];
+}
+
+osd_num_t cluster_client_t::select_nearest_osd(const std::vector<osd_num_t> & osds)
+{
+    if (!self_tree_metrics.size())
+    {
+        std::string cur_id = client_hostname;
+        int metric = 0;
+        while (self_tree_metrics.find(cur_id) == self_tree_metrics.end())
+        {
+            self_tree_metrics[cur_id] = metric++;
+            json11::Json cur_placement = st_cli.node_placement[cur_id];
+            cur_id = cur_placement["parent"].string_value();
+        }
+        if (cur_id != "")
+        {
+            self_tree_metrics[""] = metric++;
+        }
+    }
+    osd_num_t best_osd = 0;
+    int best_metric = -1;
+    for (auto & osd_num: osds)
+    {
+        int metric = -1;
+        auto met_it = osd_tree_metrics.find(osd_num);
+        if (met_it != osd_tree_metrics.end())
+        {
+            metric = met_it->second;
+        }
+        else
+        {
+            auto & peer_state = st_cli.peer_states[osd_num];
+            if (!peer_state.is_null())
+            {
+                metric = self_tree_metrics[""];
+                bool first = true;
+                std::string cur_id = std::to_string(osd_num);
+                std::set<std::string> seen;
+                while (seen.find(cur_id) == seen.end())
+                {
+                    seen.insert(cur_id);
+                    json11::Json cur_placement = st_cli.node_placement[cur_id];
+                    std::string cur_parent = cur_placement["parent"].string_value();
+                    cur_id = (!first || cur_parent != "" ? cur_parent : peer_state["host"].string_value());
+                    first = false;
+                    auto self_it = self_tree_metrics.find(cur_id);
+                    if (self_it != self_tree_metrics.end())
+                    {
+                        metric = self_it->second;
+                        break;
+                    }
+                }
+            }
+            osd_tree_metrics[osd_num] = metric;
+        }
+        if (metric >= 0 && (best_metric < 0 || metric < best_metric))
+        {
+            best_metric = metric;
+            best_osd = osd_num;
+        }
+    }
+    return best_osd;
+}
+
 void cluster_client_t::on_load_pgs_hook(bool success)
 {
    for (auto pool_item: st_cli.pool_config)
@@ -546,6 +632,7 @@ bool cluster_client_t::get_immediate_commit(uint64_t inode)

 void cluster_client_t::on_change_osd_state_hook(uint64_t peer_osd)
 {
+    osd_tree_metrics.erase(peer_osd);
    if (msgr.wanted_peers.find(peer_osd) != msgr.wanted_peers.end())
    {
        msgr.connect_peer(peer_osd, st_cli.peer_states[peer_osd]);
@@ -553,6 +640,12 @@ void cluster_client_t::on_change_osd_state_hook(uint64_t peer_osd)
    }
 }

+void cluster_client_t::on_change_node_placement_hook()
+{
+    osd_tree_metrics.clear();
+    self_tree_metrics.clear();
+}
+
 bool cluster_client_t::is_ready()
 {
    return pgs_loaded;
@@ -1221,6 +1314,17 @@ int cluster_client_t::try_send(cluster_op_t *op, int i)
        !pg_it->second.pause && pg_it->second.cur_primary)
    {
        osd_num_t primary_osd = pg_it->second.cur_primary;
+        if (pool_cfg.local_reads != POOL_LOCAL_READ_PRIMARY &&
+            pool_cfg.scheme == POOL_SCHEME_REPLICATED &&
+            (op->opcode == OSD_OP_READ || op->opcode == OSD_OP_READ_BITMAP || op->opcode == OSD_OP_READ_CHAIN_BITMAP) &&
+            (pg_it->second.cur_state == PG_ACTIVE || pg_it->second.cur_state == (PG_ACTIVE|PG_LEFT_ON_DEAD)))
+        {
+            osd_num_t nearest_osd = pool_cfg.local_reads == POOL_LOCAL_READ_NEAREST
+                ? select_nearest_osd(pg_it->second.target_set)
+                : select_random_osd(pg_it->second.target_set);
+            if (nearest_osd)
+                primary_osd = nearest_osd;
+        }
        part->osd_num = primary_osd;
        auto peer_it = msgr.osd_peer_fds.find(primary_osd);
        if (peer_it != msgr.osd_peer_fds.end())
@@ -100,6 +100,7 @@ public:
    uint64_t client_max_buffered_bytes = 0;
    uint64_t client_max_buffered_ops = 0;
    uint64_t client_max_writeback_iodepth = 0;
+    std::string conf_hostname;

    int log_level = 0;
    int client_retry_interval = 50; // ms
@@ -107,6 +108,10 @@ public:
    bool client_retry_enospc = true;
    int client_wait_up_timeout = 16; // sec (for listings)

+    std::string client_hostname;
+    std::map<std::string, int> self_tree_metrics;
+    std::map<osd_num_t, int> osd_tree_metrics;
+
    int retry_timeout_id = -1;
    int retry_timeout_duration = 0;
    std::vector<cluster_op_t*> offline_ops;
@@ -161,11 +166,14 @@ protected:
 protected:
    bool affects_osd(uint64_t inode, uint64_t offset, uint64_t len, osd_num_t osd);
    bool affects_pg(uint64_t inode, uint64_t offset, uint64_t len, pool_id_t pool_id, pg_num_t pg_num);
+
    void on_load_config_hook(json11::Json::object & config);
    void on_load_pgs_hook(bool success);
    void on_change_pool_config_hook();
    void on_change_pg_state_hook(pool_id_t pool_id, pg_num_t pg_num, osd_num_t prev_primary);
    void on_change_osd_state_hook(uint64_t peer_osd);
+    void on_change_node_placement_hook();
+
    void execute_internal(cluster_op_t *op);
    void unshift_op(cluster_op_t *op);
    int continue_rw(cluster_op_t *op);
@@ -191,5 +199,8 @@ protected:
    bool check_finish_listing(inode_list_t *lst);
    void continue_raw_ops(osd_num_t peer_osd);

+    osd_num_t select_random_osd(const std::vector<osd_num_t> & osds);
+    osd_num_t select_nearest_osd(const std::vector<osd_num_t> & osds);
+
    friend class writeback_cache_t;
 };
@@ -922,6 +922,19 @@ void etcd_state_client_t::parse_state(const etcd_kv_t & kv)
                pc.used_for_app = "fs:"+pc.used_for_app;
            else
                pc.used_for_app = pool_item.second["used_for_app"].as_string();
+            // Local Read Configuration
+            std::string local_reads = pool_item.second["local_reads"].string_value();
+            if (local_reads == "nearest")
+                pc.local_reads = POOL_LOCAL_READ_NEAREST;
+            else if (local_reads == "random")
+                pc.local_reads = POOL_LOCAL_READ_RANDOM;
+            else if (local_reads == "" || local_reads == "primary")
+                pc.local_reads = POOL_LOCAL_READ_PRIMARY;
+            else
+            {
+                pc.local_reads = POOL_LOCAL_READ_PRIMARY;
+                fprintf(stderr, "Warning: Pool %u has invalid local_reads, using 'primary'\n", pool_id);
+            }
            // Immediate Commit Mode
            pc.immediate_commit = pool_item.second["immediate_commit"].is_string()
                ? parse_immediate_commit(pool_item.second["immediate_commit"].string_value(), IMMEDIATE_ALL)
@@ -1256,6 +1269,13 @@ void etcd_state_client_t::parse_state(const etcd_kv_t & kv)
            }
        }
    }
+    else if (key == etcd_prefix+"/config/node_placement")
+    {
+        // <etcd_prefix>/config/node_placement
+        node_placement = value;
+        if (on_change_node_placement_hook)
+            on_change_node_placement_hook();
+    }
 }

 uint32_t etcd_state_client_t::parse_immediate_commit(const std::string & immediate_commit_str, uint32_t default_value)
@@ -25,6 +25,10 @@
 #define IMMEDIATE_ALL 2
 #endif

+#define POOL_LOCAL_READ_PRIMARY 0
+#define POOL_LOCAL_READ_NEAREST 1
+#define POOL_LOCAL_READ_RANDOM 2
+
 struct etcd_kv_t
 {
    std::string key;
@@ -48,21 +52,22 @@ struct pg_config_t

 struct pool_config_t
 {
-    bool exists;
-    pool_id_t id;
+    bool exists = false;
+    pool_id_t id = 0;
    std::string name;
-    uint64_t scheme;
-    uint64_t pg_size, pg_minsize, parity_chunks;
-    uint32_t data_block_size, bitmap_granularity, immediate_commit;
-    uint64_t pg_count;
-    uint64_t real_pg_count;
+    uint64_t scheme = 0;
+    uint64_t pg_size = 0, pg_minsize = 0, parity_chunks = 0;
+    uint32_t data_block_size = 0, bitmap_granularity = 0, immediate_commit = 0;
+    uint64_t pg_count = 0;
+    uint64_t real_pg_count = 0;
    std::string failure_domain;
-    uint64_t max_osd_combinations;
-    uint64_t pg_stripe_size;
+    uint64_t max_osd_combinations = 0;
+    uint64_t pg_stripe_size = 0;
    std::map<pg_num_t, pg_config_t> pg_config;
-    uint64_t scrub_interval;
+    uint64_t scrub_interval = 0;
    std::string used_for_app;
-    int backfillfull;
+    int backfillfull = 0;
+    int local_reads = 0;
 };

 struct inode_config_t
@@ -130,6 +135,7 @@ public:
    std::set<osd_num_t> seen_peers;
    std::map<inode_t, inode_config_t> inode_config;
    std::map<std::string, inode_t> inode_by_name;
+    json11::Json node_placement;

    std::function<void(std::map<std::string, etcd_kv_t> &)> on_change_hook;
    std::function<void(json11::Json::object &)> on_load_config_hook;
@@ -140,6 +146,7 @@ public:
    std::function<void(pool_id_t, pg_num_t, osd_num_t)> on_change_pg_state_hook;
    std::function<void(pool_id_t, pg_num_t)> on_change_pg_history_hook;
    std::function<void(osd_num_t)> on_change_osd_state_hook;
+    std::function<void()> on_change_node_placement_hook;
    std::function<void()> on_reload_hook;
    std::function<void(inode_t, bool)> on_inode_change_hook;
    std::function<void(http_co_t *)> on_start_watcher_hook;
@@ -185,6 +185,7 @@ static const char* help_text =
    "    --immediate_commit all        Put pool only on OSDs with this or larger immediate_commit (none < small < all)\n"
    "    --level_placement <rules>     Use additional failure domain rules (example: \"dc=112233\")\n"
    "    --raw_placement <rules>       Specify raw PG generation rules (see documentation for details)\n"
+    "    --local_reads primary         Local read policy for replicated pools: primary, nearest or random\n"
    "    --primary_affinity_tags tags  Prefer to put primary copies on OSDs with all specified tags\n"
    "    --scrub_interval <time>       Enable regular scrubbing for this pool. Format: number + unit s/m/h/d/M/y\n"
    "    --used_for_app fs:<name>      Mark pool as used for VitastorFS with metadata in image <name>\n"
@@ -91,7 +91,7 @@ std::string validate_pool_config(json11::Json::object & new_cfg, json11::Json ol
        }
        else if (key == "name" || key == "scheme" || key == "immediate_commit" ||
            key == "failure_domain" || key == "root_node" || key == "scrub_interval" || key == "used_for_app" ||
-            key == "used_for_fs" || key == "raw_placement")
+            key == "used_for_fs" || key == "raw_placement" || key == "local_reads")
        {
            if (!value.is_string())
            {
@@ -165,6 +165,10 @@ std::string validate_pool_config(json11::Json::object & new_cfg, json11::Json ol
        new_cfg["used_for_app"] = "fs:"+new_cfg["used_for_fs"].string_value();
        new_cfg.erase("used_for_fs");
    }
+    if (new_cfg.find("local_reads") != new_cfg.end() && new_cfg["local_reads"].string_value() == "primary")
+    {
+        new_cfg.erase("local_reads");
+    }

    // Prevent autovivification of object keys. Now we don't modify the config, we just check it
    json11::Json cfg = new_cfg;
@@ -340,5 +344,19 @@ std::string validate_pool_config(json11::Json::object & new_cfg, json11::Json ol
        }
    }

+    // local_reads
+    if (!cfg["local_reads"].is_null())
+    {
+        auto lr = cfg["local_reads"].string_value();
+        if (lr != "" && lr != "primary" && lr != "nearest" && lr != "random")
+        {
+            return "local_reads must be '', 'primary', 'nearest' or 'random', but it is "+cfg["local_reads"].string_value();
+        }
+        if (lr != "" && lr != "primary" && scheme != POOL_SCHEME_REPLICATED)
+        {
+            return "EC pools don't support localized reads, please clear local_reads or set it to 'primary'";
+        }
+    }
+
    return "";
 }
@@ -504,6 +504,7 @@ resume_3:
                { "failure_domain", "Failure domain" },
                { "root_node", "Root node" },
                { "osd_tags_fmt", "OSD tags" },
+                { "local_reads", "Local read policy" },
                { "primary_affinity_tags_fmt", "Primary affinity" },
                { "block_size_fmt", "Block size" },
                { "bitmap_granularity_fmt", "Bitmap granularity" },
@@ -68,17 +68,17 @@ if [ "$SCHEME" = "ec" ]; then
    PG_SIZE=${PG_SIZE:-5}
    PG_MINSIZE=${PG_MINSIZE:-4}
    PG_DATA_SIZE=${PG_DATA_SIZE:-3}
-    POOLCFG='"scheme":"ec","parity_chunks":'$((PG_SIZE-PG_DATA_SIZE))
+    POOLCFG="$POOLCFG"'"scheme":"ec","parity_chunks":'$((PG_SIZE-PG_DATA_SIZE))
 elif [ "$SCHEME" = "xor" ]; then
    PG_SIZE=${PG_SIZE:-3}
    PG_MINSIZE=${PG_MINSIZE:-3}
    PG_DATA_SIZE=$((PG_SIZE-1))
-    POOLCFG='"scheme":"xor","parity_chunks":1'
+    POOLCFG="$POOLCFG"'"scheme":"xor","parity_chunks":1'
 else
    PG_SIZE=${PG_SIZE:-2}
    PG_MINSIZE=${PG_MINSIZE:-2}
    PG_DATA_SIZE=1
-    POOLCFG='"scheme":"replicated"'
+    POOLCFG="$POOLCFG"'"scheme":"replicated"'
 fi
 POOLCFG='"name":"testpool","failure_domain":"osd",'$POOLCFG
 $ETCDCTL put /vitastor/config/pools '{"1":{'$POOLCFG',"pg_size":'$PG_SIZE',"pg_minsize":'$PG_MINSIZE',"pg_count":'$PG_COUNT'}}'
@@ -63,6 +63,7 @@ SCHEME=xor ./test_write.sh
 ./test_write_no_same.sh

 PG_SIZE=2 ./test_heal.sh
+TEST_NAME=local_read POOLCFG='"local_reads":"random",' ./test_heal.sh
 SCHEME=ec ./test_heal.sh
 ANTIETCD=1 ./test_heal.sh