Destroy qemu driver under lock (may fix crashes after attach/detach?)

docs/config/client.en.md

@@ -13,6 +13,7 @@ affect their interaction with the cluster.
 - [client_retry_interval](#client_retry_interval)
 - [client_eio_retry_interval](#client_eio_retry_interval)
 - [client_retry_enospc](#client_retry_enospc)
+- [client_wait_up_timeout](#client_wait_up_timeout)
 - [client_max_dirty_bytes](#client_max_dirty_bytes)
 - [client_max_dirty_ops](#client_max_dirty_ops)
 - [client_enable_writeback](#client_enable_writeback)
@@ -70,6 +71,19 @@ and clients are not blocked and just get EIO error code instead.
 Retry writes on out of space errors to wait until some space is freed on
 OSDs.
 
+## client_wait_up_timeout
+
+- Type: seconds
+- Default: 16
+- Can be changed online: yes
+
+Wait for this number of seconds until PGs are up when doing operations
+which require all PGs to be up. Currently only used by object listings
+in delete and merge-based commands ([vitastor-cli rm](../usage/cli.en.md#rm), merge and so on).
+
+The default value is calculated as `1 + OSD lease timeout`, which is
+`1 + etcd_report_interval + max_etcd_attempts*2*etcd_quick_timeout`.
+
 ## client_max_dirty_bytes
 
 - Type: integer

docs/config/client.ru.md

@@ -13,6 +13,7 @@
 - [client_retry_interval](#client_retry_interval)
 - [client_eio_retry_interval](#client_eio_retry_interval)
 - [client_retry_enospc](#client_retry_enospc)
+- [client_wait_up_timeout](#client_wait_up_timeout)
 - [client_max_dirty_bytes](#client_max_dirty_bytes)
 - [client_max_dirty_ops](#client_max_dirty_ops)
 - [client_enable_writeback](#client_enable_writeback)
@@ -72,6 +73,19 @@ RDMA и хотите повысить пиковую производитель
 Повторять запросы записи, завершившиеся с ошибками нехватки места, т.е.
 ожидать, пока на OSD не освободится место.
 
+## client_wait_up_timeout
+
+- Тип: секунды
+- Значение по умолчанию: 16
+- Можно менять на лету: да
+
+Время ожидания поднятия PG при операциях, требующих активности всех PG.
+В данный момент используется листингами объектов в командах, использующих
+удаление и слияние ([vitastor-cli rm](../usage/cli.ru.md#rm), merge и подобные).
+
+Значение по умолчанию вычисляется как `1 + время lease OSD`, равное
+`1 + etcd_report_interval + max_etcd_attempts*2*etcd_quick_timeout`.
+
 ## client_max_dirty_bytes
 
 - Тип: целое число

docs/config/osd.en.md

@@ -316,7 +316,7 @@ for hot data and slower disks - HDDs and maybe SATA SSDs - but will slightly
 decrease write performance for fast disks because page cache is an overhead
 itself.
 
-Choose "directsync" to use [immediate_commit](layout-cluster.ru.md#immediate_commit)
+Choose "directsync" to use [immediate_commit](layout-cluster.en.md#immediate_commit)
 (which requires disable_data_fsync) with drives having write-back cache
 which can't be turned off, for example, Intel Optane. Also note that *some*
 desktop SSDs (for example, HP EX950) may ignore O_SYNC thus making

docs/config/pool.en.md

@@ -43,7 +43,7 @@ Parameters:
 - [osd_tags](#osd_tags)
 - [primary_affinity_tags](#primary_affinity_tags)
 - [scrub_interval](#scrub_interval)
-- [used_for_fs](#used_for_fs)
+- [used_for_app](#used_for_app)
 
 Examples:
 
@@ -377,24 +377,37 @@ of the OSDs containing a data chunk for a PG.
 Automatic scrubbing interval for this pool. Overrides
 [global scrub_interval setting](osd.en.md#scrub_interval).
 
-## used_for_fs
+## used_for_app
 
 - Type: string
 
-If non-empty, the pool is marked as used for VitastorFS with metadata stored
-in block image (regular Vitastor volume) named as the value of this pool parameter.
+If non-empty, the pool is marked as used for a separate application, for example,
+VitastorFS or S3, which allocates Vitastor volume IDs by itself and does not use
+image/inode metadata in etcd.
 
-When a pool is marked as used for VitastorFS, regular block volume creation in it
+When a pool is marked as used for such app, regular block volume creation in it
 is disabled (vitastor-cli refuses to create images without --force) to protect
-the user from block volume and FS file ID collisions and data loss.
+the user from block volume and FS/S3 volume ID collisions and data loss.
 
-[vitastor-nfs](../usage/nfs.ru.md), in its turn, refuses to use pools not marked
+Also such pools do not calculate per-inode space usage statistics in etcd because
+using it for an external application implies that it may contain a very large
+number of volumes and their statistics may take too much space in etcd.
+
+Setting used_for_app to `fs:<name>` tells Vitastor that the pool is used for VitastorFS
+with VitastorKV metadata base stored in a block image (regular Vitastor volume) named
+`<name>`.
+
+[vitastor-nfs](../usage/nfs.en.md), in its turn, refuses to use pools not marked
 for the corresponding FS when starting. This also implies that you can use one
 pool only for one VitastorFS.
 
-The second thing that is disabled for VitastorFS pools is reporting per-inode space
-usage statistics in etcd because a FS pool may store a very large number of files
-and statistics for them all would take a lot of space in etcd.
+If you plan to use the pool for S3, set its used_for_app to `s3:<name>`. `<name>` may
+be basically anything you want (for example, `s3:standard`) - it's not validated
+by Vitastor S3 components in any way.
+
+All other values except prefixed with `fs:` or `s3:` may be used freely and don't
+mean anything special for Vitastor core components. For now, you can use them as
+you wish.
 
 # Examples
 

docs/config/pool.ru.md

@@ -42,7 +42,7 @@
 - [osd_tags](#osd_tags)
 - [primary_affinity_tags](#primary_affinity_tags)
 - [scrub_interval](#scrub_interval)
-- [used_for_fs](#used_for_fs)
+- [used_for_app](#used_for_app)
 
 Примеры:
 
@@ -256,7 +256,7 @@ PG в Vitastor эферемерны, то есть вы можете менят
 
 ## raw_placement
 
-- Type: string
+- Тип: строка
 
 Низкоуровневые правила генерации PG в форме DSL (доменно-специфичного языка).
 Используйте, только если действительно знаете, зачем вам это надо :)
@@ -383,26 +383,39 @@ OSD с "all".
 Интервал скраба, то есть, автоматической фоновой проверки данных для данного пула.
 Переопределяет [глобальную настройку scrub_interval](osd.ru.md#scrub_interval).
 
-## used_for_fs
+## used_for_app
 
-- Type: string
+- Тип: строка
 
-Если непусто, пул помечается как используемый для файловой системы VitastorFS с
-метаданными, хранимыми в блочном образе Vitastor с именем, равным значению
-этого параметра.
+Если непусто, пул помечается как используемый для отдельного приложения, например,
+для VitastorFS или S3, которое распределяет ID образов в пуле само и не использует
+метаданные образов/инодов в etcd.
 
-Когда пул помечается как используемый для VitastorFS, создание обычных блочных
-образов в нём отключается (vitastor-cli отказывается создавать образы без --force),
-чтобы защитить пользователя от коллизий ID файлов и блочных образов и, таким
-образом, от потери данных.
+Когда пул помечается используемым для такого приложения, создание обычных блочных
+образов в нём запрещается (vitastor-cli отказывается создавать образы без --force),
+чтобы защитить пользователя от коллизий ID блочных образов и томов ФС/S3, и,
+таким образом, от потери данных.
+
+Также для таких пулов отключается передача статистики в etcd по отдельным инодам,
+так как использование для внешнего приложения подразумевает, что пул может содержать
+очень много томов и их статистика может занять слишком много места в etcd.
+
+Установка used_for_app в значение `fs:<name>` сообщает о том, что пул используется
+для VitastorFS с базой метаданных VitastorKV, хранимой в блочном образе с именем
+`<name>`.
 
 [vitastor-nfs](../usage/nfs.ru.md), в свою очередь, при запуске отказывается
-использовать для ФС пулы, не выделенные для неё. Это также означает, что один
-пул может использоваться только для одной VitastorFS.
+использовать для ФС пулы, не помеченные, как используемые для неё. Это также
+означает, что один пул может использоваться только для одной VitastorFS.
 
-Также для ФС-пулов отключается передача статистики в etcd по отдельным инодам,
-так как ФС-пул может содержать очень много файлов и статистика по ним всем
-заняла бы очень много места в etcd.
+Если же вы планируете использовать пул для данных S3, установите его used_for_app
+в значение `s3:<name>`, где `<name>` - любое название по вашему усмотрению
+(например, `s3:standard`) - конкретное содержимое `<name>` пока никак не проверяется
+компонентами Vitastor S3.
+
+Все остальные значения used_for_app, кроме начинающихся на `fs:` или `s3:`, не
+означают ничего особенного для основных компонентов Vitastor. Поэтому сейчас вы
+можете использовать их свободно любым желаемым способом.
 
 # Примеры
 

docs/config/src/osd.yml

@@ -315,7 +315,7 @@
     decrease write performance for fast disks because page cache is an overhead
     itself.
 
-    Choose "directsync" to use [immediate_commit](layout-cluster.ru.md#immediate_commit)
+    Choose "directsync" to use [immediate_commit](layout-cluster.en.md#immediate_commit)
     (which requires disable_data_fsync) with drives having write-back cache
     which can't be turned off, for example, Intel Optane. Also note that *some*
     desktop SSDs (for example, HP EX950) may ignore O_SYNC thus making

docs/usage/cli.en.md

@@ -398,7 +398,8 @@ Optional parameters:
 | `--raw_placement <rules>`      | Specify raw PG generation rules ([details](../config/pool.en.md#raw_placement)) |
 | `--primary_affinity_tags tags` | Prefer to put primary copies on OSDs with all specified tags               |
 | `--scrub_interval <time>`      | Enable regular scrubbing for this pool. Format: number + unit s/m/h/d/M/y  |
-| `--used_for_fs <name>`         | Mark pool as used for VitastorFS with metadata in image <name>             |
+| `--used_for_app fs:<name>`     | Mark pool as used for VitastorFS with metadata in image `<name>`           |
+| `--used_for_app s3:<name>`     | Mark pool as used for S3 location with name `<name>`                       |
 | `--pg_stripe_size <number>`    | Increase object grouping stripe                                            |
 | `--max_osd_combinations 10000` | Maximum number of random combinations for LP solver input                  |
 | `--wait`                       | Wait for the new pool to come online                                       |

docs/usage/nfs.en.md

@@ -58,7 +58,7 @@ To use VitastorFS:
 2. Create an image for FS metadata, preferably in a faster (SSD or replica-HDD) pool,
    but you can create it in the data pool too if you want (image size doesn't matter):
    `vitastor-cli create -s 10G -p fastpool testfs`
-3. Mark data pool as an FS pool: `vitastor-cli modify-pool --used-for-fs testfs data-pool`
+3. Mark data pool as an FS pool: `vitastor-cli modify-pool --used-for-app fs:testfs data-pool`
 4. Either mount the FS: `vitastor-nfs mount --fs testfs --pool data-pool /mnt/vita`
 5. Or start the NFS server: `vitastor-nfs start --fs testfs --pool data-pool`
 

docs/usage/nfs.ru.md

@@ -60,7 +60,7 @@ JSON-формате :-). Для инспекции содержимого БД
    или по крайней мере на HDD, но без EC), но можно и в том же пуле, что данные
    (размер образа значения не имеет):
    `vitastor-cli create -s 10G -p fastpool testfs`
-3. Пометьте пул данных как ФС-пул: `vitastor-cli modify-pool --used-for-fs testfs data-pool`
+3. Пометьте пул данных как ФС-пул: `vitastor-cli modify-pool --used-for-app fs:testfs data-pool`
 4. Либо примонтируйте ФС: `vitastor-nfs mount --fs testfs --pool data-pool /mnt/vita`
 5. Либо запустите сетевой NFS-сервер: `vitastor-nfs start --fs testfs --pool data-pool`
 

src/client/cluster_client_wb.cpp

@@ -9,7 +9,7 @@ writeback_cache_t::~writeback_cache_t()
 {
     for (auto & bp: dirty_buffers)
     {
-        if (!--(*bp.second.refcnt))
+        if (bp.second.buf && !--(*bp.second.refcnt))
         {
             free(bp.second.refcnt); // refcnt is allocated with the buffer
         }
@@ -115,7 +115,10 @@ void writeback_cache_t::copy_write(cluster_op_t *op, int state, uint64_t new_flu
                     .flush_id = dirty_it->second.flush_id,
                     .refcnt = dirty_it->second.refcnt,
                 });
-                (*dirty_it->second.refcnt)++;
+                if (dirty_it->second.buf)
+                {
+                    (*dirty_it->second.refcnt)++;
+                }
                 if (dirty_it->second.state == CACHE_DIRTY)
                 {
                     if (dirty_it->second.buf)
@@ -193,7 +196,7 @@ void writeback_cache_t::copy_write(cluster_op_t *op, int state, uint64_t new_flu
                     writeback_queue_size++;
                 }
             }
-            if (!--(*dirty_it->second.refcnt))
+            if (dirty_it->second.buf && !--(*dirty_it->second.refcnt))
             {
                 free(dirty_it->second.refcnt);
             }
@@ -204,7 +207,10 @@ void writeback_cache_t::copy_write(cluster_op_t *op, int state, uint64_t new_flu
     bool is_del = op->opcode == OSD_OP_DELETE;
     uint64_t *refcnt = is_del ? NULL : (uint64_t*)malloc_or_die(sizeof(uint64_t) + op->len);
     uint8_t *buf = is_del ? NULL : ((uint8_t*)refcnt + sizeof(uint64_t));
-    *refcnt = 1;
+    if (!is_del)
+    {
+        *refcnt = 1;
+    }
     dirty_it = dirty_buffers.emplace_hint(dirty_it, (object_id){
         .inode = op->inode,
         .stripe = op->offset,
@@ -560,8 +566,10 @@ void writeback_cache_t::fsync_ok()
     {
         if (uw_it->second.state == CACHE_FLUSHING)
         {
-            if (!--(*uw_it->second.refcnt))
+            if (uw_it->second.buf && !--(*uw_it->second.refcnt))
+            {
                 free(uw_it->second.refcnt);
+            }
             dirty_buffers.erase(uw_it++);
         }
         else

src/client/etcd_state_client.cpp

@@ -889,7 +889,11 @@ void etcd_state_client_t::parse_state(const etcd_kv_t & kv)
             if (!pc.scrub_interval)
                 pc.scrub_interval = 0;
             // Mark pool as VitastorFS pool (disable per-inode stats and block volume creation)
-            pc.used_for_fs = pool_item.second["used_for_fs"].as_string();
+            pc.used_for_app = pool_item.second["used_for_fs"].as_string();
+            if (pc.used_for_app != "")
+                pc.used_for_app = "fs:"+pc.used_for_app;
+            else
+                pc.used_for_app = pool_item.second["used_for_app"].as_string();
             // Immediate Commit Mode
             pc.immediate_commit = pool_item.second["immediate_commit"].is_string()
                 ? parse_immediate_commit(pool_item.second["immediate_commit"].string_value(), IMMEDIATE_ALL)

src/client/etcd_state_client.h

@@ -61,7 +61,7 @@ struct pool_config_t
     uint64_t pg_stripe_size;
     std::map<pg_num_t, pg_config_t> pg_config;
     uint64_t scrub_interval;
-    std::string used_for_fs;
+    std::string used_for_app;
     int backfillfull;
 };
 

src/client/qemu_driver.c

@@ -568,6 +568,7 @@ static int vitastor_file_open(BlockDriverState *bs, QDict *options, int flags, E
 static void vitastor_close(BlockDriverState *bs)
 {
     VitastorClient *client = bs->opaque;
+    qemu_mutex_lock(&client->mutex);
     vitastor_c_destroy(client->proxy);
     if (client->fds)
     {
@@ -575,7 +576,6 @@ static void vitastor_close(BlockDriverState *bs)
         client->fds = NULL;
         client->fd_alloc = client->fd_count = 0;
     }
-    qemu_mutex_destroy(&client->mutex);
     if (client->config_path)
         g_free(client->config_path);
     if (client->etcd_host)
@@ -586,6 +586,8 @@ static void vitastor_close(BlockDriverState *bs)
         g_free(client->image);
     free(client->last_bitmap);
     client->last_bitmap = NULL;
+    qemu_mutex_unlock(&client->mutex);
+    qemu_mutex_destroy(&client->mutex);
 }
 
 #if QEMU_VERSION_MAJOR >= 3 || QEMU_VERSION_MAJOR == 2 && QEMU_VERSION_MINOR > 2

src/client/vitastor_c.cpp

@@ -127,7 +127,7 @@ vitastor_c *vitastor_c_create_qemu_uring(QEMUSetFDHandler *aio_set_fd_handler, v
     auto self = vitastor_c_create_qemu_common(aio_set_fd_handler, aio_context);
     self->ringloop = ringloop;
     self->cli = new cluster_client_t(self->ringloop, self->tfd, cfg_json);
-    ringloop->loop_continue();
+    ringloop->loop();
     return self;
 }
 
@@ -151,7 +151,7 @@ vitastor_c *vitastor_c_create_uring(const char *config_path, const char *etcd_ho
     self->ringloop = ringloop;
     self->epmgr = new epoll_manager_t(self->ringloop);
     self->cli = new cluster_client_t(self->ringloop, self->epmgr->tfd, cfg_json);
-    ringloop->loop_continue();
+    ringloop->loop();
     return self;
 }
 
@@ -185,7 +185,7 @@ vitastor_c *vitastor_c_create_uring_json(const char **options, int options_len)
     self->ringloop = ringloop;
     self->epmgr = new epoll_manager_t(self->ringloop);
     self->cli = new cluster_client_t(self->ringloop, self->epmgr->tfd, cfg_json);
-    ringloop->loop_continue();
+    ringloop->loop();
     return self;
 }
 
@@ -233,7 +233,7 @@ void vitastor_c_on_ready(vitastor_c *client, VitastorIOHandler cb, void *opaque)
     });
     if (client->ringloop)
     {
-        client->ringloop->loop_continue();
+        client->ringloop->loop();
     }
 }
 
@@ -293,7 +293,7 @@ void vitastor_c_read(vitastor_c *client, uint64_t inode, uint64_t offset, uint64
     client->cli->execute(op);
     if (client->ringloop)
     {
-        client->ringloop->loop_continue();
+        client->ringloop->loop();
     }
 }
 
@@ -318,7 +318,7 @@ void vitastor_c_write(vitastor_c *client, uint64_t inode, uint64_t offset, uint6
     client->cli->execute(op);
     if (client->ringloop)
     {
-        client->ringloop->loop_continue();
+        client->ringloop->loop();
     }
 }
 
@@ -339,7 +339,7 @@ void vitastor_c_delete(vitastor_c *client, uint64_t inode, uint64_t offset, uint
     client->cli->execute(op);
     if (client->ringloop)
     {
-        client->ringloop->loop_continue();
+        client->ringloop->loop();
     }
 }
 
@@ -365,7 +365,7 @@ void vitastor_c_read_bitmap(vitastor_c *client, uint64_t inode, uint64_t offset,
     client->cli->execute(op);
     if (client->ringloop)
     {
-        client->ringloop->loop_continue();
+        client->ringloop->loop();
     }
 }
 
@@ -381,7 +381,7 @@ void vitastor_c_sync(vitastor_c *client, VitastorIOHandler cb, void *opaque)
     client->cli->execute(op);
     if (client->ringloop)
     {
-        client->ringloop->loop_continue();
+        client->ringloop->loop();
     }
 }
 
@@ -394,7 +394,7 @@ void vitastor_c_watch_inode(vitastor_c *client, char *image, VitastorIOHandler c
     });
     if (client->ringloop)
     {
-        client->ringloop->loop_continue();
+        client->ringloop->loop();
     }
 }
 

src/cmd/cli.cpp

@@ -186,7 +186,8 @@ static const char* help_text =
     "    --raw_placement <rules>       Specify raw PG generation rules (see documentation for details)\n"
     "    --primary_affinity_tags tags  Prefer to put primary copies on OSDs with all specified tags\n"
     "    --scrub_interval <time>       Enable regular scrubbing for this pool. Format: number + unit s/m/h/d/M/y\n"
-    "    --used_for_fs <name>          Mark pool as used for VitastorFS with metadata in image <name>\n"
+    "    --used_for_app fs:<name>      Mark pool as used for VitastorFS with metadata in image <name>\n"
+    "    --used_for_app s3:<name>      Mark pool as used for S3 location with name <name>\n"
     "    --pg_stripe_size <number>     Increase object grouping stripe\n"
     "    --max_osd_combinations 10000  Maximum number of random combinations for LP solver input\n"
     "    --wait                        Wait for the new pool to come online\n"
@@ -198,7 +199,7 @@ static const char* help_text =
     "vitastor-cli modify-pool|pool-modify <id|name> [--name <new_name>] [PARAMETERS...]\n"
     "  Modify an existing pool. Modifiable parameters:\n"
     "    [-s|--pg_size <number>] [--pg_minsize <number>] [-n|--pg_count <count>]\n"
-    "    [--failure_domain <level>] [--root_node <node>] [--osd_tags <tags>] [--used_for_fs <name>]\n"
+    "    [--failure_domain <level>] [--root_node <node>] [--osd_tags <tags>] [--used_for_app <type>:<name>]\n"
     "    [--max_osd_combinations <number>] [--primary_affinity_tags <tags>] [--scrub_interval <time>]\n"
     "    [--level_placement <rules>] [--raw_placement <rules>]\n"
     "  Non-modifiable parameters (changing them WILL lead to data loss):\n"

src/cmd/cli_create.cpp

@@ -92,12 +92,12 @@ struct image_creator_t
         {
             new_pool_id = pools.begin()->first;
         }
-        if (new_pool_id && !pools.at(new_pool_id).used_for_fs.empty() && !force)
+        if (new_pool_id && !pools.at(new_pool_id).used_for_app.empty() && !force)
         {
             result = (cli_result_t){
                 .err = EINVAL,
                 .text = "Pool "+pools.at(new_pool_id).name+
-                    " is used for VitastorFS "+pools.at(new_pool_id).used_for_fs+
+                    " is used for application "+pools.at(new_pool_id).used_for_app+
                     ". Use --force if you really know what you are doing",
             };
             state = 100;

src/cmd/cli_pool_cfg.cpp

@@ -90,8 +90,8 @@ std::string validate_pool_config(json11::Json::object & new_cfg, json11::Json ol
             value = sz;
         }
         else if (key == "name" || key == "scheme" || key == "immediate_commit" ||
-            key == "failure_domain" || key == "root_node" || key == "scrub_interval" || key == "used_for_fs" ||
-            key == "raw_placement")
+            key == "failure_domain" || key == "root_node" || key == "scrub_interval" || key == "used_for_app" ||
+            key == "used_for_fs" || key == "raw_placement")
         {
             if (!value.is_string())
             {
@@ -156,8 +156,13 @@ std::string validate_pool_config(json11::Json::object & new_cfg, json11::Json ol
     {
         new_cfg.erase("parity_chunks");
     }
-    if (new_cfg.find("used_for_fs") != new_cfg.end() && new_cfg["used_for_fs"].string_value() == "")
+    if (new_cfg.find("used_for_app") != new_cfg.end() && new_cfg["used_for_app"].string_value() == "")
     {
+        new_cfg.erase("used_for_app");
+    }
+    if (new_cfg.find("used_for_app") == new_cfg.end() && new_cfg.find("used_for_fs") != new_cfg.end())
+    {
+        new_cfg["used_for_app"] = "fs:"+new_cfg["used_for_fs"].string_value();
         new_cfg.erase("used_for_fs");
     }
 

src/cmd/cli_pool_ls.cpp

@@ -199,7 +199,9 @@ resume_1:
             auto & st = pool_stats[pool_id];
             for (auto & kv: pp.second.object_items())
             {
-                if (st.find(kv.first) == st.end())
+                if (kv.first == "used_for_fs" && st.find("used_for_app") == st.end())
+                    st["used_for_app"] = "fs:"+kv.second.string_value();
+                else if (st.find(kv.first) == st.end())
                     st[kv.first] = kv.second;
             }
         }
@@ -493,7 +495,7 @@ resume_3:
                 { "name", "Name" },
                 { "id", "ID" },
                 { "scheme_name", "Scheme" },
-                { "used_for_fs", "Used for VitastorFS" },
+                { "used_for_app", "Used for app" },
                 { "status", "Status" },
                 { "backfillfull_fmt", "Backfillfull" },
                 { "pg_count_fmt", "PGs" },

src/cmd/cli_pool_modify.cpp

@@ -112,19 +112,21 @@ resume_1:
                 return;
             }
 
-            if (new_cfg.find("used_for_fs") != new_cfg.end() && !force)
+            if (new_cfg.find("used_for_app") != new_cfg.end() && !force)
             {
                 // Check that pool doesn't have images
                 auto img_it = parent->cli->st_cli.inode_config.lower_bound(INODE_WITH_POOL(pool_id, 0));
-                if (img_it != parent->cli->st_cli.inode_config.end() && INODE_POOL(img_it->first) == pool_id &&
-                    img_it->second.name == new_cfg["used_for_fs"].string_value())
+                if (img_it != parent->cli->st_cli.inode_config.end() &&
+                    INODE_POOL(img_it->first) == pool_id &&
+                    new_cfg["used_for_app"].string_value().substr(0, 3) == "fs:" &&
+                    img_it->second.name == new_cfg["used_for_app"].string_value().substr(3))
                 {
                     // Only allow metadata image to exist in the FS pool
                     img_it++;
                 }
                 if (img_it != parent->cli->st_cli.inode_config.end() && INODE_POOL(img_it->first) == pool_id)
                 {
-                    result = (cli_result_t){ .err = ENOENT, .text = "Pool "+pool_name+" has block images, delete them before using it for VitastorFS" };
+                    result = (cli_result_t){ .err = ENOENT, .text = "Pool "+pool_name+" has block images, delete them before using it for VitastorFS, S3 or another app" };
                     state = 100;
                     return;
                 }

src/nfs/nfs_kv.cpp

@@ -245,10 +245,10 @@ void kv_fs_state_t::init(nfs_proxy_t *proxy, json11::Json cfg)
     if (proxy->cli->st_cli.inode_config.find(fs_kv_inode) != proxy->cli->st_cli.inode_config.end())
     {
         auto & name = proxy->cli->st_cli.inode_config.at(fs_kv_inode).name;
-        if (pool_cfg.used_for_fs != name)
+        if (pool_cfg.used_for_app != "fs:"+name)
         {
-            fprintf(stderr, "Please mark pool as used for this file system with `vitastor-cli modify-pool --used-for-fs %s %s`\n",
-                name.c_str(), cfg["fs"].string_value().c_str());
+            fprintf(stderr, "Please mark pool as used for this file system with `vitastor-cli modify-pool --used-for-app fs:%s %s`\n",
+                name.c_str(), pool_cfg.name.c_str());
             exit(1);
         }
     }

src/osd/osd_cluster.cpp

@@ -673,7 +673,7 @@ void osd_t::apply_no_inode_stats()
     std::vector<uint64_t> no_inode_stats;
     for (auto & pool_item: st_cli.pool_config)
     {
-        if (!pool_item.second.used_for_fs.empty())
+        if (!pool_item.second.used_for_app.empty())
         {
             no_inode_stats.push_back(pool_item.first);
         }

src/osd/osd_primary.cpp

@@ -722,54 +722,58 @@ resume_3:
         cur_op->reply.rw.version = op_data->fact_ver;
         goto continue_others;
     }
-    // Save version override for parallel reads
-    pg.ver_override[op_data->oid] = op_data->fact_ver;
-    // Submit deletes
-    op_data->fact_ver++;
-    submit_primary_del_subops(cur_op, NULL, 0, op_data->object_state ? op_data->object_state->osd_set : pg.cur_loc_set);
+    // If not already deleted:
+    if (op_data->fact_ver)
+    {
+        // Save version override for parallel reads
+        pg.ver_override[op_data->oid] = op_data->fact_ver;
+        // Submit deletes
+        op_data->fact_ver++;
+        submit_primary_del_subops(cur_op, NULL, 0, op_data->object_state ? op_data->object_state->osd_set : pg.cur_loc_set);
 resume_4:
-    op_data->st = 4;
-    return;
-resume_5:
-    if (op_data->errors > 0)
-    {
-        deref_object_state(pg, &op_data->object_state, true);
-        pg_cancel_write_queue(pg, cur_op, op_data->oid, op_data->errcode);
+        op_data->st = 4;
         return;
-    }
-    // Remove version override
-    pg.ver_override.erase(op_data->oid);
-    // Adjust PG stats after "instant stabilize", because we need object_state above
-    if (!op_data->object_state)
-    {
-        pg.clean_count--;
-    }
-    else
-    {
-        remove_object_from_state(op_data->oid, &op_data->object_state, pg);
-        deref_object_state(pg, &op_data->object_state, true);
-    }
-    // Mark PG and OSDs as dirty
-    for (auto & chunk: (op_data->object_state ? op_data->object_state->osd_set : pg.cur_loc_set))
-    {
-        this->dirty_osds.insert(chunk.osd_num);
-    }
-    for (auto cl_it = msgr.clients.find(cur_op->peer_fd); cl_it != msgr.clients.end(); )
-    {
-        cl_it->second->dirty_pgs.insert({ .pool_id = pg.pool_id, .pg_num = pg.pg_num });
-        break;
-    }
-    dirty_pgs.insert({ .pool_id = pg.pool_id, .pg_num = pg.pg_num });
-    if (immediate_commit == IMMEDIATE_NONE)
-    {
-        unstable_write_count++;
-        if (unstable_write_count >= autosync_writes)
+resume_5:
+        if (op_data->errors > 0)
         {
-            unstable_write_count = 0;
-            autosync();
+            deref_object_state(pg, &op_data->object_state, true);
+            pg_cancel_write_queue(pg, cur_op, op_data->oid, op_data->errcode);
+            return;
         }
+        // Remove version override
+        pg.ver_override.erase(op_data->oid);
+        // Adjust PG stats after "instant stabilize", because we need object_state above
+        if (!op_data->object_state)
+        {
+            pg.clean_count--;
+        }
+        else
+        {
+            remove_object_from_state(op_data->oid, &op_data->object_state, pg);
+            deref_object_state(pg, &op_data->object_state, true);
+        }
+        // Mark PG and OSDs as dirty
+        for (auto & chunk: (op_data->object_state ? op_data->object_state->osd_set : pg.cur_loc_set))
+        {
+            this->dirty_osds.insert(chunk.osd_num);
+        }
+        for (auto cl_it = msgr.clients.find(cur_op->peer_fd); cl_it != msgr.clients.end(); )
+        {
+            cl_it->second->dirty_pgs.insert({ .pool_id = pg.pool_id, .pg_num = pg.pg_num });
+            break;
+        }
+        dirty_pgs.insert({ .pool_id = pg.pool_id, .pg_num = pg.pg_num });
+        if (immediate_commit == IMMEDIATE_NONE)
+        {
+            unstable_write_count++;
+            if (unstable_write_count >= autosync_writes)
+            {
+                unstable_write_count = 0;
+                autosync();
+            }
+        }
+        pg.total_count--;
     }
-    pg.total_count--;
     cur_op->reply.hdr.retval = 0;
     // indicate possibly unfinished (left_on_dead) deletions
     cur_op->reply.del.flags = OSD_DEL_SUPPORT_LEFT_ON_DEAD;

src/util/ringloop.cpp

@@ -131,17 +131,6 @@ void ring_loop_t::loop()
         }
         io_uring_cqe_seen(&ring, cqe);
     }
-    in_loop = false;
-    loop_continue();
-}
-
-void ring_loop_t::loop_continue()
-{
-    if (in_loop)
-    {
-        return;
-    }
-    in_loop = true;
     do
     {
         loop_again = false;

src/util/ringloop.h

@@ -165,7 +165,6 @@ public:
     }
 
     void loop();
-    void loop_continue();
     void wakeup();
 
     unsigned save();

tests/test_nfs.sh

@@ -4,7 +4,7 @@ PG_COUNT=16
 . `dirname $0`/run_3osds.sh
 
 build/src/cmd/vitastor-cli --etcd_address $ETCD_URL create -s 10G fsmeta
-build/src/cmd/vitastor-cli --etcd_address $ETCD_URL modify-pool --used-for-fs fsmeta testpool
+build/src/cmd/vitastor-cli --etcd_address $ETCD_URL modify-pool --used-for-app fs:fsmeta testpool
 build/src/nfs/vitastor-nfs start --fs fsmeta --etcd_address $ETCD_URL --portmap 0 --port 2050 --foreground 1 --trace 1 >>./testdata/nfs.log 2>&1 &
 NFS_PID=$!