Release 0.6.7

- Implement CLI commands for listing, viewing I/O statistics, creating, snapshotting, cloning, resizing and modifying images. All these operations are covered by 3 commands: ls, create, modify - Implement an important fix to prior OSD set tracking for PGs. The previous version had an issue which could lead to data loss due to an OSD with older copy of the data thinking it has the newest copy - Fix I/O statistics aggregation in the monitor - Several minor fixes for Cinder driver - Fix QEMU driver to be compatible with QEMU 2.x > 2.0 - Fix stalls sometimes possible in configurations without immediate_commit due to insufficient amount of automatic internal fsync operations - Add `vita` alias for `vitastor-cli`
Allow to show only specific images in listing
2021-11-13 23:23:55 +03:00 · 2021-11-13 23:23:55 +03:00 · 2021-11-13 23:23:55 +03:00 · 2021-11-13 22:39:17 +03:00 · 2021-11-13 22:39:17 +03:00 · 2021-11-13 22:39:17 +03:00
54 changed files with 2175 additions and 238 deletions
@@ -2,6 +2,6 @@ cmake_minimum_required(VERSION 2.8)
 project(vitastor)
-set(VERSION "0.6.6")
+set(VERSION "0.6.7")
 add_subdirectory(src)
@@ -40,7 +40,7 @@ Vitastor на данный момент находится в статусе п
 - Драйвер диска для QEMU (собирается вне дерева исходников QEMU)
 - Драйвер диска для утилиты тестирования производительности fio (также собирается вне дерева исходников fio)
 - NBD-прокси для монтирования образов ядром ("блочное устройство в режиме пользователя")
- Утилита для удаления образов/инодов (vitastor-cli rm)
+- Утилита для удаления образов/инодов (vitastor-cli rm-data)
 - Пакеты для Debian и CentOS
 - Статистика операций ввода/вывода и занятого места в разрезе инодов
 - Именование инодов через хранение их метаданных в etcd
@@ -50,6 +50,7 @@ Vitastor на данный момент находится в статусе п
 - CSI-плагин для Kubernetes
 - Базовая поддержка OpenStack: драйвер Cinder, патчи для Nova и libvirt
 - Слияние снапшотов (vitastor-cli {snap-rm,flatten,merge})
 - Консольный интерфейс для управления образами (vitastor-cli {ls,create,modify})
 ## Планы развития
@@ -492,10 +493,10 @@ qemu-system-x86_64 -enable-kvm -m 1024
 ### Удалить образ
-Используйте утилиту vitastor-cli rm. Например:
+Используйте утилиту vitastor-cli rm-data. Например:
 ```
-vitastor-cli rm --etcd_address 10.115.0.10:2379/v3 --pool 1 --inode 1 --parallel_osds 16 --iodepth 32
+vitastor-cli rm-data --etcd_address 10.115.0.10:2379/v3 --pool 1 --inode 1 --parallel_osds 16 --iodepth 32
 ```
 ### NBD
@@ -34,7 +34,7 @@ breaking changes in the future. However, the following is implemented:
 - QEMU driver (built out-of-tree)
 - Loadable fio engine for benchmarks (also built out-of-tree)
 - NBD proxy for kernel mounts
- Inode removal tool (vitastor-cli rm)
+- Inode removal tool (vitastor-cli rm-data)
 - Packaging for Debian and CentOS
 - Per-inode I/O and space usage statistics
 - Inode metadata storage in etcd
@@ -44,6 +44,7 @@ breaking changes in the future. However, the following is implemented:
 - CSI plugin for Kubernetes
 - Basic OpenStack support: Cinder driver, Nova and libvirt patches
 - Snapshot merge tool (vitastor-cli {snap-rm,flatten,merge})
 - Image management CLI (vitastor-cli {ls,create,modify})
 ## Roadmap
@@ -444,10 +445,10 @@ just like in qemu-img.
 ### Remove inode
-Use vitastor-rm. For example:
+Use vitastor-rm / vitastor-cli rm-data. For example:
 ```
-vitastor-rm --etcd_address 10.115.0.10:2379/v3 --pool 1 --inode 1 --parallel_osds 16 --iodepth 32
+vitastor-cli rm-data --etcd_address 10.115.0.10:2379/v3 --pool 1 --inode 1 --parallel_osds 16 --iodepth 32
 ```
 ### NBD
@@ -1,3 +1,2 @@
 vitastor-csi
 go.sum
 Dockerfile
@@ -1,7 +1,7 @@
 # Compile stage
 FROM golang:buster AS build
-ADD go.mod /app/
+ADD go.sum go.mod /app/
 RUN cd /app; CGO_ENABLED=1 GOOS=linux GOARCH=amd64 go mod download -x
 ADD . /app
 RUN perl -i -e '$/ = undef; while(<>) { s/\n\s*(\{\s*\n)/$1\n/g; s/\}(\s*\n\s*)else\b/$1} else/g; print; }' `find /app -name '*.go'`
@@ -1,4 +1,4 @@
-VERSION ?= v0.6.6
+VERSION ?= v0.6.7
 all: build push
@@ -49,7 +49,7 @@ spec:
            capabilities:
              add: ["SYS_ADMIN"]
            allowPrivilegeEscalation: true
-          image: vitalif/vitastor-csi:v0.6.6
+          image: vitalif/vitastor-csi:v0.6.7
          args:
            - "--node=$(NODE_ID)"
            - "--endpoint=$(CSI_ENDPOINT)"
@@ -116,7 +116,7 @@ spec:
            privileged: true
            capabilities:
              add: ["SYS_ADMIN"]
-          image: vitalif/vitastor-csi:v0.6.6
+          image: vitalif/vitastor-csi:v0.6.7
          args:
            - "--node=$(NODE_ID)"
            - "--endpoint=$(CSI_ENDPOINT)"
@@ -0,0 +1,448 @@
 cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
 cloud.google.com/go v0.38.0/go.mod h1:990N+gfupTy94rShfmMCWGDn0LpTmnzTp2qbd1dvSRU=
 cloud.google.com/go v0.44.1/go.mod h1:iSa0KzasP4Uvy3f1mN/7PiObzGgflwredwwASm/v6AU=
 cloud.google.com/go v0.44.2/go.mod h1:60680Gw3Yr4ikxnPRS/oxxkBccT6SA1yMk63TGekxKY=
 cloud.google.com/go v0.45.1/go.mod h1:RpBamKRgapWJb87xiFSdk4g1CME7QZg3uwTez+TSTjc=
 cloud.google.com/go v0.46.3/go.mod h1:a6bKKbmY7er1mI7TEI4lsAkts/mkhTSZK8w33B4RAg0=
 cloud.google.com/go v0.51.0/go.mod h1:hWtGJ6gnXH+KgDv+V0zFGDvpi07n3z8ZNj3T1RW0Gcw=
 cloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o=
 cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE=
 cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I=
 cloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw=
 dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
 github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78/go.mod h1:LmzpDX56iTiv29bbRTIsUNlaFfuhWRQBWjQdVyAevI8=
 github.com/Azure/go-autorest/autorest v0.9.0/go.mod h1:xyHB1BMZT0cuDHU7I0+g046+BFDTQ8rEZB0s4Yfa6bI=
 github.com/Azure/go-autorest/autorest v0.9.6/go.mod h1:/FALq9T/kS7b5J5qsQ+RSTUdAmGFqi0vUdVNNx8q630=
 github.com/Azure/go-autorest/autorest/adal v0.5.0/go.mod h1:8Z9fGy2MpX0PvDjB1pEgQTmVqjGhiHBW7RJJEciWzS0=
 github.com/Azure/go-autorest/autorest/adal v0.8.2/go.mod h1:ZjhuQClTqx435SRJ2iMlOxPYt3d2C/T/7TiQCVZSn3Q=
 github.com/Azure/go-autorest/autorest/date v0.1.0/go.mod h1:plvfp3oPSKwf2DNjlBjWF/7vwR+cUD/ELuzDCXwHUVA=
 github.com/Azure/go-autorest/autorest/date v0.2.0/go.mod h1:vcORJHLJEh643/Ioh9+vPmf1Ij9AEBM5FuBIXLmIy0g=
 github.com/Azure/go-autorest/autorest/mocks v0.1.0/go.mod h1:OTyCOPRA2IgIlWxVYxBee2F5Gr4kF2zd2J5cFRaIDN0=
 github.com/Azure/go-autorest/autorest/mocks v0.2.0/go.mod h1:OTyCOPRA2IgIlWxVYxBee2F5Gr4kF2zd2J5cFRaIDN0=
 github.com/Azure/go-autorest/autorest/mocks v0.3.0/go.mod h1:a8FDP3DYzQ4RYfVAxAN3SVSiiO77gL2j2ronKKP0syM=
 github.com/Azure/go-autorest/logger v0.1.0/go.mod h1:oExouG+K6PryycPJfVSxi/koC6LSNgds39diKLz7Vrc=
 github.com/Azure/go-autorest/tracing v0.5.0/go.mod h1:r/s2XiOKccPW3HrqB+W0TQzfbtp2fGCgRFtBroKn4Dk=
 github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
 github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
 github.com/NYTimes/gziphandler v0.0.0-20170623195520-56545f4a5d46/go.mod h1:3wb06e3pkSAbeQ52E9H9iFoQsEEwGN64994WTCIhntQ=
 github.com/PuerkitoBio/purell v1.0.0/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0=
 github.com/PuerkitoBio/urlesc v0.0.0-20160726150825-5bd2802263f2/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE=
 github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
 github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
 github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
 github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
 github.com/antihax/optional v1.0.0/go.mod h1:uupD/76wgC+ih3iEmQUL+0Ugr19nfwCT1kdvxnR2qWY=
 github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
 github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8=
 github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
 github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
 github.com/blang/semver v3.5.0+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk=
 github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
 github.com/cespare/xxhash/v2 v2.1.1 h1:6MnRN8NT7+YBpUIWxHtefFZOKTAPgGjpQSxqLNn0+qY=
 github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
 github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
 github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
 github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
 github.com/container-storage-interface/spec v1.2.0/go.mod h1:6URME8mwIBbpVyZV93Ce5St17xBiQJQY67NDsuohiy4=
 github.com/container-storage-interface/spec v1.4.0 h1:ozAshSKxpJnYUfmkpZCTYyF/4MYeYlhdXbAvPvfGmkg=
 github.com/container-storage-interface/spec v1.4.0/go.mod h1:6URME8mwIBbpVyZV93Ce5St17xBiQJQY67NDsuohiy4=
 github.com/coreos/bbolt v1.3.5 h1:XFv7xaq7701j8ZSEzR28VohFYSlyakMyqNMU5FQH6Ac=
 github.com/coreos/bbolt v1.3.5/go.mod h1:G5EMThwa9y8QZGBClrRx5EY+Yw9kAhnjy3bSjsnlVTQ=
 github.com/coreos/etcd v3.3.25+incompatible h1:0GQEw6h3YnuOVdtwygkIfJ+Omx0tZ8/QkVyXI4LkbeY=
 github.com/coreos/etcd v3.3.25+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
 github.com/coreos/go-semver v0.3.0 h1:wkHLiw0WNATZnSG7epLsujiMCgPAc9xhjJ4tgnAxmfM=
 github.com/coreos/go-semver v0.3.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
 github.com/coreos/go-systemd v0.0.0-20191104093116-d3cd4ed1dbcf h1:iW4rZ826su+pqaw19uhpSCzhj44qo35pNgKFGqzDKkU=
 github.com/coreos/go-systemd v0.0.0-20191104093116-d3cd4ed1dbcf/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
 github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f h1:lBNOc5arjvs8E5mO2tbpBpLoyyu8B6e44T7hJy6potg=
 github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA=
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/dgrijalva/jwt-go v3.2.0+incompatible h1:7qlOGliEKZXTDg6OTjfoBKDXWrumCAMpl/TFQ4/5kLM=
 github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ=
 github.com/docker/spdystream v0.0.0-20160310174837-449fdfce4d96/go.mod h1:Qh8CwZgvJUkLughtfhJv5dyTYa91l1fOUCrgjqmcifM=
 github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815/go.mod h1:WwZ+bS3ebgob9U8Nd0kOddGdZWjyMGR8Wziv+TBNwSE=
 github.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4zYo=
 github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
 github.com/elazarl/goproxy v0.0.0-20180725130230-947c36da3153/go.mod h1:/Zj4wYkgs4iZTTu3o/KG3Itv/qCCa8VVMlb3i9OVuzc=
 github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs=
 github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
 github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
 github.com/evanphx/json-patch v4.9.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk=
 github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
 github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
 github.com/ghodss/yaml v0.0.0-20150909031657-73d445a93680/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
 github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
 github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
 github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
 github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
 github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE=
 github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk=
 github.com/go-logr/logr v0.1.0/go.mod h1:ixOQHD9gLJUVQQ2ZOR7zLEifBX6tGkNJF4QyIY7sIas=
 github.com/go-logr/logr v0.2.0 h1:QvGt2nLcHH0WK9orKa+ppBPAxREcH364nPUedEpK0TY=
 github.com/go-logr/logr v0.2.0/go.mod h1:z6/tIYblkpsD+a4lm/fGIIU9mZ+XfAiaFtq7xTgseGU=
 github.com/go-openapi/jsonpointer v0.0.0-20160704185906-46af16f9f7b1/go.mod h1:+35s3my2LFTysnkMfxsJBAMHj/DoqoB9knIWoYG/Vk0=
 github.com/go-openapi/jsonreference v0.0.0-20160704190145-13c6e3589ad9/go.mod h1:W3Z9FmVs9qj+KR4zFKmDPGiLdk1D9Rlm7cyMvf57TTg=
 github.com/go-openapi/spec v0.0.0-20160808142527-6aced65f8501/go.mod h1:J8+jY1nAiCcj+friV/PDoE1/3eeccG9LYBs0tYvLOWc=
 github.com/go-openapi/swag v0.0.0-20160704191624-1d0bd113de87/go.mod h1:DXUve3Dpr1UfpPtxFw+EFuQ41HhCWZfha5jSVRG7C7I=
 github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
 github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
 github.com/gogo/protobuf v1.3.1 h1:DqDEcV5aeaTmdFBePNpYsp3FlcVH/2ISVVM9Qf8PSls=
 github.com/gogo/protobuf v1.3.1/go.mod h1:SlYgWuQ5SjCEi6WLHjHCa1yvBfUnHcTbrrZtXPKa29o=
 github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
 github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
 github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b h1:VKtxabqXZkF25pY9ekfRL6a582T4P37/31XEstQ5p58=
 github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
 github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
 github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7 h1:5ZkaAPbicIKTF2I64qf5Fh8Aa83Q/dnOafMYV0OMwjA=
 github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
 github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
 github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
 github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y=
 github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
 github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
 github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
 github.com/golang/protobuf v1.3.3/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw=
 github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8=
 github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA=
 github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs=
 github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w=
 github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0=
 github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8=
 github.com/golang/protobuf v1.4.2 h1:+Z5KGCizgyZCbGh1KZqA0fcLLkwbsjIzS4aV2v7wJX0=
 github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
 github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
 github.com/google/btree v1.0.0 h1:0udJVsspx3VBr5FwtLhQQtuAsVc79tTq0ocGIPAU6qo=
 github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
 github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
 github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
 github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
 github.com/google/go-cmp v0.4.0 h1:xsAVV57WRhGj6kEIi8ReJzQlHHqcBYCElAvkovg3B/4=
 github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
 github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
 github.com/google/gofuzz v1.1.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
 github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
 github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
 github.com/google/pprof v0.0.0-20190515194954-54271f7e092f/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
 github.com/google/pprof v0.0.0-20191218002539-d4f498aebedc/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
 github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
 github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY=
 github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
 github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg=
 github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
 github.com/googleapis/gnostic v0.4.1/go.mod h1:LRhVm6pbyptWbWbuZ38d1eyptfvIytN3ir6b65WBswg=
 github.com/gorilla/websocket v1.4.2 h1:+/TMaTYc4QFitKJxsQ7Yye35DkWvkdLcvGKqM+x0Ufc=
 github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
 github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA=
 github.com/grpc-ecosystem/go-grpc-middleware v1.3.0 h1:+9834+KizmvFV7pXQGSXQTsaWhq2GjuNUt0aUU0YBYw=
 github.com/grpc-ecosystem/go-grpc-middleware v1.3.0/go.mod h1:z0ButlSOZa5vEBq9m2m2hlwIgKw+rp3sdCBRoJY+30Y=
 github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 h1:Ovs26xHkKqVztRpIrF/92BcuyuQ/YW4NSIpoGtfXNho=
 github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0/go.mod h1:8NvIoxWQoOIhqOTXgfV/d3M/q6VIi02HzZEHgUlZvzk=
 github.com/grpc-ecosystem/grpc-gateway v1.16.0 h1:gmcG1KaJ57LophUzW0Hy8NmPhnMZb4M0+kPpLofRdBo=
 github.com/grpc-ecosystem/grpc-gateway v1.16.0/go.mod h1:BDjrQk3hbvj6Nolgz8mAMFbcEtjT1g+wF4CSlocrBnw=
 github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
 github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
 github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
 github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
 github.com/imdario/mergo v0.3.5/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA=
 github.com/jonboulle/clockwork v0.2.2 h1:UOGuzwb1PwsrDAObMuhUnj0p5ULPj8V/xJ7Kx9qUBdQ=
 github.com/jonboulle/clockwork v0.2.2/go.mod h1:Pkfl5aHPm1nk2H9h0bjmnJD/BcgbGXUBGnn1kMkgxc8=
 github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
 github.com/json-iterator/go v1.1.10 h1:Kz6Cvnvv2wGdaG/V8yMvfkmNiXq9Ya2KUv4rouJJr68=
 github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
 github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU=
 github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk=
 github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w=
 github.com/kisielk/errcheck v1.2.0/go.mod h1:/BMXB+zMLi60iA8Vv6Ksmxu/1UDYcXs4uQLJ+jE2L00=
 github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
 github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
 github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
 github.com/konsorten/go-windows-terminal-sequences v1.0.3 h1:CE8S1cTafDpPvMhIxNJKvHsGVBgn1xWYf1NbHQhywc8=
 github.com/konsorten/go-windows-terminal-sequences v1.0.3/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
 github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
 github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
 github.com/kr/pretty v0.2.0 h1:s5hAObm+yFO5uHYt5dYjxi2rXrsnmRpJx4OYvIWUaQs=
 github.com/kr/pretty v0.2.0/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
 github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
 github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
 github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
 github.com/kubernetes-csi/csi-lib-utils v0.9.1 h1:sGq6ifVujfMSkfTsMZip44Ttv8SDXvsBlFk9GdYl/b8=
 github.com/kubernetes-csi/csi-lib-utils v0.9.1/go.mod h1:8E2jVUX9j3QgspwHXa6LwyN7IHQDjW9jX3kwoWnSC+M=
 github.com/mailru/easyjson v0.0.0-20160728113105-d5b7844b561a/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
 github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
 github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369 h1:I0XW9+e1XWDxdcEniV4rQAIOPUGDq67JSCiRCgGCZLI=
 github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369/go.mod h1:BSXmuO+STAnVfrANrmjBb36TMTDstsz7MSK+HVaYKv4=
 github.com/moby/term v0.0.0-20200312100748-672ec06f55cd/go.mod h1:DdlQx2hp0Ss5/fLikoLlEeIYiATotOjgB//nb973jeo=
 github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
 github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
 github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
 github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
 github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI=
 github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
 github.com/munnerz/goautoneg v0.0.0-20120707110453-a547fc61f48d/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
 github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
 github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f/go.mod h1:ZdcZmHo+o7JKHSa8/e818NopupXU1YMK5fe1lsApnBw=
 github.com/onsi/ginkgo v0.0.0-20170829012221-11459a886d9c/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
 github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
 github.com/onsi/ginkgo v1.11.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
 github.com/onsi/gomega v0.0.0-20170829124025-dcabb60a477c/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA=
 github.com/onsi/gomega v1.7.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
 github.com/opentracing/opentracing-go v1.1.0/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o=
 github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU=
 github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
 github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
 github.com/prometheus/client_golang v1.0.0/go.mod h1:db9x61etRT2tGnBNRi70OPL5FsnadC4Ky3P0J6CfImo=
 github.com/prometheus/client_golang v1.7.1 h1:NTGy1Ja9pByO+xAeH/qiWnLrKtr3hJPNjaVUwnjpdpA=
 github.com/prometheus/client_golang v1.7.1/go.mod h1:PY5Wy2awLA44sXw4AOSfFBetzPP4j5+D6mVACh+pe2M=
 github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
 github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
 github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
 github.com/prometheus/client_model v0.2.0 h1:uq5h0d+GuxiXLJLNABMgp2qUWDPiLvgCzz2dUR+/W/M=
 github.com/prometheus/client_model v0.2.0/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
 github.com/prometheus/common v0.4.1/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4=
 github.com/prometheus/common v0.10.0 h1:RyRA7RzGXQZiW+tGMr7sxa85G1z0yOpM1qq5c8lNawc=
 github.com/prometheus/common v0.10.0/go.mod h1:Tlit/dnDKsSWFlCLTWaA1cyBgKHSMdTB80sz/V91rCo=
 github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
 github.com/prometheus/procfs v0.0.2/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
 github.com/prometheus/procfs v0.1.3 h1:F0+tqvhOksq22sc6iCHF5WGlWjdwj92p0udFh1VFBS8=
 github.com/prometheus/procfs v0.1.3/go.mod h1:lV6e/gmhEcM9IjHGsFOCxxuZ+z1YqCvr4OA4YeYWdaU=
 github.com/rogpeppe/fastuuid v1.2.0/go.mod h1:jVj6XXZzXRy/MSR5jhDC/2q6DgLz+nrA6LYCDYWNEvQ=
 github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
 github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
 github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
 github.com/sirupsen/logrus v1.6.0 h1:UBcNElsrwanuuMsnGSlYmtmgbb23qDR5dG+6X6Oo89I=
 github.com/sirupsen/logrus v1.6.0/go.mod h1:7uNnSEd1DgxDLC74fIahvMZmmYsHGZGEOFrfsX/uA88=
 github.com/soheilhy/cmux v0.1.5 h1:jjzc5WVemNEDTLwv9tlmemhC73tI08BNOIGwBOo10Js=
 github.com/soheilhy/cmux v0.1.5/go.mod h1:T7TcVDs9LWfQgPlPsdngu6I6QIoyIFZDDC6sNE1GqG0=
 github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTdifk=
 github.com/spf13/pflag v0.0.0-20170130214245-9ff6c6923cff/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
 github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
 github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
 github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
 github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
 github.com/stretchr/testify v1.5.1 h1:nOGnQDM7FYENwehXlg/kFVnos3rEvtKTjRvOWSzb6H4=
 github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
 github.com/tmc/grpc-websocket-proxy v0.0.0-20201229170055-e5319fda7802 h1:uruHq4dN7GR16kFc5fp3d1RIYzJW5onx8Ybykw2YQFA=
 github.com/tmc/grpc-websocket-proxy v0.0.0-20201229170055-e5319fda7802/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
 github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2 h1:eY9dn8+vbi4tKz5Qo6v2eYzo7kUS51QINcR5jNpbZS8=
 github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU=
 github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
 github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
 go.etcd.io/bbolt v1.3.5 h1:XAzx9gjCb0Rxj7EoqcClPD1d5ZBxZJk0jbuoPHenBt0=
 go.etcd.io/bbolt v1.3.5/go.mod h1:G5EMThwa9y8QZGBClrRx5EY+Yw9kAhnjy3bSjsnlVTQ=
 go.etcd.io/etcd v3.3.25+incompatible h1:V1RzkZJj9LqsJRy+TUBgpWSbZXITLB819lstuTFoZOY=
 go.etcd.io/etcd v3.3.25+incompatible/go.mod h1:yaeTdrJi5lOmYerz05bd8+V7KubZs8YSFZfzsF9A6aI=
 go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
 go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
 go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
 go.uber.org/atomic v1.4.0 h1:cxzIVoETapQEqDhQu3QfnvXAV4AlzcvUCxkVUFw3+EU=
 go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
 go.uber.org/multierr v1.1.0 h1:HoEmRHQPVSqub6w2z2d2EOVs2fjyFRGyofhKuyDq0QI=
 go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
 go.uber.org/zap v1.10.0 h1:ORx85nbTijNz8ljznvCMR1ZBIPKFn3jQrag10X2AsuM=
 go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
 golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
 golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
 golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20191206172530-e9b2fee46413/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
 golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9 h1:psW17arqaxU48Z5kZ0CQnkZWQJsqcURM6tKiBApRjXI=
 golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
 golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
 golang.org/x/exp v0.0.0-20190829153037-c13cbed26979/go.mod h1:86+5VVa7VpoJ4kLfm080zCjGlMRFzhUhsZKEZO7MGek=
 golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
 golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
 golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
 golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
 golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
 golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
 golang.org/x/lint v0.0.0-20190409202823-959b441ac422/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
 golang.org/x/lint v0.0.0-20190909230951-414d861bb4ac/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
 golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f/go.mod h1:5qLYkcX4OjUUV8bRuDixDT3tpyyb+LUpUlRWLxfhWrs=
 golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE=
 golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028/go.mod h1:E/iHnbuqvinMTCcRqshq8CkpyQDoeVncDDYHnLhea+o=
 golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc=
 golang.org/x/mod v0.1.0/go.mod h1:0QHyrYULN0/3qlju5TqG8bIK38QM8yzMo5ekMj3DlcY=
 golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
 golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
 golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
 golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
 golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
 golang.org/x/net v0.0.0-20190501004415-9ce7a6920f09/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
 golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
 golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
 golang.org/x/net v0.0.0-20190613194153-d28f0bde5980/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
 golang.org/x/net v0.0.0-20200707034311-ab3426394381 h1:VXak5I6aEWmAXeQjA+QSZzlgNrpq9mjcfDemuexIKsU=
 golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
 golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
 golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
 golang.org/x/net v0.0.0-20201202161906-c7110b5ffcbb h1:eBmm0M9fYhWpKZLjQUUKka/LtIxf46G4fxeEz5KJr9U=
 golang.org/x/net v0.0.0-20201202161906-c7110b5ffcbb/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
 golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
 golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190507160741-ecd444e8653b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190606165138-5da285871e9c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190624142023-c5567b49c5d0/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20191228213918-04cbcbbfeed8/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200106162015-b016eb3dc98e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200302150141-5c8b2ff67527/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200615200032-f1bc736245b1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200622214017-ed371f2e16b4 h1:5/PjkGUjvEU5Gl6BxmvKRPpqo2uNMv4rcHBMwzk/st8=
 golang.org/x/sys v0.0.0-20200622214017-ed371f2e16b4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f h1:+Nyd8tzPX9R7BWHguqsrbFdRx3WQ/1ib8I44HXV5yTA=
 golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
 golang.org/x/text v0.3.3 h1:cokOdA+Jmi5PJGXLlLllQSgYigAEfHXJAERHVMaCc2k=
 golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/time v0.0.0-20191024005414-555d28b269f0 h1:/5xXl8Y5W96D+TtHSlonuFqGHIWVuyCkGJLwGh9JJFs=
 golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20181011042414-1f849cf54d09/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20181030221726-6c7e314b6563/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
 golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
 golang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
 golang.org/x/tools v0.0.0-20190312170243-e65039ee4138/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
 golang.org/x/tools v0.0.0-20190425150028-36563e24a262/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
 golang.org/x/tools v0.0.0-20190506145303-2d16b83fe98c/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
 golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
 golang.org/x/tools v0.0.0-20190606124116-d0a3d012864b/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
 golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
 golang.org/x/tools v0.0.0-20190624222133-a101b041ded4/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
 golang.org/x/tools v0.0.0-20190628153133-6cdbf07be9d0/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
 golang.org/x/tools v0.0.0-20190816200558-6889da9d5479/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.0.0-20190911174233-4f2ddba30aff/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.0.0-20191012152004-8de300cfc20a/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.0.0-20191125144606-a911d9008d1f/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.0.0-20191227053925-7b8e75db28f4/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
 golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
 golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
 golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
 golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
 golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 google.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE=
 google.golang.org/api v0.7.0/go.mod h1:WtwebWUNSVBH/HAw79HIFXZNqEvBhG+Ra+ax0hx3E3M=
 google.golang.org/api v0.8.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg=
 google.golang.org/api v0.9.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg=
 google.golang.org/api v0.15.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=
 google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
 google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
 google.golang.org/appengine v1.6.1/go.mod h1:i06prIuMbXzDqacNJfV5OdTW448YApPu5ww/cMBSeb0=
 google.golang.org/appengine v1.6.5/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
 google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
 google.golang.org/genproto v0.0.0-20190418145605-e7d98fc518a7/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
 google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
 google.golang.org/genproto v0.0.0-20190502173448-54afdca5d873/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
 google.golang.org/genproto v0.0.0-20190801165951-fa694d86fc64/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
 google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
 google.golang.org/genproto v0.0.0-20190911173649-1774047e7e51/go.mod h1:IbNlFCBrqXvoKpeg0TB2l7cyZUmoaFKYIwrEpbDKLA8=
 google.golang.org/genproto v0.0.0-20191230161307-f3c370f40bfb/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
 google.golang.org/genproto v0.0.0-20200423170343-7949de9c1215/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
 google.golang.org/genproto v0.0.0-20200513103714-09dca8ec2884/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
 google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013 h1:+kGHl1aib/qcwaRi1CbqBZ1rk19r85MNUf8HaBghugY=
 google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo=
 google.golang.org/grpc v1.25.1 h1:wdKvqQk7IttEw92GoRyKG2IDrUIpgpj6H6m81yfeMW0=
 google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY=
 google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
 google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
 google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
 google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE=
 google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo=
 google.golang.org/protobuf v1.22.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
 google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
 google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
 google.golang.org/protobuf v1.24.0 h1:UhZDfRO8JRQru4/+LlLE0BRKGF8L+PICnvYZmx/fEGA=
 google.golang.org/protobuf v1.24.0/go.mod h1:r/3tXBNzIEhYS9I1OUVjXDlt8tc493IdKGjtUeSXeh4=
 gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo=
 gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=
 gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
 gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw=
 gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
 gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.3/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.5/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.8 h1:obN1ZagJSUGI0Ek/LBmuj4SNLPfIny3KsKFopxRdj10=
 gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gotest.tools v2.2.0+incompatible/go.mod h1:DsYFclhRJ6vuDpmuTbkuFWG+y2sxOXAzmJt81HFBacw=
 gotest.tools/v3 v3.0.2/go.mod h1:3SzNCllyD9/Y+b5r9JIKQ474KzkZyqLqEfYqMsX94Bk=
 honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
 honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
 honnef.co/go/tools v0.0.0-20190418001031-e561f6794a2a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
 honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
 honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg=
 k8s.io/api v0.19.0/go.mod h1:I1K45XlvTrDjmj5LoM5LuP/KYrhWbjUKT/SoPG0qTjw=
 k8s.io/apimachinery v0.19.0/go.mod h1:DnPGDnARWFvYa3pMHgSxtbZb7gpzzAZ1pTfaUNDVlmA=
 k8s.io/client-go v0.19.0/go.mod h1:H9E/VT95blcFQnlyShFgnFT9ZnJOAceiUHM3MlRC+mU=
 k8s.io/component-base v0.19.0/go.mod h1:dKsY8BxkA+9dZIAh2aWJLL/UdASFDNtGYTCItL4LM7Y=
 k8s.io/gengo v0.0.0-20200413195148-3a45101e95ac/go.mod h1:ezvh/TsK7cY6rbqRK0oQQ8IAqLxYwwyPxAX1Pzy0ii0=
 k8s.io/klog v1.0.0 h1:Pt+yjF5aB1xDSVbau4VsWe+dQNzA0qv1LlXdC2dF6Q8=
 k8s.io/klog v1.0.0/go.mod h1:4Bi6QPql/J/LkTDqv7R/cd3hPo4k2DG6Ptcz060Ez5I=
 k8s.io/klog/v2 v2.0.0/go.mod h1:PBfzABfn139FHAV07az/IF9Wp1bkk3vpT2XSJ76fSDE=
 k8s.io/klog/v2 v2.2.0 h1:XRvcwJozkgZ1UQJmfMGpvRthQHOvihEhYtDfAaxMz/A=
 k8s.io/klog/v2 v2.2.0/go.mod h1:Od+F08eJP+W3HUb4pSrPpgp9DGU4GzlpG/TmITuYh/Y=
 k8s.io/kube-openapi v0.0.0-20200805222855-6aeccd4b50c6/go.mod h1:UuqjUnNftUyPE5H64/qeyjQoUZhGpeFDVdxjTeEVN2o=
 k8s.io/utils v0.0.0-20200729134348-d5654de09c73/go.mod h1:jPW/WVKK9YHAvNhRxK0md/EJ228hCsBRufyofKtW8HA=
 k8s.io/utils v0.0.0-20210305010621-2afb4311ab10 h1:u5rPykqiCpL+LBfjRkXvnK71gOgIdmq3eHUEkPrbeTI=
 k8s.io/utils v0.0.0-20210305010621-2afb4311ab10/go.mod h1:jPW/WVKK9YHAvNhRxK0md/EJ228hCsBRufyofKtW8HA=
 rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8=
 sigs.k8s.io/structured-merge-diff/v4 v4.0.1/go.mod h1:bJZC9H9iH24zzfZ/41RGcq60oK1F7G282QMXDPYydCw=
 sigs.k8s.io/yaml v1.1.0/go.mod h1:UJmg0vDUVViEyp3mgSv9WPwZCDxu4rQW1olrI1uml+o=
 sigs.k8s.io/yaml v1.2.0 h1:kr/MCeFWJWTwyaHoR9c8EjH9OumOmoF9YGiZd7lFm/Q=
 sigs.k8s.io/yaml v1.2.0/go.mod h1:yfXDCHCao9+ENCvLSE62v9VSji2MKu5jeNfTrofGhJc=
@@ -5,7 +5,7 @@ package vitastor
 const (
    vitastorCSIDriverName    = "csi.vitastor.io"
-    vitastorCSIDriverVersion = "0.6.6"
+    vitastorCSIDriverVersion = "0.6.7"
 )
 // Config struct fills the parameters of request or user input
@@ -356,7 +356,7 @@ func (cs *ControllerServer) DeleteVolume(ctx context.Context, req *csi.DeleteVol
    // Delete inode data by invoking vitastor-cli
    args := []string{
-        "rm", "--etcd_address", strings.Join(etcdUrl, ","),
+        "rm-data", "--etcd_address", strings.Join(etcdUrl, ","),
        "--pool", fmt.Sprintf("%d", idx.PoolId),
        "--inode", fmt.Sprintf("%d", idx.Id),
    }
@@ -372,7 +372,7 @@ func (cs *ControllerServer) DeleteVolume(ctx context.Context, req *csi.DeleteVol
    stderrStr := string(stderr.Bytes())
    if (err != nil)
    {
-        klog.Errorf("vitastor-cli rm failed: %s, status %s\n", stderrStr, err)
+        klog.Errorf("vitastor-cli rm-data failed: %s, status %s\n", stderrStr, err)
        return nil, status.Error(codes.Internal, stderrStr+" (status "+err.Error()+")")
    }
@@ -1,7 +1,7 @@
 #!/bin/bash
-sed 's/$REL/bullseye/g' < vitastor.Dockerfile > ../Dockerfile
+cat < vitastor.Dockerfile > ../Dockerfile
 cd ..
 mkdir -p packages
-sudo podman build -v `pwd`/packages:/root/packages -f Dockerfile .
+sudo podman build --build-arg REL=bullseye -v `pwd`/packages:/root/packages -f Dockerfile .
 rm Dockerfile
@@ -1,7 +1,7 @@
 #!/bin/bash
-sed 's/$REL/buster/g' < vitastor.Dockerfile > ../Dockerfile
+cat < vitastor.Dockerfile > ../Dockerfile
 cd ..
 mkdir -p packages
-sudo podman build -v `pwd`/packages:/root/packages -f Dockerfile .
+sudo podman build --build-arg REL=buster -v `pwd`/packages:/root/packages -f Dockerfile .
 rm Dockerfile
@@ -1,4 +1,4 @@
-vitastor (0.6.6-1) unstable; urgency=medium
+vitastor (0.6.7-1) unstable; urgency=medium
  * RDMA support
  * Bugfixes
@@ -1,7 +1,9 @@
 # Build patched QEMU for Debian Buster or Bullseye/Sid inside a container
 # cd ..; podman build --build-arg REL=bullseye -v `pwd`/packages:/root/packages -f debian/patched-qemu.Dockerfile .
 ARG REL=
 FROM debian:$REL
 ARG REL=
 WORKDIR /root
@@ -11,10 +13,6 @@ RUN if [ "$REL" = "buster" ]; then \
        echo 'Package: *' >> /etc/apt/preferences; \
        echo 'Pin: release a=buster-backports' >> /etc/apt/preferences; \
        echo 'Pin-Priority: 500' >> /etc/apt/preferences; \
        echo >> /etc/apt/preferences; \
        echo 'Package: libglvnd* libgles* libglx* libgl1 libegl* libopengl* mesa*' >> /etc/apt/preferences; \
        echo 'Pin: release a=buster-backports' >> /etc/apt/preferences; \
        echo 'Pin-Priority: 50' >> /etc/apt/preferences; \
    fi; \
    grep '^deb ' /etc/apt/sources.list | perl -pe 's/^deb/deb-src/' >> /etc/apt/sources.list; \
    echo 'APT::Install-Recommends false;' >> /etc/apt/apt.conf; \
@@ -1,7 +1,9 @@
 # Build Vitastor packages for Debian Buster or Bullseye/Sid inside a container
 # cd ..; podman build --build-arg REL=bullseye -v `pwd`/packages:/root/packages -f debian/vitastor.Dockerfile .
 ARG REL=
 FROM debian:$REL
 ARG REL=
 WORKDIR /root
@@ -40,10 +42,10 @@ RUN set -e -x; \
    mkdir -p /root/packages/vitastor-$REL; \
    rm -rf /root/packages/vitastor-$REL/*; \
    cd /root/packages/vitastor-$REL; \
-    cp -r /root/vitastor vitastor-0.6.6; \
+    cp -r /root/vitastor vitastor-0.6.7; \
-    ln -s /root/packages/qemu-$REL/qemu-*/ vitastor-0.6.6/qemu; \
+    ln -s /root/packages/qemu-$REL/qemu-*/ vitastor-0.6.7/qemu; \
-    ln -s /root/fio-build/fio-*/ vitastor-0.6.6/fio; \
+    ln -s /root/fio-build/fio-*/ vitastor-0.6.7/fio; \
-    cd vitastor-0.6.6; \
+    cd vitastor-0.6.7; \
    FIO=$(head -n1 fio/debian/changelog | perl -pe 's/^.*\((.*?)\).*$/$1/'); \
    QEMU=$(head -n1 qemu/debian/changelog | perl -pe 's/^.*\((.*?)\).*$/$1/'); \
    sh copy-qemu-includes.sh; \
@@ -59,8 +61,8 @@ RUN set -e -x; \
    echo "dep:fio=$FIO" > debian/substvars; \
    echo "dep:qemu=$QEMU" >> debian/substvars; \
    cd /root/packages/vitastor-$REL; \
-    tar --sort=name --mtime='2020-01-01' --owner=0 --group=0 --exclude=debian -cJf vitastor_0.6.6.orig.tar.xz vitastor-0.6.6; \
+    tar --sort=name --mtime='2020-01-01' --owner=0 --group=0 --exclude=debian -cJf vitastor_0.6.7.orig.tar.xz vitastor-0.6.7; \
-    cd vitastor-0.6.6; \
+    cd vitastor-0.6.7; \
    V=$(head -n1 debian/changelog | perl -pe 's/^.*\((.*?)\).*$/$1/'); \
    DEBFULLNAME="Vitaliy Filippov <vitalif@yourcmc.ru>" dch -D $REL -v "$V""$REL" "Rebuild for $REL"; \
    DEB_BUILD_OPTIONS=nocheck dpkg-buildpackage --jobs=auto -sa; \
@@ -33,8 +33,10 @@ const etcd_allow = new RegExp('^'+[
    'pg/state/[1-9]\\d*/[1-9]\\d*',
    'pg/stats/[1-9]\\d*/[1-9]\\d*',
    'pg/history/[1-9]\\d*/[1-9]\\d*',
    'pool/stats/[1-9]\\d*',
    'history/last_clean_pgs',
    'inode/stats/[1-9]\\d*/[1-9]\\d*',
    'pool/stats/[1-9]\\d*',
    'stats',
    'index/image/.*',
    'index/maxid/[1-9]\\d*',
@@ -81,11 +83,12 @@ const etcd_tree = {
            osd_ping_timeout: 5, // seconds. min: 1
            up_wait_retry_interval: 500, // ms. min: 50
            // osd
-            etcd_report_interval: 30, // min: 10
+            etcd_report_interval: 5,
            run_primary: true,
            bind_address: "0.0.0.0",
            bind_port: 0,
            autosync_interval: 5,
            autosync_writes: 128,
            client_queue_depth: 128, // unused
            recovery_queue_depth: 4,
            recovery_sync_batch: 16,
@@ -229,7 +232,7 @@ const etcd_tree = {
            /* <pool_id>: {
                <pg_id>: {
                    primary: osd_num_t,
-                    state: ("starting"|"peering"|"incomplete"|"active"|"repeering"|"stopping"|"offline"|
+                    state: ("starting"|"peering"|"peered"|"incomplete"|"active"|"repeering"|"stopping"|"offline"|
                        "degraded"|"has_incomplete"|"has_degraded"|"has_misplaced"|"has_unclean"|
                        "has_invalid"|"left_on_dead")[],
                }
@@ -262,9 +265,9 @@ const etcd_tree = {
            /* <pool_id>: {
                <inode_t>: {
                    raw_used: uint64_t, // raw used bytes on OSDs
-                    read: { count: uint64_t, usec: uint64_t, bytes: uint64_t },
+                    read: { count: uint64_t, usec: uint64_t, bytes: uint64_t, bps: uint64_t, iops: uint64_t, lat: uint64_t },
-                    write: { count: uint64_t, usec: uint64_t, bytes: uint64_t },
+                    write: { count: uint64_t, usec: uint64_t, bytes: uint64_t, bps: uint64_t, iops: uint64_t, lat: uint64_t },
-                    delete: { count: uint64_t, usec: uint64_t, bytes: uint64_t },
+                    delete: { count: uint64_t, usec: uint64_t, bytes: uint64_t, bps: uint64_t, iops: uint64_t, lat: uint64_t },
                },
            }, */
        },
@@ -281,14 +284,14 @@ const etcd_tree = {
    },
    stats: {
        /* op_stats: {
-            <string>: { count: uint64_t, usec: uint64_t, bytes: uint64_t },
+            <string>: { count: uint64_t, usec: uint64_t, bytes: uint64_t, bps: uint64_t, iops: uint64_t, lat: uint64_t },
        },
        subop_stats: {
-            <string>: { count: uint64_t, usec: uint64_t },
+            <string>: { count: uint64_t, usec: uint64_t, iops: uint64_t, lat: uint64_t },
        },
        recovery_stats: {
-            degraded: { count: uint64_t, bytes: uint64_t },
+            degraded: { count: uint64_t, bytes: uint64_t, bps: uint64_t, iops: uint64_t },
-            misplaced: { count: uint64_t, bytes: uint64_t },
+            misplaced: { count: uint64_t, bytes: uint64_t, bps: uint64_t, iops: uint64_t },
        },
        object_counts: {
            object: uint64_t,
@@ -335,6 +338,8 @@ class Mon
        this.etcd_prefix = this.etcd_prefix.replace(/\/\/+/g, '/').replace(/^\/?(.*[^\/])\/?$/, '/$1');
        this.etcd_start_timeout = (config.etcd_start_timeout || 5) * 1000;
        this.state = JSON.parse(JSON.stringify(this.constructor.etcd_tree));
        this.signals_set = false;
        this.on_stop_cb = () => this.on_stop().catch(console.error);
    }
    async start()
@@ -344,7 +349,18 @@ class Mon
        await this.become_master();
        await this.load_cluster_state();
        await this.start_watcher(this.config.etcd_mon_retries);
        for (const pool_id in this.state.config.pools)
        {
            if (!this.state.pool.stats[pool_id] ||
                !this.state.pool.stats[pool_id].pg_real_size)
            {
                // Generate missing data in etcd
                this.state.config.pgs.hash = null;
                break;
            }
        }
        await this.recheck_pgs();
        this.schedule_update_stats();
    }
    async load_config()
@@ -386,11 +402,6 @@ class Mon
        {
            this.config.mon_stats_timeout = 100;
        }
        this.config.mon_stats_interval = Number(this.config.mon_stats_interval) || 5000;
        if (this.config.mon_stats_interval < 100)
        {
            this.config.mon_stats_interval = 100;
        }
        // After this number of seconds, a dead OSD will be removed from PG distribution
        this.config.osd_out_time = Number(this.config.osd_out_time) || 0;
        if (!this.config.osd_out_time)
@@ -553,7 +564,7 @@ class Mon
        const max_ttl = this.config.etcd_mon_ttl + this.config.etcd_mon_timeout/1000*this.config.etcd_mon_retries;
        const res = await this.etcd_call('/lease/grant', { TTL: max_ttl }, this.config.etcd_mon_timeout, -1);
        this.etcd_lease_id = res.ID;
-        setInterval(async () =>
+        this.lease_timer = setInterval(async () =>
        {
            const res = await this.etcd_call('/lease/keepalive', { ID: this.etcd_lease_id }, this.config.etcd_mon_timeout, this.config.etcd_mon_retries);
            if (!res.result.TTL)
@@ -561,6 +572,19 @@ class Mon
                this.die('Lease expired');
            }
        }, this.config.etcd_mon_timeout);
        if (!this.signals_set)
        {
            process.on('SIGINT', this.on_stop_cb);
            process.on('SIGTERM', this.on_stop_cb);
            this.signals_set = true;
        }
    }
    async on_stop()
    {
        clearInterval(this.lease_timer);
        await this.etcd_call('/lease/revoke', { ID: this.etcd_lease_id }, this.config.etcd_mon_timeout, this.config.etcd_mon_retries);
        process.exit(0);
    }
    async become_master()
@@ -1059,7 +1083,9 @@ class Mon
                this.state.pool.stats[pool_id] = {
                    used_raw_tb: (this.state.pool.stats[pool_id]||{}).used_raw_tb || 0,
                    total_raw_tb: optimize_result.space,
-                    raw_to_usable: pg_effsize / (pool_cfg.pg_size - (pool_cfg.parity_chunks||0)),
+                    pg_real_size: pg_effsize,
                    raw_to_usable: pg_effsize / (pool_cfg.scheme === 'replicated'
                        ? 1 : (pool_cfg.pg_size - (pool_cfg.parity_chunks||0))),
                    space_efficiency: optimize_result.space/(optimize_result.total_space||1),
                };
                etcd_request.success.push({ requestPut: {
@@ -1172,7 +1198,7 @@ class Mon
        }, this.config.mon_change_timeout || 1000);
    }
-    sum_op_stats()
+    sum_op_stats(timestamp, prev_stats)
    {
        const op_stats = {}, subop_stats = {}, recovery_stats = {};
        for (const osd in this.state.osd.stats)
@@ -1198,6 +1224,29 @@ class Mon
                recovery_stats[op].bytes += BigInt(st.recovery_stats[op].bytes||0);
            }
        }
        if (prev_stats && prev_stats.timestamp >= timestamp)
        {
            prev_stats = null;
        }
        const tm = prev_stats ? BigInt(timestamp - prev_stats.timestamp) : 0;
        for (const op in op_stats)
        {
            op_stats[op].bps = prev_stats ? (op_stats[op].bytes - prev_stats.op_stats[op].bytes) * 1000n / tm : 0;
            op_stats[op].iops = prev_stats ? (op_stats[op].count - prev_stats.op_stats[op].count) * 1000n / tm : 0;
            op_stats[op].lat = prev_stats ? (op_stats[op].usec - prev_stats.op_stats[op].usec)
                / ((op_stats[op].count - prev_stats.op_stats[op].count) || 1n) : 0;
        }
        for (const op in subop_stats)
        {
            subop_stats[op].iops = prev_stats ? (subop_stats[op].count - prev_stats.subop_stats[op].count) * 1000n / tm : 0;
            subop_stats[op].lat = prev_stats ? (subop_stats[op].usec - prev_stats.subop_stats[op].usec)
                / ((subop_stats[op].count - prev_stats.subop_stats[op].count) || 1n) : 0;
        }
        for (const op in recovery_stats)
        {
            recovery_stats[op].bps = prev_stats ? (recovery_stats[op].bytes - prev_stats.recovery_stats[op].bytes) * 1000n / tm : 0;
            recovery_stats[op].iops = prev_stats ? (recovery_stats[op].count - prev_stats.recovery_stats[op].count) * 1000n / tm : 0;
        }
        return { op_stats, subop_stats, recovery_stats };
    }
@@ -1224,7 +1273,7 @@ class Mon
        return object_counts;
    }
-    sum_inode_stats()
+    sum_inode_stats(prev_stats, timestamp, prev_timestamp)
    {
        const inode_stats = {};
        const inode_stub = () => ({
@@ -1233,8 +1282,10 @@ class Mon
            write: { count: 0n, usec: 0n, bytes: 0n },
            delete: { count: 0n, usec: 0n, bytes: 0n },
        });
        const seen_pools = {};
        for (const pool_id in this.state.config.pools)
        {
            seen_pools[pool_id] = true;
            this.state.pool.stats[pool_id] = this.state.pool.stats[pool_id] || {};
            this.state.pool.stats[pool_id].used_raw_tb = 0n;
        }
@@ -1242,7 +1293,12 @@ class Mon
        {
            for (const pool_id in this.state.osd.space[osd_num])
            {
-                this.state.pool.stats[pool_id] = this.state.pool.stats[pool_id] || { used_raw_tb: 0n };
+                this.state.pool.stats[pool_id] = this.state.pool.stats[pool_id] || {};
                if (!seen_pools[pool_id])
                {
                    this.state.pool.stats[pool_id].used_raw_tb = 0n;
                    seen_pools[pool_id] = true;
                }
                inode_stats[pool_id] = inode_stats[pool_id] || {};
                for (const inode_num in this.state.osd.space[osd_num][pool_id])
                {
@@ -1253,7 +1309,7 @@ class Mon
                }
            }
        }
-        for (const pool_id in this.state.config.pools)
+        for (const pool_id in seen_pools)
        {
            const used = this.state.pool.stats[pool_id].used_raw_tb;
            this.state.pool.stats[pool_id].used_raw_tb = Number(used)/1024/1024/1024/1024;
@@ -1276,43 +1332,31 @@ class Mon
                }
            }
        }
-        return inode_stats;
+        if (prev_stats && prev_timestamp >= timestamp)
    }
    fix_stat_overflows(obj, scratch)
    {
        for (const k in obj)
        {
-            if (typeof obj[k] == 'bigint')
+            prev_stats = null;
        }
        const tm = prev_stats ? BigInt(timestamp - prev_timestamp) : 0;
        for (const pool_id in inode_stats)
        {
            for (const inode_num in inode_stats[pool_id])
            {
-                if (obj[k] >= 0x10000000000000000n)
+                for (const op of [ 'read', 'write', 'delete' ])
                {
-                    if (scratch[k])
+                    const op_st = inode_stats[pool_id][inode_num][op];
-                    {
+                    const prev_st = prev_stats && prev_stats[pool_id] && prev_stats[pool_id][inode_num] && prev_stats[pool_id][inode_num][op];
-                        for (const k2 in scratch)
+                    op_st.bps = prev_st ? (op_st.bytes - prev_st.bytes) * 1000n / tm : 0;
-                        {
+                    op_st.iops = prev_st ? (op_st.count - prev_st.count) * 1000n / tm : 0;
-                            obj[k2] -= scratch[k2];
+                    op_st.lat = prev_st ? (op_st.usec - prev_st.usec) / ((op_st.count - prev_st.count) || 1n) : 0;
                            scratch[k2] = 0n;
                        }
                    }
                    else
                    {
                        for (const k2 in obj)
                        {
                            scratch[k2] = obj[k2];
                        }
                    }
                }
            }
            else if (typeof obj[k] == 'object')
            {
                this.fix_stat_overflows(obj[k], scratch[k] = (scratch[k] || {}));
            }
        }
        return inode_stats;
    }
    serialize_bigints(obj)
    {
        obj = { ...obj };
        for (const k in obj)
        {
            if (typeof obj[k] == 'bigint')
@@ -1321,22 +1365,26 @@ class Mon
            }
            else if (typeof obj[k] == 'object')
            {
-                this.serialize_bigints(obj[k]);
+                obj[k] = this.serialize_bigints(obj[k]);
            }
        }
        return obj;
    }
    async update_total_stats()
    {
        const txn = [];
-        const stats = this.sum_op_stats();
+        const timestamp = Date.now();
        const object_counts = this.sum_object_counts();
-        const inode_stats = this.sum_inode_stats();
+        let stats = this.sum_op_stats(timestamp, this.prev_stats);
-        this.fix_stat_overflows(stats, (this.prev_stats = this.prev_stats || {}));
+        let inode_stats = this.sum_inode_stats(
-        this.fix_stat_overflows(inode_stats, (this.prev_inode_stats = this.prev_inode_stats || {}));
+            this.prev_stats ? this.prev_stats.inode_stats : null,
            timestamp, this.prev_stats ? this.prev_stats.timestamp : null
        );
        this.prev_stats = { timestamp, ...stats, inode_stats };
        stats.object_counts = object_counts;
-        this.serialize_bigints(stats);
+        stats = this.serialize_bigints(stats);
-        this.serialize_bigints(inode_stats);
+        inode_stats = this.serialize_bigints(inode_stats);
        txn.push({ requestPut: { key: b64(this.etcd_prefix+'/stats'), value: b64(JSON.stringify(stats)) } });
        for (const pool_id in inode_stats)
        {
@@ -1350,9 +1398,11 @@ class Mon
        }
        for (const pool_id in this.state.pool.stats)
        {
            const pool_stats = { ...this.state.pool.stats[pool_id] };
            this.serialize_bigints(pool_stats);
            txn.push({ requestPut: {
                key: b64(this.etcd_prefix+'/pool/stats/'+pool_id),
-                value: b64(JSON.stringify(this.state.pool.stats[pool_id])),
+                value: b64(JSON.stringify(pool_stats)),
            } });
        }
        if (txn.length)
@@ -1365,20 +1415,13 @@ class Mon
    {
        if (this.stats_timer)
        {
-            clearTimeout(this.stats_timer);
+            return;
            this.stats_timer = null;
        }
        let sleep = (this.stats_update_next||0) - Date.now();
        if (sleep < this.config.mon_stats_timeout)
        {
            sleep = this.config.mon_stats_timeout;
        }
        this.stats_timer = setTimeout(() =>
        {
            this.stats_timer = null;
            this.stats_update_next = Date.now() + this.config.mon_stats_interval;
            this.update_total_stats().catch(console.error);
-        }, sleep);
+        }, this.config.mon_stats_timeout);
    }
    parse_kv(kv)
@@ -50,7 +50,7 @@ from cinder.volume import configuration
 from cinder.volume import driver
 from cinder.volume import volume_utils
-VERSION = '0.6.6'
+VERSION = '0.6.7'
 LOG = logging.getLogger(__name__)
@@ -266,7 +266,7 @@ class VitastorDriver(driver.CloneableImageVD,
            stats['provisioned_capacity_gb'] = round(total_provisioned/1024.0/1024.0/1024.0, 2)
            pool_stats = pool_stats['responses'][0]['kvs']
            if len(pool_stats):
-                pool_stats = pool_stats[0]
+                pool_stats = pool_stats[0]['value']
                stats['free_capacity_gb'] = round(1024.0*(pool_stats['total_raw_tb']-pool_stats['used_raw_tb'])/pool_stats['raw_to_usable'], 2)
                stats['total_capacity_gb'] = round(1024.0*pool_stats['total_raw_tb'], 2)
            stats['backend_state'] = 'up'
@@ -275,6 +275,15 @@ class VitastorDriver(driver.CloneableImageVD,
            LOG.exception('error getting vitastor pool stats: '+str(e))
        self._stats = stats
    def get_volume_stats(self, refresh=False):
        """Get volume stats.
        If 'refresh' is True, run update the stats first.
        """
        if not self._stats or refresh:
            self._update_volume_stats()
        return self._stats
    def _next_id(self, resp):
        if len(resp['kvs']) == 0:
@@ -514,7 +523,7 @@ class VitastorDriver(driver.CloneableImageVD,
        # Clear data
        for kv in layers:
            args = [
-                'vitastor-cli', 'rm', '--pool', str(kv['value']['pool_id']),
+                'vitastor-cli', 'rm-data', '--pool', str(kv['value']['pool_id']),
                '--inode', str(kv['value']['id']), '--progress', '0',
                *(self._vitastor_args())
            ]
@@ -48,4 +48,4 @@ FIO=`rpm -qi fio | perl -e 'while(<>) { /^Epoch[\s:]+(\S+)/ && print "$1:"; /^Ve
 QEMU=`rpm -qi qemu qemu-kvm | perl -e 'while(<>) { /^Epoch[\s:]+(\S+)/ && print "$1:"; /^Version[\s:]+(\S+)/ && print $1; /^Release[\s:]+(\S+)/ && print "-$1"; }'`
 perl -i -pe 's/(Requires:\s*fio)([^\n]+)?/$1 = '$FIO'/' $VITASTOR/rpm/vitastor-el$EL.spec
 perl -i -pe 's/(Requires:\s*qemu(?:-kvm)?)([^\n]+)?/$1 = '$QEMU'/' $VITASTOR/rpm/vitastor-el$EL.spec
-tar --transform 's#^#vitastor-0.6.6/#' --exclude 'rpm/*.rpm' -czf $VITASTOR/../vitastor-0.6.6$(rpm --eval '%dist').tar.gz *
+tar --transform 's#^#vitastor-0.6.7/#' --exclude 'rpm/*.rpm' -czf $VITASTOR/../vitastor-0.6.7$(rpm --eval '%dist').tar.gz *
@@ -38,7 +38,7 @@ ADD . /root/vitastor
 RUN set -e; \
    cd /root/vitastor/rpm; \
    sh build-tarball.sh; \
-    cp /root/vitastor-0.6.6.el7.tar.gz ~/rpmbuild/SOURCES; \
+    cp /root/vitastor-0.6.7.el7.tar.gz ~/rpmbuild/SOURCES; \
    cp vitastor-el7.spec ~/rpmbuild/SPECS/vitastor.spec; \
    cd ~/rpmbuild/SPECS/; \
    rpmbuild -ba vitastor.spec; \
@@ -1,11 +1,11 @@
 Name:           vitastor
-Version:        0.6.6
+Version:        0.6.7
 Release:        1%{?dist}
 Summary:        Vitastor, a fast software-defined clustered block storage
 License:        Vitastor Network Public License 1.1
 URL:            https://vitastor.io/
-Source0:        vitastor-0.6.6.el7.tar.gz
+Source0:        vitastor-0.6.7.el7.tar.gz
 BuildRequires:  liburing-devel >= 0.6
 BuildRequires:  gperftools-devel
@@ -59,6 +59,7 @@ cp -r mon %buildroot/usr/lib/vitastor/mon
 %_bindir/vitastor-osd
 %_bindir/vitastor-cli
 %_bindir/vitastor-rm
 %_bindir/vita
 %_libdir/qemu-kvm/block-vitastor.so
 %_libdir/libfio_vitastor.so
 %_libdir/libfio_vitastor_blk.so
@@ -36,7 +36,7 @@ ADD . /root/vitastor
 RUN set -e; \
    cd /root/vitastor/rpm; \
    sh build-tarball.sh; \
-    cp /root/vitastor-0.6.6.el8.tar.gz ~/rpmbuild/SOURCES; \
+    cp /root/vitastor-0.6.7.el8.tar.gz ~/rpmbuild/SOURCES; \
    cp vitastor-el8.spec ~/rpmbuild/SPECS/vitastor.spec; \
    cd ~/rpmbuild/SPECS/; \
    rpmbuild -ba vitastor.spec; \
@@ -1,11 +1,11 @@
 Name:           vitastor
-Version:        0.6.6
+Version:        0.6.7
 Release:        1%{?dist}
 Summary:        Vitastor, a fast software-defined clustered block storage
 License:        Vitastor Network Public License 1.1
 URL:            https://vitastor.io/
-Source0:        vitastor-0.6.6.el8.tar.gz
+Source0:        vitastor-0.6.7.el8.tar.gz
 BuildRequires:  liburing-devel >= 0.6
 BuildRequires:  gperftools-devel
@@ -56,6 +56,7 @@ cp -r mon %buildroot/usr/lib/vitastor
 %_bindir/vitastor-osd
 %_bindir/vitastor-cli
 %_bindir/vitastor-rm
 %_bindir/vita
 %_libdir/qemu-kvm/block-vitastor.so
 %_libdir/libfio_vitastor.so
 %_libdir/libfio_vitastor_blk.so
@@ -15,7 +15,7 @@ if("${CMAKE_INSTALL_PREFIX}" MATCHES "^/usr/local/?$")
 	set(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_PREFIX}/${CMAKE_INSTALL_LIBDIR}")
 endif()
-add_definitions(-DVERSION="0.6.6")
+add_definitions(-DVERSION="0.6.7")
 add_definitions(-Wall -Wno-sign-compare -Wno-comment -Wno-parentheses -Wno-pointer-arith -fdiagnostics-color=always -I ${CMAKE_SOURCE_DIR}/src)
 if (${WITH_ASAN})
 	add_definitions(-fsanitize=address -fno-omit-frame-pointer)
@@ -37,7 +37,7 @@ string(REGEX REPLACE "([\\/\\-]D) *NDEBUG" "" CMAKE_C_FLAGS_MINSIZEREL "${CMAKE_
 string(REGEX REPLACE "([\\/\\-]D) *NDEBUG" "" CMAKE_C_FLAGS_RELWITHDEBINFO "${CMAKE_C_FLAGS_RELWITHDEBINFO}")
 macro(install_symlink filepath sympath)
-	install(CODE "execute_process(COMMAND ${CMAKE_COMMAND} -E create_symlink ${filepath} ${sympath})")
+	install(CODE "execute_process(COMMAND ${CMAKE_COMMAND} -E create_symlink ${filepath} \$ENV{DESTDIR}${sympath})")
 	install(CODE "message(\"-- Created symlink: ${sympath} -> ${filepath}\")")
 endmacro(install_symlink)
@@ -153,7 +153,7 @@ target_link_libraries(vitastor-nbd
 # vitastor-cli
 add_executable(vitastor-cli
-	cli.cpp cli_flatten.cpp cli_merge.cpp cli_rm.cpp cli_snap_rm.cpp
+	cli.cpp cli_ls.cpp cli_create.cpp cli_modify.cpp cli_flatten.cpp cli_merge.cpp cli_rm.cpp cli_snap_rm.cpp
 )
 target_link_libraries(vitastor-cli
 	vitastor_client
@@ -241,7 +241,8 @@ target_include_directories(test_cluster_client PUBLIC ${CMAKE_SOURCE_DIR}/src/mo
 ### Install
 install(TARGETS vitastor-osd vitastor-dump-journal vitastor-nbd vitastor-cli RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR})
-install_symlink(${CMAKE_INSTALL_BINDIR}/vitastor-rm vitastor-cli)
+install_symlink(vitastor-cli ${CMAKE_INSTALL_PREFIX}/${CMAKE_INSTALL_BINDIR}/vitastor-rm)
 install_symlink(vitastor-cli ${CMAKE_INSTALL_PREFIX}/${CMAKE_INSTALL_BINDIR}/vita)
 install(
 	TARGETS vitastor_blk vitastor_client
 	LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
@@ -68,6 +68,11 @@ uint64_t blockstore_t::get_free_block_count()
    return impl->get_free_block_count();
 }
 uint64_t blockstore_t::get_journal_size()
 {
    return impl->get_journal_size();
 }
 uint32_t blockstore_t::get_bitmap_granularity()
 {
    return impl->get_bitmap_granularity();
@@ -194,5 +194,7 @@ public:
    uint64_t get_block_count();
    uint64_t get_free_block_count();
    uint64_t get_journal_size();
    uint32_t get_bitmap_granularity();
 };
@@ -368,4 +368,5 @@ public:
    inline uint64_t get_block_count() { return block_count; }
    inline uint64_t get_free_block_count() { return data_alloc->get_free_count(); }
    inline uint32_t get_bitmap_granularity() { return disk_alignment; }
    inline uint64_t get_journal_size() { return journal.len; }
 };
@@ -28,10 +28,38 @@ json11::Json::object cli_tool_t::parse_args(int narg, const char *args[])
        {
            help();
        }
        else if (args[i][0] == '-' && args[i][1] == 'l')
        {
            cfg["long"] = "1";
        }
        else if (args[i][0] == '-' && args[i][1] == 'n')
        {
            cfg["count"] = args[++i];
        }
        else if (args[i][0] == '-' && args[i][1] == 'p')
        {
            cfg["pool"] = args[++i];
        }
        else if (args[i][0] == '-' && args[i][1] == 's')
        {
            cfg["size"] = args[++i];
        }
        else if (args[i][0] == '-' && args[i][1] == 'r')
        {
            cfg["reverse"] = "1";
        }
        else if (args[i][0] == '-' && args[i][1] == 'f')
        {
            cfg["force"] = "1";
        }
        else if (args[i][0] == '-' && args[i][1] == '-')
        {
            const char *opt = args[i]+2;
-            cfg[opt] = !strcmp(opt, "json") || !strcmp(opt, "wait-list") || i == narg-1 ? "1" : args[++i];
+            cfg[opt] = i == narg-1 || !strcmp(opt, "json") || !strcmp(opt, "wait-list") ||
                !strcmp(opt, "long") || !strcmp(opt, "del") || !strcmp(opt, "no-color") ||
                !strcmp(opt, "force") || !strcmp(opt, "reverse") ||
                !strcmp(opt, "writers-stopped") && strcmp("1", args[i+1]) != 0
                ? "1" : args[++i];
        }
        else
        {
@@ -54,35 +82,64 @@ void cli_tool_t::help()
 {
    printf(
        "Vitastor command-line tool\n"
-        "(c) Vitaliy Filippov, 2019+ (VNPL-1.1)\n\n"
+        "(c) Vitaliy Filippov, 2019+ (VNPL-1.1)\n"
        "USAGE:\n"
        "%s rm-data [OPTIONS] --pool <pool> --inode <inode> [--wait-list]\n"
        "  Remove inode data without changing metadata.\n"
        "  --wait-list means first retrieve objects listings and then remove it.\n"
        "  --wait-list requires more memory, but allows to show correct stats.\n"
        "\n"
-        "%s merge-data [OPTIONS] <from> <to> [--target <target>]\n"
+        "USAGE:\n"
        "%s ls [-l] [-p POOL] [--sort FIELD] [-r] [-n N] [<name> ...]\n"
        "  List images (only specified if <name> passed).\n"
        "  -p|--pool POOL  Filter images by pool ID or name\n"
        "  -l|--long       Also report allocated size and I/O statistics\n"
        "  --del           Also include delete operation statistics\n"
        "  --sort FIELD    Sort by specified field (name, size, used_size, <read|write|delete>_<iops|bps|lat|queue>)\n"
        "  -r|--reverse    Sort in descending order\n"
        "  -n|--count N    Only list first N items\n"
        "\n"
        "%s create -s|--size <size> [-p|--pool <id|name>] [--parent <parent_name>[@<snapshot>]] <name>\n"
        "  Create an image. You may use K/M/G/T suffixes for <size>. If --parent is specified,\n"
        "  a copy-on-write image clone is created. Parent must be a snapshot (readonly image).\n"
        "  Pool must be specified if there is more than one pool.\n"
        "\n"
        "%s create --snapshot <snapshot> [-p|--pool <id|name>] <image>\n"
        "%s snap-create [-p|--pool <id|name>] <image>@<snapshot>\n"
        "  Create a snapshot of image <name>. May be used live if only a single writer is active.\n"
        "\n"
        "%s modify <name> [-s|--size <size>] [--readonly | --readwrite] [-f|--force]\n"
        "  Resize image or change its readonly status. Images with children can't be made read-write.\n"
        "  If the new size is smaller than the old size, extra data will be purged.\n"
        "  You should resize file system in the image, if present, before shrinking it.\n"
        "  -f|--force  Proceed with shrinking or setting readwrite flag even if the image has children.\n"
        "\n"
        "%s rm <from> [<to>] [--writers-stopped]\n"
        "  Remove <from> or all layers between <from> and <to> (<to> must be a child of <from>),\n"
        "  rebasing all their children accordingly. --writers-stopped allows merging to be a bit\n"
        "  more effective in case of a single 'slim' read-write child and 'fat' removed parent:\n"
        "  the child is merged into parent and parent is renamed to child in that case.\n"
        "  In other cases parent layers are always merged into children.\n"
        "\n"
        "%s flatten <layer>\n"
        "  Flatten a layer, i.e. merge data and detach it from parents.\n"
        "\n"
        "%s rm-data --pool <pool> --inode <inode> [--wait-list] [--min-offset <offset>]\n"
        "  Remove inode data without changing metadata.\n"
        "  --wait-list   Retrieve full objects listings before starting to remove objects.\n"
        "                Requires more memory, but allows to show correct removal progress.\n"
        "  --min-offset  Purge only data starting with specified offset.\n"
        "\n"
        "%s merge-data <from> <to> [--target <target>]\n"
        "  Merge layer data without changing metadata. Merge <from>..<to> to <target>.\n"
        "  <to> must be a child of <from> and <target> may be one of the layers between\n"
        "  <from> and <to>, including <from> and <to>.\n"
        "\n"
-        "%s flatten [OPTIONS] <layer>\n"
+        "GLOBAL OPTIONS:\n"
        "  Flatten a layer, i.e. merge data and detach it from parents\n"
        "\n"
        "%s rm [OPTIONS] <from> [<to>] [--writers-stopped 1]\n"
        "  Remove <from> or all layers between <from> and <to> (<to> must be a child of <from>),\n"
        "  rebasing all their children accordingly. One of deleted parents may be renamed to one\n"
        "  of children \"to be rebased\", but only if that child itself is readonly or if\n"
        "  --writers-stopped 1 is specified\n"
        "\n"
        "OPTIONS (global):\n"
        "  --etcd_address <etcd_address>\n"
        "  --iodepth N         Send N operations in parallel to each OSD when possible (default 32)\n"
        "  --parallel_osds M   Work with M osds in parallel when possible (default 4)\n"
        "  --progress 1|0      Report progress (default 1)\n"
        "  --cas 1|0           Use online CAS writes when possible (default auto)\n"
        "  --no-color          Disable colored output\n"
        "  --json              JSON output\n"
        ,
-        exe_name, exe_name, exe_name, exe_name
+        exe_name, exe_name, exe_name, exe_name, exe_name, exe_name, exe_name, exe_name, exe_name
    );
    exit(0);
 }
@@ -176,6 +233,21 @@ void cli_tool_t::run(json11::Json cfg)
        fprintf(stderr, "command is missing\n");
        exit(1);
    }
    else if (cmd[0] == "ls")
    {
        // List images
        action_cb = start_ls(cfg);
    }
    else if (cmd[0] == "create" || cmd[0] == "snap-create")
    {
        // Create image/snapshot
        action_cb = start_create(cfg);
    }
    else if (cmd[0] == "modify")
    {
        // Modify image
        action_cb = start_modify(cfg);
    }
    else if (cmd[0] == "rm-data")
    {
        // Delete inode data
@@ -201,6 +273,8 @@ void cli_tool_t::run(json11::Json cfg)
        fprintf(stderr, "unknown command: %s\n", cmd[0].string_value().c_str());
        exit(1);
    }
    color = !cfg["no-color"].bool_value();
    json_output = cfg["json"].bool_value();
    iodepth = cfg["iodepth"].uint64_value();
    if (!iodepth)
        iodepth = 32;
@@ -236,7 +310,8 @@ void cli_tool_t::run(json11::Json cfg)
    while (action_cb != NULL)
    {
        ringloop->loop();
-        ringloop->wait();
+        if (action_cb != NULL)
            ringloop->wait();
    }
 }
@@ -25,8 +25,9 @@ public:
    uint64_t iodepth = 0, parallel_osds = 0;
    bool progress = true;
    bool list_first = false;
    bool json_output = false;
    int log_level = 0;
-    int mode = 0;
+    bool color = false;
    ring_loop_t *ringloop = NULL;
    epoll_manager_t *epmgr = NULL;
@@ -49,6 +50,9 @@ public:
    friend struct snap_flattener_t;
    friend struct snap_remover_t;
    std::function<bool(void)> start_ls(json11::Json cfg);
    std::function<bool(void)> start_create(json11::Json cfg);
    std::function<bool(void)> start_modify(json11::Json cfg);
    std::function<bool(void)> start_rm(json11::Json);
    std::function<bool(void)> start_merge(json11::Json);
    std::function<bool(void)> start_flatten(json11::Json);
@@ -0,0 +1,514 @@
 // Copyright (c) Vitaliy Filippov, 2019+
 // License: VNPL-1.1 (see README.md for details)
 #include <ctype.h>
 #include "cli.h"
 #include "cluster_client.h"
 #include "base64.h"
 // Create an image, snapshot or clone
 //
 // Snapshot creation performs a etcd transaction which:
 // - Checks that the image exists
 // - Checks that the snapshot doesn't exist
 // - Renames the inode to a new name with snapshot (say, testimg -> testimg@0)
 // - Sets the readonly flag for the old inode
 // - Creates a new inode with the same name pointing to the old inode as parent
 // - Adjusts /index/image/*
 //
 // The same algorithm can be easily implemented in any other language or even via etcdctl,
 // however we have it here for completeness
 struct image_creator_t
 {
    cli_tool_t *parent;
    pool_id_t new_pool_id = 0;
    std::string new_pool_name;
    std::string image_name, new_snap, new_parent;
    uint64_t size;
    pool_id_t old_pool_id = 0;
    inode_t new_parent_id = 0;
    inode_t new_id = 0, old_id = 0;
    uint64_t max_id_mod_rev = 0, cfg_mod_rev = 0, idx_mod_rev = 0;
    json11::Json result;
    int state = 0;
    bool is_done()
    {
        return state == 100;
    }
    void loop()
    {
        if (state >= 1)
            goto resume_1;
        if (new_pool_id)
        {
            auto & pools = parent->cli->st_cli.pool_config;
            if (pools.find(new_pool_id) == pools.end())
            {
                new_pool_id = 0;
            }
        }
        else if (new_pool_name != "")
        {
            for (auto & ic: parent->cli->st_cli.pool_config)
            {
                if (ic.second.name == new_pool_name)
                {
                    new_pool_id = ic.first;
                    break;
                }
            }
        }
        else if (parent->cli->st_cli.pool_config.size() == 1)
        {
            auto it = parent->cli->st_cli.pool_config.begin();
            new_pool_id = it->first;
        }
        if (!new_pool_id)
        {
            if (new_pool_name == "")
            {
                fprintf(stderr, "Pool name or ID is missing\n");
            }
            else
            {
                fprintf(stderr, "Pool %s does not exist\n", new_pool_name.c_str());
            }
            exit(1);
        }
        state = 1;
    resume_1:
        if (new_snap == "")
            create_image();
        else
            create_snapshot();
    }
    void create_image()
    {
        if (state == 2)
            goto resume_2;
        else if (state == 3)
            goto resume_3;
        for (auto & ic: parent->cli->st_cli.inode_config)
        {
            if (ic.second.name == image_name)
            {
                fprintf(stderr, "Image %s already exists\n", image_name.c_str());
                exit(1);
            }
        }
        do
        {
            etcd_txn(json11::Json::object {
                { "success", json11::Json::array { get_next_id() } }
            });
            state = 2;
 resume_2:
            if (parent->waiting > 0)
                return;
            extract_next_id(result["responses"][0]);
            attempt_create();
            state = 3;
 resume_3:
            if (parent->waiting > 0)
                return;
            if (!result["succeeded"].bool_value() &&
                result["responses"][0]["response_range"]["kvs"].array_items().size() > 0)
            {
                fprintf(stderr, "Image %s already exists\n", image_name.c_str());
                exit(1);
            }
        } while (!result["succeeded"].bool_value());
        if (parent->progress)
        {
            printf("Image %s created\n", image_name.c_str());
        }
        state = 100;
    }
    void create_snapshot()
    {
        if (state == 2)
            goto resume_2;
        else if (state == 3)
            goto resume_3;
        else if (state == 4)
            goto resume_4;
        for (auto & ic: parent->cli->st_cli.inode_config)
        {
            if (ic.second.name == image_name+"@"+new_snap)
            {
                fprintf(stderr, "Snapshot %s@%s already exists\n", image_name.c_str(), new_snap.c_str());
                exit(1);
            }
        }
        do
        {
            // In addition to next_id, get: size, old_id, old_pool_id, new_parent, cfg_mod_rev, idx_mod_rev
 resume_2:
 resume_3:
            get_image_details();
            if (parent->waiting > 0)
                return;
            if (!old_id)
            {
                fprintf(stderr, "Image %s does not exist\n", image_name.c_str());
                exit(1);
            }
            attempt_create();
            state = 4;
 resume_4:
            if (parent->waiting > 0)
                return;
            if (!result["succeeded"].bool_value() &&
                result["responses"][0]["response_range"]["kvs"].array_items().size() > 0)
            {
                fprintf(stderr, "Snapshot %s@%s already exists\n", image_name.c_str(), new_snap.c_str());
                exit(1);
            }
        } while (!result["succeeded"].bool_value());
        if (parent->progress)
        {
            printf("Snapshot %s@%s created\n", image_name.c_str(), new_snap.c_str());
        }
        state = 100;
    }
    json11::Json::object get_next_id()
    {
        return json11::Json::object {
            { "request_range", json11::Json::object {
                { "key", base64_encode(
                    parent->cli->st_cli.etcd_prefix+"/index/maxid/"+std::to_string(new_pool_id)
                ) },
            } },
        };
    }
    void extract_next_id(json11::Json response)
    {
        new_id = 1;
        max_id_mod_rev = 0;
        if (response["response_range"]["kvs"].array_items().size() > 0)
        {
            auto kv = parent->cli->st_cli.parse_etcd_kv(response["response_range"]["kvs"][0]);
            new_id = 1+INODE_NO_POOL(kv.value.uint64_value());
            max_id_mod_rev = kv.mod_revision;
        }
        auto ino_it = parent->cli->st_cli.inode_config.lower_bound(INODE_WITH_POOL(new_pool_id, 0));
        if (ino_it != parent->cli->st_cli.inode_config.begin())
        {
            ino_it--;
            if (INODE_POOL(ino_it->first) == new_pool_id && new_id < 1+INODE_NO_POOL(ino_it->first))
                new_id = 1+INODE_NO_POOL(ino_it->first);
        }
    }
    void get_image_details()
    {
        if (state == 2)
            goto resume_2;
        else if (state == 3)
            goto resume_3;
        etcd_txn(json11::Json::object { { "success", json11::Json::array {
            get_next_id(),
            json11::Json::object {
                { "request_range", json11::Json::object {
                    { "key", base64_encode(
                        parent->cli->st_cli.etcd_prefix+"/index/image/"+image_name
                    ) },
                } },
            },
        } } });
        state = 2;
 resume_2:
        if (parent->waiting > 0)
            return;
        extract_next_id(result["responses"][0]);
        old_id = 0;
        old_pool_id = 0;
        cfg_mod_rev = idx_mod_rev = 0;
        if (result["responses"][1]["response_range"]["kvs"].array_items().size() == 0)
        {
            for (auto & ic: parent->cli->st_cli.inode_config)
            {
                if (ic.second.name == image_name)
                {
                    old_id = INODE_NO_POOL(ic.first);
                    old_pool_id = INODE_POOL(ic.first);
                    size = ic.second.size;
                    new_parent_id = ic.second.parent_id;
                    cfg_mod_rev = ic.second.mod_revision;
                    break;
                }
            }
        }
        else
        {
            // FIXME: Parse kvs in etcd_state_client automatically
            {
                auto kv = parent->cli->st_cli.parse_etcd_kv(result["responses"][1]["response_range"]["kvs"][0]);
                old_id = INODE_NO_POOL(kv.value["id"].uint64_value());
                old_pool_id = (pool_id_t)kv.value["pool_id"].uint64_value();
                idx_mod_rev = kv.mod_revision;
                if (!old_id || !old_pool_id || old_pool_id >= POOL_ID_MAX)
                {
                    fprintf(stderr, "Invalid pool or inode ID in etcd key %s\n", kv.key.c_str());
                    exit(1);
                }
            }
            etcd_txn(json11::Json::object {
                { "success", json11::Json::array {
                    json11::Json::object {
                        { "request_range", json11::Json::object {
                            { "key", base64_encode(
                                parent->cli->st_cli.etcd_prefix+"/config/inode/"+
                                std::to_string(old_pool_id)+"/"+std::to_string(old_id)
                            ) },
                        } },
                    },
                } },
            });
            state = 3;
 resume_3:
            if (parent->waiting > 0)
                return;
            {
                auto kv = parent->cli->st_cli.parse_etcd_kv(result["responses"][0]["response_range"]["kvs"][0]);
                size = kv.value["size"].uint64_value();
                new_parent_id = kv.value["parent_id"].uint64_value();
                uint64_t parent_pool_id = kv.value["parent_pool_id"].uint64_value();
                if (new_parent_id)
                {
                    new_parent_id = INODE_WITH_POOL(parent_pool_id ? parent_pool_id : old_pool_id, new_parent_id);
                }
                cfg_mod_rev = kv.mod_revision;
            }
        }
    }
    void attempt_create()
    {
        inode_config_t new_cfg = {
            .num = INODE_WITH_POOL(new_pool_id, new_id),
            .name = image_name,
            .size = size,
            .parent_id = (new_snap != "" ? INODE_WITH_POOL(old_pool_id, old_id) : new_parent_id),
            .readonly = false,
        };
        json11::Json::array checks = json11::Json::array {
            json11::Json::object {
                { "target", "VERSION" },
                { "version", 0 },
                { "key", base64_encode(
                    parent->cli->st_cli.etcd_prefix+"/config/inode/"+
                    std::to_string(new_pool_id)+"/"+std::to_string(new_id)
                ) },
            },
            json11::Json::object {
                { "target", "VERSION" },
                { "version", 0 },
                { "key", base64_encode(
                    parent->cli->st_cli.etcd_prefix+"/index/image/"+image_name+
                    (new_snap != "" ? "@"+new_snap : "")
                ) },
            },
            json11::Json::object {
                { "target", "MOD" },
                { "mod_revision", max_id_mod_rev },
                { "key", base64_encode(parent->cli->st_cli.etcd_prefix+"/index/maxid/"+std::to_string(new_pool_id)) },
            },
        };
        json11::Json::array success = json11::Json::array {
            json11::Json::object {
                { "request_put", json11::Json::object {
                    { "key", base64_encode(
                        parent->cli->st_cli.etcd_prefix+"/config/inode/"+
                        std::to_string(new_pool_id)+"/"+std::to_string(new_id)
                    ) },
                    { "value", base64_encode(
                        json11::Json(parent->cli->st_cli.serialize_inode_cfg(&new_cfg)).dump()
                    ) },
                } },
            },
            json11::Json::object {
                { "request_put", json11::Json::object {
                    { "key", base64_encode(parent->cli->st_cli.etcd_prefix+"/index/image/"+image_name) },
                    { "value", base64_encode(json11::Json(json11::Json::object{
                        { "id", new_id },
                        { "pool_id", (uint64_t)new_pool_id },
                    }).dump()) },
                } },
            },
            json11::Json::object {
                { "request_put", json11::Json::object {
                    { "key", base64_encode(
                        parent->cli->st_cli.etcd_prefix+"/index/maxid/"+
                        std::to_string(new_pool_id)
                    ) },
                    { "value", base64_encode(std::to_string(new_id)) }
                } },
            },
        };
        json11::Json::array failure = json11::Json::array {
            json11::Json::object {
                { "request_range", json11::Json::object {
                    { "key", base64_encode(
                        parent->cli->st_cli.etcd_prefix+"/index/image/"+
                        image_name+(new_snap != "" ? "@"+new_snap : "")
                    ) },
                } },
            },
        };
        if (new_snap != "")
        {
            inode_config_t snap_cfg = {
                .num = INODE_WITH_POOL(old_pool_id, old_id),
                .name = image_name+"@"+new_snap,
                .size = size,
                .parent_id = new_parent_id,
                .readonly = true,
            };
            checks.push_back(json11::Json::object {
                { "target", "MOD" },
                { "mod_revision", cfg_mod_rev },
                { "key", base64_encode(
                    parent->cli->st_cli.etcd_prefix+"/config/inode/"+
                    std::to_string(old_pool_id)+"/"+std::to_string(old_id)
                ) },
            });
            checks.push_back(json11::Json::object {
                { "target", "MOD" },
                { "mod_revision", idx_mod_rev },
                { "key", base64_encode(parent->cli->st_cli.etcd_prefix+"/index/image/"+image_name) }
            });
            success.push_back(json11::Json::object {
                { "request_put", json11::Json::object {
                    { "key", base64_encode(
                        parent->cli->st_cli.etcd_prefix+"/config/inode/"+
                        std::to_string(old_pool_id)+"/"+std::to_string(old_id)
                    ) },
                    { "value", base64_encode(
                        json11::Json(parent->cli->st_cli.serialize_inode_cfg(&snap_cfg)).dump()
                    ) },
                } },
            });
            success.push_back(json11::Json::object {
                { "request_put", json11::Json::object {
                    { "key", base64_encode(parent->cli->st_cli.etcd_prefix+"/index/image/"+image_name+"@"+new_snap) },
                    { "value", base64_encode(json11::Json(json11::Json::object{
                        { "id", old_id },
                        { "pool_id", (uint64_t)old_pool_id },
                    }).dump()) },
                } },
            });
        };
        etcd_txn(json11::Json::object {
            { "compare", checks },
            { "success", success },
            { "failure", failure },
        });
    }
    void etcd_txn(json11::Json txn)
    {
        parent->waiting++;
        parent->cli->st_cli.etcd_txn(txn, ETCD_SLOW_TIMEOUT, [this](std::string err, json11::Json res)
        {
            parent->waiting--;
            if (err != "")
            {
                fprintf(stderr, "Error reading from etcd: %s\n", err.c_str());
                exit(1);
            }
            this->result = res;
            parent->ringloop->wakeup();
        });
    }
 };
 std::function<bool(void)> cli_tool_t::start_create(json11::Json cfg)
 {
    json11::Json::array cmd = cfg["command"].array_items();
    auto image_creator = new image_creator_t();
    image_creator->parent = this;
    image_creator->image_name = cmd.size() > 1 ? cmd[1].string_value() : "";
    image_creator->new_pool_id = cfg["pool"].uint64_value();
    image_creator->new_pool_name = cfg["pool"].string_value();
    if (cfg["snapshot"].string_value() != "")
    {
        image_creator->new_snap = cfg["snapshot"].string_value();
    }
    else if (cmd[0] == "snap-create")
    {
        int p = image_creator->image_name.find('@');
        if (p == std::string::npos || p == image_creator->image_name.length()-1)
        {
            fprintf(stderr, "Please specify new snapshot name after @\n");
            exit(1);
        }
        image_creator->new_snap = image_creator->image_name.substr(p + 1);
        image_creator->image_name = image_creator->image_name.substr(0, p);
    }
    image_creator->new_parent = cfg["parent"].string_value();
    if (cfg["size"].string_value() != "")
    {
        std::string size_str = cfg["size"].string_value();
        uint64_t mul = 1;
        char type_char = tolower(size_str[size_str.length()-1]);
        if (type_char == 'k' || type_char == 'm' || type_char == 'g' || type_char == 't')
        {
            if (type_char == 'k')
                mul = 1l<<10;
            else if (type_char == 'm')
                mul = 1l<<20;
            else if (type_char == 'g')
                mul = 1l<<30;
            else /*if (type_char == 't')*/
                mul = 1l<<40;
            size_str = size_str.substr(0, size_str.length()-1);
        }
        uint64_t size = json11::Json(size_str).uint64_value() * mul;
        if (size == 0)
        {
            fprintf(stderr, "Invalid syntax for size: %s\n", cfg["size"].string_value().c_str());
            exit(1);
        }
        if (size % 4096)
        {
            fprintf(stderr, "Image size should be a multiple of 4096\n");
            exit(1);
        }
        image_creator->size = size;
        if (image_creator->new_snap != "")
        {
            fprintf(stderr, "--size can't be specified for snapshots\n");
            exit(1);
        }
    }
    if (image_creator->image_name == "")
    {
        fprintf(stderr, "Image name is missing\n");
        exit(1);
    }
    if (image_creator->image_name.find('@') != std::string::npos)
    {
        fprintf(stderr, "Image name can't contain @ character\n");
        exit(1);
    }
    return [image_creator]()
    {
        image_creator->loop();
        if (image_creator->is_done())
        {
            delete image_creator;
            return true;
        }
        return false;
    };
 }
@@ -0,0 +1,522 @@
 // Copyright (c) Vitaliy Filippov, 2019+
 // License: VNPL-1.1 (see README.md for details)
 #include <algorithm>
 #include "cli.h"
 #include "cluster_client.h"
 #include "base64.h"
 #define MIN(a, b) ((a) < (b) ? (b) : (a))
 std::string print_table(json11::Json items, json11::Json header, bool use_esc);
 std::string format_size(uint64_t size);
 std::string format_lat(uint64_t lat);
 std::string format_q(double depth);
 // List existing images
 //
 // Again, you can just look into etcd, but this console tool incapsulates it
 struct image_lister_t
 {
    cli_tool_t *parent;
    pool_id_t list_pool_id = 0;
    std::string list_pool_name;
    std::string sort_field;
    std::set<std::string> only_names;
    bool reverse = false;
    int max_count = 0;
    bool show_stats = false, show_delete = false;
    int state = 0;
    std::map<inode_t, json11::Json::object> stats;
    json11::Json space_info;
    bool is_done()
    {
        return state == 100;
    }
    void get_list()
    {
        if (list_pool_name != "")
        {
            for (auto & ic: parent->cli->st_cli.pool_config)
            {
                if (ic.second.name == list_pool_name)
                {
                    list_pool_id = ic.first;
                    break;
                }
            }
            if (!list_pool_id)
            {
                fprintf(stderr, "Pool %s does not exist\n", list_pool_name.c_str());
                exit(1);
            }
        }
        for (auto & ic: parent->cli->st_cli.inode_config)
        {
            if (list_pool_id && INODE_POOL(ic.second.num) != list_pool_id)
            {
                continue;
            }
            auto & pool_cfg = parent->cli->st_cli.pool_config.at(INODE_POOL(ic.second.num));
            auto item = json11::Json::object {
                { "name", ic.second.name },
                { "size", ic.second.size },
                { "used_size", 0 },
                { "readonly", ic.second.readonly },
                { "pool_id", (uint64_t)INODE_POOL(ic.second.num) },
                { "pool_name", pool_cfg.name },
                { "inode_num", INODE_NO_POOL(ic.second.num) },
                { "inode_id", ic.second.num },
            };
            if (ic.second.parent_id)
            {
                auto p_it = parent->cli->st_cli.inode_config.find(ic.second.parent_id);
                item["parent_name"] = p_it != parent->cli->st_cli.inode_config.end()
                    ? p_it->second.name : "";
                item["parent_pool_id"] = (uint64_t)INODE_POOL(ic.second.parent_id);
                item["parent_inode_num"] = INODE_NO_POOL(ic.second.parent_id);
            }
            stats[ic.second.num] = item;
        }
    }
    void get_stats()
    {
        if (state == 1)
            goto resume_1;
        // Space statistics
        // inode/stats/<pool>/<inode>::raw_used divided by pool/stats/<pool>::pg_real_size
        // multiplied by 1 or number of data drives
        parent->waiting++;
        parent->cli->st_cli.etcd_txn(json11::Json::object {
            { "success", json11::Json::array {
                json11::Json::object {
                    { "request_range", json11::Json::object {
                        { "key", base64_encode(
                            parent->cli->st_cli.etcd_prefix+"/pool/stats"+
                            (list_pool_id ? "/"+std::to_string(list_pool_id) : "")+"/"
                        ) },
                        { "range_end", base64_encode(
                            parent->cli->st_cli.etcd_prefix+"/pool/stats"+
                            (list_pool_id ? "/"+std::to_string(list_pool_id) : "")+"0"
                        ) },
                    } },
                },
                json11::Json::object {
                    { "request_range", json11::Json::object {
                        { "key", base64_encode(
                            parent->cli->st_cli.etcd_prefix+"/inode/stats"+
                            (list_pool_id ? "/"+std::to_string(list_pool_id) : "")+"/"
                        ) },
                        { "range_end", base64_encode(
                            parent->cli->st_cli.etcd_prefix+"/inode/stats"+
                            (list_pool_id ? "/"+std::to_string(list_pool_id) : "")+"0"
                        ) },
                    } },
                },
            } },
        }, ETCD_SLOW_TIMEOUT, [this](std::string err, json11::Json res)
        {
            parent->waiting--;
            if (err != "")
            {
                fprintf(stderr, "Error reading from etcd: %s\n", err.c_str());
                exit(1);
            }
            space_info = res;
            parent->ringloop->wakeup();
        });
        state = 1;
 resume_1:
        if (parent->waiting > 0)
            return;
        std::map<pool_id_t, uint64_t> pool_pg_real_size;
        for (auto & kv_item: space_info["responses"][0]["response_range"]["kvs"].array_items())
        {
            auto kv = parent->cli->st_cli.parse_etcd_kv(kv_item);
            // pool ID
            pool_id_t pool_id;
            char null_byte = 0;
            sscanf(kv.key.substr(parent->cli->st_cli.etcd_prefix.length()).c_str(), "/pool/stats/%u%c", &pool_id, &null_byte);
            if (!pool_id || pool_id >= POOL_ID_MAX || null_byte != 0)
            {
                fprintf(stderr, "Invalid key in etcd: %s\n", kv.key.c_str());
                continue;
            }
            // pg_real_size
            pool_pg_real_size[pool_id] = kv.value["pg_real_size"].uint64_value();
        }
        for (auto & kv_item: space_info["responses"][1]["response_range"]["kvs"].array_items())
        {
            auto kv = parent->cli->st_cli.parse_etcd_kv(kv_item);
            // pool ID & inode number
            pool_id_t pool_id;
            inode_t only_inode_num;
            char null_byte = 0;
            sscanf(kv.key.substr(parent->cli->st_cli.etcd_prefix.length()).c_str(),
                "/inode/stats/%u/%lu%c", &pool_id, &only_inode_num, &null_byte);
            if (!pool_id || pool_id >= POOL_ID_MAX || INODE_POOL(only_inode_num) != 0 || null_byte != 0)
            {
                fprintf(stderr, "Invalid key in etcd: %s\n", kv.key.c_str());
                continue;
            }
            inode_t inode_num = INODE_WITH_POOL(pool_id, only_inode_num);
            uint64_t used_size = kv.value["raw_used"].uint64_value();
            // save stats
            auto pool_it = parent->cli->st_cli.pool_config.find(pool_id);
            if (pool_it != parent->cli->st_cli.pool_config.end())
            {
                auto & pool_cfg = pool_it->second;
                used_size = used_size / pool_pg_real_size[pool_id]
                    * (pool_cfg.scheme == POOL_SCHEME_REPLICATED ? 1 : pool_cfg.pg_size-pool_cfg.parity_chunks);
            }
            auto stat_it = stats.find(inode_num);
            if (stat_it == stats.end())
            {
                stats[inode_num] = json11::Json::object {
                    { "name", "Pool:"+std::to_string(pool_id)+",ID:"+std::to_string(only_inode_num) },
                    { "size", 0 },
                    { "readonly", false },
                    { "pool_id", (uint64_t)INODE_POOL(inode_num) },
                    { "pool_name", pool_it == parent->cli->st_cli.pool_config.end()
                        ? (pool_it->second.name == "" ? "<Unnamed>" : pool_it->second.name) : "?" },
                    { "inode_num", INODE_NO_POOL(inode_num) },
                    { "inode_id", inode_num },
                };
                stat_it = stats.find(inode_num);
            }
            stat_it->second["used_size"] = used_size;
            stat_it->second["read_iops"] = kv.value["read"]["iops"];
            stat_it->second["read_bps"] = kv.value["read"]["bps"];
            stat_it->second["read_lat"] = kv.value["read"]["lat"];
            stat_it->second["read_queue"] = kv.value["read"]["iops"].number_value() * kv.value["read"]["lat"].number_value() / 1000000;
            stat_it->second["write_iops"] = kv.value["write"]["iops"];
            stat_it->second["write_bps"] = kv.value["write"]["bps"];
            stat_it->second["write_lat"] = kv.value["write"]["lat"];
            stat_it->second["write_queue"] = kv.value["write"]["iops"].number_value() * kv.value["write"]["lat"].number_value() / 1000000;
            stat_it->second["delete_iops"] = kv.value["delete"]["iops"];
            stat_it->second["delete_bps"] = kv.value["delete"]["bps"];
            stat_it->second["delete_lat"] = kv.value["delete"]["lat"];
            stat_it->second["delete_queue"] = kv.value["delete"]["iops"].number_value() * kv.value["delete"]["lat"].number_value() / 1000000;
        }
    }
    json11::Json::array to_list()
    {
        json11::Json::array list;
        for (auto & kv: stats)
        {
            if (!only_names.size() || only_names.find(kv.second["name"].string_value()) != only_names.end())
            {
                list.push_back(kv.second);
            }
        }
        if (sort_field == "name" || sort_field == "pool_name")
        {
            std::sort(list.begin(), list.end(), [this](json11::Json a, json11::Json b)
            {
                auto av = a[sort_field].as_string();
                auto bv = b[sort_field].as_string();
                return reverse ? av > bv : av < bv;
            });
        }
        else
        {
            std::sort(list.begin(), list.end(), [this](json11::Json a, json11::Json b)
            {
                auto av = a[sort_field].number_value();
                auto bv = b[sort_field].number_value();
                return reverse ? av > bv : av < bv;
            });
        }
        if (max_count > 0 && list.size() > max_count)
        {
            list.resize(max_count);
        }
        return list;
    }
    void loop()
    {
        if (state == 1)
            goto resume_1;
        get_list();
        if (show_stats)
        {
 resume_1:
            get_stats();
            if (parent->waiting > 0)
                return;
        }
        if (parent->json_output)
        {
            // JSON output
            printf("%s\n", json11::Json(to_list()).dump().c_str());
            state = 100;
            return;
        }
        // Table output: name, size_fmt, [used_size_fmt], ro, parent_name
        json11::Json::array cols;
        cols.push_back(json11::Json::object{
            { "key", "name" },
            { "title", "NAME" },
        });
        if (!list_pool_id)
        {
            cols.push_back(json11::Json::object{
                { "key", "pool_name" },
                { "title", "POOL" },
            });
        }
        cols.push_back(json11::Json::object{
            { "key", "size_fmt" },
            { "title", "SIZE" },
        });
        if (show_stats)
        {
            cols.push_back(json11::Json::object{
                { "key", "used_size_fmt" },
                { "title", "USED" },
            });
            cols.push_back(json11::Json::object{
                { "key", "read_bw" },
                { "title", "READ" },
            });
            cols.push_back(json11::Json::object{
                { "key", "read_iops" },
                { "title", "IOPS" },
            });
            cols.push_back(json11::Json::object{
                { "key", "read_q" },
                { "title", "QUEUE" },
            });
            cols.push_back(json11::Json::object{
                { "key", "read_lat_f" },
                { "title", "LAT" },
            });
            cols.push_back(json11::Json::object{
                { "key", "write_bw" },
                { "title", "WRITE" },
            });
            cols.push_back(json11::Json::object{
                { "key", "write_iops" },
                { "title", "IOPS" },
            });
            cols.push_back(json11::Json::object{
                { "key", "write_q" },
                { "title", "QUEUE" },
            });
            cols.push_back(json11::Json::object{
                { "key", "write_lat_f" },
                { "title", "LAT" },
            });
            if (show_delete)
            {
                cols.push_back(json11::Json::object{
                    { "key", "delete_bw" },
                    { "title", "DEL" },
                });
                cols.push_back(json11::Json::object{
                    { "key", "delete_iops" },
                    { "title", "IOPS" },
                });
                cols.push_back(json11::Json::object{
                    { "key", "delete_q" },
                    { "title", "QUEUE" },
                });
                cols.push_back(json11::Json::object{
                    { "key", "delete_lat_f" },
                    { "title", "LAT" },
                });
            }
        }
        cols.push_back(json11::Json::object{
            { "key", "ro" },
            { "title", "FLAGS" },
            { "right", true },
        });
        cols.push_back(json11::Json::object{
            { "key", "parent_name" },
            { "title", "PARENT" },
        });
        json11::Json::array list;
        for (auto & kv: stats)
        {
            if (show_stats)
            {
                kv.second["used_size_fmt"] = format_size(kv.second["used_size"].uint64_value());
                kv.second["read_bw"] = format_size(kv.second["read_bps"].uint64_value())+"/s";
                kv.second["write_bw"] = format_size(kv.second["write_bps"].uint64_value())+"/s";
                kv.second["delete_bw"] = format_size(kv.second["delete_bps"].uint64_value())+"/s";
                kv.second["read_lat_f"] = format_lat(kv.second["read_lat"].uint64_value());
                kv.second["write_lat_f"] = format_lat(kv.second["write_lat"].uint64_value());
                kv.second["delete_lat_f"] = format_lat(kv.second["delete_lat"].uint64_value());
                kv.second["read_q"] = format_q(kv.second["read_queue"].number_value());
                kv.second["write_q"] = format_q(kv.second["write_queue"].number_value());
                kv.second["delete_q"] = format_q(kv.second["delete_queue"].number_value());
            }
            kv.second["size_fmt"] = format_size(kv.second["size"].uint64_value());
            kv.second["ro"] = kv.second["readonly"].bool_value() ? "RO" : "-";
        }
        printf("%s", print_table(to_list(), cols, parent->color).c_str());
        state = 100;
    }
 };
 std::string print_table(json11::Json items, json11::Json header, bool use_esc)
 {
    std::vector<int> sizes;
    for (int i = 0; i < header.array_items().size(); i++)
    {
        sizes.push_back(header[i]["title"].string_value().length());
    }
    for (auto & item: items.array_items())
    {
        for (int i = 0; i < header.array_items().size(); i++)
        {
            int l = item[header[i]["key"].string_value()].as_string().length();
            sizes[i] = sizes[i] < l ? l : sizes[i];
        }
    }
    std::string str = use_esc ? "\033[1m" : "";
    for (int i = 0; i < header.array_items().size(); i++)
    {
        if (i > 0)
        {
            // Separator
            str += "  ";
        }
        int pad = sizes[i]-header[i]["title"].string_value().length();
        if (header[i]["right"].bool_value())
        {
            // Align right
            for (int j = 0; j < pad; j++)
                str += ' ';
            str += header[i]["title"].string_value();
        }
        else
        {
            // Align left
            str += header[i]["title"].string_value();
            for (int j = 0; j < pad; j++)
                str += ' ';
        }
    }
    if (use_esc)
        str += "\033[0m";
    str += "\n";
    for (auto & item: items.array_items())
    {
        for (int i = 0; i < header.array_items().size(); i++)
        {
            if (i > 0)
            {
                // Separator
                str += "  ";
            }
            int pad = sizes[i] - item[header[i]["key"].string_value()].as_string().length();
            if (header[i]["right"].bool_value())
            {
                // Align right
                for (int j = 0; j < pad; j++)
                    str += ' ';
                str += item[header[i]["key"].string_value()].as_string();
            }
            else
            {
                // Align left
                str += item[header[i]["key"].string_value()].as_string();
                for (int j = 0; j < pad; j++)
                    str += ' ';
            }
        }
        str += "\n";
    }
    return str;
 }
 static uint64_t size_thresh[] = { 1024l*1024*1024*1024, 1024l*1024*1024, 1024l*1024, 1024, 0 };
 static const char *size_unit = "TGMKB";
 std::string format_size(uint64_t size)
 {
    char buf[256];
    for (int i = 0; i < sizeof(size_thresh)/sizeof(size_thresh[0]); i++)
    {
        if (size >= size_thresh[i] || i >= sizeof(size_thresh)/sizeof(size_thresh[0])-1)
        {
            double value = size_thresh[i] ? (double)size/size_thresh[i] : size;
            int l = snprintf(buf, sizeof(buf), "%.1f", value);
            assert(l < sizeof(buf)-2);
            if (buf[l-1] == '0')
                l -= 2;
            buf[l] = ' ';
            buf[l+1] = size_unit[i];
            buf[l+2] = 0;
            break;
        }
    }
    return std::string(buf);
 }
 std::string format_lat(uint64_t lat)
 {
    char buf[256];
    int l = 0;
    if (lat < 100)
        l = snprintf(buf, sizeof(buf), "%lu us", lat);
    else if (lat < 500000)
        l = snprintf(buf, sizeof(buf), "%.2f ms", (double)lat/1000);
    else
        l = snprintf(buf, sizeof(buf), "%.2f s", (double)lat/1000000);
    assert(l < sizeof(buf));
    return std::string(buf);
 }
 std::string format_q(double depth)
 {
    char buf[256];
    int l = snprintf(buf, sizeof(buf), "%.2f", depth);
    assert(l < sizeof(buf));
    if (buf[l-1] == '0')
        l--;
    if (buf[l-1] == '0')
        l -= 2;
    buf[l] = 0;
    return std::string(buf);
 }
 std::function<bool(void)> cli_tool_t::start_ls(json11::Json cfg)
 {
    json11::Json::array cmd = cfg["command"].array_items();
    auto lister = new image_lister_t();
    lister->parent = this;
    lister->list_pool_id = cfg["pool"].uint64_value();
    lister->list_pool_name = lister->list_pool_id ? "" : cfg["pool"].as_string();
    lister->show_stats = cfg["long"].bool_value();
    lister->show_delete = cfg["del"].bool_value();
    lister->sort_field = cfg["sort"].string_value();
    lister->reverse = cfg["reverse"].bool_value();
    lister->max_count = cfg["count"].uint64_value();
    for (int i = 0; i < cmd.size(); i++)
    {
        lister->only_names.insert(cmd[i].string_value());
    }
    return [lister]()
    {
        lister->loop();
        if (lister->is_done())
        {
            delete lister;
            return true;
        }
        return false;
    };
 }
@@ -0,0 +1,182 @@
 // Copyright (c) Vitaliy Filippov, 2019+
 // License: VNPL-1.1 (see README.md for details)
 #include "cli.h"
 #include "cluster_client.h"
 #include "base64.h"
 // Resize image (purging extra data on shrink) or change its readonly status
 struct image_changer_t
 {
    cli_tool_t *parent;
    std::string image_name;
    uint64_t new_size = 0;
    bool set_readonly = false, set_readwrite = false, force = false;
    // interval between fsyncs
    int fsync_interval = 128;
    uint64_t inode_num = 0;
    inode_config_t cfg;
    std::string cur_cfg_key;
    bool has_children = false;
    int state = 0;
    std::function<bool(void)> cb;
    bool is_done()
    {
        return state == 100;
    }
    void loop()
    {
        if (state == 1)
            goto resume_1;
        else if (state == 2)
            goto resume_2;
        for (auto & ic: parent->cli->st_cli.inode_config)
        {
            if (ic.second.name == image_name)
            {
                inode_num = ic.first;
                cfg = ic.second;
                break;
            }
        }
        if (!inode_num)
        {
            fprintf(stderr, "Image %s does not exist\n", image_name.c_str());
            exit(1);
        }
        for (auto & ic: parent->cli->st_cli.inode_config)
        {
            if (ic.second.parent_id == inode_num)
            {
                has_children = true;
                break;
            }
        }
        if (new_size != 0)
        {
            if (cfg.size >= new_size)
            {
                // Check confirmation if trimming an image with children
                if (has_children && !force)
                {
                    fprintf(stderr, "Image %s has children. Refusing to shrink it without --force", image_name.c_str());
                    exit(1);
                }
                // Shrink the image first
                cb = parent->start_rm(json11::Json::object {
                    { "inode", INODE_NO_POOL(inode_num) },
                    { "pool", (uint64_t)INODE_POOL(inode_num) },
                    { "fsync-interval", fsync_interval },
                    { "min-offset", new_size },
                });
 resume_1:
                while (!cb())
                {
                    state = 1;
                    return;
                }
                cb = NULL;
            }
            cfg.size = new_size;
        }
        if (set_readonly)
        {
            cfg.readonly = true;
        }
        if (set_readwrite)
        {
            cfg.readonly = false;
            // Check confirmation if trimming an image with children
            if (!force)
            {
                fprintf(stderr, "Image %s has children. Refusing to make it read-write without --force", image_name.c_str());
                exit(1);
            }
        }
        cur_cfg_key = base64_encode(parent->cli->st_cli.etcd_prefix+
            "/config/inode/"+std::to_string(INODE_POOL(inode_num))+
            "/"+std::to_string(INODE_NO_POOL(inode_num)));
        parent->waiting++;
        parent->cli->st_cli.etcd_txn(json11::Json::object {
            { "compare", json11::Json::array {
                json11::Json::object {
                    { "target", "MOD" },
                    { "key", cur_cfg_key },
                    { "result", "LESS" },
                    { "mod_revision", cfg.mod_revision+1 },
                },
            } },
            { "success", json11::Json::array {
                json11::Json::object {
                    { "request_put", json11::Json::object {
                        { "key", cur_cfg_key },
                        { "value", base64_encode(json11::Json(
                            parent->cli->st_cli.serialize_inode_cfg(&cfg)
                        ).dump()) },
                    } }
                },
            } },
        }, ETCD_SLOW_TIMEOUT, [this](std::string err, json11::Json res)
        {
            if (err != "")
            {
                fprintf(stderr, "Error changing %s: %s\n", image_name.c_str(), err.c_str());
                exit(1);
            }
            if (!res["succeeded"].bool_value())
            {
                fprintf(stderr, "Image %s was modified by someone else, please repeat your request\n", image_name.c_str());
                exit(1);
            }
            parent->waiting--;
            parent->ringloop->wakeup();
        });
        state = 2;
 resume_2:
        if (parent->waiting > 0)
            return;
        printf("Image %s changed\n", image_name.c_str());
        state = 100;
    }
 };
 std::function<bool(void)> cli_tool_t::start_modify(json11::Json cfg)
 {
    json11::Json::array cmd = cfg["command"].array_items();
    auto changer = new image_changer_t();
    changer->parent = this;
    changer->image_name = cmd.size() > 1 ? cmd[1].string_value() : "";
    if (changer->image_name == "")
    {
        fprintf(stderr, "Image name is missing\n");
        exit(1);
    }
    changer->new_size = cfg["size"].uint64_value();
    if (changer->new_size != 0 && (changer->new_size % 4096))
    {
        fprintf(stderr, "Image size should be a multiple of 4096\n");
        exit(1);
    }
    changer->force = cfg["force"].bool_value();
    changer->set_readonly = cfg["readonly"].bool_value();
    changer->set_readwrite = cfg["readwrite"].bool_value();
    changer->fsync_interval = cfg["fsync-interval"].uint64_value();
    if (!changer->fsync_interval)
        changer->fsync_interval = 128;
    // FIXME Check that the image doesn't have children when shrinking
    return [changer]()
    {
        changer->loop();
        if (changer->is_done())
        {
            delete changer;
            return true;
        }
        return false;
    };
 }
@@ -14,7 +14,7 @@ struct rm_pg_t
    osd_num_t rm_osd_num;
    std::set<object_id> objects;
    std::set<object_id>::iterator obj_pos;
-    uint64_t obj_count = 0, obj_done = 0, obj_prev_done = 0;
+    uint64_t obj_count = 0, obj_done = 0;
    int state = 0;
    int in_flight = 0;
 };
@@ -23,6 +23,7 @@ struct rm_inode_t
 {
    uint64_t inode = 0;
    pool_id_t pool_id = 0;
    uint64_t min_offset = 0;
    cli_tool_t *parent = NULL;
    inode_list_t *lister = NULL;
@@ -43,8 +44,21 @@ struct rm_inode_t
                .objects = objects,
                .obj_count = objects.size(),
                .obj_done = 0,
                .obj_prev_done = 0,
            });
            if (min_offset == 0)
            {
                total_count += objects.size();
            }
            else
            {
                for (object_id oid: objects)
                {
                    if (oid.stripe >= min_offset)
                    {
                        total_count++;
                    }
                }
            }
            rm->obj_pos = rm->objects.begin();
            lists.push_back(rm);
            if (parent->list_first)
@@ -78,38 +92,41 @@ struct rm_inode_t
        }
        while (cur_list->in_flight < parent->iodepth && cur_list->obj_pos != cur_list->objects.end())
        {
-            osd_op_t *op = new osd_op_t();
+            if (cur_list->obj_pos->stripe >= min_offset)
            op->op_type = OSD_OP_OUT;
            op->peer_fd = parent->cli->msgr.osd_peer_fds[cur_list->rm_osd_num];
            op->req = (osd_any_op_t){
                .rw = {
                    .header = {
                        .magic = SECONDARY_OSD_OP_MAGIC,
                        .id = parent->cli->next_op_id(),
                        .opcode = OSD_OP_DELETE,
                    },
                    .inode = cur_list->obj_pos->inode,
                    .offset = cur_list->obj_pos->stripe,
                    .len = 0,
                },
            };
            op->callback = [this, cur_list](osd_op_t *op)
            {
-                cur_list->in_flight--;
+                osd_op_t *op = new osd_op_t();
-                if (op->reply.hdr.retval < 0)
+                op->op_type = OSD_OP_OUT;
                op->peer_fd = parent->cli->msgr.osd_peer_fds[cur_list->rm_osd_num];
                op->req = (osd_any_op_t){
                    .rw = {
                        .header = {
                            .magic = SECONDARY_OSD_OP_MAGIC,
                            .id = parent->cli->next_op_id(),
                            .opcode = OSD_OP_DELETE,
                        },
                        .inode = cur_list->obj_pos->inode,
                        .offset = cur_list->obj_pos->stripe,
                        .len = 0,
                    },
                };
                op->callback = [this, cur_list](osd_op_t *op)
                {
-                    fprintf(stderr, "Failed to remove object %lx:%lx from PG %u (OSD %lu) (retval=%ld)\n",
+                    cur_list->in_flight--;
-                        op->req.rw.inode, op->req.rw.offset,
+                    if (op->reply.hdr.retval < 0)
-                        cur_list->pg_num, cur_list->rm_osd_num, op->reply.hdr.retval);
+                    {
-                }
+                        fprintf(stderr, "Failed to remove object %lx:%lx from PG %u (OSD %lu) (retval=%ld)\n",
-                delete op;
+                            op->req.rw.inode, op->req.rw.offset,
-                cur_list->obj_done++;
+                            cur_list->pg_num, cur_list->rm_osd_num, op->reply.hdr.retval);
-                total_done++;
+                    }
-                continue_delete();
+                    delete op;
-            };
+                    cur_list->obj_done++;
                    total_done++;
                    continue_delete();
                };
                cur_list->in_flight++;
                parent->cli->msgr.outbox_push(op);
            }
            cur_list->obj_pos++;
            cur_list->in_flight++;
            parent->cli->msgr.outbox_push(op);
        }
    }
@@ -183,6 +200,7 @@ std::function<bool(void)> cli_tool_t::start_rm(json11::Json cfg)
        fprintf(stderr, "pool is missing\n");
        exit(1);
    }
    remover->min_offset = cfg["min-offset"].uint64_value();
    return [remover]()
    {
        if (remover->loop())
@@ -766,7 +766,7 @@ void etcd_state_client_t::close_watch(inode_watch_t* watch)
    delete watch;
 }
-json11::Json::object & etcd_state_client_t::serialize_inode_cfg(inode_config_t *cfg)
+json11::Json::object etcd_state_client_t::serialize_inode_cfg(inode_config_t *cfg)
 {
    json11::Json::object new_cfg = json11::Json::object {
        { "name", cfg->name },
@@ -99,7 +99,7 @@ public:
    std::function<void(pool_id_t, pg_num_t)> on_change_pg_history_hook;
    std::function<void(osd_num_t)> on_change_osd_state_hook;
-    json11::Json::object & serialize_inode_cfg(inode_config_t *cfg);
+    json11::Json::object serialize_inode_cfg(inode_config_t *cfg);
    etcd_kv_t parse_etcd_kv(const json11::Json & kv_json);
    void etcd_call(std::string api, json11::Json payload, int timeout, std::function<void(std::string, json11::Json)> callback);
    void etcd_txn(json11::Json txn, int timeout, std::function<void(std::string, json11::Json)> callback);
@@ -17,11 +17,12 @@ void osd_messenger_t::init()
    {
        rdma_context = msgr_rdma_context_t::create(
            rdma_device != "" ? rdma_device.c_str() : NULL,
-            rdma_port_num, rdma_gid_index, rdma_mtu
+            rdma_port_num, rdma_gid_index, rdma_mtu, log_level
        );
        if (!rdma_context)
        {
-            fprintf(stderr, "[OSD %lu] Couldn't initialize RDMA, proceeding with TCP only\n", osd_num);
+            if (log_level > 0)
                fprintf(stderr, "[OSD %lu] Couldn't initialize RDMA, proceeding with TCP only\n", osd_num);
        }
        else
        {
@@ -51,7 +51,7 @@ msgr_rdma_connection_t::~msgr_rdma_connection_t()
            free(b);
 }
-msgr_rdma_context_t *msgr_rdma_context_t::create(const char *ib_devname, uint8_t ib_port, uint8_t gid_index, uint32_t mtu)
+msgr_rdma_context_t *msgr_rdma_context_t::create(const char *ib_devname, uint8_t ib_port, uint8_t gid_index, uint32_t mtu, int log_level)
 {
    int res;
    ibv_device **dev_list = NULL;
@@ -70,7 +70,8 @@ msgr_rdma_context_t *msgr_rdma_context_t::create(const char *ib_devname, uint8_t
        ctx->dev = *dev_list;
        if (!ctx->dev)
        {
-            fprintf(stderr, "No RDMA devices found\n");
+            if (log_level > 0)
                fprintf(stderr, "No RDMA devices found\n");
            goto cleanup;
        }
    }
@@ -35,7 +35,7 @@ struct msgr_rdma_context_t
    int max_cqe = 0;
    int used_max_cqe = 0;
-    static msgr_rdma_context_t *create(const char *ib_devname, uint8_t ib_port, uint8_t gid_index, uint32_t mtu);
+    static msgr_rdma_context_t *create(const char *ib_devname, uint8_t ib_port, uint8_t gid_index, uint32_t mtu, int log_level);
    ~msgr_rdma_context_t();
 };
@@ -100,7 +100,7 @@ public:
            }
            unmap(cfg["dev_num"].uint64_value());
        }
-        else if (cfg["command"] == "list" || cfg["command"] == "list-mapped")
+        else if (cfg["command"] == "ls" || cfg["command"] == "list" || cfg["command"] == "list-mapped")
        {
            auto mapped = list_mapped();
            print_mapped(mapped, !cfg["json"].is_null());
@@ -119,7 +119,7 @@ public:
            "USAGE:\n"
            "  %s map [--etcd_address <etcd_address>] (--image <image> | --pool <pool> --inode <inode> --size <size in bytes>)\n"
            "  %s unmap /dev/nbd0\n"
-            "  %s list [--json]\n",
+            "  %s ls [--json]\n",
            exe_name, exe_name, exe_name
        );
        exit(0);
@@ -402,7 +402,7 @@ public:
                printf("%s\n", dev.first.c_str());
                for (auto & k: dev.second.object_items())
                {
-                    printf("%s: %s\n", k.first.c_str(), k.second.string_value().c_str());
+                    printf("%s: %s\n", k.first.c_str(), k.second.as_string().c_str());
                }
                printf("\n");
            }
@@ -45,6 +45,12 @@ osd_t::osd_t(const json11::Json & config, ring_loop_t *ringloop)
    // FIXME: Create Blockstore from on-disk superblock config and check it against the OSD cluster config
    auto bs_cfg = json_to_bs(this->config);
    this->bs = new blockstore_t(bs_cfg, ringloop, tfd);
    {
        // Autosync based on the number of unstable writes to prevent stalls due to insufficient journal space
        uint64_t max_autosync = bs->get_journal_size() / bs->get_block_size() / 2;
        if (autosync_writes > max_autosync)
            autosync_writes = max_autosync;
    }
    this->tfd->set_timer(print_stats_interval*1000, true, [this](int timer_id)
    {
@@ -104,7 +110,7 @@ void osd_t::parse_config(const json11::Json & config)
    log_level = config["log_level"].uint64_value();
    etcd_report_interval = config["etcd_report_interval"].uint64_value();
    if (etcd_report_interval <= 0)
-        etcd_report_interval = 30;
+        etcd_report_interval = 5;
    readonly = config["readonly"] == "true" || config["readonly"] == "1" || config["readonly"] == "yes";
    run_primary = config["run_primary"] != "false" && config["run_primary"] != "0" && config["run_primary"] != "no";
    no_rebalance = config["no_rebalance"] == "true" || config["no_rebalance"] == "1" || config["no_rebalance"] == "yes";
@@ -123,6 +129,11 @@ void osd_t::parse_config(const json11::Json & config)
        if (autosync_interval > MAX_AUTOSYNC_INTERVAL)
            autosync_interval = DEFAULT_AUTOSYNC_INTERVAL;
    }
    if (!config["autosync_writes"].is_null())
    {
        // Allow to set it to 0
        autosync_writes = config["autosync_writes"].uint64_value();
    }
    if (!config["client_queue_depth"].is_null())
    {
        client_queue_depth = config["client_queue_depth"].uint64_value();
@@ -35,6 +35,7 @@
 #define MAX_AUTOSYNC_INTERVAL 3600
 #define DEFAULT_AUTOSYNC_INTERVAL 5
 #define DEFAULT_AUTOSYNC_WRITES 128
 #define MAX_RECOVERY_QUEUE 2048
 #define DEFAULT_RECOVERY_QUEUE 4
 #define DEFAULT_RECOVERY_BATCH 16
@@ -93,7 +94,7 @@ class osd_t
    // config
    json11::Json::object config;
-    int etcd_report_interval = 30;
+    int etcd_report_interval = 5;
    bool readonly = false;
    osd_num_t osd_num = 1; // OSD numbers start with 1
@@ -108,7 +109,8 @@ class osd_t
    int print_stats_interval = 3;
    int slow_log_interval = 10;
    int immediate_commit = IMMEDIATE_NONE;
-    int autosync_interval = DEFAULT_AUTOSYNC_INTERVAL; // sync every 5 seconds
+    int autosync_interval = DEFAULT_AUTOSYNC_INTERVAL; // "emergency" sync every 5 seconds
    int autosync_writes = DEFAULT_AUTOSYNC_WRITES;
    int recovery_queue_depth = DEFAULT_RECOVERY_QUEUE;
    int recovery_sync_batch = DEFAULT_RECOVERY_BATCH;
    int log_level = 0;
@@ -140,6 +142,7 @@ class osd_t
    osd_op_t *autosync_op = NULL;
    // Unstable writes
    uint64_t unstable_write_count = 0;
    std::map<osd_object_id_t, uint64_t> unstable_writes;
    std::deque<osd_op_t*> syncs_in_progress;
@@ -615,7 +615,7 @@ void osd_t::apply_pg_config()
                }
                if (currently_taken)
                {
-                    if (pg_it->second.state & (PG_ACTIVE | PG_INCOMPLETE | PG_PEERING | PG_REPEERING))
+                    if (pg_it->second.state & (PG_ACTIVE | PG_INCOMPLETE | PG_PEERING | PG_REPEERING | PG_PEERED))
                    {
                        if (pg_it->second.target_set == pg_cfg.target_set)
                        {
@@ -703,13 +703,19 @@ void osd_t::apply_pg_config()
    this->pg_config_applied = all_applied;
 }
 struct reporting_pg_t
 {
    pool_pg_num_t pool_pg_num;
    bool history_changed;
 };
 void osd_t::report_pg_states()
 {
    if (etcd_reporting_pg_state || !this->pg_state_dirty.size() || !st_cli.etcd_addresses.size())
    {
        return;
    }
-    std::vector<std::pair<pool_pg_num_t,bool>> reporting_pgs;
+    std::vector<reporting_pg_t> reporting_pgs;
    json11::Json::array checks;
    json11::Json::array success;
    json11::Json::array failure;
@@ -721,7 +727,7 @@ void osd_t::report_pg_states()
            continue;
        }
        auto & pg = pg_it->second;
-        reporting_pgs.push_back({ *it, pg.history_changed });
+        reporting_pgs.push_back((reporting_pg_t){ *it, pg.history_changed });
        std::string state_key_base64 = base64_encode(st_cli.etcd_prefix+"/pg/state/"+std::to_string(pg.pool_id)+"/"+std::to_string(pg.pg_num));
        bool pg_state_exists = false;
        if (pg.state != PG_STARTING)
@@ -827,10 +833,10 @@ void osd_t::report_pg_states()
            // One of PG state updates failed, put dirty flags back
            for (auto pp: reporting_pgs)
            {
-                this->pg_state_dirty.insert(pp.first);
+                this->pg_state_dirty.insert(pp.pool_pg_num);
-                if (pp.second)
+                if (pp.history_changed)
                {
-                    auto pg_it = this->pgs.find(pp.first);
+                    auto pg_it = this->pgs.find(pp.pool_pg_num);
                    if (pg_it != this->pgs.end())
                    {
                        pg_it->second.history_changed = true;
@@ -870,17 +876,27 @@ void osd_t::report_pg_states()
            // Success. We'll get our changes back via the watcher and react to them
            for (auto pp: reporting_pgs)
            {
-                auto pg_it = this->pgs.find(pp.first);
+                auto pg_it = this->pgs.find(pp.pool_pg_num);
                if (pg_it != this->pgs.end() &&
-                    pg_it->second.state == PG_OFFLINE &&
+                    pg_state_dirty.find(pp.pool_pg_num) == pg_state_dirty.end())
                    pg_state_dirty.find(pp.first) == pg_state_dirty.end())
                {
-                    // Forget offline PGs after reporting their state
+                    if (pg_it->second.state == PG_OFFLINE)
                    if (pg_it->second.scheme == POOL_SCHEME_JERASURE)
                    {
-                        use_jerasure(pg_it->second.pg_size, pg_it->second.pg_data_size, false);
+                        // Forget offline PGs after reporting their state
                        // (if the state wasn't changed again)
                        if (pg_it->second.scheme == POOL_SCHEME_JERASURE)
                        {
                            use_jerasure(pg_it->second.pg_size, pg_it->second.pg_data_size, false);
                        }
                        this->pgs.erase(pg_it);
                    }
                    else if (pg_it->second.state & PG_PEERED)
                    {
                        // Activate PG after PG PEERED state is reported along with history
                        // (if the state wasn't changed again)
                        pg_it->second.state = pg_it->second.state & ~PG_PEERED | PG_ACTIVE;
                        report_pg_state(pg_it->second);
                    }
                    this->pgs.erase(pg_it);
                }
            }
            // Push other PG state updates, if any
@@ -37,6 +37,10 @@ void osd_t::handle_peers()
                    still = true;
                }
            }
            else if (p.second.state & PG_PEERED)
            {
                still = true;
            }
        }
        if (!still)
        {
@@ -57,6 +61,10 @@ void osd_t::handle_peers()
                }
                still = true;
            }
            else if (p.second.state & PG_PEERED)
            {
                still = true;
            }
        }
        if (!still)
        {
@@ -79,7 +87,7 @@ void osd_t::repeer_pgs(osd_num_t peer_osd)
    {
        auto & pg = p.second;
        bool repeer = false;
-        if (pg.state & (PG_PEERING | PG_ACTIVE | PG_INCOMPLETE))
+        if (pg.state & (PG_PEERING | PG_PEERED | PG_ACTIVE | PG_INCOMPLETE))
        {
            for (osd_num_t pg_osd: pg.all_peers)
            {
@@ -86,9 +86,24 @@ void pg_obj_state_check_t::walk()
    }
    if (pg->pg_cursize < pg->pg_size)
    {
-        pg->state |= PG_DEGRADED;
+        // Report PG history and activate
        pg->state |= PG_DEGRADED | PG_PEERED;
        std::vector<osd_num_t> history_set;
        for (auto peer_osd: pg->cur_set)
        {
            if (peer_osd != 0)
            {
                history_set.push_back(peer_osd);
            }
        }
        pg->target_history.push_back(history_set);
        pg->history_changed = true;
    }
    else
    {
        // Just activate
        pg->state |= PG_ACTIVE;
    }
    pg->state |= PG_ACTIVE;
    if (pg->state == PG_ACTIVE && pg->cur_peers.size() < pg->all_peers.size())
    {
        pg->state |= PG_LEFT_ON_DEAD;
@@ -430,10 +445,11 @@ void pg_t::calc_object_states(int log_level)
 void pg_t::print_state()
 {
    printf(
-        "[PG %u/%u] is %s%s%s%s%s%s%s%s%s%s%s%s%s%s (%lu objects)\n", pool_id, pg_num,
+        "[PG %u/%u] is %s%s%s%s%s%s%s%s%s%s%s%s%s%s%s (%lu objects)\n", pool_id, pg_num,
        (state & PG_STARTING) ? "starting" : "",
        (state & PG_OFFLINE) ? "offline" : "",
        (state & PG_PEERING) ? "peering" : "",
        (state & PG_PEERED) ? "peered" : "",
        (state & PG_INCOMPLETE) ? "incomplete" : "",
        (state & PG_ACTIVE) ? "active" : "",
        (state & PG_REPEERING) ? "repeering" : "",
@@ -5,8 +5,6 @@
 void osd_t::autosync()
 {
    // FIXME Autosync based on the number of unstable writes to prevent
    // "journal_sector_buffer_count is too low for this batch" errors
    if (immediate_commit != IMMEDIATE_ALL && !autosync_op)
    {
        autosync_op = new osd_op_t();
@@ -274,6 +274,11 @@ continue_others:
    }
    // finish_op would invalidate next_it if it cleared pg.write_queue, but it doesn't do that :)
    finish_op(cur_op, cur_op->reply.hdr.retval);
    if (unstable_write_count >= autosync_writes)
    {
        unstable_write_count = 0;
        autosync();
    }
    if (next_op)
    {
        // Continue next write to the same object
@@ -353,6 +358,7 @@ resume_7:
    else
    {
 lazy:
        unstable_write_count++;
        if (op_data->scheme != POOL_SCHEME_REPLICATED)
        {
            // Remember version as unstable for EC/XOR
@@ -3,11 +3,12 @@
 #include "pg_states.h"
-const int pg_state_bit_count = 15;
+const int pg_state_bit_count = 16;
-const int pg_state_bits[15] = {
+const int pg_state_bits[16] = {
    PG_STARTING,
    PG_PEERING,
    PG_PEERED,
    PG_INCOMPLETE,
    PG_ACTIVE,
    PG_REPEERING,
@@ -22,9 +23,10 @@ const int pg_state_bits[15] = {
    PG_LEFT_ON_DEAD,
 };
-const char *pg_state_names[15] = {
+const char *pg_state_names[16] = {
    "starting",
    "peering",
    "peered",
    "incomplete",
    "active",
    "repeering",
@@ -4,23 +4,27 @@
 #pragma once
 // Placement group states
-// STARTING -> [acquire lock] -> PEERING -> INCOMPLETE|ACTIVE -> STOPPING -> OFFLINE -> [release lock]
+// STARTING -> [acquire lock] -> PEERING -> PEERED
 // PEERED -> [report history if required!] -> INCOMPLETE|ACTIVE
 // ACTIVE -> REPEERING -> PEERING
 // ACTIVE -> STOPPING -> OFFLINE -> [release lock]
 // Exactly one of these:
 #define PG_STARTING (1<<0)
 #define PG_PEERING (1<<1)
-#define PG_INCOMPLETE (1<<2)
+#define PG_PEERED (1<<2)
-#define PG_ACTIVE (1<<3)
+#define PG_INCOMPLETE (1<<3)
-#define PG_REPEERING (1<<4)
+#define PG_ACTIVE (1<<4)
-#define PG_STOPPING (1<<5)
+#define PG_REPEERING (1<<5)
-#define PG_OFFLINE (1<<6)
+#define PG_STOPPING (1<<6)
 #define PG_OFFLINE (1<<7)
 // Plus any of these:
-#define PG_DEGRADED (1<<7)
+#define PG_DEGRADED (1<<8)
-#define PG_HAS_INCOMPLETE (1<<8)
+#define PG_HAS_INCOMPLETE (1<<9)
-#define PG_HAS_DEGRADED (1<<9)
+#define PG_HAS_DEGRADED (1<<10)
-#define PG_HAS_MISPLACED (1<<10)
+#define PG_HAS_MISPLACED (1<<11)
-#define PG_HAS_UNCLEAN (1<<11)
+#define PG_HAS_UNCLEAN (1<<12)
-#define PG_HAS_INVALID (1<<12)
+#define PG_HAS_INVALID (1<<13)
-#define PG_LEFT_ON_DEAD (1<<13)
+#define PG_LEFT_ON_DEAD (1<<14)
 // Lower bits that represent object role (EC 0/1/2... or always 0 with replication)
 // 12 bits is a safe default that doesn't depend on pg_stripe_size or pg_block_size
@@ -19,6 +19,10 @@
 #include "qemu/units.h"
 #include "block/qdict.h"
 #include "qemu/cutils.h"
 #elif QEMU_VERSION_MAJOR == 2 && QEMU_VERSION_MINOR >= 10
 #include "qemu/cutils.h"
 #include "qapi/qmp/qstring.h"
 #include "qapi/qmp/qjson.h"
 #else
 #include "qapi/qmp/qint.h"
 #define qdict_put_int(options, name, num_val) qdict_put_obj(options, name, QOBJECT(qint_from_int(num_val)))
@@ -290,7 +294,7 @@ static void vitastor_close(BlockDriverState *bs)
        g_free(client->image);
 }
-#if QEMU_VERSION_MAJOR >= 3
+#if QEMU_VERSION_MAJOR >= 3 || QEMU_VERSION_MAJOR == 2 && QEMU_VERSION_MINOR > 2
 static int vitastor_probe_blocksizes(BlockDriverState *bs, BlockSizes *bsz)
 {
    bsz->phys = 4096;
@@ -299,6 +303,7 @@ static int vitastor_probe_blocksizes(BlockDriverState *bs, BlockSizes *bsz)
 }
 #endif
 #if QEMU_VERSION_MAJOR >= 3
 static int coroutine_fn vitastor_co_create_opts(
 #if QEMU_VERSION_MAJOR >= 4
    BlockDriver *drv,
@@ -323,6 +328,7 @@ out:
    qobject_unref(options);
    return ret;
 }
 #endif
 #if QEMU_VERSION_MAJOR >= 3
 static int coroutine_fn vitastor_co_truncate(BlockDriverState *bs, int64_t offset,
@@ -362,20 +368,18 @@ static int64_t vitastor_getlength(BlockDriverState *bs)
    return client->size;
 }
-#if QEMU_VERSION_MAJOR >= 3
+#if QEMU_VERSION_MAJOR >= 3 || QEMU_VERSION_MAJOR == 2 && QEMU_VERSION_MINOR > 0
 static void vitastor_refresh_limits(BlockDriverState *bs, Error **errp)
 #else
 static int vitastor_refresh_limits(BlockDriverState *bs)
 #endif
 {
 #if QEMU_VERSION_MAJOR >= 4
    bs->bl.request_alignment = 4096;
 #if QEMU_VERSION_MAJOR >= 3 || QEMU_VERSION_MAJOR == 2 && QEMU_VERSION_MINOR > 3
    bs->bl.min_mem_alignment = 4096;
 #else
    bs->request_alignment = 4096;
 #endif
    bs->bl.opt_mem_alignment = 4096;
-#if QEMU_VERSION_MAJOR < 3
+#if QEMU_VERSION_MAJOR < 2 || QEMU_VERSION_MAJOR == 2 && QEMU_VERSION_MINOR == 0
    return 0;
 #endif
 }
@@ -400,7 +404,7 @@ static void vitastor_co_generic_bh_cb(void *opaque, long retval)
    task->complete = 1;
    if (qemu_coroutine_self() != task->co)
    {
-#if QEMU_VERSION_MAJOR >= 3
+#if QEMU_VERSION_MAJOR >= 3 || QEMU_VERSION_MAJOR == 2 && QEMU_VERSION_MINOR > 8
        aio_co_wake(task->co);
 #else
        qemu_coroutine_enter(task->co, NULL);
@@ -484,7 +488,7 @@ static int coroutine_fn vitastor_co_flush(BlockDriverState *bs)
    return task.ret;
 }
-#if QEMU_VERSION_MAJOR >= 3
+#if QEMU_VERSION_MAJOR >= 3 || QEMU_VERSION_MAJOR == 2 && QEMU_VERSION_MINOR > 0
 static QemuOptsList vitastor_create_opts = {
    .name = "vitastor-create-opts",
    .head = QTAILQ_HEAD_INITIALIZER(vitastor_create_opts.head),
@@ -528,7 +532,7 @@ static BlockDriver bdrv_vitastor = {
    .bdrv_has_zero_init             = bdrv_has_zero_init_1,
    .bdrv_get_info                  = vitastor_get_info,
    .bdrv_getlength                 = vitastor_getlength,
-#if QEMU_VERSION_MAJOR >= 3
+#if QEMU_VERSION_MAJOR >= 3 || QEMU_VERSION_MAJOR == 2 && QEMU_VERSION_MINOR > 2
    .bdrv_probe_blocksizes          = vitastor_probe_blocksizes,
 #endif
    .bdrv_refresh_limits            = vitastor_refresh_limits,
@@ -540,7 +544,7 @@ static BlockDriver bdrv_vitastor = {
    .bdrv_close                     = vitastor_close,
    // Option list for the create operation
-#if QEMU_VERSION_MAJOR >= 3
+#if QEMU_VERSION_MAJOR >= 3 || QEMU_VERSION_MAJOR == 2 && QEMU_VERSION_MINOR > 0
    .create_opts                    = &vitastor_create_opts,
 #else
    .create_options                 = vitastor_create_opts,
@@ -4,17 +4,15 @@
 OSD_SIZE=${OSD_SIZE:-1024}
 PG_COUNT=${PG_COUNT:-1}
 PG_SIZE=${PG_SIZE:-3}
 OSD_COUNT=${OSD_COUNT:-3}
 SCHEME=${SCHEME:-ec}
-dd if=/dev/zero of=./testdata/test_osd1.bin bs=1024 count=1 seek=$((OSD_SIZE*1024-1))
+for i in $(seq 1 $OSD_COUNT); do
-dd if=/dev/zero of=./testdata/test_osd2.bin bs=1024 count=1 seek=$((OSD_SIZE*1024-1))
+    dd if=/dev/zero of=./testdata/test_osd$i.bin bs=1024 count=1 seek=$((OSD_SIZE*1024-1))
-dd if=/dev/zero of=./testdata/test_osd3.bin bs=1024 count=1 seek=$((OSD_SIZE*1024-1))
+    build/src/vitastor-osd --osd_num $i --bind_address 127.0.0.1 $OSD_ARGS --etcd_address $ETCD_URL $(node mon/simple-offsets.js --format options --device ./testdata/test_osd$i.bin 2>/dev/null) &>./testdata/osd$i.log &
-
+    eval OSD${i}_PID=$!
-build/src/vitastor-osd --osd_num 1 --bind_address 127.0.0.1 $OSD_ARGS --etcd_address $ETCD_URL $(node mon/simple-offsets.js --format options --device ./testdata/test_osd1.bin 2>/dev/null) &>./testdata/osd1.log &
+done
 OSD1_PID=$!
 build/src/vitastor-osd --osd_num 2 --bind_address 127.0.0.1 $OSD_ARGS --etcd_address $ETCD_URL $(node mon/simple-offsets.js --format options --device ./testdata/test_osd2.bin 2>/dev/null) &>./testdata/osd2.log &
 OSD2_PID=$!
 build/src/vitastor-osd --osd_num 3 --bind_address 127.0.0.1 $OSD_ARGS --etcd_address $ETCD_URL $(node mon/simple-offsets.js --format options --device ./testdata/test_osd3.bin 2>/dev/null) &>./testdata/osd3.log &
 OSD3_PID=$!
 cd mon
 npm install
@@ -26,11 +24,15 @@ if [ -n "$GLOBAL_CONF" ]; then
    $ETCDCTL put /vitastor/config/global "$GLOBAL_CONF"
 fi
-$ETCDCTL put /vitastor/config/pools '{"1":{"name":"testpool","scheme":"xor","pg_size":3,"pg_minsize":2,"parity_chunks":1,"pg_count":'$PG_COUNT',"failure_domain":"osd"}}'
+if [ "$SCHEME" = "replicated" ]; then
    $ETCDCTL put /vitastor/config/pools '{"1":{"name":"testpool","scheme":"replicated","pg_size":'$PG_SIZE',"pg_minsize":'$((PG_SIZE-1))',"pg_count":'$PG_COUNT',"failure_domain":"osd"}}'
 else
    $ETCDCTL put /vitastor/config/pools '{"1":{"name":"testpool","scheme":"xor","pg_size":'$PG_SIZE',"pg_minsize":'$((PG_SIZE-1))',"parity_chunks":1,"pg_count":'$PG_COUNT',"failure_domain":"osd"}}'
 fi
 sleep 2
-if ! ($ETCDCTL get /vitastor/config/pgs --print-value-only | jq -s -e '(. | length) != 0 and ([ .[0].items["1"][] | select((.osd_set | sort) == ["1","2","3"]) ] | length) == '$PG_COUNT); then
+if ! ($ETCDCTL get /vitastor/config/pgs --print-value-only | jq -s -e '(. | length) != 0 and ([ .[0].items["1"][] | select(((.osd_set | select(. != 0) | sort | unique) | length) == '$PG_SIZE') ] | length) == '$PG_COUNT); then
    format_error "FAILED: $PG_COUNT PG(s) NOT CONFIGURED"
 fi
@@ -18,19 +18,19 @@ dd if=/dev/zero of=./testdata/test_osd5.bin bs=1024 count=1 seek=$((1024*1024-1)
 dd if=/dev/zero of=./testdata/test_osd6.bin bs=1024 count=1 seek=$((1024*1024-1))
 dd if=/dev/zero of=./testdata/test_osd7.bin bs=1024 count=1 seek=$((1024*1024-1))
-build/src/vitastor-osd --osd_num 1 --bind_address 127.0.0.1 $NO_SAME --etcd_address $ETCD_URL $(node mon/simple-offsets.js --format options --device ./testdata/test_osd1.bin 2>/dev/null) 2>&1 >>./testdata/osd1.log &
+build/src/vitastor-osd --osd_num 1 --bind_address 127.0.0.1 $NO_SAME --etcd_address $ETCD_URL $(node mon/simple-offsets.js --format options --device ./testdata/test_osd1.bin 2>/dev/null) &>./testdata/osd1.log &
 OSD1_PID=$!
-build/src/vitastor-osd --osd_num 2 --bind_address 127.0.0.1 $NO_SAME --etcd_address $ETCD_URL $(node mon/simple-offsets.js --format options --device ./testdata/test_osd2.bin 2>/dev/null) 2>&1 >>./testdata/osd2.log &
+build/src/vitastor-osd --osd_num 2 --bind_address 127.0.0.1 $NO_SAME --etcd_address $ETCD_URL $(node mon/simple-offsets.js --format options --device ./testdata/test_osd2.bin 2>/dev/null) &>./testdata/osd2.log &
 OSD2_PID=$!
-build/src/vitastor-osd --osd_num 3 --bind_address 127.0.0.1 $NO_SAME --etcd_address $ETCD_URL $(node mon/simple-offsets.js --format options --device ./testdata/test_osd3.bin 2>/dev/null) 2>&1 >>./testdata/osd3.log &
+build/src/vitastor-osd --osd_num 3 --bind_address 127.0.0.1 $NO_SAME --etcd_address $ETCD_URL $(node mon/simple-offsets.js --format options --device ./testdata/test_osd3.bin 2>/dev/null) &>./testdata/osd3.log &
 OSD3_PID=$!
-build/src/vitastor-osd --osd_num 4 --bind_address 127.0.0.1 $NO_SAME --etcd_address $ETCD_URL $(node mon/simple-offsets.js --format options --device ./testdata/test_osd4.bin 2>/dev/null) 2>&1 >>./testdata/osd4.log &
+build/src/vitastor-osd --osd_num 4 --bind_address 127.0.0.1 $NO_SAME --etcd_address $ETCD_URL $(node mon/simple-offsets.js --format options --device ./testdata/test_osd4.bin 2>/dev/null) &>./testdata/osd4.log &
 OSD4_PID=$!
-build/src/vitastor-osd --osd_num 5 --bind_address 127.0.0.1 $NO_SAME --etcd_address $ETCD_URL $(node mon/simple-offsets.js --format options --device ./testdata/test_osd5.bin 2>/dev/null) 2>&1 >>./testdata/osd5.log &
+build/src/vitastor-osd --osd_num 5 --bind_address 127.0.0.1 $NO_SAME --etcd_address $ETCD_URL $(node mon/simple-offsets.js --format options --device ./testdata/test_osd5.bin 2>/dev/null) &>./testdata/osd5.log &
 OSD5_PID=$!
-build/src/vitastor-osd --osd_num 6 --bind_address 127.0.0.1 $NO_SAME --etcd_address $ETCD_URL $(node mon/simple-offsets.js --format options --device ./testdata/test_osd6.bin 2>/dev/null) 2>&1 >>./testdata/osd6.log &
+build/src/vitastor-osd --osd_num 6 --bind_address 127.0.0.1 $NO_SAME --etcd_address $ETCD_URL $(node mon/simple-offsets.js --format options --device ./testdata/test_osd6.bin 2>/dev/null) &>./testdata/osd6.log &
 OSD6_PID=$!
-build/src/vitastor-osd --osd_num 7 --bind_address 127.0.0.1 $NO_SAME --etcd_address $ETCD_URL $(node mon/simple-offsets.js --format options --device ./testdata/test_osd7.bin 2>/dev/null) 2>&1 >>./testdata/osd7.log &
+build/src/vitastor-osd --osd_num 7 --bind_address 127.0.0.1 $NO_SAME --etcd_address $ETCD_URL $(node mon/simple-offsets.js --format options --device ./testdata/test_osd7.bin 2>/dev/null) &>./testdata/osd7.log &
 OSD7_PID=$!
 cd mon
@@ -0,0 +1,32 @@
 #!/bin/bash -ex
 OSD_COUNT=2
 PG_SIZE=2
 SCHEME=replicated
 . `dirname $0`/run_3osds.sh
 # Kill OSD 1
 kill $OSD1_PID
 sleep 2
 # Write
 LD_PRELOAD=libasan.so.5 \
    fio -thread -name=test -ioengine=build/src/libfio_vitastor.so -bs=4k -direct=1 -iodepth=1 -fsync=1 \
        -rw=randwrite -etcd=$ETCD_URL -pool=1 -inode=1 -size=128M -runtime=10 -number_ios=100
 # Kill OSD 2, start OSD 1
 kill $OSD2_PID
 build/src/vitastor-osd --osd_num 1 --bind_address 127.0.0.1 $OSD_ARGS --etcd_address $ETCD_URL $(node mon/simple-offsets.js --format options --device ./testdata/test_osd2.bin 2>/dev/null) >>./testdata/osd2.log 2>&1 &
 sleep 2
 # Check PG state - it should NOT become active
 if ($ETCDCTL get --prefix /vitastor/pg/state/ --print-value-only | grep -q active); then
    format_error "FAILED: PG STILL ACTIVE AFTER SPLITBRAIN"
 fi
 format_green OK
@@ -5,6 +5,12 @@
 #LD_PRELOAD=libasan.so.5 \
 #    fio -thread -name=test -ioengine=build/src/libfio_vitastor_sec.so -bs=4k -fsync=128 `$ETCDCTL get /vitastor/osd/state/1 --print-value-only | jq -r '"-host="+.addresses[0]+" -port="+(.port|tostring)'` -rw=write -size=32M
 # Random writes without immediate_commit were stalling OSDs
 LD_PRELOAD=libasan.so.5 \
    fio -thread -name=test -ioengine=build/src/libfio_vitastor.so -bs=124k -direct=1 -numjobs=16 -iodepth=4 \
        -rw=randwrite -etcd=$ETCD_URL -pool=1 -inode=1 -size=128M -runtime=10
 # A lot of parallel syncs was crashing the primary OSD at some point
 LD_PRELOAD=libasan.so.5 \
Author	SHA1	Message	Date
Vitaliy Filippov	7eabc364bf	Release 0.6.7 - Implement CLI commands for listing, viewing I/O statistics, creating, snapshotting, cloning, resizing and modifying images. All these operations are covered by 3 commands: ls, create, modify - Implement an important fix to prior OSD set tracking for PGs. The previous version had an issue which could lead to data loss due to an OSD with older copy of the data thinking it has the newest copy - Fix I/O statistics aggregation in the monitor - Several minor fixes for Cinder driver - Fix QEMU driver to be compatible with QEMU 2.x > 2.0 - Fix stalls sometimes possible in configurations without immediate_commit due to insufficient amount of automatic internal fsync operations - Add `vita` alias for `vitastor-cli`	2021-11-13 23:23:55 +03:00
Vitaliy Filippov	a346f84c69	Allow to show only specific images in listing	2021-11-13 23:23:55 +03:00
Vitaliy Filippov	71a0c1a7b9	Fix list sorting	2021-11-13 23:23:55 +03:00
Vitaliy Filippov	20e86c7d84	Note about vitastor-cli	2021-11-13 22:39:17 +03:00
Vitaliy Filippov	110b39900b	Rename the new "set" command to "modify"	2021-11-13 22:39:17 +03:00
Vitaliy Filippov	697ee30a26	Add ARG REL into build Dockerfiles	2021-11-13 22:39:17 +03:00
Vitaliy Filippov	42479b4590	Fix vitastor-nbd list, add ls alias	2021-11-13 22:39:17 +03:00
Vitaliy Filippov	6e82044e84	Add `vita` symlink	2021-11-13 22:39:17 +03:00
Vitaliy Filippov	2cb3e84882	Implement CLI set (resize, change readonly status) command	2021-11-13 22:39:17 +03:00
Vitaliy Filippov	32614c5bc8	Add a test for the last change ("split-brain avoidance", lol)	2021-11-13 22:39:17 +03:00
Vitaliy Filippov	aa436027c8	Report pg/history from OSD on every degraded activation Required to prevent data loss due to activation of an OSD with older data when PG OSD set change doesn't occur. I.e. fixes the simplest case: - Run 2 OSDs with 1 PG - Start writing into the PG - Stop OSD 2 - Stop OSD 1 - Start OSD 2 After this change the PG will refuse to start after the last step.	2021-11-13 22:39:17 +03:00
Vitaliy Filippov	577a563b91	Allow to disable colored output	2021-11-11 01:41:58 +03:00
Vitaliy Filippov	e4efa2c08a	Improve vitastor-cli ls - show I/O statistics, allow to sort & limit output	2021-11-11 01:41:58 +03:00
Vitaliy Filippov	0f3f0a9d29	Calculate average statistics in mon, remove buggy "fix_stat_overflows"	2021-11-11 00:20:57 +03:00
Vitaliy FilippovandGitHub	0544a16f95	Merge pull request #30 from moly7x/moly7x-fix-No-weighed-backend-found Fix No weighed backend found	2021-11-10 17:13:17 +03:00
Tân LêandGitHub	30d8930958	Revert spacing in comment	2021-11-10 10:47:58 +07:00
Tân LêandGitHub	baf003fbd3	Revert newline in comment	2021-11-10 10:43:24 +07:00
Tân LêandGitHub	ba39a38dc4	Fix No weighed backend found	2021-11-10 10:27:01 +07:00
Vitaliy Filippov	d528cd77f1	Fix install_symlink	2021-11-09 16:42:29 +03:00
Vitaliy Filippov	6e6f407df3	Simplify & fix monitor stats aggregation	2021-11-09 01:41:22 +03:00
Vitaliy Filippov	4d43774cbb	Use 5s etcd_report_interval by default	2021-11-09 01:27:12 +03:00
Vitaliy Filippov	a1488f7217	Fix qemu_driver to build with QEMU 2.x (previously it was only correct for QEMU 2.0)	2021-11-08 23:07:31 +03:00
Vitaliy Filippov	404e07d365	Implement image/snapshot/clone creation and listing by pool	2021-11-07 01:01:07 +03:00
Vitaliy Filippov	b3dcee0d43	Also print "bare" inodes with missing config if they occupy space	2021-11-06 14:56:41 +03:00
Vitaliy Filippov	609bd4eb59	Remove naggy RDMA messages when log level is zero	2021-11-06 14:36:23 +03:00
Vitaliy Filippov	8e445ddc9a	Begin to implement CLI: implement listing, add help, add create stub	2021-11-06 14:32:19 +03:00
Vitaliy Filippov	ffb06536ff	Revoke lease in mon on SIGINT & SIGTERM, fix raw_to_usable calculation	2021-11-06 13:54:35 +03:00
Vitaliy FilippovandGitHub	eeecab20c2	Merge pull request #28 from moly7x/fix Fix building QEMU 3.1	2021-11-06 11:12:17 +03:00
Tân LêandGitHub	e889ac4209	Fix building QEMU 3.1	2021-11-05 13:45:51 +07:00
Vitaliy Filippov	cfe8de9b84	Autosync based on number of unstable ops to prevent journal stalls	2021-10-30 14:26:48 +03:00
Vitaliy Filippov	24b9b19066	Change rm to rm-data as it was renamed	2021-10-23 10:39:20 +03:00
Vitaliy FilippovandGitHub	ef645ee0c2	Merge pull request #26 from moly7x/fix FIX ERROR cinder.volume.drivers.vitastor KeyError: 'total_raw_tb'	2021-10-22 21:29:52 +03:00
Tân Lê	8a9bae5216	FIX ERROR cinder.volume.drivers.vitastor KeyError: 'total_raw_tb'	2021-10-22 11:38:50 +07:00
Vitaliy Filippov	da99686a15	Correctly aggregate pool statistics for unknown pools	2021-10-21 18:58:56 +03:00
Vitaliy Filippov	dcc03ee41f	Add go.sum	2021-10-19 02:42:10 +03:00
`@@ -1,4 +1,4 @@`
	`VERSION ?= v0.6.6`	`VERSION ?= v0.6.7`

	`all: build push`	`all: build push`