Support multiple OSD networks and separate OSD cluster network

2025-03-31 21:01:15 +03:00
parent 5d2e28d4a9
commit a103065d12
20 changed files with 288 additions and 200 deletions
@@ -10,13 +10,14 @@ These parameters only apply to OSDs, are not fixed at the moment of OSD drive
 initialization and can be changed - in /etc/vitastor/vitastor.conf or [vitastor-disk update-sb](../usage/disk.en.md#update-sb)
 with an OSD restart or, for some of them, even without restarting by updating configuration in etcd.

+- [osd_network](#osd_network)
+- [osd_cluster_network](#osd_cluster_network)
+- [bind_address](#bind_address)
+- [bind_port](#bind_port)
 - [osd_iothread_count](#osd_iothread_count)
 - [etcd_report_interval](#etcd_report_interval)
 - [etcd_stats_interval](#etcd_stats_interval)
 - [run_primary](#run_primary)
- [osd_network](#osd_network)
- [bind_address](#bind_address)
- [bind_port](#bind_port)
 - [autosync_interval](#autosync_interval)
 - [autosync_writes](#autosync_writes)
 - [recovery_queue_depth](#recovery_queue_depth)
@@ -65,6 +66,42 @@ with an OSD restart or, for some of them, even without restarting by updating co
 - [min_discard_size](#min_discard_size)
 - [allow_net_split](#allow_net_split)

+## osd_network
+
+- Type: string or array of strings
+
+Network mask of public OSD network(s) (IPv4 or IPv6). Each OSD listens on all
+addresses of UP + RUNNING interfaces matching one of these networks, on the
+same port. Port is auto-selected except if [bind_port](#bind_port) is
+explicitly specified. Bind address(es) may also be overridden manually by
+specifying [bind_address](#bind_address). If OSD networks are not specified
+at all, OSD just listens on a wildcard address (0.0.0.0).
+
+## osd_cluster_network
+
+- Type: string or array of strings
+
+Network mask of separate network(s) (IPv4 or IPv6) to use for OSD
+cluster connections. I.e. OSDs will always attempt to use these networks
+to connect to other OSDs, while clients will attempt to use networks from
+[osd_network](#osd_network).
+
+## bind_address
+
+- Type: string or array of strings
+
+Instead of the network mask, you can also set OSD listen addresses explicitly
+using this parameter. May be useful if you want to start OSDs on interfaces
+that are not UP + RUNNING.
+
+## bind_port
+
+- Type: integer
+
+By default, OSDs pick random ports to use for incoming connections
+automatically. With this option you can set a specific port for a specific
+OSD by hand.
+
 ## osd_iothread_count

 - Type: integer
@@ -107,34 +144,6 @@ debugging purposes. It's possible to implement additional feature for the
 monitor which may allow to separate primary and secondary OSDs, but it's
 unclear why anyone could need it, so it's not implemented.

-## osd_network
-
- Type: string or array of strings
-
-Network mask of the network (IPv4 or IPv6) to use for OSDs. Note that
-although it's possible to specify multiple networks here, this does not
-mean that OSDs will create multiple listening sockets - they'll only
-pick the first matching address of an UP + RUNNING interface. Separate
-networks for cluster and client connections are also not implemented, but
-they are mostly useless anyway, so it's not a big deal.
-
-## bind_address
-
- Type: string
- Default: 0.0.0.0
-
-Instead of the network mask, you can also set OSD listen address explicitly
-using this parameter. May be useful if you want to start OSDs on interfaces
-that are not UP + RUNNING.
-
-## bind_port
-
- Type: integer
-
-By default, OSDs pick random ports to use for incoming connections
-automatically. With this option you can set a specific port for a specific
-OSD by hand.
-
 ## autosync_interval

 - Type: seconds