feat(regionserver): add graceful shutdown configuration

CHANGELOG.md

@@ -7,6 +7,7 @@
 - Reduce CRD size from `1.4MB` to `96KB` by accepting arbitrary YAML input instead of the underlying schema for the following fields ([#548]):
   - `podOverrides`
   - `affinity`
+- Support moving regions to other Pods during graceful shutdown of region servers ([#570]).
 
 ### Fixed
 
@@ -22,6 +23,7 @@
 [#550]: https://github.com/stackabletech/hbase-operator/pull/550
 [#556]: https://github.com/stackabletech/hbase-operator/pull/556
 [#558]: https://github.com/stackabletech/hbase-operator/pull/558
+[#570]: https://github.com/stackabletech/hbase-operator/pull/570
 
 ## [24.7.0] - 2024-07-24
 

Cargo.toml

@@ -20,6 +20,7 @@ rstest = "0.22"
 serde = { version = "1.0", features = ["derive"] }
 serde_json = "1.0"
 serde_yaml = "0.9"
+shell-escape = "0.1"
 snafu = "0.8"
 stackable-operator = { git = "https://github.com/stackabletech/operator-rs.git", tag = "stackable-operator-0.76.0" }
 product-config = { git = "https://github.com/stackabletech/product-config.git", tag = "0.7.0" }

deploy/helm/hbase-operator/crds/crds.yaml

@@ -724,6 +724,32 @@ spec:
                               nullable: true
                               type: boolean
                           type: object
+                        regionMover:
+                          description: Before terminating a region server pod, the RegionMover tool can be invoked to transfer local regions to other servers. This may cause a lot of network traffic in the Kubernetes cluster if the entire HBase stacklet is being restarted. The operator will compute a timeout period for the region move that will not exceed the graceful shutdown timeout.
+                          nullable: true
+                          properties:
+                            ack:
+                              description: If enabled (default), the region mover will confirm that regions are available on the source as well as the target pods before and after the move.
+                              type: boolean
+                            extraOpts:
+                              default: []
+                              description: Additional options to pass to the region mover.
+                              items:
+                                type: string
+                              type: array
+                            maxThreads:
+                              description: Maximum number of threads to use for moving regions.
+                              format: uint16
+                              minimum: 0.0
+                              type: integer
+                            runBeforeShutdown:
+                              description: Move local regions to other servers before terminating a region server's pod.
+                              type: boolean
+                          required:
+                            - ack
+                            - maxThreads
+                            - runBeforeShutdown
+                          type: object
                         resources:
                           default:
                             cpu:
@@ -947,6 +973,32 @@ spec:
                                     nullable: true
                                     type: boolean
                                 type: object
+                              regionMover:
+                                description: Before terminating a region server pod, the RegionMover tool can be invoked to transfer local regions to other servers. This may cause a lot of network traffic in the Kubernetes cluster if the entire HBase stacklet is being restarted. The operator will compute a timeout period for the region move that will not exceed the graceful shutdown timeout.
+                                nullable: true
+                                properties:
+                                  ack:
+                                    description: If enabled (default), the region mover will confirm that regions are available on the source as well as the target pods before and after the move.
+                                    type: boolean
+                                  extraOpts:
+                                    default: []
+                                    description: Additional options to pass to the region mover.
+                                    items:
+                                      type: string
+                                    type: array
+                                  maxThreads:
+                                    description: Maximum number of threads to use for moving regions.
+                                    format: uint16
+                                    minimum: 0.0
+                                    type: integer
+                                  runBeforeShutdown:
+                                    description: Move local regions to other servers before terminating a region server's pod.
+                                    type: boolean
+                                required:
+                                  - ack
+                                  - maxThreads
+                                  - runBeforeShutdown
+                                type: object
                               resources:
                                 default:
                                   cpu:

docs/modules/hbase/pages/usage-guide/operations/graceful-shutdown.adoc

@@ -1,6 +1,6 @@
 = Graceful shutdown
 
-You can configure the graceful shutdown as described in xref:concepts:operations/graceful_shutdown.adoc[].
+You can configure the graceful shutdown grace period as described in xref:concepts:operations/graceful_shutdown.adoc[].
 
 == Masters
 
@@ -15,7 +15,7 @@
 
 == RegionServers
 
-As a default, RegionServers have `60 minutes` to shut down gracefully.
+By default, RegionServers have `60 minutes` to shut down gracefully.
 
 They use the same mechanism described above.
 In contrast to the Master servers, they will, however, acknowledge the graceful shutdown with a message in the logs:
@@ -26,6 +26,61 @@
 2023-10-11 12:38:05,060 INFO  [shutdown-hook-0] regionserver.HRegionServer: ***** STOPPING region server 'test-hbase-regionserver-default-0.test-hbase-regionserver-default.kuttl-test-topical-parakeet.svc.cluster.local,16020,1697027870348' *****
 ----
 
+The operator allows for finer control over the shutdown process of region servers.
+For each region server pod, the region mover tool may be invoked before terminating the region server's pod.
+The affected regions are transferred to other pods thus ensuring that the data is not lost.
+
+Here is a an example:
+
+[source,yaml]
+----
+spec:
+  regionServers:
+    config:
+      regionMover
+        runBeforeShutdown: true # <1>
+        maxThreads: 5 # <2>
+        ack: false # <3>
+        extraOpts: ["--designatedFile", "/path/to/designatedFile"] # <4>
+----
+<1>: Run the region mover tool before shutting down the region server. Default is `false`.
+<2>: Maximum number of threads to use for moving regions. Default is 1.
+<3>: Enable or disable region confirmation on the present and target servers. Default is `true`.
+<4>: Extra options to pass to the region mover tool.
+
+For a list of additional options accepted by the region mover use the `--help` option first:
+
+[source,bash]
+----
+$ /stackable/hbase/bin/hbase org.apache.hadoop.hbase.util.RegionMover --help
+usage: hbase org.apache.hadoop.hbase.util.RegionMover <options>
+Options:
+ -r,--regionserverhost <arg>   region server <hostname>|<hostname:port>
+ -o,--operation <arg>          Expected: load/unload/unload_from_rack/isolate_regions
+ -m,--maxthreads <arg>         Define the maximum number of threads to use to unload and reload the regions
+ -i,--isolateRegionIds <arg>   Comma separated list of Region IDs hash to isolate on a RegionServer and put region
+                               server in draining mode. This option should only be used with '-o isolate_regions'. By
+                               putting region server in decommission/draining mode, master can't assign any new region
+                               on this server. If one or more regions are not found OR failed to isolate successfully,
+                               utility will exist without putting RS in draining/decommission mode. Ex.
+                               --isolateRegionIds id1,id2,id3 OR -i id1,id2,id3
+ -x,--excludefile <arg>        File with <hostname:port> per line to exclude as unload targets; default excludes only
+                               target host; useful for rack decommisioning.
+ -d,--designatedfile <arg>     File with <hostname:port> per line as unload targets;default is all online hosts
+ -f,--filename <arg>           File to save regions list into unloading, or read from loading; default
+                               /tmp/<usernamehostname:port>
+ -n,--noack                    Turn on No-Ack mode(default: false) which won't check if region is online on target
+                               RegionServer, hence best effort. This is more performant in unloading and loading but
+                               might lead to region being unavailable for some time till master reassigns it in case the
+                               move failed
+ -t,--timeout <arg>            timeout in seconds after which the tool will exit irrespective of whether it finished or
+                               not;default Integer.MAX_VALUE
+----
+
+NOTE: There is no need to explicitly specify a timeout for the region movement. The operator will compute an appropriate timeout that cannot exceed the `gracefulShutdownTimeout` for region servers.
+
+IMPORTANT: The ZooKeeper connection must be available during the time the region mover is running for the graceful shutdown process to succeed.
+
 == RestServers
 
 As a default, RestServers have `5 minutes` to shut down gracefully.

rust/crd/Cargo.toml

@@ -12,6 +12,7 @@ publish = false
 product-config.workspace = true
 serde.workspace = true
 serde_json.workspace = true
+shell-escape.workspace = true
 snafu.workspace = true
 stackable-operator.workspace = true
 strum.workspace = true