https://issues.redhat.com/browse/ACM-13050 improving the topic for 2.…

…11 (#7322)

troubleshooting/acm_thanos_compactor.adoc

@@ -1,22 +1,59 @@
 [#troubleshooting-thanos-compactor]
-= Troubleshooting a block error for Thanos compactor
+= Troubleshooting Thanos compactor halts
 
-You might receive a block error message that indicates that the block for Thanos compactor is corrupted.
+You might receive an error message that the compactor is halted. This can occur when there are corrupted blocks or when there is insufficient space on the Thanos compactor persistent volume claim (PVC).
 
 [#symptom-thanos-compactor]
-== Symptom: Block error for Thanos compactor
+== Symptom: Thanos compactor halts
 
-After you upgrade {acm} and check the logs for the Thanos compactor by using the `oc logs observability-thanos-compact-0` command, the logs display the following error message:
+The Thanos compactor halts because there is no space left on your persistent volume claim (PVC). You receive the following message:
 
+[source,terminal]
 ----
-ts=2024-01-24T15:34:51.948653839Z caller=compact.go:491 level=error msg="critical error detected; halting" err="compaction: group 0@15699422364132557315: compact blocks [/var/thanos/compact/compact/0@15699422364132557315/01HKZGQGJCKQWF3XMA8EXAMPLE /var/thanos/compact/compact/0@15699422364132557315/01HKZQK7TD06J2XWGR5EXAMPLE /var/thanos/compact/compact/0@15699422364132557315/01HKZYEZ2DVDQXF1STVEXAMPLE /var/thanos/compact/compact/0@15699422364132557315/01HM05APAHXBQSNC0N5EXAMPLE]: populate block: chunk iter: cannot populate chunk 8 from block 01HKZYEZ2DVDQXF1STVEXAMPLE: segment index 0 out of range"
+ts=2024-01-24T15:34:51.948653839Z caller=compact.go:491 level=error msg="critical error detected; halting" err="compaction: group 0@5827190780573537664: compact blocks [ /var/thanos/compact/compact/0@15699422364132557315/01HKZGQGJCKQWF3XMA8EXAMPLE]: 2 errors: populate block: add series: write series data: write /var/thanos/compact/compact/0@15699422364132557315/01HKZGQGJCKQWF3XMA8EXAMPLE.tmp-for-creation/index: no space left on device; write /var/thanos/compact/compact/0@15699422364132557315/01HKZGQGJCKQWF3XMA8EXAMPLE.tmp-for-creation/index: no space left on device"
 ----
 
 [#resolving-thanos-compactor]
-== Resolving the problem: Add the _thanos bucket verify_ command
+== Resolving the problem: Thanos compactor halts
+
+To resolve the problem, increase the storage space of the Thanos compactor PVC. Complete the following steps:
+
+
+. Increase the storage space for the `data-observability-thanos-compact-0` PVC. See link:../observability/customize_observability.adoc#increase-decrease-pv-pvc[Increasing and decreasing persistent volumes and persistent volume claims] for more information.
+
+
+. Restart the `observability-thanos-compact` pod by deleting the pod. The new pod is automatically created and started.
+
++
+[source,bash]
+----
+oc delete pod observability-thanos-compact-0 -n open-cluster-management-observability
+----
+
+. After you restart the `observability-thanos-compact` pod, check the `acm_thanos_compact_todo_compactions` metric. As the Thanos compactor works through the backlog, the metric value decreases. 
+
+. Confirm that the metric changes in a consistent cycle and check the disk usage. Then you can reattempt to decrease the PVC again.
+
++
+*Note:* This might take several weeks. 
+
+[#symptom-thanos-compactor-two]
+== Symptom: Thanos compactor halts
+
+The Thanos compactor halts because you have corrupted blocks. You might receive the following output where the `01HKZYEZ2DVDQXF1STVEXAMPLE` block is corrupted:
+
++
+[source,terminal]
+----
+ts=2024-01-24T15:34:51.948653839Z caller=compact.go:491 level=error msg="critical error detected; halting" err="compaction: group 0@15699422364132557315: compact blocks [/var/thanos/compact/compact/0@15699422364132557315/01HKZGQGJCKQWF3XMA8EXAMPLE /var/thanos/compact/compact/0@15699422364132557315/01HKZQK7TD06J2XWGR5EXAMPLE /var/thanos/compact/compact/0@15699422364132557315/01HKZYEZ2DVDQXF1STVEXAMPLE /var/thanos/compact/compact/0@15699422364132557315/01HM05APAHXBQSNC0N5EXAMPLE]: populate block: chunk iter: cannot populate chunk 8 from block 01HKZYEZ2DVDQXF1STVEXAMPLE: segment index 0 out of range"
+----
+
+[#resolving-thanos-compactor-two]
+== Resolving the problem: Thanos compactor halts
 
 Add the `thanos bucket verify` command to the object storage configuration. Complete the following steps:
 
+
 . Resolve the block error by adding the `thanos bucket verify` command to the object storage configuration. Set the configuration in the `observability-thanos-compact` pod by using the following commands:
 
 +
@@ -35,7 +72,7 @@ thanos tools bucket verify -r --objstore.config="$OBJSTORE_CONFIG" --objstore-ba
 thanos tools bucket mark --id "01HKZYEZ2DVDQXF1STVEXAMPLE" --objstore.config="$OBJSTORE_CONFIG" --marker=deletion-mark.json --details=DELETE 
 ----
 
-. If you blocked for deletion, clean up the marked blocks by running the following command:
+. If you are blocked for deletion, clean up the marked blocks by running the following command:
 
 +
 [source,bash]