Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/indexer monitoring #377

Merged
merged 13 commits into from
Oct 2, 2023
Merged

Feat/indexer monitoring #377

Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
7628eb4
update monitor script to print log
starknetdev Oct 2, 2023
4644c78
edit
starknetdev Oct 2, 2023
d034f60
edit
starknetdev Oct 2, 2023
723ab35
edit
starknetdev Oct 2, 2023
5acc857
fix
starknetdev Oct 2, 2023
f7ce37e
edit
starknetdev Oct 2, 2023
a3401d6
edit
starknetdev Oct 2, 2023
9ac0c5a
edit
starknetdev Oct 2, 2023
15b8860
force reset to check works
starknetdev Oct 2, 2023
738096c
edit
starknetdev Oct 2, 2023
d5fd737
edit
starknetdev Oct 2, 2023
f457409
edit
starknetdev Oct 2, 2023
ca36da1
fix
starknetdev Oct 2, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 8 additions & 5 deletions indexer/restart-compose.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#!/bin/bash

# Path to your docker-compose.yml file
COMPOSE_PATH="/indexer/goerli-compose.yml"
COMPOSE_PATH="goerli-compose.yaml"

# Container names (as specified in docker-compose.yml)
CONTAINER_NAMES=("indexer-adventurers_indexer-1" "indexer-battles_indexer-1" "indexer-beasts_indexer-1" "indexer-discoveries_indexer-1" "indexer-items_indexer-1" "indexer-scores_indexer-1")
Expand All @@ -12,13 +12,16 @@ INTERVAL=300
# Check if a block has been processed in the last minute
check_container_activity() {
local CONTAINER_NAME="$1"
LAST_LOG=$(docker logs "$CONTAINER_NAME" 2>&1 | grep "Processed block at" | tail -1)
LAST_TIMESTAMP=$(echo "$LAST_LOG" | awk '{print $1}' | tr -d 'T')
LAST_LOG=$(docker logs "$CONTAINER_NAME" 2>&1 | tail -1) # Assuming you want the last log entry
echo "Extracted log: $LAST_LOG" # Print timestamp for verification
LAST_TIMESTAMP=$(echo "$LAST_LOG" | sed -E 's/\x1B\[[0-9;]*[JKmsu]//g' | cut -d' ' -f1)

echo "Extracted timestamp: $LAST_TIMESTAMP" # Print timestamp for verification

# Convert timestamp to epoch (seconds since 1970)
LAST_EPOCH=$(date -d "$LAST_TIMESTAMP" +%s)
CURRENT_EPOCH=$(date +%s)
echo "Current time difference: $((CURRENT_EPOCH - LAST_EPOCH)) seconds"

# If the difference is more than the interval, return 1 (unhealthy)
if (( CURRENT_EPOCH - LAST_EPOCH > INTERVAL )); then
Expand All @@ -32,8 +35,8 @@ while true; do
for CONTAINER_NAME in "${CONTAINER_NAMES[@]}"; do
if ! check_container_activity "$CONTAINER_NAME"; then
echo "Container $CONTAINER_NAME seems to be stuck. Restarting..."
# docker-compose -f $COMPOSE_PATH stop "$CONTAINER_NAME"
# docker-compose -f $COMPOSE_PATH up -d "$CONTAINER_NAME"
docker-compose -f $COMPOSE_PATH stop "$CONTAINER_NAME"
docker-compose -f $COMPOSE_PATH up -d "$CONTAINER_NAME"
# # Add a notification mechanism here if needed
fi
done
Expand Down