[Mithril] script updates (#1785)

## Description 1. Changes to the nginx mithril relay LB and squid mithril relay * Nginx mithril relay LB: * Uses stream protocol * proxy_connect_timeout of 10 seconds * max_fails set to 1 * fail_timeout set to 10 seconds per relay (20 seconds for 2 relay configurations, 30 for 3, etc.) * Squid mithril relay: * Change to how ACL's and allow rulesare written for multiple block producers * Includes rules for non block producers (other relays, local relay IP, simplifies testing from other than block producer) etc. 2. Enhances the verify_signer_registered function * Continues to check if mithril-signer has registered in the current epoch. * If the signer has registered for the current epoch it will also check if the signer registered two epochs earlier and reports if the signer is valid to sign certificates in the current epoch or not. 3. Adds a new MITHRIL_HOME variable to the `scripts/cnode-helper-scripts/env` file. * Defaults to the current path of `${CNODE_HOME}/mithril`. * Provides the SPO a way to set a unique path for where mithril environment and mithril data-stores will be located. 4. Changes to a status code for checks on mithril minimum versions. Sourced from past conversation w/ @Scitz0 5. Adds functions **semantic_version_check** and **check_mithril_upgrade_safe** * GitHub Actions workflow now uses **check_mithril_upgrade_safe** against the repo `node-latest.txt` to determine if changing `mithril-latest.xt` is acceptable. * Not yet implemented in `guild-deploy.sh`, but the **check_mithril_upgrade_safe** should support both CI Actions and production node/signer environments. Open for implementation discussions. ## Where to start? I'd suggest review of each commit vs. reviewing the files changed. The commit for **Refactor mithril functions into mithril library.** will be quite large. There is a general section of functions used by multiple scripts, and then functions are sorted into blocks by the script that uses them, client, relay or signer. Each section is alphabetically sorted. ## Motivation and context 1. Nginx relay/lb * Issues with current nginx config reported in Koios support channel. * Simplified testing of squid proxy when additional IPs are provided. * Offline relays were adding 60 second delays until nginx timeouts occured and LB moved onto the next relay. * Tuned the max fails to 1 and fail_timeout 10 seconds * # of relays. * Temporarily prevents re-using an offline relay if many requests get sent through the same LB 2. Reduce confusion for new signer users who were not aware signing had a 2 epoch delay. 3. Flexibility for SPO's to choose their own directories. 4. Past discussions with @Scitz0 5. Past discussions with @rdlrt ## How has this been tested? 1. Nginx relay/squid * Direct curl requests on the block producer: `curl -4 --proxy http://127.0.0.1:3132 https://aggregator.release-mainnet.api.mithril.network/aggregator` * Enabling tcpdump on port 3132 for all (squid) mithril relays, disabling the first relay in the mithril_relays upstream definition of the (nginx) mithril relay lb. Then timing the request while watching traffic from each mithril relay: `time curl -4 --proxy http://127.0.0.1:3132 https://aggregator.release-mainnet.api.mithril.network/aggregator`. * The first request takes 10.0x seconds, to succeed, subsequent requests take sub 1 second and traffic shows round robin over the remaining online mithril relays. If round robin reaches the offline mithril relay before the fail_timout has expired it gets skipped and the traffic is again sent to the first online (second in the list) upstream mithril relay. 2. Manually 3. Test virtual machines altering path to `/opt/cardano/mithril` 4. Testing locally and in forked repository github actions 5. Testing locally and in forked repository github actions ![image](https://github.com/user-attachments/assets/3838f427-1f38-4808-8188-eb517def0153) --------- Co-authored-by: RdLrT <[email protected]>
cardano-community · Aug 13, 2024 · 52e5f1a · 52e5f1a
1 parent 0479167
commit 52e5f1a
Show file tree

Hide file tree

Showing 8 changed files with 905 additions and 695 deletions.
diff --git a/.github/workflows/mithril-latest.yml b/.github/workflows/mithril-latest.yml
@@ -7,7 +7,7 @@ jobs:
   get-version:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v4
         with:
           token: ${{ secrets.REPO_SCOPED_TOKEN }}
           fetch-depth: 0
@@ -18,13 +18,19 @@ jobs:
       - name: Assigns release version
         run: |
           VERSION=$(cat ./files/docker/node/release-versions/mithril-latest.txt)
+      - name: Source mithril.library and check upgrade safety
+        run: |
+          set -e
+          . scripts/cnode-helper-scripts/mithril.library workflow
+          check_mithril_upgrade_safe
+          echo "MITHRIL_UPGRADE_SAFE=$MITHRIL_UPGRADE_SAFE" >> $GITHUB_ENV
       - name: Check for modified files
         id: git-check
         run: echo ::set-output name=modified::$([ -z "`git status --porcelain`" ] && echo "false" || echo "true")
       - name: Commit latest release version
-        if: steps.git-check.outputs.modified == 'true'
+        if: steps.git-check.outputs.modified == 'true' && env.MITHRIL_UPGRADE_SAFE == 'Y'
         run: |
           git config --global user.name ${{ secrets.REPO_SCOPED_USER }}
           git config --global user.email ${{ secrets.REPO_SCOPED_EMAIL }}
           git commit -am "New mithril release version ${VERSION}"
-          git push
+          git push
diff --git a/scripts/cnode-helper-scripts/env b/scripts/cnode-helper-scripts/env
@@ -110,6 +110,7 @@
 #CIP0094_POLL_URL="https://raw.githubusercontent.com/cardano-foundation/CIP-0094-polls/main/networks/polls.json" # URL for polls to vote against
 
 #MITHRIL_DOWNLOAD="N"                                   # (Y|N) Download latest Mithril snapshot
+#MITHRIL_HOME="${CNODE_HOME}/mithril"                   # Override default Mithril path
 #MITHRIL_SIGNER_ENABLED="N"                             # (Y|N) Enable the gLiveView Mithril Signer Section
 
 #STRICT_VERSION_CHECK="Y"                               # (Y|N) Restrict operation to supported major.minor version leaving patch version open. If disabled, any version will be accepted (unsupported)
@@ -950,6 +951,7 @@ set_default_vars() {
   [[ -z ${ASSET_POLICY_ID_FILENAME} ]] && ASSET_POLICY_ID_FILENAME="policy.id"
   [[ -z ${CIP0094_POLL_URL} ]] && CIP0094_POLL_URL="https://raw.githubusercontent.com/cardano-foundation/CIP-0094-polls/main/networks/polls.json"
   [[ -z ${MITHRIL_DOWNLOAD} ]] && MITHRIL_DOWNLOAD="N"
+  [[ -z ${MITHRIL_HOME} ]] && MITHRIL_HOME="${CNODE_HOME}/mithril"
   [[ -z ${MITHRIL_SIGNER_ENABLED} ]] && MITHRIL_SIGNER_ENABLED="N"
   [[ -z ${STRICT_VERSION_CHECK} ]] && STRICT_VERSION_CHECK="Y"
   FG_BLACK='\e[30m'

diff --git a/scripts/cnode-helper-scripts/gLiveView.sh b/scripts/cnode-helper-scripts/gLiveView.sh
@@ -778,8 +778,8 @@ unset cpu_now cpu_last
 
 mithrilSignerVars() {
   # mithril.env sourcing needed to have values in ${METRICS_SERVER_IP} and ${METRICS_SERVER_PORT}
-  . ${CNODE_HOME}/mithril/mithril.env
-  signerMetricsEnabled=$(grep -q "ENABLE_METRICS_SERVER=true" ${CNODE_HOME}/mithril/mithril.env && echo "true" || echo "false")
+  . ${MITHRIL_HOME}/mithril.env
+  signerMetricsEnabled=$(grep -q "ENABLE_METRICS_SERVER=true" ${MITHRIL_HOME}/mithril.env && echo "true" || echo "false")
   if [[ "${signerMetricsEnabled}" == "true" ]] ; then
     mithrilSignerMetrics=$(curl -s "http://${METRICS_SERVER_IP}:${METRICS_SERVER_PORT}/metrics" 2>/dev/null | grep -v -E "HELP|TYPE" | sed 's/mithril_signer_//g')
     SIGNER_METRICS_HTTP_RESPONSE=$(curl --write-out "%{http_code}" --silent --output /dev/null --connect-timeout 2 http://${METRICS_SERVER_IP}:${METRICS_SERVER_PORT}/metrics)

diff --git a/scripts/cnode-helper-scripts/guild-deploy.sh b/scripts/cnode-helper-scripts/guild-deploy.sh
@@ -112,6 +112,7 @@ set_defaults() {
   [[ -z "${CARDANO_NODE_VERSION}" ]] && CARDANO_NODE_VERSION="$(curl -sfk "https://raw.githubusercontent.com/${G_ACCOUNT}/guild-operators/${BRANCH}/files/docker/node/release-versions/cardano-node-latest.txt")"
   CNODE_HOME="${CNODE_PATH}/${CNODE_NAME}"
   CNODE_VNAME=$(echo "$CNODE_NAME" | awk '{print toupper($0)}')
+  [[ -z ${MITHRIL_HOME} ]] && MITHRIL_HOME="${CNODE_HOME}/mithril"
   REPO="https://github.com/${G_ACCOUNT}/guild-operators"
   REPO_RAW="https://raw.githubusercontent.com/${G_ACCOUNT}/guild-operators"
   URL_RAW="${REPO_RAW}/${BRANCH}"
@@ -407,9 +408,7 @@ download_cnodebins() {
   if [[ "${SKIP_DBSYNC_DOWNLOAD}" == "N" ]]; then
     echo -e "\n  Downloading Cardano DB Sync archive created from GitHub.."
 
-    # TODO: Replace CI Build artifact against 13.2.0.2 tag with release from github artefacts once available
-    #curl -m 200 -sfL https://github.com/IntersectMBO/cardano-db-sync/releases/download/13.2.0.2/cardano-db-sync-13.2.0.1-linux.tar.gz -o cnodedbsync.tar.gz || err_exit "  Could not download cardano-db-sync release 13.2.0.2 from GitHub!"
-    curl -m 200 -sfL https://github.com/IntersectMBO/cardano-db-sync/releases/download/13.3.0.0/cardano-db-sync-13.3.0.0-linux.tar.gz -o cnodedbsync.tar.gz || err_exit "  Could not download cardano-db-sync release 13.3.0.0 from GitHub!"
+    curl -m 200 -sfL https://github.com/IntersectMBO/cardano-db-sync/releases/download/13.4.0.0/cardano-db-sync-13.4.0.0-linux.tar.gz -o cnodedbsync.tar.gz || err_exit "  Could not download cardano-db-sync release 13.3.0.0 from GitHub!"
     tar zxf cnodedbsync.tar.gz --strip-components 1 ./cardano-db-sync &>/dev/null
     [[ -f cardano-db-sync ]] || err_exit " cardano-db-sync archive downloaded but binary (cardano-db-sync) not found after extracting package!"
     rm -f cnodedbsync.tar.gz
@@ -588,7 +587,7 @@ setup_folder() {
     echo -e "\nexport ${CNODE_VNAME}_HOME=${CNODE_HOME}" >> "${HOME}"/.bashrc
   fi
 
-  $sudo mkdir -p "${CNODE_HOME}"/files "${CNODE_HOME}"/db "${CNODE_HOME}"/guild-db "${CNODE_HOME}"/logs "${CNODE_HOME}"/scripts "${CNODE_HOME}"/scripts/archive "${CNODE_HOME}"/sockets "${CNODE_HOME}"/priv "${CNODE_HOME}"/mithril/data-stores
+  $sudo mkdir -p "${CNODE_HOME}"/files "${CNODE_HOME}"/db "${CNODE_HOME}"/guild-db "${CNODE_HOME}"/logs "${CNODE_HOME}"/scripts "${CNODE_HOME}"/scripts/archive "${CNODE_HOME}"/sockets "${CNODE_HOME}"/priv "${MITHRIL_HOME}"/data-stores
   $sudo chown -R "$U_ID":"$G_ID" "${CNODE_HOME}" 2>/dev/null
 
 }

diff --git a/scripts/cnode-helper-scripts/mithril-client.sh b/scripts/cnode-helper-scripts/mithril-client.sh
@@ -48,216 +48,100 @@ EOF
 SKIP_UPDATE=N
 [[ $1 = "-u" ]] && export SKIP_UPDATE=Y && shift
 
-## mithril environment subcommands
-
-environment_override() {
-  local var_to_override="$1"
-  local new_value="$2"
-  local env_file="${CNODE_HOME}/mithril/mithril.env"
-
-  # Check if the variable exists in the environment file
-  if ! grep -q "^${var_to_override}=" "$env_file"; then
-    echo "Error: Variable $var_to_override does not exist in $env_file" >&2
-    return 1
-  fi
-
-  # Use sed to replace the variable's value in the environment file
-  sed -i "s|^${var_to_override}=.*|${var_to_override}=${new_value}|" "$env_file"
-}
-
-mithril_init() {
-  [[ ! -f "${CNODE_HOME}"/mithril/mithril.env ]] && generate_environment_file
-  . "${CNODE_HOME}"/mithril/mithril.env
-}
-
-
-check_db_dir() {
-  # If the DB directory does not exist then set DOWNLOAD_SNAPSHOT to Y
-  if [[ ! -d "${DB_DIRECTORY}" ]]; then
-    echo "INFO: The db directory does not exist.."
-    DOWNLOAD_SNAPSHOT="Y"
-  # If the DB directory is empty then set DOWNLOAD_SNAPSHOT to Y
-  elif [[ -d "${DB_DIRECTORY}" ]] && [[ -z "$(ls -A "${DB_DIRECTORY}")" ]] && [[ $(du -cs "${DB_DIRECTORY}"/* 2>/dev/null | awk '/total$/ {print $1}') -eq 0 ]]; then
-    echo "INFO: The db directory is empty.."
-    DOWNLOAD_SNAPSHOT="Y"
-  else
-    echo "INFO: The db directory is not empty, skipping Cardano DB download.."
-  fi
-}
-
-cleanup_db_directory() {
-  echo "WARNING: Download failure, cleaning up DB directory.."
-  # Safety check to prevent accidental deletion of system files
-  if [[ -z "${DB_DIRECTORY}" ]]; then
-    echo "ERROR: DB_DIRECTORY is unset or null."
-  elif [[ -n "${DB_DIRECTORY}" && "${DB_DIRECTORY}" != "/" && "${DB_DIRECTORY}" != "${CNODE_HOME}" ]]; then
-    # :? Safety check to prevent accidental deletion of system files, even though initial if block should already prevent this
-    rm -rf "${DB_DIRECTORY:?}/"*
-  else
-    echo "INFO: Skipping cleanup of DB directory: ${DB_DIRECTORY}."
-  fi
-}
-
-## mithril snapshot subcommands
-
-download_snapshot() {
-  if [[ "${DOWNLOAD_SNAPSHOT}" == "Y" ]]; then
-    echo "INFO: Downloading latest mithril snapshot.."
-    trap 'cleanup_db_directory' INT
-    if ! "${MITHRILBIN}" -v --aggregator-endpoint ${AGGREGATOR_ENDPOINT} cardano-db download --download-dir $(dirname ${DB_DIRECTORY}) --genesis-verification-key ${GENESIS_VERIFICATION_KEY} ${SNAPSHOT_DIGEST} ; then
-      cleanup_db_directory
-      exit 1
-    fi
-  else
-    echo "INFO: Skipping Cardano DB download.."
-  fi
-}
-
-list_snapshots() {
-  local json_flag=""
-
-  for arg in "$@"; do
-    if [[ $arg == "json" ]]; then
-      json_flag="--json"
-    fi
-  done
-
-  "${MITHRILBIN}" -v --aggregator-endpoint ${AGGREGATOR_ENDPOINT} cardano-db snapshot list $json_flag
-}
-
-show_snapshot() {
-  local digest=""
-  local json_flag=""
-
-  for arg in "$@"; do
-    if [[ $arg == "json" ]]; then
-      json_flag="--json"
-    else
-      digest="$arg"
-    fi
-  done
-
-  if [[ -z $digest ]]; then
-    echo "ERROR: Snapshot digest is required for the 'show' subcommand" >&2
-    exit 1
-  fi
-
-  "${MITHRILBIN}" -v --aggregator-endpoint ${AGGREGATOR_ENDPOINT} cardano-db snapshot show $digest $json_flag
-}
-
-## mithril-stake-distribution subcommands
-
-download_stake_distribution() {
-  if [[ "${DOWNLOAD_STAKE_DISTRIBUTION}" == "Y" ]]; then
-    echo "INFO: Downloading latest mithril stake distribution.."
-    "${MITHRILBIN}" -v --aggregator-endpoint ${AGGREGATOR_ENDPOINT} mithril-stake-distribution download --download-dir "${CNODE_HOME}/mithril/" ${STAKE_DISTRIBUTION_DIGEST}
-  else
-    echo "INFO: Skipping stake distribution download.."
-  fi
-}
-
-list_stake_distributions() {
-  local json_flag=""
-
-  for arg in "$@"; do
-    if [[ $arg == "json" ]]; then
-      json_flag="--json"
-    fi
-  done
-
-  "${MITHRILBIN}" -v --aggregator-endpoint ${AGGREGATOR_ENDPOINT} mithril-stake-distribution list $json_flag
-
-}
 
 #####################
 # Execution/Main    #
 #####################
 
-. "$(dirname $0)"/mithril.library
-
-update_check "$@"
-
-set_defaults
+function main() {
+  . "$(dirname $0)"/mithril.library
+
+  update_check "$@"
+
+  set_defaults
+
+  # Parse command line options
+  case $1 in
+    environment)
+      case $2 in
+        setup)
+          generate_environment_file
+          ;;
+        override)
+          environment_override $3 $4
+          ;;
+        update)
+          export UPDATE_ENVIRONMENT="Y"
+          generate_environment_file
+          ;;
+        *)
+          echo "Invalid environment subcommand: $2" >&2
+          usage
+          exit 1
+          ;;
+      esac
+      ;;
+    cardano-db)
+      mithril_init client
+      case $2 in
+        download)
+          check_db_dir
+          download_snapshot
+          ;;
+        snapshot)
+          case $3 in
+            list)
+              case $4 in
+                json)
+                  list_snapshots json
+                  ;;
+                *)
+                  list_snapshots
+                  ;;
+              esac
+              ;;
+            show)
+              show_snapshot $4 $5
+              ;;
+            *)
+              echo "Invalid snapshot subcommand: $3" >&2
+              usage
+              exit 1
+              ;;
+          esac
+      esac
+      ;;
+    stake-distribution)
+      mithril_init client
+      case $2 in
+        download)
+          download_stake_distribution
+          ;;
+        list)
+          case $3 in
+            json)
+              list_stake_distributions json
+              ;;
+            *)
+              list_stake_distributions
+              ;;
+          esac
+          ;;
+        *)
+          echo "Invalid mithril-stake-distribution subcommand: $2" >&2
+          usage
+          exit 1
+          ;;
+      esac
+      ;;
+    *)
+      echo "Invalid $(basename "$0") command: $1" >&2
+      usage
+      exit 1
+      ;;
+  esac
 
-# Parse command line options
-case $1 in
-  environment)
-    case $2 in
-      setup)    
-        generate_environment_file
-        ;;
-      override)
-        environment_override $3 $4
-        ;;
-      update)
-        export UPDATE_ENVIRONMENT="Y"
-        generate_environment_file
-        ;;
-      *)
-        echo "Invalid environment subcommand: $2" >&2
-        usage
-        exit 1
-        ;;
-    esac
-    ;;
-  cardano-db)
-    mithril_init
-    case $2 in
-      download)
-        check_db_dir
-        download_snapshot
-        ;;
-      snapshot)
-        case $3 in 
-          list)
-            case $4 in
-              json)
-                list_snapshots json
-                ;;
-              *)
-                list_snapshots
-                ;;
-            esac
-            ;;
-          show)
-            show_snapshot $4 $5
-            ;;
-          *)
-            echo "Invalid snapshot subcommand: $3" >&2
-            usage
-            exit 1
-            ;;
-        esac
-    esac
-    ;;
-  stake-distribution)
-    mithril_init
-    case $2 in
-      download)
-        download_stake_distribution
-        ;;
-      list)
-        case $3 in
-          json)
-            list_stake_distributions json
-            ;;
-          *)
-            list_stake_distributions
-            ;;
-        esac
-        ;;
-      *)
-        echo "Invalid mithril-stake-distribution subcommand: $2" >&2
-        usage
-        exit 1
-        ;;
-    esac
-    ;;
-  *)
-    echo "Invalid $(basename "$0") command: $1" >&2
-    usage
-    exit 1
-    ;;
-esac
+  exit 0
+}
 
-exit 0
+main "$@"