[Mithril] script updates (#1785)

## Description
1. Changes to the nginx mithril relay LB and squid mithril relay
   * Nginx mithril relay LB:
     * Uses stream protocol
     * proxy_connect_timeout of 10 seconds
     * max_fails set to 1
* fail_timeout set to 10 seconds per relay (20 seconds for 2 relay
configurations, 30 for 3, etc.)
   * Squid mithril relay:
* Change to how ACL's and allow rulesare written for multiple block
producers
* Includes rules for non block producers (other relays, local relay IP,
simplifies testing from other than block producer) etc.
2. Enhances the verify_signer_registered function
* Continues to check if mithril-signer has registered in the current
epoch.
* If the signer has registered for the current epoch it will also check
if the signer registered two epochs earlier and reports if the signer is
valid to sign certificates in the current epoch or not.
3. Adds a new MITHRIL_HOME variable to the
`scripts/cnode-helper-scripts/env` file.
   * Defaults to the current path of `${CNODE_HOME}/mithril`.
* Provides the SPO a way to set a unique path for where mithril
environment and mithril data-stores will be located.
4. Changes to a status code for checks on mithril minimum versions.
Sourced from past conversation w/ @Scitz0
5. Adds functions **semantic_version_check** and
**check_mithril_upgrade_safe**
* GitHub Actions workflow now uses **check_mithril_upgrade_safe**
against the repo `node-latest.txt` to determine if changing
`mithril-latest.xt` is acceptable.
* Not yet implemented in `guild-deploy.sh`, but the
**check_mithril_upgrade_safe** should support both CI Actions and
production node/signer environments. Open for implementation
discussions.

## Where to start?
I'd suggest review of each commit vs. reviewing the files changed. The
commit for **Refactor mithril functions into mithril library.** will be
quite large. There is a general section of functions used by multiple
scripts, and then functions are sorted into blocks by the script that
uses them, client, relay or signer. Each section is alphabetically
sorted.

## Motivation and context
1. Nginx relay/lb
 * Issues with current nginx config reported in Koios support channel.
 * Simplified testing of squid proxy when additional IPs are provided.
* Offline relays were adding 60 second delays until nginx timeouts
occured and LB moved onto the next relay.
 * Tuned the max fails to 1 and fail_timeout 10 seconds * # of relays.
* Temporarily prevents re-using an offline relay if many requests get
sent through the same LB
2. Reduce confusion for new signer users who were not aware signing had
a 2 epoch delay.
3. Flexibility for SPO's to choose their own directories.
4. Past discussions with @Scitz0 
5. Past discussions with @rdlrt 

## How has this been tested?
1. Nginx relay/squid
   * Direct curl requests on the block producer:
`curl -4 --proxy http://127.0.0.1:3132
https://aggregator.release-mainnet.api.mithril.network/aggregator`

* Enabling tcpdump on port 3132 for all (squid) mithril relays,
disabling the first relay in the mithril_relays upstream definition of
the (nginx) mithril relay lb. Then timing the request while watching
traffic from each mithril relay:
`time curl -4 --proxy http://127.0.0.1:3132
https://aggregator.release-mainnet.api.mithril.network/aggregator`.

* The first request takes 10.0x seconds, to succeed, subsequent requests
take sub 1 second and traffic shows round robin over the remaining
online mithril relays. If round robin reaches the offline mithril relay
before the fail_timout has expired it gets skipped and the traffic is
again sent to the first online (second in the list) upstream mithril
relay.

2. Manually
3. Test virtual machines altering path to `/opt/cardano/mithril`
4. Testing locally and in forked repository github actions
5. Testing locally and in forked repository github actions

![image](https://github.com/user-attachments/assets/3838f427-1f38-4808-8188-eb517def0153)

---------

Co-authored-by: RdLrT <[email protected]>

.github/workflows/mithril-latest.yml

@@ -7,7 +7,7 @@ jobs:
   get-version:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v4
         with:
           token: ${{ secrets.REPO_SCOPED_TOKEN }}
           fetch-depth: 0
@@ -18,13 +18,19 @@ jobs:
       - name: Assigns release version
         run: |
           VERSION=$(cat ./files/docker/node/release-versions/mithril-latest.txt)
+      - name: Source mithril.library and check upgrade safety
+        run: |
+          set -e
+          . scripts/cnode-helper-scripts/mithril.library workflow
+          check_mithril_upgrade_safe
+          echo "MITHRIL_UPGRADE_SAFE=$MITHRIL_UPGRADE_SAFE" >> $GITHUB_ENV
       - name: Check for modified files
         id: git-check
         run: echo ::set-output name=modified::$([ -z "`git status --porcelain`" ] && echo "false" || echo "true")
       - name: Commit latest release version
-        if: steps.git-check.outputs.modified == 'true'
+        if: steps.git-check.outputs.modified == 'true' && env.MITHRIL_UPGRADE_SAFE == 'Y'
         run: |
           git config --global user.name ${{ secrets.REPO_SCOPED_USER }}
           git config --global user.email ${{ secrets.REPO_SCOPED_EMAIL }}
           git commit -am "New mithril release version ${VERSION}"
-          git push
+          git push

scripts/cnode-helper-scripts/env

@@ -110,6 +110,7 @@
 #CIP0094_POLL_URL="https://raw.githubusercontent.com/cardano-foundation/CIP-0094-polls/main/networks/polls.json" # URL for polls to vote against
 
 #MITHRIL_DOWNLOAD="N"                                   # (Y|N) Download latest Mithril snapshot
+#MITHRIL_HOME="${CNODE_HOME}/mithril"                   # Override default Mithril path
 #MITHRIL_SIGNER_ENABLED="N"                             # (Y|N) Enable the gLiveView Mithril Signer Section
 
 #STRICT_VERSION_CHECK="Y"                               # (Y|N) Restrict operation to supported major.minor version leaving patch version open. If disabled, any version will be accepted (unsupported)
@@ -950,6 +951,7 @@ set_default_vars() {
   [[ -z ${ASSET_POLICY_ID_FILENAME} ]] && ASSET_POLICY_ID_FILENAME="policy.id"
   [[ -z ${CIP0094_POLL_URL} ]] && CIP0094_POLL_URL="https://raw.githubusercontent.com/cardano-foundation/CIP-0094-polls/main/networks/polls.json"
   [[ -z ${MITHRIL_DOWNLOAD} ]] && MITHRIL_DOWNLOAD="N"
+  [[ -z ${MITHRIL_HOME} ]] && MITHRIL_HOME="${CNODE_HOME}/mithril"
   [[ -z ${MITHRIL_SIGNER_ENABLED} ]] && MITHRIL_SIGNER_ENABLED="N"
   [[ -z ${STRICT_VERSION_CHECK} ]] && STRICT_VERSION_CHECK="Y"
   FG_BLACK='\e[30m'

scripts/cnode-helper-scripts/gLiveView.sh

@@ -778,8 +778,8 @@ unset cpu_now cpu_last
 
 mithrilSignerVars() {
   # mithril.env sourcing needed to have values in ${METRICS_SERVER_IP} and ${METRICS_SERVER_PORT}
-  . ${CNODE_HOME}/mithril/mithril.env
-  signerMetricsEnabled=$(grep -q "ENABLE_METRICS_SERVER=true" ${CNODE_HOME}/mithril/mithril.env && echo "true" || echo "false")
+  . ${MITHRIL_HOME}/mithril.env
+  signerMetricsEnabled=$(grep -q "ENABLE_METRICS_SERVER=true" ${MITHRIL_HOME}/mithril.env && echo "true" || echo "false")
   if [[ "${signerMetricsEnabled}" == "true" ]] ; then
     mithrilSignerMetrics=$(curl -s "http://${METRICS_SERVER_IP}:${METRICS_SERVER_PORT}/metrics" 2>/dev/null | grep -v -E "HELP|TYPE" | sed 's/mithril_signer_//g')
     SIGNER_METRICS_HTTP_RESPONSE=$(curl --write-out "%{http_code}" --silent --output /dev/null --connect-timeout 2 http://${METRICS_SERVER_IP}:${METRICS_SERVER_PORT}/metrics)

scripts/cnode-helper-scripts/guild-deploy.sh

@@ -112,6 +112,7 @@ set_defaults() {
   [[ -z "${CARDANO_NODE_VERSION}" ]] && CARDANO_NODE_VERSION="$(curl -sfk "https://raw.githubusercontent.com/${G_ACCOUNT}/guild-operators/${BRANCH}/files/docker/node/release-versions/cardano-node-latest.txt")"
   CNODE_HOME="${CNODE_PATH}/${CNODE_NAME}"
   CNODE_VNAME=$(echo "$CNODE_NAME" | awk '{print toupper($0)}')
+  [[ -z ${MITHRIL_HOME} ]] && MITHRIL_HOME="${CNODE_HOME}/mithril"
   REPO="https://github.com/${G_ACCOUNT}/guild-operators"
   REPO_RAW="https://raw.githubusercontent.com/${G_ACCOUNT}/guild-operators"
   URL_RAW="${REPO_RAW}/${BRANCH}"
@@ -588,7 +589,7 @@ setup_folder() {
     echo -e "\nexport ${CNODE_VNAME}_HOME=${CNODE_HOME}" >> "${HOME}"/.bashrc
   fi
 
-  $sudo mkdir -p "${CNODE_HOME}"/files "${CNODE_HOME}"/db "${CNODE_HOME}"/guild-db "${CNODE_HOME}"/logs "${CNODE_HOME}"/scripts "${CNODE_HOME}"/scripts/archive "${CNODE_HOME}"/sockets "${CNODE_HOME}"/priv "${CNODE_HOME}"/mithril/data-stores
+  $sudo mkdir -p "${CNODE_HOME}"/files "${CNODE_HOME}"/db "${CNODE_HOME}"/guild-db "${CNODE_HOME}"/logs "${CNODE_HOME}"/scripts "${CNODE_HOME}"/scripts/archive "${CNODE_HOME}"/sockets "${CNODE_HOME}"/priv "${MITHRIL_HOME}"/data-stores
   $sudo chown -R "$U_ID":"$G_ID" "${CNODE_HOME}" 2>/dev/null
 
 }

scripts/cnode-helper-scripts/mithril-client.sh

@@ -48,216 +48,100 @@ EOF
 SKIP_UPDATE=N
 [[ $1 = "-u" ]] && export SKIP_UPDATE=Y && shift
 
-## mithril environment subcommands
-
-environment_override() {
-  local var_to_override="$1"
-  local new_value="$2"
-  local env_file="${CNODE_HOME}/mithril/mithril.env"
-
-  # Check if the variable exists in the environment file
-  if ! grep -q "^${var_to_override}=" "$env_file"; then
-    echo "Error: Variable $var_to_override does not exist in $env_file" >&2
-    return 1
-  fi
-
-  # Use sed to replace the variable's value in the environment file
-  sed -i "s|^${var_to_override}=.*|${var_to_override}=${new_value}|" "$env_file"
-}
-
-mithril_init() {
-  [[ ! -f "${CNODE_HOME}"/mithril/mithril.env ]] && generate_environment_file
-  . "${CNODE_HOME}"/mithril/mithril.env
-}
-
-
-check_db_dir() {
-  # If the DB directory does not exist then set DOWNLOAD_SNAPSHOT to Y
-  if [[ ! -d "${DB_DIRECTORY}" ]]; then
-    echo "INFO: The db directory does not exist.."
-    DOWNLOAD_SNAPSHOT="Y"
-  # If the DB directory is empty then set DOWNLOAD_SNAPSHOT to Y
-  elif [[ -d "${DB_DIRECTORY}" ]] && [[ -z "$(ls -A "${DB_DIRECTORY}")" ]] && [[ $(du -cs "${DB_DIRECTORY}"/* 2>/dev/null | awk '/total$/ {print $1}') -eq 0 ]]; then
-    echo "INFO: The db directory is empty.."
-    DOWNLOAD_SNAPSHOT="Y"
-  else
-    echo "INFO: The db directory is not empty, skipping Cardano DB download.."
-  fi
-}
-
-cleanup_db_directory() {
-  echo "WARNING: Download failure, cleaning up DB directory.."
-  # Safety check to prevent accidental deletion of system files
-  if [[ -z "${DB_DIRECTORY}" ]]; then
-    echo "ERROR: DB_DIRECTORY is unset or null."
-  elif [[ -n "${DB_DIRECTORY}" && "${DB_DIRECTORY}" != "/" && "${DB_DIRECTORY}" != "${CNODE_HOME}" ]]; then
-    # :? Safety check to prevent accidental deletion of system files, even though initial if block should already prevent this
-    rm -rf "${DB_DIRECTORY:?}/"*
-  else
-    echo "INFO: Skipping cleanup of DB directory: ${DB_DIRECTORY}."
-  fi
-}
-
-## mithril snapshot subcommands
-
-download_snapshot() {
-  if [[ "${DOWNLOAD_SNAPSHOT}" == "Y" ]]; then
-    echo "INFO: Downloading latest mithril snapshot.."
-    trap 'cleanup_db_directory' INT
-    if ! "${MITHRILBIN}" -v --aggregator-endpoint ${AGGREGATOR_ENDPOINT} cardano-db download --download-dir $(dirname ${DB_DIRECTORY}) --genesis-verification-key ${GENESIS_VERIFICATION_KEY} ${SNAPSHOT_DIGEST} ; then
-      cleanup_db_directory
-      exit 1
-    fi
-  else
-    echo "INFO: Skipping Cardano DB download.."
-  fi
-}
-
-list_snapshots() {
-  local json_flag=""
-
-  for arg in "$@"; do
-    if [[ $arg == "json" ]]; then
-      json_flag="--json"
-    fi
-  done
-
-  "${MITHRILBIN}" -v --aggregator-endpoint ${AGGREGATOR_ENDPOINT} cardano-db snapshot list $json_flag
-}
-
-show_snapshot() {
-  local digest=""
-  local json_flag=""
-
-  for arg in "$@"; do
-    if [[ $arg == "json" ]]; then
-      json_flag="--json"
-    else
-      digest="$arg"
-    fi
-  done
-
-  if [[ -z $digest ]]; then
-    echo "ERROR: Snapshot digest is required for the 'show' subcommand" >&2
-    exit 1
-  fi
-
-  "${MITHRILBIN}" -v --aggregator-endpoint ${AGGREGATOR_ENDPOINT} cardano-db snapshot show $digest $json_flag
-}
-
-## mithril-stake-distribution subcommands
-
-download_stake_distribution() {
-  if [[ "${DOWNLOAD_STAKE_DISTRIBUTION}" == "Y" ]]; then
-    echo "INFO: Downloading latest mithril stake distribution.."
-    "${MITHRILBIN}" -v --aggregator-endpoint ${AGGREGATOR_ENDPOINT} mithril-stake-distribution download --download-dir "${CNODE_HOME}/mithril/" ${STAKE_DISTRIBUTION_DIGEST}
-  else
-    echo "INFO: Skipping stake distribution download.."
-  fi
-}
-
-list_stake_distributions() {
-  local json_flag=""
-
-  for arg in "$@"; do
-    if [[ $arg == "json" ]]; then
-      json_flag="--json"
-    fi
-  done
-
-  "${MITHRILBIN}" -v --aggregator-endpoint ${AGGREGATOR_ENDPOINT} mithril-stake-distribution list $json_flag
-
-}
 
 #####################
 # Execution/Main    #
 #####################
 
-. "$(dirname $0)"/mithril.library
-
-update_check "$@"
-
-set_defaults
+function main() {
+  . "$(dirname $0)"/mithril.library
+
+  update_check "$@"
+
+  set_defaults
+
+  # Parse command line options
+  case $1 in
+    environment)
+      case $2 in
+        setup)
+          generate_environment_file
+          ;;
+        override)
+          environment_override $3 $4
+          ;;
+        update)
+          export UPDATE_ENVIRONMENT="Y"
+          generate_environment_file
+          ;;
+        *)
+          echo "Invalid environment subcommand: $2" >&2
+          usage
+          exit 1
+          ;;
+      esac
+      ;;
+    cardano-db)
+      mithril_init client
+      case $2 in
+        download)
+          check_db_dir
+          download_snapshot
+          ;;
+        snapshot)
+          case $3 in
+            list)
+              case $4 in
+                json)
+                  list_snapshots json
+                  ;;
+                *)
+                  list_snapshots
+                  ;;
+              esac
+              ;;
+            show)
+              show_snapshot $4 $5
+              ;;
+            *)
+              echo "Invalid snapshot subcommand: $3" >&2
+              usage
+              exit 1
+              ;;
+          esac
+      esac
+      ;;
+    stake-distribution)
+      mithril_init client
+      case $2 in
+        download)
+          download_stake_distribution
+          ;;
+        list)
+          case $3 in
+            json)
+              list_stake_distributions json
+              ;;
+            *)
+              list_stake_distributions
+              ;;
+          esac
+          ;;
+        *)
+          echo "Invalid mithril-stake-distribution subcommand: $2" >&2
+          usage
+          exit 1
+          ;;
+      esac
+      ;;
+    *)
+      echo "Invalid $(basename "$0") command: $1" >&2
+      usage
+      exit 1
+      ;;
+  esac
 
-# Parse command line options
-case $1 in
-  environment)
-    case $2 in
-      setup)    
-        generate_environment_file
-        ;;
-      override)
-        environment_override $3 $4
-        ;;
-      update)
-        export UPDATE_ENVIRONMENT="Y"
-        generate_environment_file
-        ;;
-      *)
-        echo "Invalid environment subcommand: $2" >&2
-        usage
-        exit 1
-        ;;
-    esac
-    ;;
-  cardano-db)
-    mithril_init
-    case $2 in
-      download)
-        check_db_dir
-        download_snapshot
-        ;;
-      snapshot)
-        case $3 in 
-          list)
-            case $4 in
-              json)
-                list_snapshots json
-                ;;
-              *)
-                list_snapshots
-                ;;
-            esac
-            ;;
-          show)
-            show_snapshot $4 $5
-            ;;
-          *)
-            echo "Invalid snapshot subcommand: $3" >&2
-            usage
-            exit 1
-            ;;
-        esac
-    esac
-    ;;
-  stake-distribution)
-    mithril_init
-    case $2 in
-      download)
-        download_stake_distribution
-        ;;
-      list)
-        case $3 in
-          json)
-            list_stake_distributions json
-            ;;
-          *)
-            list_stake_distributions
-            ;;
-        esac
-        ;;
-      *)
-        echo "Invalid mithril-stake-distribution subcommand: $2" >&2
-        usage
-        exit 1
-        ;;
-    esac
-    ;;
-  *)
-    echo "Invalid $(basename "$0") command: $1" >&2
-    usage
-    exit 1
-    ;;
-esac
+  exit 0
+}
 
-exit 0
+main "$@"