Merge pull request #274 from cisagov/v23.08.0_merge_cisagov

Malcolm v23.08.0 is a minor release with a few improvements, bug fixes and component updates. v23.07.1...v23.08.0 * Features and enhancements + Rewrote the [Network Traffic Artifact Upload](https://cisagov.github.io/Malcolm/docs/upload.html#Upload) interface and backend, replacing the defunct [jQuery-File-Upload](https://github.com/blueimp/jQuery-File-Upload) with [FilePond](https://pqina.nl/filepond/). This was mainly due to jQuery-File-Upload no longer receiving security fixes and having some known vulnerabilities. see idaholab#235 + Use [netbox-initializers](https://github.com/tobiasge/netbox-initializers) plugin, adding the ability to drop YAML files for various NetBox obects to be [preloaded](https://cisagov.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPreload) at startup. see idaholab#228 + handle changes to ICSNPP parsers with source_ip/destination_ip fields (idaholab#233 and idaholab#226) * Bug fixes + Fixed extracting Malcolm version during ISO build + Workaround for wireshark no longer publishing raw manuf (OUI) list (idaholab#230) + Remove news feed from default NetBox dashboard (as it would try to reach out to the web for RSS updates) * Component version updates + Rebased Docker and ISO images to Debian 12 (bookworm) + live-build tool for building ISO images to debian/1%20230131 + Arkime to [v4.4.0](https://github.com/arkime/arkime/blob/6f667600596e8a2252555640933f424730c258d5/CHANGELOG#L33-L55) + supercronic to [v0.2.26](https://github.com/aptible/supercronic/releases/tag/v0.2.26) + FileBeat to [v8.9.0](https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-8.9.0.html) + LogStash to [v8.9.0](https://www.elastic.co/guide/en/logstash/8.9/logstash-8-9-0.html) (idaholab#234) + NetBox to [v3.5.7](https://github.com/netbox-community/netbox/releases/tag/v3.5.7) + PostgreSQL (used by NetBox) to [v15](https://www.postgresql.org/docs/release/15.0/) + opensearch-py to [v2.3.0](https://github.com/opensearch-project/opensearch-py/releases/tag/v2.3.0) + PHP (as used by Upload interface) to [v8.2](https://www.php.net/ChangeLog-8.php#PHP_8_2) + Fluent Bit to [v2.1.8](https://github.com/fluent/fluent-bit/releases/tag/v2.1.8) + certifi to [v2023.7.22](https://github.com/certifi/python-certifi/releases/tag/2023.07.22) (idaholab#229)
cisagov · Aug 15, 2023 · d5ee03e · d5ee03e
2 parents 06b9562 + bfd945c
commit d5ee03e
Show file tree

Hide file tree

Showing 157 changed files with 1,880 additions and 1,905 deletions.
diff --git a/.github/workflows/malcolm-iso-build-docker-wrap-push-ghcr.yml b/.github/workflows/malcolm-iso-build-docker-wrap-push-ghcr.yml
@@ -70,12 +70,12 @@ jobs:
                virt-what \
                xorriso \
                xz-utils
-          git clone --depth=1 --single-branch --recurse-submodules --shallow-submodules --branch='debian/1%20210407' 'https://salsa.debian.org/live-team/live-build.git' /tmp/live-build
+          git clone --depth=1 --single-branch --recurse-submodules --shallow-submodules --branch='debian/1%20230131' 'https://salsa.debian.org/live-team/live-build.git' /tmp/live-build
           cd /tmp/live-build
           dpkg-buildpackage -b -uc -us
           cd /tmp
           sudo dpkg -i /tmp/live-build*.deb
-          rm -rf /tmp/live-build
+          sudo rm -rf /tmp/live-build /tmp/live-build*.deb
       -
         name: Checkout
         uses: actions/checkout@v3
@@ -92,7 +92,7 @@ jobs:
       -
         name: Extract Malcolm version
         shell: bash
-        run: echo "mversion=$(grep -P "^\s+image:\s*malcolm" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
         id: extract_malcolm_version
       -
         name: Build image

diff --git a/.github/workflows/sensor-iso-build-docker-wrap-push-ghcr.yml b/.github/workflows/sensor-iso-build-docker-wrap-push-ghcr.yml
@@ -66,10 +66,12 @@ jobs:
                squashfs-tools \
                virt-what \
                xorriso
-            git clone --depth=1 --single-branch --recurse-submodules --shallow-submodules --branch='debian/1%20210407' 'https://salsa.debian.org/live-team/live-build.git' /tmp/live-build
-            cd /tmp/live-build
-            dpkg-buildpackage -b -uc -us
-            sudo dpkg -i /tmp/live-build*.deb
+          git clone --depth=1 --single-branch --recurse-submodules --shallow-submodules --branch='debian/1%20230131' 'https://salsa.debian.org/live-team/live-build.git' /tmp/live-build
+          cd /tmp/live-build
+          dpkg-buildpackage -b -uc -us
+          cd /tmp
+          sudo dpkg -i /tmp/live-build*.deb
+          sudo rm -rf /tmp/live-build /tmp/live-build*.deb
       -
         name: Checkout
         uses: actions/checkout@v3
@@ -86,7 +88,7 @@ jobs:
       -
         name: Extract Malcolm version
         shell: bash
-        run: echo "mversion=$(grep -P "^\s+image:\s*malcolm" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
         id: extract_malcolm_version
       -
         name: Build image

diff --git a/Dockerfiles/api.Dockerfile b/Dockerfiles/api.Dockerfile
@@ -1,4 +1,4 @@
-FROM python:3-slim-bullseye as builder
+FROM python:3-slim-bookworm as builder
 
 ENV DEBIAN_FRONTEND noninteractive
 ENV TERM xterm
@@ -8,8 +8,8 @@ ENV PYTHONUNBUFFERED 1
 RUN    apt-get update -q \
     && apt-get -y -q upgrade \
     && apt-get install -y gcc \
-    && python3 -m pip install --upgrade pip \
-    && python3 -m pip install flake8
+    && python3 -m pip install --break-system-packages --no-cache-dir --upgrade pip \
+    && python3 -m pip install --break-system-packages --no-cache-dir flake8
 
 COPY ./api /usr/src/app/
 COPY scripts/malcolm_utils.py /usr/src/app/
@@ -18,7 +18,7 @@ WORKDIR /usr/src/app
 RUN python3 -m pip wheel --no-cache-dir --no-deps --wheel-dir /usr/src/app/wheels -r requirements.txt \
     && flake8 --ignore=E203,E501,F401,W503
 
-FROM python:3-slim-bullseye
+FROM python:3-slim-bookworm
 
 # Copyright (c) 2023 Battelle Energy Alliance, LLC.  All rights reserved.
 LABEL maintainer="[email protected]"
@@ -79,7 +79,7 @@ COPY --from=ghcr.io/mmguero-dev/gostatic --chmod=755 /goStatic /usr/bin/goStatic
 
 RUN    apt-get -q update \
     && apt-get -y -q --no-install-recommends upgrade \
-    && apt-get -y -q --no-install-recommends install curl netcat rsync tini \
+    && apt-get -y -q --no-install-recommends install curl netcat-openbsd rsync tini \
     && python3 -m pip install --upgrade pip \
     && python3 -m pip install --no-cache /wheels/* \
     && groupadd --gid ${DEFAULT_GID} ${PGROUP} \

diff --git a/Dockerfiles/arkime.Dockerfile b/Dockerfiles/arkime.Dockerfile
@@ -1,10 +1,10 @@
-FROM debian:11-slim AS build
+FROM debian:12-slim AS build
 
 # Copyright (c) 2023 Battelle Energy Alliance, LLC.  All rights reserved.
 
 ENV DEBIAN_FRONTEND noninteractive
 
-ENV ARKIME_VERSION "v4.3.2"
+ENV ARKIME_VERSION "v4.4.0"
 ENV ARKIME_DIR "/opt/arkime"
 ENV ARKIME_URL "https://github.com/arkime/arkime.git"
 ENV ARKIME_LOCALELASTICSEARCH no
@@ -33,6 +33,7 @@ RUN apt-get -q update && \
         libkrb5-dev \
         libmaxminddb-dev \
         libpcap0.8-dev \
+        libpcre3-dev \
         libssl-dev \
         libtool \
         libwww-perl \
@@ -49,7 +50,7 @@ RUN apt-get -q update && \
         swig \
         wget \
         zlib1g-dev && \
-  pip3 install --no-cache-dir beautifulsoup4 && \
+  python3 -m pip install --break-system-packages --no-cache-dir beautifulsoup4 meson && \
   cd /opt && \
     git clone --recurse-submodules --branch="$ARKIME_VERSION" "$ARKIME_URL" "./arkime-"$ARKIME_VERSION && \
     cd "./arkime-"$ARKIME_VERSION && \
@@ -61,7 +62,7 @@ RUN apt-get -q update && \
     python3 /opt/bs4_remove_div.py -i ./viewer/vueapp/src/components/users/Users.vue -o ./viewer/vueapp/src/components/users/Users.new -c "new-user-form" && \
     mv -vf ./viewer/vueapp/src/components/users/Users.new ./viewer/vueapp/src/components/users/Users.vue && \
     sed -i 's/v-if.*password.*"/v-if="false"/g' ./viewer/vueapp/src/components/settings/Settings.vue && \
-    rm -rf ./viewer/vueapp/src/components/upload ./capture/plugins/suricata* && \
+    rm -rf ./capture/plugins/suricata* && \
     sed -i "s/^\(ARKIME_LOCALELASTICSEARCH=\).*/\1"$ARKIME_LOCALELASTICSEARCH"/" ./release/Configure && \
     sed -i "s/^\(ARKIME_INET=\).*/\1"$ARKIME_INET"/" ./release/Configure && \
     ./easybutton-build.sh && \
@@ -71,7 +72,7 @@ RUN apt-get -q update && \
     rm -f ${ARKIME_DIR}/wiseService/source.* ${ARKIME_DIR}/etc/*.systemd.service && \
     bash -c "file ${ARKIME_DIR}/bin/* ${ARKIME_DIR}/node-v*/bin/* | grep 'ELF 64-bit' | sed 's/:.*//' | xargs -l -r strip -v --strip-unneeded"
 
-FROM debian:11-slim
+FROM debian:12-slim
 
 LABEL maintainer="[email protected]"
 LABEL org.opencontainers.image.authors='[email protected]'
@@ -133,7 +134,7 @@ ENV PCAP_MONITOR_HOST $PCAP_MONITOR_HOST
 
 COPY --from=build $ARKIME_DIR $ARKIME_DIR
 
-RUN sed -i "s/bullseye main/bullseye main contrib non-free/g" /etc/apt/sources.list && \
+RUN sed -i "s/main$/main contrib non-free/g" /etc/apt/sources.list.d/debian.sources && \
     apt-get -q update && \
     apt-get -y -q --no-install-recommends upgrade && \
     apt-get install -q -y --no-install-recommends \
@@ -148,14 +149,14 @@ RUN sed -i "s/bullseye main/bullseye main contrib non-free/g" /etc/apt/sources.l
       libkrb5-3 \
       libmaxminddb0 \
       libpcap0.8 \
-      libssl1.0 \
+      libpcre3 \
+      libssl3 \
       libtool \
       libwww-perl \
       libyaml-0-2 \
       libzmq5 \
       procps \
       psmisc \
-      python \
       python3 \
       python3-pip \
       python3-setuptools \
@@ -164,15 +165,15 @@ RUN sed -i "s/bullseye main/bullseye main contrib non-free/g" /etc/apt/sources.l
       rsync \
       sudo \
       supervisor \
+      tini \
       vim-tiny \
       wget \
-      tini \
       tar gzip unzip cpio bzip2 lzma xz-utils p7zip-full unrar zlib1g && \
-    pip3 install --no-cache-dir beautifulsoup4 pyzmq watchdog && \
+    python3 -m pip install --break-system-packages --no-cache-dir beautifulsoup4 pyzmq watchdog && \
     ln -sfr $ARKIME_DIR/bin/npm /usr/local/bin/npm && \
       ln -sfr $ARKIME_DIR/bin/node /usr/local/bin/node && \
       ln -sfr $ARKIME_DIR/bin/npx /usr/local/bin/npx && \
-    apt-get -q -y --purge remove gcc gcc-10 cpp cpp-10 libssl-dev && \
+    apt-get -q -y --purge remove gcc gcc-12 cpp cpp-12 libssl-dev && \
       apt-get -q -y autoremove && \
       apt-get clean && \
       rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
@@ -206,7 +207,7 @@ RUN [ ${#MAXMIND_GEOIP_DB_LICENSE_KEY} -gt 1 ] && for DB in ASN Country City; do
       rm -f "GeoLite2-$DB*"; \
     done; \
   curl -s -S -L -o $ARKIME_DIR/etc/ipv4-address-space.csv "https://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.csv" && \
-  curl -s -S -L -o $ARKIME_DIR/etc/oui.txt "https://raw.githubusercontent.com/wireshark/wireshark/master/manuf"
+  curl -s -S -L -o $ARKIME_DIR/etc/oui.txt "https://gitlab.com/wireshark/wireshark/raw/release-4.0/manuf"
 
 RUN groupadd --gid $DEFAULT_GID $PGROUP && \
     useradd -M --uid $DEFAULT_UID --gid $DEFAULT_GID --home $ARKIME_DIR $PUSER && \

diff --git a/Dockerfiles/dashboards-helper.Dockerfile b/Dockerfiles/dashboards-helper.Dockerfile
@@ -47,10 +47,10 @@ ENV DASHBOARDS_URL $DASHBOARDS_URL
 ENV DASHBOARDS_DARKMODE $DASHBOARDS_DARKMODE
 ENV PATH="/data:${PATH}"
 
-ENV SUPERCRONIC_VERSION "0.2.25"
+ENV SUPERCRONIC_VERSION "0.2.26"
 ENV SUPERCRONIC_URL "https://github.com/aptible/supercronic/releases/download/v$SUPERCRONIC_VERSION/supercronic-linux-amd64"
 ENV SUPERCRONIC "supercronic-linux-amd64"
-ENV SUPERCRONIC_SHA1SUM "642f4f5a2b67f3400b5ea71ff24f18c0a7d77d49"
+ENV SUPERCRONIC_SHA1SUM "7a79496cf8ad899b99a719355d4db27422396735"
 ENV SUPERCRONIC_CRONTAB "/etc/crontab"
 
 ENV ECS_RELEASES_URL "https://api.github.com/repos/elastic/ecs/releases/latest"

diff --git a/Dockerfiles/dashboards.Dockerfile b/Dockerfiles/dashboards.Dockerfile
@@ -128,8 +128,8 @@ RUN yum upgrade -y && \
     /usr/share/opensearch-dashboards/bin/opensearch-dashboards-plugin install file:///tmp/kbnSankeyVis.zip --allow-root && \
     cd /tmp && \
         # unzip transformVis.zip opensearch-dashboards/transformVis/opensearch_dashboards.json opensearch-dashboards/transformVis/package.json && \
-        # sed -i "s/2\.7\.0/2\.8\.0/g" opensearch-dashboards/transformVis/opensearch_dashboards.json && \
-        # sed -i "s/2\.7\.0/2\.8\.0/g" opensearch-dashboards/transformVis/package.json && \
+        # sed -i "s/2\.9\.0/2\.9\.0/g" opensearch-dashboards/transformVis/opensearch_dashboards.json && \
+        # sed -i "s/2\.9\.0/2\.9\.0/g" opensearch-dashboards/transformVis/package.json && \
         # zip transformVis.zip opensearch-dashboards/transformVis/opensearch_dashboards.json opensearch-dashboards/transformVis/package.json && \
         cd /usr/share/opensearch-dashboards/plugins && \
         /usr/share/opensearch-dashboards/bin/opensearch-dashboards-plugin install file:///tmp/transformVis.zip --allow-root && \

diff --git a/Dockerfiles/file-monitor.Dockerfile b/Dockerfiles/file-monitor.Dockerfile
@@ -1,4 +1,4 @@
-FROM debian:11-slim
+FROM debian:12-slim
 
 # Copyright (c) 2023 Battelle Energy Alliance, LLC.  All rights reserved.
 LABEL maintainer="[email protected]"
@@ -91,15 +91,15 @@ ENV EXTRACTED_FILE_HTTP_SERVER_ENCRYPT $EXTRACTED_FILE_HTTP_SERVER_ENCRYPT
 ENV EXTRACTED_FILE_HTTP_SERVER_KEY $EXTRACTED_FILE_HTTP_SERVER_KEY
 ENV EXTRACTED_FILE_HTTP_SERVER_PORT $EXTRACTED_FILE_HTTP_SERVER_PORT
 
-ENV SUPERCRONIC_VERSION "0.2.25"
+ENV SUPERCRONIC_VERSION "0.2.26"
 ENV SUPERCRONIC_URL "https://github.com/aptible/supercronic/releases/download/v$SUPERCRONIC_VERSION/supercronic-linux-amd64"
 ENV SUPERCRONIC "supercronic-linux-amd64"
-ENV SUPERCRONIC_SHA1SUM "642f4f5a2b67f3400b5ea71ff24f18c0a7d77d49"
+ENV SUPERCRONIC_SHA1SUM "7a79496cf8ad899b99a719355d4db27422396735"
 ENV SUPERCRONIC_CRONTAB "/etc/crontab"
 
 COPY --chmod=755 shared/bin/yara_rules_setup.sh /usr/local/bin/
 
-RUN sed -i "s/bullseye main/bullseye main contrib non-free/g" /etc/apt/sources.list && \
+RUN sed -i "s/main$/main contrib non-free/g" /etc/apt/sources.list.d/debian.sources && \
     apt-get -q update && \
     apt-get -y -q --no-install-recommends upgrade && \
     apt-get install --no-install-recommends -y -q \
@@ -112,13 +112,13 @@ RUN sed -i "s/bullseye main/bullseye main contrib non-free/g" /etc/apt/sources.l
       gcc \
       git \
       jq \
-      libclamunrar9 \
+      libclamunrar11 \
       libjansson-dev \
       libjansson4 \
       libmagic-dev \
       libmagic1 \
       libssl-dev \
-      libssl1.1 \
+      libssl3 \
       libtool \
       make \
       pkg-config \
@@ -135,7 +135,7 @@ RUN sed -i "s/bullseye main/bullseye main contrib non-free/g" /etc/apt/sources.l
       python3-requests \
       python3-zmq \
       rsync && \
-    pip3 install clamd supervisor yara-python python-magic psutil pycryptodome watchdog && \
+    python3 -m pip install --break-system-packages --no-cache-dir clamd supervisor yara-python python-magic psutil pycryptodome watchdog && \
     curl -fsSLO "$SUPERCRONIC_URL" && \
       echo "${SUPERCRONIC_SHA1SUM}  ${SUPERCRONIC}" | sha1sum -c - && \
       chmod +x "$SUPERCRONIC" && \
@@ -167,9 +167,9 @@ RUN sed -i "s/bullseye main/bullseye main contrib non-free/g" /etc/apt/sources.l
         automake \
         build-essential \
         gcc \
-        gcc-8 \
+        gcc-12 \
         libc6-dev \
-        libgcc-8-dev \
+        libgcc-12-dev \
         libjansson-dev \
         libmagic-dev \
         libssl-dev \