Replace Debian with Ubuntu 24.04 LTS

README.md

@@ -27,20 +27,28 @@ Notable features:
 
 The underlying spirit of this project is to allow "repeatable deployments", and all pull requests in this direction will be merged post-haste.
 
+## Warning 
+
+As part of our recent efforts to reduce the number of CVEs affecting the Docker images, we recently changed the base image from Debian Bookworm to Ubuntu 24.04.
+
+While the transition did not affect MISP and MISP modules, the GitHub Action triggered a bug affecting `libcurl`  and Ubuntu 24.04 when running on `linux/arm64` and establishing TLS connections to `api.github.com` when the server decides toreturn a 302. The issue is being discussed here https://github.com/curl/curl/issues/14154 and being further investigated here https://bugs.launchpad.net/ubuntu/+source/curl/+bug/2073448.
+
+To allow the build to complete, we temporarily disabled TLS verification (see `core/Dockerfile` when using `composer` to install PHP dependencies; the temporary workaround affects only the build when the target platform is `linux/arm64`, leaving the `linux/amd64` build unaffected.
+
 ## Getting Started
 
 -   Copy the `template.env` to `.env` 
 -   Customize `.env` based on your needs (optional step)
 
 ### Run
 
--   `docker-compose pull` if you want to use pre-built images or `docker-compose build` if you want to build your own (see the `Troubleshooting` section in case of errors)
--   `docker-compose up`
+-   `docker compose pull` if you want to use pre-built images or `docker compose build` if you want to build your own (see the `Troubleshooting` section in case of errors)
+-   `docker compose up`
 -   Login to `https://localhost`
     -   User: `[email protected]`
     -   Password: `admin`
 
-Keeping the image up-to-date with upstream should be as simple as running `docker-compose pull`.
+Keeping the image up-to-date with upstream should be as simple as running `docker compose pull`.
 
 ### Configuration
 
@@ -64,21 +72,21 @@ New options are added on a regular basis.
 
 #### Environment variable behaviour
 
-Set environment variables in .env to configure settings instead of in docker-compose.yml where possible. Setting the variables in .env will allow you to pull updates from Github without issues caused by a modified docker-compose.yml file, should there be an update for it.
+Set environment variables in .env to configure settings instead of in `docker-compose.yml` where possible. Setting the variables in `.env` will allow you to pull updates from Github without issues caused by a modified `docker-compose.yml` file, should there be an update for it.
 
 Environment variable driven settings are enforced every time the misp-core container starts. This means that if you change the config.php file or database for a setting that has a set environment variable, it will be changed to the environment variable value upon next container start. Empty environment variables may have a safe default which is enforced instead.
 
-If you push a change to add or remove an environment variable, please look in "core/files/etc/misp-docker/" for json files with "envars" in the name and adjust there.
+If you push a change to add or remove an environment variable, please look in `core/files/etc/misp-docker/` for json files with "envars" in the name and adjust there.
 
 #### Unset safe default settings behaviour
 
-The misp-core container has definitions for minimum safe default settings which are set if needed each time the container starts. They will only be set if there is no existing entry in the config.php file or database for these settings. If you specify a custom value for any of these settings it will be respected. See the definitions of these in "core/files/etc/misp-docker" where the filenames contain the word "defaults".
+The misp-core container has definitions for minimum safe default settings which are set if needed each time the container starts. They will only be set if there is no existing entry in the `config.php` file or database for these settings. If you specify a custom value for any of these settings it will be respected. See the definitions of these in `core/files/etc/misp-docker` where the filenames contain the word "defaults".
 
 #### Storing system settings in the DB
 
-This container includes the "ENABLE_DB_SETTINGS" environment variable, which can be used to set "MISP.system_setting_db" to true or false. This changes the behaviour of where MISP chooses to store operator made settings changes; in config.php or in the system_settings database table. By default this is set to false.
+This container includes the `ENABLE_DB_SETTINGS` environment variable, which can be used to set `MISP.system_setting_db` to true or false. This changes the behaviour of where MISP chooses to store operator made settings changes; in `config.php` or in the system_settings database table. By default this is set to false.
 
-If a setting is not defined in the DB, but is defined in config.php, it will be read out of config.php and used. This can sometimes lead to operator confusion, so please check both locations for values when troubleshooting.
+If a setting is not defined in the DB, but is defined in `config.php`, it will be read out of `config.php` and used. This can sometimes lead to operator confusion, so please check both locations for values when troubleshooting.
 
 If you change this setting from false to true, settings are not migrated from config.php to the database, but rather the above behaviour is relied upon.
 
@@ -88,7 +96,7 @@ While storing system settings in the DB works as expected most of the time, you
 
 If you are trying to accomplish something and the above behaviours get in the way, please let us know as this is not intended.
 
-To override these behaviours edit the docker-compose.yml file's misp-core volume definitions to enable the "customize_misp.sh" behaviour (see the bottom of the Production section for details). The "customize_misp.sh" script triggers after the above behaviours complete and is an appropriate place to override a setting. It is suggested that you use the "/var/www/MISP/app/cake Admin setSetting" command to override a setting, as this tool is config.php file and database setting aware.
+To override these behaviours edit the `docker-compose.yml` file's misp-core volume definitions to enable the `customize_misp.sh` behaviour (see the bottom of the Production section for details). The `customize_misp.sh` script triggers after the above behaviours complete and is an appropriate place to override a setting. It is suggested that you use the `/var/www/MISP/app/cake Admin setSetting` command to override a setting, as this tool is `config.php` file and database setting aware.
 
 #### Adding a new setting and unsure what files to edit?
 
@@ -133,7 +141,6 @@ Custom root CA certificates can be mounted under `/usr/local/share/ca-certificat
 ## Troubleshooting
 
 -   Make sure you run a fairly recent version of Docker and Docker Compose (if in doubt, update following the steps outlined in https://docs.docker.com/engine/install/ubuntu/)
--   Some Linux distributions provide a recent version of Docker but a legacy version of Docker Compose, so you can try running `docker compose` instead of `docker-compose`
 -   Make sure you are not running an old image or container; when in doubt run `docker system prune --volumes` and clone this repository into an empty directory
 
 ## Versioning

core/Dockerfile

@@ -1,20 +1,24 @@
 ARG DOCKER_HUB_PROXY=""
 
 
-FROM "${DOCKER_HUB_PROXY}python:3.12-slim-bookworm" AS php-base
+FROM "${DOCKER_HUB_PROXY}ubuntu:24.04" AS php-base
     ENV DEBIAN_FRONTEND noninteractive
 
     # Uncomment when building in corporate environments
     # COPY ./rootca.crt /usr/local/share/ca-certificates/rootca.pem
     # COPY ./rootca.crt /usr/lib/ssl/cert.pem
 
     RUN apt-get update; apt-get install -y --no-install-recommends \
-        lsb-release \
-        ca-certificates \
-        curl
-    RUN curl -sSLo /tmp/debsuryorg-archive-keyring.deb https://packages.sury.org/debsuryorg-archive-keyring.deb
-    RUN dpkg -i /tmp/debsuryorg-archive-keyring.deb
-    RUN echo "deb [signed-by=/usr/share/keyrings/deb.sury.org-php.gpg] https://packages.sury.org/php/ $(lsb_release -sc) main" > /etc/apt/sources.list.d/php.list
+        ca-certificates
+
+    COPY files/etc/apt/sources.list.d/ondrej-ubuntu-php-noble.sources /etc/apt/sources.list.d/ondrej-ubuntu-php-noble.sources
+    COPY files/etc/apt/sources.list.d/ondrej-ubuntu-nginx-mainline-noble.sources /etc/apt/sources.list.d/ondrej-ubuntu-nginx-mainline-noble.sources
+
+    # RUN apt-get update; apt-get install -y --no-install-recommends \
+    #    software-properties-common
+    #    # && apt-get autoremove -y && apt-get clean -y && rm -rf /var/lib/apt/lists/*
+    # RUN add-apt-repository ppa:ondrej/php
+    # RUN add-apt-repository ppa:ondrej/nginx-mainline
     RUN apt-get update
 
 
@@ -23,6 +27,7 @@ FROM php-base AS composer-build
     ENV COMPOSER_ALLOW_SUPERUSER 1
     ARG CORE_TAG
     ARG CORE_COMMIT
+    ARG TARGETPLATFORM
 
     RUN apt-get install -y --no-install-recommends \
         php7.4 \
@@ -42,10 +47,28 @@ FROM php-base AS composer-build
 
     WORKDIR /tmp
     ADD https://raw.githubusercontent.com/MISP/MISP/${CORE_COMMIT:-${CORE_TAG}}/app/composer.json /tmp
-    COPY --from=composer:latest /usr/bin/composer /usr/bin/composer
-    RUN composer config --no-interaction allow-plugins.composer/installers true
-    RUN composer install
-    RUN composer require --with-all-dependencies --no-interaction \
+    COPY --from=composer:2.7.7 /usr/bin/composer /usr/bin/composer
+
+    # See:
+    #   - https://github.com/curl/curl/issues/14154
+    #   - https://bugs.launchpad.net/ubuntu/+source/curl/+bug/2073448
+    RUN <<-EOF
+        if [ "$TARGETPLATFORM" = "linux/arm64" ]; then
+            cp /usr/bin/composer /composer.phar
+            mkdir /out/
+            php -r '$phar = new Phar("/composer.phar"); $phar->extractTo("/out/");'
+            sed -i "/'verify_peer_name' =>.*/a 'verify_peer_status' => CURLOPT_SSL_VERIFYSTATUS," /out/src/Composer/Util/Http/CurlDownloader.php
+            sed -i "/\$options = StreamContextFactory.*/a \$options['ssl']['verify_peer'] = false;" /out/src/Composer/Util/Http/CurlDownloader.php
+            sed -i "/\$options = StreamContextFactory.*/a \$options['ssl']['verify_peer_name'] = false;" /out/src/Composer/Util/Http/CurlDownloader.php
+            sed -i "/\$options = StreamContextFactory.*/a \$options['ssl']['verify_peer_status'] = false;" /out/src/Composer/Util/Http/CurlDownloader.php
+            rm /usr/bin/composer
+            ln -s /out/bin/composer /usr/bin/composer
+        fi
+EOF
+
+    RUN php /usr/bin/composer config --no-interaction allow-plugins.composer/installers true
+    RUN php /usr/bin/composer install
+    RUN php /usr/bin/composer require --with-all-dependencies --no-interaction \
         supervisorphp/supervisor:^4.0 \
         guzzlehttp/guzzle:^7.4.5 \
         lstrojny/fxmlrpc \
@@ -105,6 +128,7 @@ FROM php-base AS python-build
 
     RUN apt-get install -y --no-install-recommends \
         git \
+        python3-pip \
         && apt-get autoremove -y && apt-get clean -y && rm -rf /var/lib/apt/lists/*
 
     # Download MISP using git in the /var/www/ directory. Remove unnecessary items.
@@ -177,6 +201,7 @@ FROM php-base
         gpg-agent \
         mariadb-client \
         rsync \
+        python3-pip \
         # PHP Requirements
         php7.4 \
         php7.4-apcu \
@@ -195,7 +220,7 @@ FROM php-base
         libldap-common \
         librdkafka1 \
         libbrotli1 \
-        libsimdjson14 \
+        libsimdjson19 \
         libzstd1 \
         ssdeep \
         libfuzzy2 \
@@ -209,8 +234,7 @@ FROM php-base
 
     # Install python modules
     COPY --from=python-build /wheels /wheels
-    RUN pip install --no-cache-dir /wheels/*.whl && rm -rf /wheels
-    RUN pip uninstall -y pip
+    RUN pip install --break-system-packages --no-cache-dir /wheels/*.whl && rm -rf /wheels
 
     # PHP: install prebuilt libraries, then install the app's PHP deps
     COPY --from=php-build ["/usr/lib/php/${PHP_VER}/ssdeep.so", "/usr/lib/php/${PHP_VER}/rdkafka.so", "/usr/lib/php/${PHP_VER}/brotli.so", "/usr/lib/php/${PHP_VER}/simdjson.so", "/usr/lib/php/${PHP_VER}/zstd.so", "/usr/lib/php/${PHP_VER}/"]
@@ -221,6 +245,15 @@ FROM php-base
     COPY --from=composer-build --chown=www-data:www-data --chmod=0550 /tmp/Vendor /var/www/MISP/app/Vendor
     COPY --from=composer-build --chown=www-data:www-data --chmod=0550 /tmp/Plugin /var/www/MISP/app/Plugin
 
+    # python3-setuptools (distutils.version) is needed by 'mixbox'
+    RUN cp /usr/lib/python3/dist-packages/setuptools/_distutils/version.py \
+        /usr/local/lib/python3.12/dist-packages/mixbox/distutils_version.py
+    RUN cp /usr/lib/python3/dist-packages/setuptools/_distutils/version.py \
+        /var/www/MISP/app/files/scripts/mixbox/mixbox/distutils_version.py
+    RUN sed -i 's/from distutils\.version/from mixbox.distutils_version/' /usr/local/lib/python3.12/dist-packages/mixbox/parser.py
+    RUN sed -i 's/from distutils\.version/from mixbox.distutils_version/' /var/www/MISP/app/files/scripts/mixbox/mixbox/parser.py
+    RUN apt-get remove --purge python3-pip python3-setuptools -y
+
     # Gather these in one layer, only act on actual directories under /etc/php/
     RUN <<-EOF
         set -- "ssdeep" "rdkafka" "brotli" "simdjson" "zstd"

core/files/etc/apt/sources.list.d/ondrej-ubuntu-nginx-mainline-noble.sources

@@ -0,0 +1,24 @@
+Types: deb
+URIs: https://ppa.launchpadcontent.net/ondrej/nginx-mainline/ubuntu/
+Suites: noble
+Components: main
+Signed-By:
+ -----BEGIN PGP PUBLIC KEY BLOCK-----
+ .
+ mI0ESX35nAEEALKDCUDVXvmW9n+T/+3G1DnTpoWh9/1xNaz/RrUH6fQKhHr568F8
+ hfnZP/2CGYVYkW9hxP9LVW9IDvzcmnhgIwK+ddeaPZqh3T/FM4OTA7Q78HSvR81m
+ Jpf2iMLm/Zvh89ZsmP2sIgZuARiaHo8lxoTSLtmKXsM3FsJVlusyewHfABEBAAG0
+ H0xhdW5jaHBhZCBQUEEgZm9yIE9uZMWZZWogU3Vyw72ItgQTAQIAIAUCSX35nAIb
+ AwYLCQgHAwIEFQIIAwQWAgMBAh4BAheAAAoJEE9OoKrlJnpsQjYD/jW1NlIFAlT6
+ EvF2xfVbkhERii9MapjaUsSso4XLCEmZdEGX54GQ01svXnrivwnd/kmhKvyxCqiN
+ LDY/dOaK8MK//bDI6mqdKmG8XbP2vsdsxhifNC+GH/OwaDPvn1TyYB653kwyruCG
+ FjEnCreZTcRUu2oBQyolORDl+BmF4DjLiQEzBBABCgAdFiEECvaBvTqO/UqmWMI/
+ thEcm0xImQEFAmXTV0AACgkQthEcm0xImQGTTggAhuMHGeBZlRUAsZE7jJM7Mf06
+ /WIhcgUfBfSFnJFlFH+xdEe/GFYyVk9kingDsPh90Ecnt4n8DJHTlsuUV1+SPBIO
+ JfbQTUjx1n/+Ck+TVKzRByvrpRXtiZQ214m3zbhZpme2eBBMItZByjG7g925NUIq
+ rL+R5ZoEcZvVlYscfsA0Sr8yJTsGJPefuLYI6eJkNDa1QkzBkSSW4XaCfNIxNBRs
+ zN/qGe3xy0bibOaC4T2TcbZPSAVP855ahNbLAdqkyfAutiEWcKZmQpR9qNh4482k
+ 0pXVlQJ8UB860gVFHjwjFm/MsCeX8yfeAi38ZyInWL2OSG2pDx5ZzNESwnCPIg==
+ =N1rh
+ -----END PGP PUBLIC KEY BLOCK-----
+

core/files/etc/apt/sources.list.d/ondrej-ubuntu-php-noble.sources

@@ -0,0 +1,23 @@
+Types: deb
+URIs: https://ppa.launchpadcontent.net/ondrej/php/ubuntu/
+Suites: noble
+Components: main
+Signed-By: -----BEGIN PGP PUBLIC KEY BLOCK-----
+ .
+ mI0ESX35nAEEALKDCUDVXvmW9n+T/+3G1DnTpoWh9/1xNaz/RrUH6fQKhHr568F8
+ hfnZP/2CGYVYkW9hxP9LVW9IDvzcmnhgIwK+ddeaPZqh3T/FM4OTA7Q78HSvR81m
+ Jpf2iMLm/Zvh89ZsmP2sIgZuARiaHo8lxoTSLtmKXsM3FsJVlusyewHfABEBAAG0
+ H0xhdW5jaHBhZCBQUEEgZm9yIE9uZMWZZWogU3Vyw72ItgQTAQIAIAUCSX35nAIb
+ AwYLCQgHAwIEFQIIAwQWAgMBAh4BAheAAAoJEE9OoKrlJnpsQjYD/jW1NlIFAlT6
+ EvF2xfVbkhERii9MapjaUsSso4XLCEmZdEGX54GQ01svXnrivwnd/kmhKvyxCqiN
+ LDY/dOaK8MK//bDI6mqdKmG8XbP2vsdsxhifNC+GH/OwaDPvn1TyYB653kwyruCG
+ FjEnCreZTcRUu2oBQyolORDl+BmF4DjLiQEzBBABCgAdFiEECvaBvTqO/UqmWMI/
+ thEcm0xImQEFAmXTV0AACgkQthEcm0xImQGTTggAhuMHGeBZlRUAsZE7jJM7Mf06
+ /WIhcgUfBfSFnJFlFH+xdEe/GFYyVk9kingDsPh90Ecnt4n8DJHTlsuUV1+SPBIO
+ JfbQTUjx1n/+Ck+TVKzRByvrpRXtiZQ214m3zbhZpme2eBBMItZByjG7g925NUIq
+ rL+R5ZoEcZvVlYscfsA0Sr8yJTsGJPefuLYI6eJkNDa1QkzBkSSW4XaCfNIxNBRs
+ zN/qGe3xy0bibOaC4T2TcbZPSAVP855ahNbLAdqkyfAutiEWcKZmQpR9qNh4482k
+ 0pXVlQJ8UB860gVFHjwjFm/MsCeX8yfeAi38ZyInWL2OSG2pDx5ZzNESwnCPIg==
+ =N1rh
+ -----END PGP PUBLIC KEY BLOCK-----
+

core/files/etc/nginx/sites-available/misp443

@@ -1,6 +1,7 @@
 server {
-    listen 443 ssl http2;
-    listen [::]:443 ssl http2;
+    listen 443 ssl;
+    listen [::]:443 ssl;
+    http2  on;
 
     # disable access logs
     access_log off;

modules/Dockerfile

@@ -1,6 +1,6 @@
 ARG DOCKER_HUB_PROXY=""
 
-FROM "${DOCKER_HUB_PROXY}python:3.12-slim-bookworm" AS python-build
+FROM "${DOCKER_HUB_PROXY}ubuntu:24.04" AS python-build
     ENV DEBIAN_FRONTEND noninteractive
     ARG MODULES_TAG
     ARG MODULES_COMMIT
@@ -9,6 +9,10 @@ FROM "${DOCKER_HUB_PROXY}python:3.12-slim-bookworm" AS python-build
     RUN apt-get update && apt-get install -y --no-install-recommends \
         cmake \
         git \
+        python3-dev \
+        python3-pip \
+        python3-wheel \
+        pipenv \
         build-essential \
         libpoppler-cpp-dev \
         libfuzzy-dev \
@@ -30,7 +34,6 @@ FROM "${DOCKER_HUB_PROXY}python:3.12-slim-bookworm" AS python-build
 EOF
 
     WORKDIR /srv/misp-modules
-    RUN pip install pipenv
     RUN pipenv requirements > requirements.txt
     RUN pip wheel -r requirements.txt --no-cache-dir -w /wheels/
 
@@ -56,7 +59,7 @@ EOF
     RUN rm -rf /srv/faup
 
 
-FROM "${DOCKER_HUB_PROXY}python:3.12-slim-bookworm"
+FROM "${DOCKER_HUB_PROXY}ubuntu:24.04"
     ENV DEBIAN_FRONTEND noninteractive
 
     RUN apt-get update && apt-get install -y --no-install-recommends \
@@ -69,18 +72,19 @@ FROM "${DOCKER_HUB_PROXY}python:3.12-slim-bookworm"
         libxml2 \
         libxslt1.1  \
         libzbar0 \
+        python3-pip \
         && apt-get autoremove -y && apt-get clean -y && rm -rf /var/lib/apt/lists/*
 
     COPY --from=python-build /wheels /wheels
     COPY --from=python-build /usr/local/lib/libfaupl* /usr/local/lib/
     RUN ldconfig
-    RUN pip install --no-cache-dir --use-deprecated=legacy-resolver /wheels/*.whl && rm -rf /wheels
-    RUN pip uninstall -y pip
+    RUN pip install --break-system-packages --no-cache-dir --use-deprecated=legacy-resolver /wheels/*.whl && rm -rf /wheels
+    RUN apt-get remove --purge python3-pip python3-setuptools -y
 
     # Since we compile faup ourselves and lua is not required anymore, we can load our own library 
     #   and skip the pre-compiled blob to improve compatibility with other architectures like ARM
     RUN sed -i s/LoadLibrary\(LOAD_LIB\)/LoadLibrary\(\"\\/usr\\/local\\/lib\\/libfaupl.so\"\)/ \
-        /usr/local/lib/python3.12/site-packages/pyfaup/__init__.py
+        /usr/local/lib/python3.12/dist-packages/pyfaup/__init__.py
 
     # Disable (all) warnings raised when using 'future'
     RUN sed -i '/import sys/a import warnings\nwarnings.warn = lambda *args, **kwargs: None' \