From a644672f9e0347069befc6cd5e3f6cb3ff4f07fa Mon Sep 17 00:00:00 2001 From: Stefano Ortolani Date: Fri, 12 Jul 2024 17:41:37 +0100 Subject: [PATCH] Replace Debian with Ubuntu 24.04 LTS --- README.md | 27 +++++--- core/Dockerfile | 61 ++++++++++++++----- ...ondrej-ubuntu-nginx-mainline-noble.sources | 24 ++++++++ .../ondrej-ubuntu-php-noble.sources | 23 +++++++ core/files/etc/nginx/sites-available/misp443 | 5 +- modules/Dockerfile | 16 +++-- 6 files changed, 124 insertions(+), 32 deletions(-) create mode 100644 core/files/etc/apt/sources.list.d/ondrej-ubuntu-nginx-mainline-noble.sources create mode 100644 core/files/etc/apt/sources.list.d/ondrej-ubuntu-php-noble.sources diff --git a/README.md b/README.md index 7577857..2327a22 100644 --- a/README.md +++ b/README.md @@ -27,6 +27,14 @@ Notable features: The underlying spirit of this project is to allow "repeatable deployments", and all pull requests in this direction will be merged post-haste. +## Warning + +As part of our recent efforts to reduce the number of CVEs affecting the Docker images, we recently changed the base image from Debian Bookworm to Ubuntu 24.04. + +While the transition did not affect MISP and MISP modules, the GitHub Action triggered a bug affecting `libcurl` and Ubuntu 24.04 when running on `linux/arm64` and establishing TLS connections to `api.github.com` when the server decides toreturn a 302. The issue is being discussed here https://github.com/curl/curl/issues/14154 and being further investigated here https://bugs.launchpad.net/ubuntu/+source/curl/+bug/2073448. + +To allow the build to complete, we temporarily disabled TLS verification (see `core/Dockerfile` when using `composer` to install PHP dependencies; the temporary workaround affects only the build when the target platform is `linux/arm64`, leaving the `linux/amd64` build unaffected. + ## Getting Started - Copy the `template.env` to `.env` @@ -34,13 +42,13 @@ The underlying spirit of this project is to allow "repeatable deployments", and ### Run -- `docker-compose pull` if you want to use pre-built images or `docker-compose build` if you want to build your own (see the `Troubleshooting` section in case of errors) -- `docker-compose up` +- `docker compose pull` if you want to use pre-built images or `docker compose build` if you want to build your own (see the `Troubleshooting` section in case of errors) +- `docker compose up` - Login to `https://localhost` - User: `admin@admin.test` - Password: `admin` -Keeping the image up-to-date with upstream should be as simple as running `docker-compose pull`. +Keeping the image up-to-date with upstream should be as simple as running `docker compose pull`. ### Configuration @@ -64,21 +72,21 @@ New options are added on a regular basis. #### Environment variable behaviour -Set environment variables in .env to configure settings instead of in docker-compose.yml where possible. Setting the variables in .env will allow you to pull updates from Github without issues caused by a modified docker-compose.yml file, should there be an update for it. +Set environment variables in .env to configure settings instead of in `docker-compose.yml` where possible. Setting the variables in `.env` will allow you to pull updates from Github without issues caused by a modified `docker-compose.yml` file, should there be an update for it. Environment variable driven settings are enforced every time the misp-core container starts. This means that if you change the config.php file or database for a setting that has a set environment variable, it will be changed to the environment variable value upon next container start. Empty environment variables may have a safe default which is enforced instead. -If you push a change to add or remove an environment variable, please look in "core/files/etc/misp-docker/" for json files with "envars" in the name and adjust there. +If you push a change to add or remove an environment variable, please look in `core/files/etc/misp-docker/` for json files with "envars" in the name and adjust there. #### Unset safe default settings behaviour -The misp-core container has definitions for minimum safe default settings which are set if needed each time the container starts. They will only be set if there is no existing entry in the config.php file or database for these settings. If you specify a custom value for any of these settings it will be respected. See the definitions of these in "core/files/etc/misp-docker" where the filenames contain the word "defaults". +The misp-core container has definitions for minimum safe default settings which are set if needed each time the container starts. They will only be set if there is no existing entry in the `config.php` file or database for these settings. If you specify a custom value for any of these settings it will be respected. See the definitions of these in `core/files/etc/misp-docker` where the filenames contain the word "defaults". #### Storing system settings in the DB -This container includes the "ENABLE_DB_SETTINGS" environment variable, which can be used to set "MISP.system_setting_db" to true or false. This changes the behaviour of where MISP chooses to store operator made settings changes; in config.php or in the system_settings database table. By default this is set to false. +This container includes the `ENABLE_DB_SETTINGS` environment variable, which can be used to set `MISP.system_setting_db` to true or false. This changes the behaviour of where MISP chooses to store operator made settings changes; in `config.php` or in the system_settings database table. By default this is set to false. -If a setting is not defined in the DB, but is defined in config.php, it will be read out of config.php and used. This can sometimes lead to operator confusion, so please check both locations for values when troubleshooting. +If a setting is not defined in the DB, but is defined in `config.php`, it will be read out of `config.php` and used. This can sometimes lead to operator confusion, so please check both locations for values when troubleshooting. If you change this setting from false to true, settings are not migrated from config.php to the database, but rather the above behaviour is relied upon. @@ -88,7 +96,7 @@ While storing system settings in the DB works as expected most of the time, you If you are trying to accomplish something and the above behaviours get in the way, please let us know as this is not intended. -To override these behaviours edit the docker-compose.yml file's misp-core volume definitions to enable the "customize_misp.sh" behaviour (see the bottom of the Production section for details). The "customize_misp.sh" script triggers after the above behaviours complete and is an appropriate place to override a setting. It is suggested that you use the "/var/www/MISP/app/cake Admin setSetting" command to override a setting, as this tool is config.php file and database setting aware. +To override these behaviours edit the `docker-compose.yml` file's misp-core volume definitions to enable the `customize_misp.sh` behaviour (see the bottom of the Production section for details). The `customize_misp.sh` script triggers after the above behaviours complete and is an appropriate place to override a setting. It is suggested that you use the `/var/www/MISP/app/cake Admin setSetting` command to override a setting, as this tool is `config.php` file and database setting aware. #### Adding a new setting and unsure what files to edit? @@ -133,7 +141,6 @@ Custom root CA certificates can be mounted under `/usr/local/share/ca-certificat ## Troubleshooting - Make sure you run a fairly recent version of Docker and Docker Compose (if in doubt, update following the steps outlined in https://docs.docker.com/engine/install/ubuntu/) -- Some Linux distributions provide a recent version of Docker but a legacy version of Docker Compose, so you can try running `docker compose` instead of `docker-compose` - Make sure you are not running an old image or container; when in doubt run `docker system prune --volumes` and clone this repository into an empty directory ## Versioning diff --git a/core/Dockerfile b/core/Dockerfile index 84ce324..73ca5fd 100644 --- a/core/Dockerfile +++ b/core/Dockerfile @@ -1,7 +1,7 @@ ARG DOCKER_HUB_PROXY="" -FROM "${DOCKER_HUB_PROXY}python:3.12-slim-bookworm" AS php-base +FROM "${DOCKER_HUB_PROXY}ubuntu:24.04" AS php-base ENV DEBIAN_FRONTEND noninteractive # Uncomment when building in corporate environments @@ -9,12 +9,16 @@ FROM "${DOCKER_HUB_PROXY}python:3.12-slim-bookworm" AS php-base # COPY ./rootca.crt /usr/lib/ssl/cert.pem RUN apt-get update; apt-get install -y --no-install-recommends \ - lsb-release \ - ca-certificates \ - curl - RUN curl -sSLo /tmp/debsuryorg-archive-keyring.deb https://packages.sury.org/debsuryorg-archive-keyring.deb - RUN dpkg -i /tmp/debsuryorg-archive-keyring.deb - RUN echo "deb [signed-by=/usr/share/keyrings/deb.sury.org-php.gpg] https://packages.sury.org/php/ $(lsb_release -sc) main" > /etc/apt/sources.list.d/php.list + ca-certificates + + COPY files/etc/apt/sources.list.d/ondrej-ubuntu-php-noble.sources /etc/apt/sources.list.d/ondrej-ubuntu-php-noble.sources + COPY files/etc/apt/sources.list.d/ondrej-ubuntu-nginx-mainline-noble.sources /etc/apt/sources.list.d/ondrej-ubuntu-nginx-mainline-noble.sources + + # RUN apt-get update; apt-get install -y --no-install-recommends \ + # software-properties-common + # # && apt-get autoremove -y && apt-get clean -y && rm -rf /var/lib/apt/lists/* + # RUN add-apt-repository ppa:ondrej/php + # RUN add-apt-repository ppa:ondrej/nginx-mainline RUN apt-get update @@ -23,6 +27,7 @@ FROM php-base AS composer-build ENV COMPOSER_ALLOW_SUPERUSER 1 ARG CORE_TAG ARG CORE_COMMIT + ARG TARGETPLATFORM RUN apt-get install -y --no-install-recommends \ php7.4 \ @@ -42,10 +47,28 @@ FROM php-base AS composer-build WORKDIR /tmp ADD https://raw.githubusercontent.com/MISP/MISP/${CORE_COMMIT:-${CORE_TAG}}/app/composer.json /tmp - COPY --from=composer:latest /usr/bin/composer /usr/bin/composer - RUN composer config --no-interaction allow-plugins.composer/installers true - RUN composer install - RUN composer require --with-all-dependencies --no-interaction \ + COPY --from=composer:2.7.7 /usr/bin/composer /usr/bin/composer + + # See: + # - https://github.com/curl/curl/issues/14154 + # - https://bugs.launchpad.net/ubuntu/+source/curl/+bug/2073448 + RUN <<-EOF + if [ "$TARGETPLATFORM" = "linux/arm64" ]; then + cp /usr/bin/composer /composer.phar + mkdir /out/ + php -r '$phar = new Phar("/composer.phar"); $phar->extractTo("/out/");' + sed -i "/'verify_peer_name' =>.*/a 'verify_peer_status' => CURLOPT_SSL_VERIFYSTATUS," /out/src/Composer/Util/Http/CurlDownloader.php + sed -i "/\$options = StreamContextFactory.*/a \$options['ssl']['verify_peer'] = false;" /out/src/Composer/Util/Http/CurlDownloader.php + sed -i "/\$options = StreamContextFactory.*/a \$options['ssl']['verify_peer_name'] = false;" /out/src/Composer/Util/Http/CurlDownloader.php + sed -i "/\$options = StreamContextFactory.*/a \$options['ssl']['verify_peer_status'] = false;" /out/src/Composer/Util/Http/CurlDownloader.php + rm /usr/bin/composer + ln -s /out/bin/composer /usr/bin/composer + fi +EOF + + RUN php /usr/bin/composer config --no-interaction allow-plugins.composer/installers true + RUN php /usr/bin/composer install + RUN php /usr/bin/composer require --with-all-dependencies --no-interaction \ supervisorphp/supervisor:^4.0 \ guzzlehttp/guzzle:^7.4.5 \ lstrojny/fxmlrpc \ @@ -105,6 +128,7 @@ FROM php-base AS python-build RUN apt-get install -y --no-install-recommends \ git \ + python3-pip \ && apt-get autoremove -y && apt-get clean -y && rm -rf /var/lib/apt/lists/* # Download MISP using git in the /var/www/ directory. Remove unnecessary items. @@ -177,6 +201,7 @@ FROM php-base gpg-agent \ mariadb-client \ rsync \ + python3-pip \ # PHP Requirements php7.4 \ php7.4-apcu \ @@ -195,7 +220,7 @@ FROM php-base libldap-common \ librdkafka1 \ libbrotli1 \ - libsimdjson14 \ + libsimdjson19 \ libzstd1 \ ssdeep \ libfuzzy2 \ @@ -209,8 +234,7 @@ FROM php-base # Install python modules COPY --from=python-build /wheels /wheels - RUN pip install --no-cache-dir /wheels/*.whl && rm -rf /wheels - RUN pip uninstall -y pip + RUN pip install --break-system-packages --no-cache-dir /wheels/*.whl && rm -rf /wheels # PHP: install prebuilt libraries, then install the app's PHP deps COPY --from=php-build ["/usr/lib/php/${PHP_VER}/ssdeep.so", "/usr/lib/php/${PHP_VER}/rdkafka.so", "/usr/lib/php/${PHP_VER}/brotli.so", "/usr/lib/php/${PHP_VER}/simdjson.so", "/usr/lib/php/${PHP_VER}/zstd.so", "/usr/lib/php/${PHP_VER}/"] @@ -221,6 +245,15 @@ FROM php-base COPY --from=composer-build --chown=www-data:www-data --chmod=0550 /tmp/Vendor /var/www/MISP/app/Vendor COPY --from=composer-build --chown=www-data:www-data --chmod=0550 /tmp/Plugin /var/www/MISP/app/Plugin + # python3-setuptools (distutils.version) is needed by 'mixbox' + RUN cp /usr/lib/python3/dist-packages/setuptools/_distutils/version.py \ + /usr/local/lib/python3.12/dist-packages/mixbox/distutils_version.py + RUN cp /usr/lib/python3/dist-packages/setuptools/_distutils/version.py \ + /var/www/MISP/app/files/scripts/mixbox/mixbox/distutils_version.py + RUN sed -i 's/from distutils\.version/from mixbox.distutils_version/' /usr/local/lib/python3.12/dist-packages/mixbox/parser.py + RUN sed -i 's/from distutils\.version/from mixbox.distutils_version/' /var/www/MISP/app/files/scripts/mixbox/mixbox/parser.py + RUN apt-get remove --purge python3-pip python3-setuptools -y + # Gather these in one layer, only act on actual directories under /etc/php/ RUN <<-EOF set -- "ssdeep" "rdkafka" "brotli" "simdjson" "zstd" diff --git a/core/files/etc/apt/sources.list.d/ondrej-ubuntu-nginx-mainline-noble.sources b/core/files/etc/apt/sources.list.d/ondrej-ubuntu-nginx-mainline-noble.sources new file mode 100644 index 0000000..0536632 --- /dev/null +++ b/core/files/etc/apt/sources.list.d/ondrej-ubuntu-nginx-mainline-noble.sources @@ -0,0 +1,24 @@ +Types: deb +URIs: https://ppa.launchpadcontent.net/ondrej/nginx-mainline/ubuntu/ +Suites: noble +Components: main +Signed-By: + -----BEGIN PGP PUBLIC KEY BLOCK----- + . + mI0ESX35nAEEALKDCUDVXvmW9n+T/+3G1DnTpoWh9/1xNaz/RrUH6fQKhHr568F8 + hfnZP/2CGYVYkW9hxP9LVW9IDvzcmnhgIwK+ddeaPZqh3T/FM4OTA7Q78HSvR81m + Jpf2iMLm/Zvh89ZsmP2sIgZuARiaHo8lxoTSLtmKXsM3FsJVlusyewHfABEBAAG0 + H0xhdW5jaHBhZCBQUEEgZm9yIE9uZMWZZWogU3Vyw72ItgQTAQIAIAUCSX35nAIb + AwYLCQgHAwIEFQIIAwQWAgMBAh4BAheAAAoJEE9OoKrlJnpsQjYD/jW1NlIFAlT6 + EvF2xfVbkhERii9MapjaUsSso4XLCEmZdEGX54GQ01svXnrivwnd/kmhKvyxCqiN + LDY/dOaK8MK//bDI6mqdKmG8XbP2vsdsxhifNC+GH/OwaDPvn1TyYB653kwyruCG + FjEnCreZTcRUu2oBQyolORDl+BmF4DjLiQEzBBABCgAdFiEECvaBvTqO/UqmWMI/ + thEcm0xImQEFAmXTV0AACgkQthEcm0xImQGTTggAhuMHGeBZlRUAsZE7jJM7Mf06 + /WIhcgUfBfSFnJFlFH+xdEe/GFYyVk9kingDsPh90Ecnt4n8DJHTlsuUV1+SPBIO + JfbQTUjx1n/+Ck+TVKzRByvrpRXtiZQ214m3zbhZpme2eBBMItZByjG7g925NUIq + rL+R5ZoEcZvVlYscfsA0Sr8yJTsGJPefuLYI6eJkNDa1QkzBkSSW4XaCfNIxNBRs + zN/qGe3xy0bibOaC4T2TcbZPSAVP855ahNbLAdqkyfAutiEWcKZmQpR9qNh4482k + 0pXVlQJ8UB860gVFHjwjFm/MsCeX8yfeAi38ZyInWL2OSG2pDx5ZzNESwnCPIg== + =N1rh + -----END PGP PUBLIC KEY BLOCK----- + diff --git a/core/files/etc/apt/sources.list.d/ondrej-ubuntu-php-noble.sources b/core/files/etc/apt/sources.list.d/ondrej-ubuntu-php-noble.sources new file mode 100644 index 0000000..18d8fe1 --- /dev/null +++ b/core/files/etc/apt/sources.list.d/ondrej-ubuntu-php-noble.sources @@ -0,0 +1,23 @@ +Types: deb +URIs: https://ppa.launchpadcontent.net/ondrej/php/ubuntu/ +Suites: noble +Components: main +Signed-By: -----BEGIN PGP PUBLIC KEY BLOCK----- + . + mI0ESX35nAEEALKDCUDVXvmW9n+T/+3G1DnTpoWh9/1xNaz/RrUH6fQKhHr568F8 + hfnZP/2CGYVYkW9hxP9LVW9IDvzcmnhgIwK+ddeaPZqh3T/FM4OTA7Q78HSvR81m + Jpf2iMLm/Zvh89ZsmP2sIgZuARiaHo8lxoTSLtmKXsM3FsJVlusyewHfABEBAAG0 + H0xhdW5jaHBhZCBQUEEgZm9yIE9uZMWZZWogU3Vyw72ItgQTAQIAIAUCSX35nAIb + AwYLCQgHAwIEFQIIAwQWAgMBAh4BAheAAAoJEE9OoKrlJnpsQjYD/jW1NlIFAlT6 + EvF2xfVbkhERii9MapjaUsSso4XLCEmZdEGX54GQ01svXnrivwnd/kmhKvyxCqiN + LDY/dOaK8MK//bDI6mqdKmG8XbP2vsdsxhifNC+GH/OwaDPvn1TyYB653kwyruCG + FjEnCreZTcRUu2oBQyolORDl+BmF4DjLiQEzBBABCgAdFiEECvaBvTqO/UqmWMI/ + thEcm0xImQEFAmXTV0AACgkQthEcm0xImQGTTggAhuMHGeBZlRUAsZE7jJM7Mf06 + /WIhcgUfBfSFnJFlFH+xdEe/GFYyVk9kingDsPh90Ecnt4n8DJHTlsuUV1+SPBIO + JfbQTUjx1n/+Ck+TVKzRByvrpRXtiZQ214m3zbhZpme2eBBMItZByjG7g925NUIq + rL+R5ZoEcZvVlYscfsA0Sr8yJTsGJPefuLYI6eJkNDa1QkzBkSSW4XaCfNIxNBRs + zN/qGe3xy0bibOaC4T2TcbZPSAVP855ahNbLAdqkyfAutiEWcKZmQpR9qNh4482k + 0pXVlQJ8UB860gVFHjwjFm/MsCeX8yfeAi38ZyInWL2OSG2pDx5ZzNESwnCPIg== + =N1rh + -----END PGP PUBLIC KEY BLOCK----- + diff --git a/core/files/etc/nginx/sites-available/misp443 b/core/files/etc/nginx/sites-available/misp443 index d38b810..06492d0 100644 --- a/core/files/etc/nginx/sites-available/misp443 +++ b/core/files/etc/nginx/sites-available/misp443 @@ -1,6 +1,7 @@ server { - listen 443 ssl http2; - listen [::]:443 ssl http2; + listen 443 ssl; + listen [::]:443 ssl; + http2 on; # disable access logs access_log off; diff --git a/modules/Dockerfile b/modules/Dockerfile index d8201a4..7868c39 100644 --- a/modules/Dockerfile +++ b/modules/Dockerfile @@ -1,6 +1,6 @@ ARG DOCKER_HUB_PROXY="" -FROM "${DOCKER_HUB_PROXY}python:3.12-slim-bookworm" AS python-build +FROM "${DOCKER_HUB_PROXY}ubuntu:24.04" AS python-build ENV DEBIAN_FRONTEND noninteractive ARG MODULES_TAG ARG MODULES_COMMIT @@ -9,6 +9,10 @@ FROM "${DOCKER_HUB_PROXY}python:3.12-slim-bookworm" AS python-build RUN apt-get update && apt-get install -y --no-install-recommends \ cmake \ git \ + python3-dev \ + python3-pip \ + python3-wheel \ + pipenv \ build-essential \ libpoppler-cpp-dev \ libfuzzy-dev \ @@ -30,7 +34,6 @@ FROM "${DOCKER_HUB_PROXY}python:3.12-slim-bookworm" AS python-build EOF WORKDIR /srv/misp-modules - RUN pip install pipenv RUN pipenv requirements > requirements.txt RUN pip wheel -r requirements.txt --no-cache-dir -w /wheels/ @@ -56,7 +59,7 @@ EOF RUN rm -rf /srv/faup -FROM "${DOCKER_HUB_PROXY}python:3.12-slim-bookworm" +FROM "${DOCKER_HUB_PROXY}ubuntu:24.04" ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && apt-get install -y --no-install-recommends \ @@ -69,18 +72,19 @@ FROM "${DOCKER_HUB_PROXY}python:3.12-slim-bookworm" libxml2 \ libxslt1.1 \ libzbar0 \ + python3-pip \ && apt-get autoremove -y && apt-get clean -y && rm -rf /var/lib/apt/lists/* COPY --from=python-build /wheels /wheels COPY --from=python-build /usr/local/lib/libfaupl* /usr/local/lib/ RUN ldconfig - RUN pip install --no-cache-dir --use-deprecated=legacy-resolver /wheels/*.whl && rm -rf /wheels - RUN pip uninstall -y pip + RUN pip install --break-system-packages --no-cache-dir --use-deprecated=legacy-resolver /wheels/*.whl && rm -rf /wheels + RUN apt-get remove --purge python3-pip python3-setuptools -y # Since we compile faup ourselves and lua is not required anymore, we can load our own library # and skip the pre-compiled blob to improve compatibility with other architectures like ARM RUN sed -i s/LoadLibrary\(LOAD_LIB\)/LoadLibrary\(\"\\/usr\\/local\\/lib\\/libfaupl.so\"\)/ \ - /usr/local/lib/python3.12/site-packages/pyfaup/__init__.py + /usr/local/lib/python3.12/dist-packages/pyfaup/__init__.py # Disable (all) warnings raised when using 'future' RUN sed -i '/import sys/a import warnings\nwarnings.warn = lambda *args, **kwargs: None' \