From 45350bba1fae9f94621ff6dede81a9224aa55ab8 Mon Sep 17 00:00:00 2001 From: "renovate[bot]" <29139614+renovate[bot]@users.noreply.github.com> Date: Thu, 28 Nov 2024 16:02:06 +0000 Subject: [PATCH] Add support for XNAT 1.9 (#135) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fixes #140 Fixes #146 - update default XNAT version to 1.9.1 - update plugin versions - update docker role to add option to run without TLS - don't use TLS for docker in XNAT Container Service (due to bug in current version of the Container Service) - fix protocol used to connect to Container Service host (now needs be tcp rather than http / https) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [xnatdev/container-service](https://bitbucket.org/xnatdev/container-service) | minor | `3.4.3` -> `3.5.0` | --- > [!WARNING] > Some dependencies could not be looked up. Check the warning logs for more information. --- ### Release Notes
xnatdev/container-service (xnatdev/container-service) ### [`v3.5.0`](https://bitbucket.org/xnatdev/container-service/blob/HEAD/CHANGELOG.md#350) [Compare Source](https://bitbucket.org/xnatdev/container-service/branches/compare/3.5.0%0D3.4.3) [Released](https://bitbucket.org/xnatdev/container-service/src/3.5.0/). - **Improvement** [CS-946][] Prevent setting mutually distinct k8s PVC mounting options - **Bugfix** [CS-966][] Ensure tracking of container IDs in workflow tables in a Kubernetes environment - **Bugfix** [CS-968][] Switch the docker API library we use from [docker-client][] to [docker-java][]. This should restore CS functionality on docker engine v25 and higher. ##### A Note About Our Docker Library Originally we used the [spotify/docker-client][] library to wrap the docker remote API in java method calls. They stopped updating that and put out their final release [v6.1.1][] in 2016. We switched the Container Service to use a fork of that client, [dmandalidis/docker-client][] in CS version 3.0.0. Given that this was a fork of the client we already used, it was a simple drop-in replacement with no changes needed. But that library maintainer did continue to make changes. In 2023 they released a major version upgrade, [v7.0.0][], which dropped support for Java 8. That is the version of Java we use in XNAT (at time of writing) so this change meant we weren't able to update our version of this library. That was fine for a while...\ ...Until version 25 of the docker engine, in which they made an API change which caused an error in the version we used of `docker-client`. The library (presumably) fixed their issue but we weren't able to use that fix because our version of the library was frozen by their decision to drop Java 8 support. This forced us to switch our library from `docker-client` to [docker-java][]. This was not a drop-in replacement, and did require a migration. All the same docker API endpoints were supported in a 1:1 replacement—which took a little effort but was straightforward—except for one. The `docker-java` library did not support requesting `GenericResources` on a swarm service, which is the mechanism by which we allow commands to specify that they need a GPU. We opened a ticket reporting that lack of support ([https://github.com/docker-java/docker-java/issues/2320](https://redirect.github.com/docker-java/docker-java/issues/2320)), but at time of writing there has been no response. I created a fork (https://github.com/johnflavin/docker-java) and fixed the issue myself ([https://github.com/docker-java/docker-java/pull/2327](https://redirect.github.com/docker-java/docker-java/pull/2327)), but at time of writing that also has no response. I built a custom version of `docker-java` `3.4.0.1` and pushed that to the XNAT artifactory ([ext-release-local/com/github/docker-java][]). Long story short, as of CS version `3.5.0` we depend on `docker-java` version `3.4.0.1` for our docker (and swarm) API support. [CS-946]: https://radiologics.atlassian.net/browse/CS-946 [CS-966]: https://radiologics.atlassian.net/browse/CS-966 [CS-968]: https://radiologics.atlassian.net/browse/CS-968 [docker-client]: https://redirect.github.com/spotify/docker-client [spotify/docker-client]: https://redirect.github.com/spotify/docker-client [v6.1.1]: https://redirect.github.com/spotify/docker-client/releases/tag/v6.1.1 [dmandalidis/docker-client]: https://redirect.github.com/dmandalidis/docker-client [v7.0.0]: https://redirect.github.com/dmandalidis/docker-client/tree/v7.0.0 [docker-java]: https://redirect.github.com/docker-java/docker-java [ext-release-local/com/github/docker-java]: https://nrgxnat.jfrog.io/ui/repos/tree/General/ext-release-local/com/github/docker-java
--- ### Configuration 📅 **Schedule**: Branch creation - "before 4am on the first day of the month" (UTC), Automerge - "every weekday" (UTC). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/UCL-MIRSG/ansible-collection-infra). --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Patrick J. Roddy Co-authored-by: Daniel Matthews Co-authored-by: Haroon Chughtai Co-authored-by: ruaridhg Co-authored-by: ruaridhg <32329546+ruaridhg@users.noreply.github.com> Co-authored-by: Paul Smith --- .../group_vars/container_service_host.yml | 6 +-- playbooks/molecule/resources/xnat/verify.yml | 21 --------- roles/docker/README.md | 2 +- roles/docker/tasks/main.yml | 44 +++++++------------ roles/docker/templates/daemon.json.j2 | 4 ++ roles/provision/tasks/main.yml | 4 -- roles/xnat/defaults/main.yml | 17 +++---- .../xnat_container_service/defaults/main.yml | 1 + .../tasks/copy_certs_to_client.yml | 36 +++++++++++++++ roles/xnat_container_service/tasks/main.yml | 44 +++---------------- 10 files changed, 77 insertions(+), 102 deletions(-) create mode 100644 roles/xnat_container_service/tasks/copy_certs_to_client.yml diff --git a/playbooks/molecule/resources/xnat/inventory/group_vars/container_service_host.yml b/playbooks/molecule/resources/xnat/inventory/group_vars/container_service_host.yml index cb1747d0..0d13e6f2 100644 --- a/playbooks/molecule/resources/xnat/inventory/group_vars/container_service_host.yml +++ b/playbooks/molecule/resources/xnat/inventory/group_vars/container_service_host.yml @@ -1,13 +1,13 @@ --- # mirsg.infrastructure.docker -docker_generate_certificates: true # generate TLS certs for clients +docker_generate_certificates: false # generate TLS certs for clients docker_client_hostnames: - "{{ hostvars['xnat_web']['hostname'] }}" docker_tls_verify: false -# docker<25 is required for XNAT +# docker<25 is required for XNAT for Container Service plugin < 3.5.0 # see: https://groups.google.com/g/xnat_discussion/c/yyPBkN4kayE/m/LUe5GQH5AAAJ -docker_version: 24.0.9 +docker_version: latest # mirsg.infrastructure.firewalld firewalld_internal_zone_sources: diff --git a/playbooks/molecule/resources/xnat/verify.yml b/playbooks/molecule/resources/xnat/verify.yml index 50e6ee34..8a99f92a 100644 --- a/playbooks/molecule/resources/xnat/verify.yml +++ b/playbooks/molecule/resources/xnat/verify.yml @@ -17,27 +17,6 @@ - response.server == "nginx" - response.content is search('MIRSG XNAT') -- name: Verify correct Docker version is installed - hosts: xnat_cserv - tasks: - - name: Get Docker server version - ansible.builtin.command: - cmd: "{% raw %}docker version -f '{{.Server.Version}}'{% endraw %}" - changed_when: false - register: server_version - - - name: Get Docker client version - ansible.builtin.command: - cmd: "{% raw %}docker version -f '{{.Client.Version}}'{% endraw %}" - changed_when: false - register: client_version - - - name: Check Docker server version - ansible.builtin.assert: - that: - - server_version.stdout is version( docker_version ) - - client_version.stdout is version( docker_version ) - - name: Verify container service is running hosts: localhost tasks: diff --git a/roles/docker/README.md b/roles/docker/README.md index e59c310d..ef28ced9 100644 --- a/roles/docker/README.md +++ b/roles/docker/README.md @@ -31,7 +31,7 @@ used to configure certificate creation and signing: | `docker_server_hostname` | Hostname of your Docker server. Used for the `commonName` field of the certificate signing request subject. Defaults to `"{{ ansible_host }}"` | | `docker_server_ip` | IP address of your Docker server. Defaults to `0.0.0.0` | | `docker_server_port` | Port the Docker Daemon will listen on. Defaults to `2376`. | -| `docker_tls_verify` | If `true`, require that TLS certificates can be verified by a root authority. Defaults to `true` | +| `docker_tls_verify` | If `true`, require that TLS certificates can be verified by a root authority. Defaults to `false` | | `docker_ca_key` | Filename for the CA certificate key. Defaults to `/home/docker/.docker/ca.key` | | `docker_ca_csr` | Filename for the CA certificate signing request. Defaults to `/home/docker/.docker/ca.csr` | | `docker_ca_cert` | Filename for the CA certificate. Defaults to `/home/docker/.docker/ca.pem` | diff --git a/roles/docker/tasks/main.yml b/roles/docker/tasks/main.yml index 0dfe0816..78403f33 100644 --- a/roles/docker/tasks/main.yml +++ b/roles/docker/tasks/main.yml @@ -43,18 +43,6 @@ - docker-buildx-plugin - docker-compose-plugin -- name: Ensure yum-plugin-versionlock is installed - ansible.builtin.yum: - name: yum-plugin-versionlock - state: installed - -- name: Prevent docker from being updated - community.general.yum_versionlock: - state: "{{ 'absent' if docker_version == 'latest' else 'present' }}" - name: - - docker-ce - - docker-ce-cli - - name: Ensure docker service directory exists ansible.builtin.file: path: "{{ docker_service_directory }}" @@ -72,27 +60,27 @@ mode: "0644" notify: Reload docker +- name: Ensure docker config directory exists - {{ docker_config_dir }} + ansible.builtin.file: + path: "{{ docker_config_dir }}" + owner: "{{ docker_owner }}" + group: "{{ docker_group }}" + state: directory + mode: "0700" + +- name: Write docker daemon configuration file + ansible.builtin.template: + src: daemon.json.j2 + dest: "{{ docker_daemon_conf_file }}" + owner: "{{ docker_owner }}" + group: "{{ docker_group }}" + mode: "0640" + - name: Generate CA, server, and client certificates when: docker_generate_certificates notify: - Restart docker block: - - name: Ensure docker config directory exists - {{ docker_config_dir }} - ansible.builtin.file: - path: "{{ docker_config_dir }}" - owner: "{{ docker_owner }}" - group: "{{ docker_group }}" - state: directory - mode: "0700" - - - name: Write docker daemon configuration file - ansible.builtin.template: - src: daemon.json.j2 - dest: "{{ docker_daemon_conf_file }}" - owner: "{{ docker_owner }}" - group: "{{ docker_group }}" - mode: "0640" - - name: Generate CA certificate ansible.builtin.import_tasks: ca-cert.yml diff --git a/roles/docker/templates/daemon.json.j2 b/roles/docker/templates/daemon.json.j2 index 3769115e..3fc8ac4b 100644 --- a/roles/docker/templates/daemon.json.j2 +++ b/roles/docker/templates/daemon.json.j2 @@ -1,7 +1,11 @@ { + {% if docker_generate_certificates %} "hosts": ["tcp://{{ docker_server_ip }}:{{ docker_server_port }}", "unix:///var/run/docker.sock"], "tlsverify": {{ docker_tls_verify | lower }}, "tlscacert": "{{ docker_ca_cert }}", "tlscert": "{{ docker_server_cert }}", "tlskey": "{{ docker_server_key }}" + {% else %} + "hosts": ["tcp://{{ docker_server_ip }}:{{ docker_server_port }}", "unix:///var/run/docker.sock"] + {% endif %} } diff --git a/roles/provision/tasks/main.yml b/roles/provision/tasks/main.yml index 7e84e9c3..c8a80d18 100644 --- a/roles/provision/tasks/main.yml +++ b/roles/provision/tasks/main.yml @@ -18,10 +18,6 @@ ansible.builtin.yum: name: "*" state: latest - exclude: - # https://groups.google.com/g/xnat_discussion/c/yyPBkN4kayE/m/LUe5GQH5AAAJ - - docker-ce - - docker-ce-cli tags: - molecule-idempotence-notest diff --git a/roles/xnat/defaults/main.yml b/roles/xnat/defaults/main.yml index 4ea1371b..c60e6ff5 100644 --- a/roles/xnat/defaults/main.yml +++ b/roles/xnat/defaults/main.yml @@ -1,5 +1,5 @@ --- -xnat_version: 1.8.10.1 +xnat_version: 1.9.1 xnat_pipeline_version: 1.8.10 xnat_archive_dir: "{{ xnat_root_dir }}/archive" xnat_prearchive_dir: "{{ xnat_root_dir }}/prearchive" @@ -39,16 +39,17 @@ xnat_ldap_keystore_alias: "" # Plugins xnat_plugin_urls: - - https://api.bitbucket.org/2.0/repositories/xnatdev/xsync/downloads/xsync-plugin-all-1.7.0.jar - - https://api.bitbucket.org/2.0/repositories/xnatx/ldap-auth-plugin/downloads/ldap-auth-plugin-1.1.0.jar - - https://api.bitbucket.org/2.0/repositories/xnatdev/container-service/downloads/container-service-3.4.3-fat.jar - - https://api.bitbucket.org/2.0/repositories/xnatx/xnatx-batch-launch-plugin/downloads/batch-launch-0.6.0.jar + - https://api.bitbucket.org/2.0/repositories/xnatdev/xsync/downloads/xsync-plugin-all-1.8.1.jar + - https://api.bitbucket.org/2.0/repositories/xnatx/ldap-auth-plugin/downloads/ldap-auth-plugin-1.2.1.jar + - https://api.bitbucket.org/2.0/repositories/xnatdev/container-service/downloads/container-service-3.6.1-fat.jar + - https://api.bitbucket.org/2.0/repositories/xnatx/xnatx-batch-launch-plugin/downloads/batch-launch-0.7.0.jar - https://github.com/VUIIS/dax/raw/main/misc/xnat-plugins/dax-plugin-genProcData-1.4.2.jar - - https://api.bitbucket.org/2.0/repositories/icrimaginginformatics/ohif-viewer-xnat-plugin/downloads/ohif-viewer-3.6.2.jar - - https://api.bitbucket.org/2.0/repositories/xnatx/ml-plugin/downloads/ml-plugin-1.0.2.jar - - https://api.bitbucket.org/2.0/repositories/xnatx/datasets-plugin/downloads/datasets-plugin-1.0.3.jar + - https://api.bitbucket.org/2.0/repositories/icrimaginginformatics/ohif-viewer-xnat-plugin/downloads/ohif-viewer-3.7.0-XNAT-1.8.10.jar + - https://api.bitbucket.org/2.0/repositories/xnatx/ml-schema-plugin/downloads/ml-schema-plugin-1.0.0.jar + - https://api.bitbucket.org/2.0/repositories/xnatx/datasets-schema-plugin/downloads/datasets-schema-plugin-1.0.0.jar - https://api.bitbucket.org/2.0/repositories/xnatdev/xnat-image-viewer-plugin/downloads/ximgview-plugin-1.0.2.jar - https://api.bitbucket.org/2.0/repositories/xnatx/xnatx-dxm-settings-plugin/downloads/dxm-settings-plugin-1.0.jar + - https://api.bitbucket.org/2.0/repositories/xnatx/pipeline_engine_plugin/downloads/pipeline_engine_ui-1.1.0-xpl.jar xnat_plugin_bundle_urls: [] # yamllint disable-line rule:brackets xnat_plugin_packages: [] # yamllint disable-line rule:brackets diff --git a/roles/xnat_container_service/defaults/main.yml b/roles/xnat_container_service/defaults/main.yml index 0dd3dfa5..ac752a62 100644 --- a/roles/xnat_container_service/defaults/main.yml +++ b/roles/xnat_container_service/defaults/main.yml @@ -1,6 +1,7 @@ --- xnat_container_service_owner: tomcat xnat_container_service_group: tomcat +xnat_container_service_use_ssl: false xnat_container_service_certificate_directory: /usr/share/tomcat/.docker xnat_container_service_key: /usr/share/tomcat/.docker/key.pem xnat_container_service_csr: /usr/share/tomcat/.docker/docker.csr diff --git a/roles/xnat_container_service/tasks/copy_certs_to_client.yml b/roles/xnat_container_service/tasks/copy_certs_to_client.yml new file mode 100644 index 00000000..d8afea70 --- /dev/null +++ b/roles/xnat_container_service/tasks/copy_certs_to_client.yml @@ -0,0 +1,36 @@ +--- +- name: Ensure Docker certificate directory exists on the client + ansible.builtin.file: + path: "{{ xnat_container_service_certificate_directory }}" + state: directory + owner: "{{ xnat_container_service_owner }}" + group: "{{ xnat_container_service_group }}" + mode: "0700" + +- name: Copy Docker server certificate from Ansible Controller cache to client + ansible.builtin.copy: + src: "{{ xnat_container_service_certificate_cache_directory }}/ca.pem" + dest: "{{ xnat_container_service_server_ca_cert }}" + owner: "{{ xnat_container_service_owner }}" + group: "{{ xnat_container_service_group }}" + mode: "0600" + +- name: + Copy signed Docker client certificate from Ansible Controller cache to + client + ansible.builtin.copy: + src: + "{{ xnat_container_service_certificate_cache_directory }}/{{ + xnat_container_service_client_hostname }}.cert" + dest: "{{ xnat_container_service_cert }}" + owner: "{{ xnat_container_service_owner }}" + group: "{{ xnat_container_service_group }}" + mode: "0600" + +- name: Copy private key from Ansible Controller cache to client + ansible.builtin.copy: + src: "{{ xnat_container_service_certificate_cache_directory }}/key.pem" + dest: "{{ xnat_container_service_key }}" + owner: "{{ xnat_container_service_owner }}" + group: "{{ xnat_container_service_group }}" + mode: "0600" diff --git a/roles/xnat_container_service/tasks/main.yml b/roles/xnat_container_service/tasks/main.yml index af50cc5f..9da7b8a0 100644 --- a/roles/xnat_container_service/tasks/main.yml +++ b/roles/xnat_container_service/tasks/main.yml @@ -1,39 +1,7 @@ --- -- name: Ensure Docker certificate directory exists on the client - ansible.builtin.file: - path: "{{ xnat_container_service_certificate_directory }}" - state: directory - owner: "{{ xnat_container_service_owner }}" - group: "{{ xnat_container_service_group }}" - mode: "0700" - -- name: Copy Docker server certificate from Ansible Controller cache to client - ansible.builtin.copy: - src: "{{ xnat_container_service_certificate_cache_directory }}/ca.pem" - dest: "{{ xnat_container_service_server_ca_cert }}" - owner: "{{ xnat_container_service_owner }}" - group: "{{ xnat_container_service_group }}" - mode: "0600" - -- name: - Copy signed Docker client certificate from Ansible Controller cache to - client - ansible.builtin.copy: - src: - "{{ xnat_container_service_certificate_cache_directory }}/{{ - xnat_container_service_client_hostname }}.cert" - dest: "{{ xnat_container_service_cert }}" - owner: "{{ xnat_container_service_owner }}" - group: "{{ xnat_container_service_group }}" - mode: "0600" - -- name: Copy private key from Ansible Controller cache to client - ansible.builtin.copy: - src: "{{ xnat_container_service_certificate_cache_directory }}/key.pem" - dest: "{{ xnat_container_service_key }}" - owner: "{{ xnat_container_service_owner }}" - group: "{{ xnat_container_service_group }}" - mode: "0600" +- name: Copy SSL certificates to client + ansible.builtin.include_tasks: copy_certs_to_client.yml + when: xnat_container_service_use_ssl - name: Configure XNAT to talk to container service ansible.builtin.uri: @@ -45,9 +13,11 @@ body: name: "{{ xnat_container_service_name }}" host: - https://{{ xnat_container_service_hostname }}:{{ + tcp://{{ xnat_container_service_hostname }}:{{ xnat_container_service_port }} - cert-path: "{{ xnat_container_service_certificate_directory }}" + cert-path: + "{{ xnat_container_service_certificate_directory }} if {{ + xnat_container_service_use_ssl }} else ''" swarm-mode: false path-translation-xnat-prefix: "{{ xnat_container_service_path_translation_xnat_prefix }}"