Add support for XNAT 1.9 (#135)

Fixes #140 Fixes #146 - update default XNAT version to 1.9.1 - update plugin versions - update docker role to add option to run without TLS - don't use TLS for docker in XNAT Container Service (due to bug in current version of the Container Service) - fix protocol used to connect to Container Service host (now needs be tcp rather than http / https) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [xnatdev/container-service](https://bitbucket.org/xnatdev/container-service) | minor | `3.4.3` -> `3.5.0` | --- > [!WARNING] > Some dependencies could not be looked up. Check the warning logs for more information. --- ### Release Notes <details> <summary>xnatdev/container-service (xnatdev/container-service)</summary> ### [`v3.5.0`](https://bitbucket.org/xnatdev/container-service/blob/HEAD/CHANGELOG.md#350) [Compare Source](https://bitbucket.org/xnatdev/container-service/branches/compare/3.5.0%0D3.4.3) [Released](https://bitbucket.org/xnatdev/container-service/src/3.5.0/). - **Improvement** [CS-946][] Prevent setting mutually distinct k8s PVC mounting options - **Bugfix** [CS-966][] Ensure tracking of container IDs in workflow tables in a Kubernetes environment - **Bugfix** [CS-968][] Switch the docker API library we use from [docker-client][] to [docker-java][]. This should restore CS functionality on docker engine v25 and higher. ##### A Note About Our Docker Library Originally we used the [spotify/docker-client][] library to wrap the docker remote API in java method calls. They stopped updating that and put out their final release [v6.1.1][] in 2016. We switched the Container Service to use a fork of that client, [dmandalidis/docker-client][] in CS version 3.0.0. Given that this was a fork of the client we already used, it was a simple drop-in replacement with no changes needed. But that library maintainer did continue to make changes. In 2023 they released a major version upgrade, [v7.0.0][], which dropped support for Java 8. That is the version of Java we use in XNAT (at time of writing) so this change meant we weren't able to update our version of this library. That was fine for a while...\ ...Until version 25 of the docker engine, in which they made an API change which caused an error in the version we used of `docker-client`. The library (presumably) fixed their issue but we weren't able to use that fix because our version of the library was frozen by their decision to drop Java 8 support. This forced us to switch our library from `docker-client` to [docker-java][]. This was not a drop-in replacement, and did require a migration. All the same docker API endpoints were supported in a 1:1 replacement—which took a little effort but was straightforward—except for one. The `docker-java` library did not support requesting `GenericResources` on a swarm service, which is the mechanism by which we allow commands to specify that they need a GPU. We opened a ticket reporting that lack of support ([https://github.com/docker-java/docker-java/issues/2320](https://redirect.github.com/docker-java/docker-java/issues/2320)), but at time of writing there has been no response. I created a fork (https://github.com/johnflavin/docker-java) and fixed the issue myself ([https://github.com/docker-java/docker-java/pull/2327](https://redirect.github.com/docker-java/docker-java/pull/2327)), but at time of writing that also has no response. I built a custom version of `docker-java` `3.4.0.1` and pushed that to the XNAT artifactory ([ext-release-local/com/github/docker-java][]). Long story short, as of CS version `3.5.0` we depend on `docker-java` version `3.4.0.1` for our docker (and swarm) API support. [CS-946]: https://radiologics.atlassian.net/browse/CS-946 [CS-966]: https://radiologics.atlassian.net/browse/CS-966 [CS-968]: https://radiologics.atlassian.net/browse/CS-968 [docker-client]: https://redirect.github.com/spotify/docker-client [spotify/docker-client]: https://redirect.github.com/spotify/docker-client [v6.1.1]: https://redirect.github.com/spotify/docker-client/releases/tag/v6.1.1 [dmandalidis/docker-client]: https://redirect.github.com/dmandalidis/docker-client [v7.0.0]: https://redirect.github.com/dmandalidis/docker-client/tree/v7.0.0 [docker-java]: https://redirect.github.com/docker-java/docker-java [ext-release-local/com/github/docker-java]: https://nrgxnat.jfrog.io/ui/repos/tree/General/ext-release-local/com/github/docker-java </details> --- ### Configuration 📅 **Schedule**: Branch creation - "before 4am on the first day of the month" (UTC), Automerge - "every weekday" (UTC). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/UCL-MIRSG/ansible-collection-infra).  --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Patrick J. Roddy <[email protected]> Co-authored-by: Daniel Matthews <[email protected]> Co-authored-by: Haroon Chughtai <[email protected]> Co-authored-by: ruaridhg <[email protected]> Co-authored-by: ruaridhg <[email protected]> Co-authored-by: Paul Smith <[email protected]>
UCL-MIRSG · Nov 28, 2024 · 45350bb · 45350bb
1 parent 18bd517
commit 45350bb
Show file tree

Hide file tree

Showing 10 changed files with 77 additions and 102 deletions.
diff --git a/playbooks/molecule/resources/xnat/inventory/group_vars/container_service_host.yml b/playbooks/molecule/resources/xnat/inventory/group_vars/container_service_host.yml
@@ -1,13 +1,13 @@
 ---
 # mirsg.infrastructure.docker
-docker_generate_certificates: true # generate TLS certs for clients
+docker_generate_certificates: false # generate TLS certs for clients
 docker_client_hostnames:
   - "{{ hostvars['xnat_web']['hostname'] }}"
 docker_tls_verify: false
 
-# docker<25 is required for XNAT
+# docker<25 is required for XNAT for Container Service plugin < 3.5.0
 # see: https://groups.google.com/g/xnat_discussion/c/yyPBkN4kayE/m/LUe5GQH5AAAJ
-docker_version: 24.0.9
+docker_version: latest
 
 # mirsg.infrastructure.firewalld
 firewalld_internal_zone_sources:

diff --git a/playbooks/molecule/resources/xnat/verify.yml b/playbooks/molecule/resources/xnat/verify.yml
@@ -17,27 +17,6 @@
           - response.server == "nginx"
           - response.content is search('MIRSG XNAT')
 
-- name: Verify correct Docker version is installed
-  hosts: xnat_cserv
-  tasks:
-    - name: Get Docker server version
-      ansible.builtin.command:
-        cmd: "{% raw %}docker version -f '{{.Server.Version}}'{% endraw %}"
-      changed_when: false
-      register: server_version
-
-    - name: Get Docker client version
-      ansible.builtin.command:
-        cmd: "{% raw %}docker version -f '{{.Client.Version}}'{% endraw %}"
-      changed_when: false
-      register: client_version
-
-    - name: Check Docker server version
-      ansible.builtin.assert:
-        that:
-          - server_version.stdout is version( docker_version )
-          - client_version.stdout is version( docker_version )
-
 - name: Verify container service is running
   hosts: localhost
   tasks:

diff --git a/roles/docker/README.md b/roles/docker/README.md
@@ -31,7 +31,7 @@ used to configure certificate creation and signing:
 | `docker_server_hostname`                    | Hostname of your Docker server. Used for the `commonName` field of the certificate signing request subject. Defaults to `"{{ ansible_host }}"` |
 | `docker_server_ip`                          | IP address of your Docker server. Defaults to `0.0.0.0`                                                                                        |
 | `docker_server_port`                        | Port the Docker Daemon will listen on. Defaults to `2376`.                                                                                     |
-| `docker_tls_verify`                         | If `true`, require that TLS certificates can be verified by a root authority. Defaults to `true`                                               |
+| `docker_tls_verify`                         | If `true`, require that TLS certificates can be verified by a root authority. Defaults to `false`                                              |
 | `docker_ca_key`                             | Filename for the CA certificate key. Defaults to `/home/docker/.docker/ca.key`                                                                 |
 | `docker_ca_csr`                             | Filename for the CA certificate signing request. Defaults to `/home/docker/.docker/ca.csr`                                                     |
 | `docker_ca_cert`                            | Filename for the CA certificate. Defaults to `/home/docker/.docker/ca.pem`                                                                     |

diff --git a/roles/docker/tasks/main.yml b/roles/docker/tasks/main.yml
@@ -43,18 +43,6 @@
     - docker-buildx-plugin
     - docker-compose-plugin
 
-- name: Ensure yum-plugin-versionlock is installed
-  ansible.builtin.yum:
-    name: yum-plugin-versionlock
-    state: installed
-
-- name: Prevent docker from being updated
-  community.general.yum_versionlock:
-    state: "{{ 'absent' if docker_version == 'latest' else 'present' }}"
-    name:
-      - docker-ce
-      - docker-ce-cli
-
 - name: Ensure docker service directory exists
   ansible.builtin.file:
     path: "{{ docker_service_directory }}"
@@ -72,27 +60,27 @@
     mode: "0644"
   notify: Reload docker
 
+- name: Ensure docker config directory exists - {{ docker_config_dir }}
+  ansible.builtin.file:
+    path: "{{ docker_config_dir }}"
+    owner: "{{ docker_owner }}"
+    group: "{{ docker_group }}"
+    state: directory
+    mode: "0700"
+
+- name: Write docker daemon configuration file
+  ansible.builtin.template:
+    src: daemon.json.j2
+    dest: "{{ docker_daemon_conf_file }}"
+    owner: "{{ docker_owner }}"
+    group: "{{ docker_group }}"
+    mode: "0640"
+
 - name: Generate CA, server, and client certificates
   when: docker_generate_certificates
   notify:
     - Restart docker
   block:
-    - name: Ensure docker config directory exists - {{ docker_config_dir }}
-      ansible.builtin.file:
-        path: "{{ docker_config_dir }}"
-        owner: "{{ docker_owner }}"
-        group: "{{ docker_group }}"
-        state: directory
-        mode: "0700"
-
-    - name: Write docker daemon configuration file
-      ansible.builtin.template:
-        src: daemon.json.j2
-        dest: "{{ docker_daemon_conf_file }}"
-        owner: "{{ docker_owner }}"
-        group: "{{ docker_group }}"
-        mode: "0640"
-
     - name: Generate CA certificate
       ansible.builtin.import_tasks: ca-cert.yml
 

diff --git a/roles/docker/templates/daemon.json.j2 b/roles/docker/templates/daemon.json.j2
@@ -1,7 +1,11 @@
 {
+  {% if docker_generate_certificates %}
    "hosts": ["tcp://{{ docker_server_ip }}:{{ docker_server_port }}", "unix:///var/run/docker.sock"],
    "tlsverify": {{ docker_tls_verify | lower }},
    "tlscacert": "{{ docker_ca_cert }}",
    "tlscert": "{{ docker_server_cert }}",
    "tlskey": "{{ docker_server_key }}"
+  {% else %}
+   "hosts": ["tcp://{{ docker_server_ip }}:{{ docker_server_port }}", "unix:///var/run/docker.sock"]
+  {% endif %}
  }
diff --git a/roles/provision/tasks/main.yml b/roles/provision/tasks/main.yml
@@ -18,10 +18,6 @@
   ansible.builtin.yum:
     name: "*"
     state: latest
-    exclude:
-      # https://groups.google.com/g/xnat_discussion/c/yyPBkN4kayE/m/LUe5GQH5AAAJ
-      - docker-ce
-      - docker-ce-cli
 
   tags:
     - molecule-idempotence-notest

diff --git a/roles/xnat/defaults/main.yml b/roles/xnat/defaults/main.yml
@@ -1,5 +1,5 @@
 ---
-xnat_version: 1.8.10.1
+xnat_version: 1.9.1
 xnat_pipeline_version: 1.8.10
 xnat_archive_dir: "{{ xnat_root_dir }}/archive"
 xnat_prearchive_dir: "{{ xnat_root_dir }}/prearchive"
@@ -39,16 +39,17 @@ xnat_ldap_keystore_alias: ""
 
 # Plugins
 xnat_plugin_urls:
-  - https://api.bitbucket.org/2.0/repositories/xnatdev/xsync/downloads/xsync-plugin-all-1.7.0.jar
-  - https://api.bitbucket.org/2.0/repositories/xnatx/ldap-auth-plugin/downloads/ldap-auth-plugin-1.1.0.jar
-  - https://api.bitbucket.org/2.0/repositories/xnatdev/container-service/downloads/container-service-3.4.3-fat.jar
-  - https://api.bitbucket.org/2.0/repositories/xnatx/xnatx-batch-launch-plugin/downloads/batch-launch-0.6.0.jar
+  - https://api.bitbucket.org/2.0/repositories/xnatdev/xsync/downloads/xsync-plugin-all-1.8.1.jar
+  - https://api.bitbucket.org/2.0/repositories/xnatx/ldap-auth-plugin/downloads/ldap-auth-plugin-1.2.1.jar
+  - https://api.bitbucket.org/2.0/repositories/xnatdev/container-service/downloads/container-service-3.6.1-fat.jar
+  - https://api.bitbucket.org/2.0/repositories/xnatx/xnatx-batch-launch-plugin/downloads/batch-launch-0.7.0.jar
   - https://github.com/VUIIS/dax/raw/main/misc/xnat-plugins/dax-plugin-genProcData-1.4.2.jar
-  - https://api.bitbucket.org/2.0/repositories/icrimaginginformatics/ohif-viewer-xnat-plugin/downloads/ohif-viewer-3.6.2.jar
-  - https://api.bitbucket.org/2.0/repositories/xnatx/ml-plugin/downloads/ml-plugin-1.0.2.jar
-  - https://api.bitbucket.org/2.0/repositories/xnatx/datasets-plugin/downloads/datasets-plugin-1.0.3.jar
+  - https://api.bitbucket.org/2.0/repositories/icrimaginginformatics/ohif-viewer-xnat-plugin/downloads/ohif-viewer-3.7.0-XNAT-1.8.10.jar
+  - https://api.bitbucket.org/2.0/repositories/xnatx/ml-schema-plugin/downloads/ml-schema-plugin-1.0.0.jar
+  - https://api.bitbucket.org/2.0/repositories/xnatx/datasets-schema-plugin/downloads/datasets-schema-plugin-1.0.0.jar
   - https://api.bitbucket.org/2.0/repositories/xnatdev/xnat-image-viewer-plugin/downloads/ximgview-plugin-1.0.2.jar
   - https://api.bitbucket.org/2.0/repositories/xnatx/xnatx-dxm-settings-plugin/downloads/dxm-settings-plugin-1.0.jar
+  - https://api.bitbucket.org/2.0/repositories/xnatx/pipeline_engine_plugin/downloads/pipeline_engine_ui-1.1.0-xpl.jar
 
 xnat_plugin_bundle_urls: [] # yamllint disable-line rule:brackets
 xnat_plugin_packages: [] # yamllint disable-line rule:brackets

diff --git a/roles/xnat_container_service/defaults/main.yml b/roles/xnat_container_service/defaults/main.yml
@@ -1,6 +1,7 @@
 ---
 xnat_container_service_owner: tomcat
 xnat_container_service_group: tomcat
+xnat_container_service_use_ssl: false
 xnat_container_service_certificate_directory: /usr/share/tomcat/.docker
 xnat_container_service_key: /usr/share/tomcat/.docker/key.pem
 xnat_container_service_csr: /usr/share/tomcat/.docker/docker.csr

diff --git a/roles/xnat_container_service/tasks/copy_certs_to_client.yml b/roles/xnat_container_service/tasks/copy_certs_to_client.yml
@@ -0,0 +1,36 @@
+---
+- name: Ensure Docker certificate directory exists on the client
+  ansible.builtin.file:
+    path: "{{ xnat_container_service_certificate_directory }}"
+    state: directory
+    owner: "{{ xnat_container_service_owner }}"
+    group: "{{ xnat_container_service_group }}"
+    mode: "0700"
+
+- name: Copy Docker server certificate from Ansible Controller cache to client
+  ansible.builtin.copy:
+    src: "{{ xnat_container_service_certificate_cache_directory }}/ca.pem"
+    dest: "{{ xnat_container_service_server_ca_cert }}"
+    owner: "{{ xnat_container_service_owner }}"
+    group: "{{ xnat_container_service_group }}"
+    mode: "0600"
+
+- name:
+    Copy signed Docker client certificate from Ansible Controller cache to
+    client
+  ansible.builtin.copy:
+    src:
+      "{{ xnat_container_service_certificate_cache_directory }}/{{
+      xnat_container_service_client_hostname }}.cert"
+    dest: "{{ xnat_container_service_cert }}"
+    owner: "{{ xnat_container_service_owner }}"
+    group: "{{ xnat_container_service_group }}"
+    mode: "0600"
+
+- name: Copy private key from Ansible Controller cache to client
+  ansible.builtin.copy:
+    src: "{{ xnat_container_service_certificate_cache_directory }}/key.pem"
+    dest: "{{ xnat_container_service_key }}"
+    owner: "{{ xnat_container_service_owner }}"
+    group: "{{ xnat_container_service_group }}"
+    mode: "0600"
diff --git a/roles/xnat_container_service/tasks/main.yml b/roles/xnat_container_service/tasks/main.yml
@@ -1,39 +1,7 @@
 ---
-- name: Ensure Docker certificate directory exists on the client
-  ansible.builtin.file:
-    path: "{{ xnat_container_service_certificate_directory }}"
-    state: directory
-    owner: "{{ xnat_container_service_owner }}"
-    group: "{{ xnat_container_service_group }}"
-    mode: "0700"
-
-- name: Copy Docker server certificate from Ansible Controller cache to client
-  ansible.builtin.copy:
-    src: "{{ xnat_container_service_certificate_cache_directory }}/ca.pem"
-    dest: "{{ xnat_container_service_server_ca_cert }}"
-    owner: "{{ xnat_container_service_owner }}"
-    group: "{{ xnat_container_service_group }}"
-    mode: "0600"
-
-- name:
-    Copy signed Docker client certificate from Ansible Controller cache to
-    client
-  ansible.builtin.copy:
-    src:
-      "{{ xnat_container_service_certificate_cache_directory }}/{{
-      xnat_container_service_client_hostname }}.cert"
-    dest: "{{ xnat_container_service_cert }}"
-    owner: "{{ xnat_container_service_owner }}"
-    group: "{{ xnat_container_service_group }}"
-    mode: "0600"
-
-- name: Copy private key from Ansible Controller cache to client
-  ansible.builtin.copy:
-    src: "{{ xnat_container_service_certificate_cache_directory }}/key.pem"
-    dest: "{{ xnat_container_service_key }}"
-    owner: "{{ xnat_container_service_owner }}"
-    group: "{{ xnat_container_service_group }}"
-    mode: "0600"
+- name: Copy SSL certificates to client
+  ansible.builtin.include_tasks: copy_certs_to_client.yml
+  when: xnat_container_service_use_ssl
 
 - name: Configure XNAT to talk to container service
   ansible.builtin.uri:
@@ -45,9 +13,11 @@
     body:
       name: "{{ xnat_container_service_name }}"
       host:
-        https://{{ xnat_container_service_hostname }}:{{
+        tcp://{{ xnat_container_service_hostname }}:{{
         xnat_container_service_port }}
-      cert-path: "{{ xnat_container_service_certificate_directory }}"
+      cert-path:
+        "{{ xnat_container_service_certificate_directory }} if {{
+        xnat_container_service_use_ssl }} else ''"
       swarm-mode: false
       path-translation-xnat-prefix:
         "{{ xnat_container_service_path_translation_xnat_prefix }}"