Skip to content

Commit

Permalink
add monitoring setup (#26)
Browse files Browse the repository at this point in the history 
Changes:

- Update `monitoring_client` role
- Update `monitoring_server` role
- Add molecule scenarios for testing `install_monitoring` playbook

This has been tested by running the `install_monitoring` playbook with
`mirsg-linux` as the monitoring server and `ucl-test-xnat.cs.ucl.ac.uk`
as the client.

---------

Co-authored-by: Paul Smith <[email protected]>
  • Loading branch information
@drmatthews @p-j-smith
drmatthews and p-j-smith authored Jan 3, 2024
1 parent 91753ae commit c5af6bd
Show file tree
Hide file tree
Showing 42 changed files with 954 additions and 435 deletions.
53 changes: 53 additions & 0 deletions .github/workflows/molecule-monitoring.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
name: Test install_monitoring playbook
on:
pull_request:
paths:
- "roles/monitoring_client/**"
- "roles/monitoring_server/**"
- ".github/workflows/molecule.yml"
- ".github/workflows/molecule-monitoring.yml"

jobs:
molecule-monitoring:
runs-on: ubuntu-latest
strategy:
fail-fast: true
matrix:
scenario:
- centos7_monitoring
- rocky9_monitoring
env:
MOLECULE_RUN_TAGS: monitoring
PY_COLORS: 1
ANSIBLE_FORCE_COLOR: 1

steps:
- name: Check out the codebase
uses: actions/checkout@v4
with:
path: ansible_collections/mirsg/infrastructure

- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: 3.11

- name: Install test dependencies
shell: bash
run: |
sudo apt-get update && sudo apt-get -y install rsync
python3 -m pip install --upgrade pip
python3 -m pip install \
ansible \
molecule \
molecule-plugins[docker] \
docker \
passlib \
bcrypt==4.0.1 \
requests
- name: Test with molecule
shell: bash
run: |
cd ansible_collections/mirsg/infrastructure/tests
molecule test --scenario-name ${{ matrix.scenario }}
146 changes: 98 additions & 48 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# mirsg.infrastructure Ansible Collection

This repository contains the `mirsg.infrastructure` Ansible Collection. This collection can be used to
configure infrastructure for deploying XNAT and OMERO.
This repository contains the `mirsg.infrastructure` Ansible Collection. This
collection can be used to configure infrastructure for deploying XNAT and OMERO.

## Included content

Expand All @@ -13,16 +13,17 @@ configure infrastructure for deploying XNAT and OMERO.

## External requirements

Before using this collection and its playbooks, you must install the
[necessary Ansible collections and roles](meta/requirements.yml).
Before using this collection and its playbooks, you must install the [necessary
Ansible collections and roles](meta/requirements.yml).

## Using this collection

You can install this collection using the `ansible-galaxy` command-line tool:
This collection can be installed using the `ansible-galaxy` command-line tool:

ansible-galaxy collection install https://github.com/UCL-MIRSG/ansible-collection-infra.git
ansible-galaxy collection install
https://github.com/UCL-MIRSG/ansible-collection-infra.git

You can also include it in a `requirements.yml` file and install it via
It can also be included in a `requirements.yml` file and install it via
`ansible-galaxy collection install -r requirements.yml` using the format:

```yaml
Expand All @@ -35,114 +36,163 @@ collections:
## Testing this collection
We use [Ansible Molecule](https://ansible.readthedocs.io/projects/molecule/) and its
[Docker plugin](https://github.com/ansible-community/molecule-plugins) to test the roles
and playbooks in this collection.
[Ansible Molecule](https://ansible.readthedocs.io/projects/molecule/) and its
[Docker plugin](https://github.com/ansible-community/molecule-plugins) is used
to test the roles and playbooks in this collection.
If you would like to run the tests locally you will need to:
To run the tests locally you will need to:
- clone this repository
- install Ansible Molecule and other test requirements
- install and start the [Docker Engine](https://docs.docker.com/engine/install/) if you have not done so already
- if necessary, add your user to the docker group in order to access the Docker Engine
- install and start the [Docker Engine](https://docs.docker.com/engine/install/)
or [Docker Desktop](https://www.docker.com/products/docker-desktop/) on MacOS,
if you have not done so already
- if necessary (i.e. you are running Docker on Linux), add your user to the
docker group in order to access the Docker Engine
- run the tests using Molecule
### Clone this repository
To test a collection, Molecule requires that it is in the path
`ansible_collections/<namespace>/<collection name>`. This means when you clone this repository you
must ensure it is in the path `ansible_collections/mirsg/infrastructure`. The simplest way to do this is:
`ansible_collections/<namespace>/<collection name>`. This means when you clone
this repository you must ensure it is in the path
`ansible_collections/mirsg/infrastructure`. The simplest way to do this is:

```
```shell
git clone [email protected]:UCL-MIRSG/ansible-collection-infra.git ansible_collections/mirsg/infrastructure
```

### Install Ansible Molecule

Before running the tests you'll need to install Molecule, the Docker plugin, and the Python Docker
Engine API using `pip`:
Before running the tests install Molecule, the Docker plugin, and the Python
Docker Engine API using `pip`:

```
```shell
python -m pip install molecule 'molecule-plugins[docker]' docker
```

### Run the tests using Molecule
### Testing the roles using Molecule

Molecue 6.0 requires that the test configuration is not in the top-level directory of the
collection. To support running the tests with Molecule 6, the Molecule configuration is in
`ansible_collections/mirsg/infrastructure/tests`. To run the tests you must be in this directory:
Molecue 6.0 requires that the test configuration is not in the top-level
directory of the collection. To support running the tests with Molecule 6, the
Molecule configuration is in `ansible_collections/mirsg/infrastructure/tests`.
To run the tests you must be in this directory:

```
```shell
cd ansible_collections/mirsg/infrastructure/tests
```

This collection is tested using a
[Molecule scenario](https://ansible.readthedocs.io/projects/molecule/getting-started/#molecule-scenarios) -
that runs on CentOS 7 and Rocky 9. The available roles are in the molecule subdirectory.
This collection contains [molecule
scenarios](https://ansible.readthedocs.io/projects/molecule/getting-started/#molecule-scenarios)
that allow testing of the individual roles. There are scenarios that run the
tests on both CentOS 7 and Rocky 9.

To run the tests for this CentOS 7 sceneraio:
To run the CentOS 7 tests for the roles:

```
```shell
molecule test -s centos7_roles
```

This command will:

- install the required Ansible roles and collections
- create a CentOS 7 container
- `tests/molecule/resources/prepare.yml` playbook to do any required setup for the roles
- run the `tests/molecule/resources/converge.yml` playbook, which will run the roles in this collection
- run `tests/molecule/resources/converge.yml` a second time to check the roles are
- `tests/molecule/resources/shared/prepare.yml` playbook to do any required
setup for the roles
- run the `tests/molecule/resources/roles/converge.yml` playbook, which will run
the roles in this collection
- run `tests/molecule/resources/roles/converge.yml` a second time to check the
roles are
[idempotent](https://docs.ansible.com/ansible/latest/reference_appendices/glossary.html#term-Idempotency)
- run `tests/molecule/resources/verify.yml`
- run `tests/molecule/resources/roles/verify.yml`
- destroy the CentOS 7 container

### Test a single role in the collection

Individual roles in the collection can be tested by setting the
`MOLECULE_RUN_TAGS` environment variable:

```shell
export MOLECULE_RUN_TAGS=provision
molecule test -s centos7_roles
```

### Inspecting the Container

If the `molecule test` command fails at any stage, the container is immediately destroyed.
This is due to the pre-defined sequence of actions the Molecule take when the `molecule test`
command is invoked.
If the `molecule test` command fails at any stage, the container is immediately
destroyed. This is due to the pre-defined sequence of actions the Molecule take
when the `molecule test` command is invoked.

If you would like to be able to access the test container, you should instead
use the `molecule converge` command. To run this on CentOS 7:

```
```shell
molecule converge -s centos7_roles
```

This will install necessary Ansible roles and collections, create the test container, and run the
molecule playbooks. If the deployment fails, the container is not destroyed.
This will install necessary Ansible roles and collections, create the test
container, and run the molecule playbooks. If the deployment fails, the
container is not destroyed.

#### Access the container

Once the command has finished running, you can access the container using the name
of the scenario. To access the container for the `centos7` scenario:
Once the command has finished running, you can access the container using the
name of the scenario. To access the container for the `centos7_roles` scenario:

```
```shell
molecule login -s centos7_roles
```

If testing a role or playbook where Molecule creates multiple containers,
individual hosts can be accessed using the `--host` flag:

```shell
molecule login -s centos7_monitoring --host mserv
```

#### Destroy the container

If you use the `molecule converge` command, you must remember to destroy the container, network,
and volumes yourself. You can do this using the `molecule destroy` command:
If you use the `molecule converge` command, you must remember to destroy the
container, network, and volumes yourself. You can do this using the `molecule
destroy` command:

```
```shell
molecule destroy -s centos7_roles
```

### Test a playbook

Playbooks in the collection can also be tested using Molecule. An example of how
this can be done can be seen by looking at the tests for the
`mirsg.install_monitoring` playbook in this collection. This is tested on CentOS
7 and RockyLinux 9 using the
[centos7_monitoring](./tests/molecule/centos7_monitoring/) and
[rocky9_monitoring](./tests/molecule/rocky9_monitoring/) scenarios. An inventory
and associated group variables can be found in
[resources/monitoring/inventory](./tests/molecule/resources/monitoring/inventory/).
Testing the playbook also requires its own
[converge.yml](./tests/molecule/resources/monitoring/converge.yml) playbook but
it uses the shared [prepare.yml](./tests/molecule/resources/shared/prepare.yml)
playbook. Running the tests then proceeds as with testing the roles:

```shell
export MOLECULE_RUN_TAGS=monitoring
molecule test -s centos7_monitoring
```

### Integration tests

When a PR that modifies a role is opened, the changes are
[tested](.github/workflows/) by deploying that role using GitHub Actions. See the
[`molecule-firewalld` workflow](.github/workflows/molecule-firewalld.yml)
[tested](.github/workflows/) by deploying that role using GitHub Actions. See
the [`molecule-firewalld` workflow](.github/workflows/molecule-firewalld.yml)
for an example.

## Code style and formatting

[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://github.com/pre-commit/pre-commit)

This repo has `pre-commit` hooks enabled, for instructions see <https://github.com/UCL-MIRSG/.github/tree/main/precommit>.
This repo has `pre-commit` hooks enabled, for instructions see
<https://github.com/UCL-MIRSG/.github/tree/main/precommit>.

## License

Expand Down
40 changes: 40 additions & 0 deletions playbooks/install_monitoring.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
---
- name: Set up monitoring server
hosts: monitoring_host
become: true
gather_facts: true
pre_tasks:
- name: Generate list of docker clients from `monitoring_client` group
ansible.builtin.set_fact:
docker_client_hostnames: >
{{
query('inventory_hostnames', ansible_limit | default('')) |
intersect(groups['monitoring_client']) |
map('extract', hostvars, monitoring_server_hostname_extractor) |
list | default([])
}}
failed_when: docker_client_hostnames | length == 0

roles:
- role: mirsg.infrastructure.provision
- role: mirsg.infrastructure.install_python
- role: mirsg.infrastructure.docker
- role: mirsg.infrastructure.monitoring_server

- name: Set up monitoring clients
hosts: monitoring_client
become: true
gather_facts: true

roles:
- role: mirsg.infrastructure.monitoring_client
- role: mirsg.infrastructure.firewalld
vars:
internal_zone_sources:
- "{{ monitoring_client_monitoring_server_ip }}"
rich_rules:
- zone: "internal"
rule: "family=ipv4 source \
address={{ monitoring_client_monitoring_server_ip }}/32 \
port protocol=tcp \
port={{ monitoring_client_node_exporter_port }} accept"
29 changes: 0 additions & 29 deletions roles/create_monitoring_client_csr/tasks/main.yml
View file

This file was deleted.

24 changes: 24 additions & 0 deletions roles/monitoring_client/defaults/main.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
---
monitoring_client_node_exporter_version: 1.7.0

monitoring_client_node_exporter_binary:
"https://github.com/prometheus/node_exporter/releases/download/v\
{{ monitoring_client_node_exporter_version }}/node_exporter-\
{{ monitoring_client_node_exporter_version }}.linux-amd64.tar.gz"
monitoring_client_node_exporter_install_dir: /usr/bin/node_exporter
monitoring_client_node_exporter_service: /etc/systemd/system/node_exporter.service
monitoring_client_node_exporter_web_config: /usr/bin/node_exporter/web.yml
monitoring_client_node_exporter_port: 9100
monitoring_client_node_exporter_ssl_key: /usr/bin/node_exporter/node_exporter.key
monitoring_client_node_exporter_ssl_cert: /usr/bin/node_exporter/node_exporter.crt

monitoring_client_owner: root
monitoring_client_group: root
monitoring_client_cert_dir: /root/monitoring_certs
monitoring_client_ssl_key_file: /root/monitoring_certs/key.pem
monitoring_client_ssl_csr_file: /root/monitoring_certs/monitoring.csr
monitoring_client_ssl_pk8_file: /root/monitoring_certs/monitoring.pk8
monitoring_client_ssl_cert_file: /root/monitoring_certs/cert.pem
monitoring_client_server_ca_cert_file: /root/monitoring_certs/ca.pem
monitoring_client_exporter_username: prometheus
monitoring_client_exporter_password: ""
Loading

0 comments on commit c5af6bd

Please sign in to comment.