From 01f0b676cba98e156e4ea2bb9b5d7f1fb058dbf3 Mon Sep 17 00:00:00 2001 From: Seunghun Lee <45145778+seunghun1ee@users.noreply.github.com> Date: Thu, 5 Dec 2024 12:02:21 +0000 Subject: [PATCH] Doc from openstack admin guide (#1083) --- doc/source/configuration/cephadm.rst | 8 +- doc/source/configuration/ci-cd.rst | 14 +- doc/source/configuration/lvm.rst | 4 +- doc/source/configuration/monitoring.rst | 2 + doc/source/configuration/release-train.rst | 2 +- doc/source/configuration/swap.rst | 4 +- doc/source/configuration/vault.rst | 5 + doc/source/configuration/wazuh.rst | 71 +- doc/source/contributor/pre-commit.rst | 6 +- .../bifrost-hardware-inventory-management.rst | 282 ++++++ doc/source/operations/ceph-management.rst | 294 +++++++ .../operations/control-plane-operation.rst | 386 +++++++++ doc/source/operations/customising-horizon.rst | 118 +++ doc/source/operations/gpu-in-openstack.rst | 807 ++++++++++++++++++ doc/source/operations/hotfix-playbook.rst | 4 +- doc/source/operations/index.rst | 12 +- doc/source/operations/migrating-vm.rst | 22 + doc/source/operations/nova-compute-ironic.rst | 8 +- doc/source/operations/octavia.rst | 4 +- ...penstack-projects-and-users-management.rst | 38 + .../operations/openstack-reconfiguration.rst | 150 ++++ doc/source/operations/secret-rotation.rst | 4 +- doc/source/operations/upgrading-openstack.rst | 2 +- 23 files changed, 2197 insertions(+), 50 deletions(-) create mode 100644 doc/source/operations/bifrost-hardware-inventory-management.rst create mode 100644 doc/source/operations/ceph-management.rst create mode 100644 doc/source/operations/control-plane-operation.rst create mode 100644 doc/source/operations/customising-horizon.rst create mode 100644 doc/source/operations/gpu-in-openstack.rst create mode 100644 doc/source/operations/migrating-vm.rst create mode 100644 doc/source/operations/openstack-projects-and-users-management.rst create mode 100644 doc/source/operations/openstack-reconfiguration.rst diff --git a/doc/source/configuration/cephadm.rst b/doc/source/configuration/cephadm.rst index bcb13cd6c..19112a083 100644 --- a/doc/source/configuration/cephadm.rst +++ b/doc/source/configuration/cephadm.rst @@ -1,6 +1,8 @@ -==== -Ceph -==== +.. _cephadm-kayobe: + +================ +Cephadm & Kayobe +================ This section describes how to use the Cephadm integration included in StackHPC Kayobe configuration to deploy Ceph. diff --git a/doc/source/configuration/ci-cd.rst b/doc/source/configuration/ci-cd.rst index 6e495c2e8..dcf86350e 100644 --- a/doc/source/configuration/ci-cd.rst +++ b/doc/source/configuration/ci-cd.rst @@ -57,26 +57,26 @@ Runner Deployment Ideally an Infra VM could be used here or failing that the control host. Wherever it is deployed the host will need access to the :code:`admin_network`, :code:`public_network` and the :code:`pulp registry` on the seed. -2. Edit the environment's :code:`${KAYOBE_CONFIG_PATH}/environments/${KAYOBE_ENVIRONMENT}/inventory/groups` to add the predefined :code:`github-runners` group to :code:`infra-vms` +2. Edit the environment's :code:`$KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/inventory/groups` to add the predefined :code:`github-runners` group to :code:`infra-vms` .. code-block:: ini [infra-vms:children] github-runners -3. Edit the environment's :code:`${KAYOBE_CONFIG_PATH}/environments/${KAYOBE_ENVIRONMENT}/inventory/hosts` to define the host(s) that will host the runners. +3. Edit the environment's :code:`$KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/inventory/hosts` to define the host(s) that will host the runners. .. code-block:: ini [github-runners] prod-runner-01 -4. Provide all the relevant Kayobe :code:`group_vars` for :code:`github-runners` under :code:`${KAYOBE_CONFIG_PATH}/environments/${KAYOBE_ENVIRONMENT}/inventory/group_vars/github-runners` +4. Provide all the relevant Kayobe :code:`group_vars` for :code:`github-runners` under :code:`$KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/inventory/group_vars/github-runners` * `infra-vms` ensuring all required `infra_vm_extra_network_interfaces` are defined * `network-interfaces` * `python-interpreter.yml` ensuring that `ansible_python_interpreter: /usr/bin/python3` has been set -5. Edit the ``${KAYOBE_CONFIG_PATH}/inventory/group_vars/github-runners/runners.yml`` file which will contain the variables required to deploy a series of runners. +5. Edit the ``$KAYOBE_CONFIG_PATH/inventory/group_vars/github-runners/runners.yml`` file which will contain the variables required to deploy a series of runners. Below is a core set of variables that will require consideration and modification for successful deployment of the runners. The number of runners deployed can be configured by removing and extending the dict :code:`github-runners`. As for how many runners present three is suitable number as this would prevent situations where long running jobs could halt progress other tasks whilst waiting for a free runner. @@ -120,7 +120,7 @@ Runner Deployment 7. If the host is an actual Infra VM then please refer to upstream `Infrastructure VMs `__ documentation for additional configuration and steps. -8. Run :code:`kayobe playbook run ${KAYOBE_CONFIG_PATH}/ansible/deploy-github-runner.yml` +8. Run :code:`kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/deploy-github-runner.yml` 9. Check runners have registered properly by visiting the repository's :code:`Action` tab -> :code:`Runners` -> :code:`Self-hosted runners` @@ -130,9 +130,9 @@ Runner Deployment Workflow Deployment ------------------- -1. Edit :code:`${KAYOBE_CONFIG_PATH}/inventory/group_vars/github-writer/writer.yml` in the base configuration making the appropriate changes to your deployments specific needs. See documentation for `stackhpc.kayobe_workflows.github `__. +1. Edit :code:`$KAYOBE_CONFIG_PATH/inventory/group_vars/github-writer/writer.yml` in the base configuration making the appropriate changes to your deployments specific needs. See documentation for `stackhpc.kayobe_workflows.github `__. -2. Run :code:`kayobe playbook run ${KAYOBE_CONFIG_PATH}/ansible/write-github-workflows.yml` +2. Run :code:`kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/write-github-workflows.yml` 3. Add all required secrets and variables to repository either via the GitHub UI or GitHub CLI (may require repository owner) diff --git a/doc/source/configuration/lvm.rst b/doc/source/configuration/lvm.rst index a96ca8db9..bb2b7862c 100644 --- a/doc/source/configuration/lvm.rst +++ b/doc/source/configuration/lvm.rst @@ -93,6 +93,6 @@ hosts: .. code-block:: console - mkdir -p ${KAYOBE_CONFIG_PATH}/hooks/overcloud-host-configure/pre.d - cd ${KAYOBE_CONFIG_PATH}/hooks/overcloud-host-configure/pre.d + mkdir -p $KAYOBE_CONFIG_PATH/hooks/overcloud-host-configure/pre.d + cd $KAYOBE_CONFIG_PATH/hooks/overcloud-host-configure/pre.d ln -s ../../../ansible/growroot.yml 30-growroot.yml diff --git a/doc/source/configuration/monitoring.rst b/doc/source/configuration/monitoring.rst index 8215dc48b..77c5e47f7 100644 --- a/doc/source/configuration/monitoring.rst +++ b/doc/source/configuration/monitoring.rst @@ -2,6 +2,8 @@ Monitoring ========== +.. _monitoring-service-configuration: + Monitoring Configuration ======================== diff --git a/doc/source/configuration/release-train.rst b/doc/source/configuration/release-train.rst index f77109aff..5ed9b50c7 100644 --- a/doc/source/configuration/release-train.rst +++ b/doc/source/configuration/release-train.rst @@ -1,4 +1,4 @@ -.. _stackhpc_release_train: +.. _stackhpc-release-train: ====================== StackHPC Release Train diff --git a/doc/source/configuration/swap.rst b/doc/source/configuration/swap.rst index 58545e906..241919574 100644 --- a/doc/source/configuration/swap.rst +++ b/doc/source/configuration/swap.rst @@ -23,6 +23,6 @@ hosts: .. code-block:: console - mkdir -p ${KAYOBE_CONFIG_PATH}/hooks/overcloud-host-configure/post.d - cd ${KAYOBE_CONFIG_PATH}/hooks/overcloud-host-configure/post.d + mkdir -p $KAYOBE_CONFIG_PATH/hooks/overcloud-host-configure/post.d + cd $KAYOBE_CONFIG_PATH/hooks/overcloud-host-configure/post.d ln -s ../../../ansible/swap.yml 10-swap.yml diff --git a/doc/source/configuration/vault.rst b/doc/source/configuration/vault.rst index 893af246c..61a5ab1c9 100644 --- a/doc/source/configuration/vault.rst +++ b/doc/source/configuration/vault.rst @@ -1,3 +1,5 @@ +.. _hashicorp-vault: + ================================ Hashicorp Vault for internal PKI ================================ @@ -111,6 +113,9 @@ Certificates generation Create the external TLS certificates (testing only) --------------------------------------------------- +This method should only be used for testing. For external TLS on production systems, +See `Installing External TLS Certificates `__. + Typically external API TLS certificates should be generated by a organisation's trusted internal or third-party CA. For test and development purposes it is possible to use Vault as a CA for the external API. diff --git a/doc/source/configuration/wazuh.rst b/doc/source/configuration/wazuh.rst index cd57716d3..40b8a973c 100644 --- a/doc/source/configuration/wazuh.rst +++ b/doc/source/configuration/wazuh.rst @@ -2,13 +2,20 @@ Wazuh ===== +`Wazuh `_ is a security monitoring platform. +It monitors for: + +* Security-related system events. +* Known vulnerabilities (CVEs) in versions of installed software. +* Misconfigurations in system security. + The short version ================= #. Create an infrastructure VM for the Wazuh manager, and add it to the wazuh-manager group #. Configure the infrastructure VM with kayobe: ``kayobe infra vm host configure`` #. Edit your config under - ``etc/kayobe/inventory/group_vars/wazuh-manager/wazuh-manager``, in + ``$KAYOBE_CONFIG_PATH/inventory/group_vars/wazuh-manager/wazuh-manager``, in particular the defaults assume that the ``provision_oc_net`` network will be used. #. Generate secrets: ``kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/wazuh-secrets.yml`` @@ -27,14 +34,14 @@ Provisioning an infra VM for Wazuh Manager. Kayobe supports :kayobe-doc:`provisioning infra VMs `. The following configuration may be used as a guide. Config for infra VMs is documented :kayobe-doc:`here `. -Add a Wazuh Manager host to the ``wazuh-manager`` group in ``etc/kayobe/inventory/hosts``. +Add a Wazuh Manager host to the ``wazuh-manager`` group in ``$KAYOBE_CONFIG_PATH/inventory/hosts``. .. code-block:: ini [wazuh-manager] os-wazuh -Add the ``wazuh-manager`` group to the ``infra-vms`` group in ``etc/kayobe/inventory/groups``. +Add the ``wazuh-manager`` group to the ``infra-vms`` group in ``$KAYOBE_CONFIG_PATH/inventory/groups``. .. code-block:: ini @@ -43,7 +50,7 @@ Add the ``wazuh-manager`` group to the ``infra-vms`` group in ``etc/kayobe/inven [infra-vms:children] wazuh-manager -Define VM sizing in ``etc/kayobe/inventory/group_vars/wazuh-manager/infra-vms``: +Define VM sizing in ``$KAYOBE_CONFIG_PATH/inventory/group_vars/wazuh-manager/infra-vms``: .. code-block:: yaml @@ -57,7 +64,7 @@ Define VM sizing in ``etc/kayobe/inventory/group_vars/wazuh-manager/infra-vms``: # Capacity of the infra VM data volume. infra_vm_data_capacity: "200G" -Optional: define LVM volumes in ``etc/kayobe/inventory/group_vars/wazuh-manager/lvm``. +Optional: define LVM volumes in ``$KAYOBE_CONFIG_PATH/inventory/group_vars/wazuh-manager/lvm``. ``/var/ossec`` often requires greater storage space, and ``/var/lib/wazuh-indexer`` may be beneficial too. @@ -79,7 +86,7 @@ may be beneficial too. create: true -Define network interfaces ``etc/kayobe/inventory/group_vars/wazuh-manager/network-interfaces``: +Define network interfaces ``$KAYOBE_CONFIG_PATH/inventory/group_vars/wazuh-manager/network-interfaces``: (The following is an example - the names will depend on your particular network configuration.) @@ -91,7 +98,7 @@ Define network interfaces ``etc/kayobe/inventory/group_vars/wazuh-manager/networ The Wazuh manager may need to be exposed externally, in which case it may require another interface. -This can be done as follows in ``etc/kayobe/inventory/group_vars/wazuh-manager/network-interfaces``, +This can be done as follows in ``$KAYOBE_CONFIG_PATH/inventory/group_vars/wazuh-manager/network-interfaces``, with the network defined in ``networks.yml`` as usual. .. code-block:: yaml @@ -183,7 +190,7 @@ Deploying Wazuh Manager services Setup ----- -To install a specific version modify the wazuh-ansible entry in ``etc/kayobe/ansible/requirements.yml``: +To install a specific version modify the wazuh-ansible entry in ``$KAYOBE_CONFIG_PATH/ansible/requirements.yml``: .. code-block:: yaml @@ -204,7 +211,7 @@ Edit the playbook and variables to your needs: Wazuh manager configuration --------------------------- -Wazuh manager playbook is located in ``etc/kayobe/ansible/wazuh-manager.yml``. +Wazuh manager playbook is located in ``$KAYOBE_CONFIG_PATH/ansible/wazuh-manager.yml``. Running this playbook will: * generate certificates for wazuh-manager @@ -214,7 +221,7 @@ Running this playbook will: * setup and deploy wazuh-dashboard on wazuh-manager vm * copy certificates over to wazuh-manager vm -Wazuh manager variables file is located in ``etc/kayobe/inventory/group_vars/wazuh-manager/wazuh-manager``. +Wazuh manager variables file is located in ``$KAYOBE_CONFIG_PATH/inventory/group_vars/wazuh-manager/wazuh-manager``. You may need to modify some of the variables, including: @@ -225,24 +232,27 @@ You may need to modify some of the variables, including: If you are using multiple environments, and you need to customise Wazuh in each environment, create override files in an appropriate directory, - for example ``etc/kayobe/environments/production/inventory/group_vars/``. + for example ``$KAYOBE_CONFIG_PATH/environments/production/inventory/group_vars/``. Files which values can be overridden (in the context of Wazuh): - - etc/kayobe/inventory/group_vars/wazuh/wazuh-manager/wazuh-manager - - etc/kayobe/wazuh-manager.yml - - etc/kayobe/inventory/group_vars/wazuh/wazuh-agent/wazuh-agent + - $KAYOBE_CONFIG_PATH/inventory/group_vars/wazuh/wazuh-manager/wazuh-manager + - $KAYOBE_CONFIG_PATH/wazuh-manager.yml + - $KAYOBE_CONFIG_PATH/inventory/group_vars/wazuh/wazuh-agent/wazuh-agent + +You'll need to run ``wazuh-manager.yml`` playbook again to apply customisation. Secrets ------- -Wazuh secrets playbook is located in ``etc/kayobe/ansible/wazuh-secrets.yml``. +Wazuh requires that secrets or passwords are set for itself and the services with which it communiticates. +Wazuh secrets playbook is located in ``$KAYOBE_CONFIG_PATH/ansible/wazuh-secrets.yml``. Running this playbook will generate and put pertinent security items into secrets vault file which will be placed in ``$KAYOBE_CONFIG_PATH/wazuh-secrets.yml``. If using environments it ends up in ``$KAYOBE_CONFIG_PATH/environments//wazuh-secrets.yml`` Remember to encrypt! -Wazuh secrets template is located in ``etc/kayobe/ansible/templates/wazuh-secrets.yml.j2``. +Wazuh secrets template is located in ``$KAYOBE_CONFIG_PATH/ansible/templates/wazuh-secrets.yml.j2``. It will be used by wazuh secrets playbook to generate wazuh secrets vault file. @@ -250,6 +260,10 @@ It will be used by wazuh secrets playbook to generate wazuh secrets vault file. kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/wazuh-secrets.yml +.. note:: Use ``ansible-vault`` to view the secrets: + + ``ansible-vault view --vault-password-file ~/vault.password $KAYOBE_CONFIG_PATH/inventory/group_vars/wazuh-manager/wazuh-secrets.yml`` + Configure Wazuh Dashboard's Server Host --------------------------------------- @@ -366,7 +380,7 @@ Verification ------------ The Wazuh portal should be accessible on port 443 of the Wazuh -manager’s IPs (using HTTPS, with the root CA cert in ``etc/kayobe/ansible/wazuh/certificates/wazuh-certificates/root-ca.pem``). +manager’s IPs (using HTTPS, with the root CA cert in ``$KAYOBE_CONFIG_PATH/ansible/wazuh/certificates/wazuh-certificates/root-ca.pem``). The first login should be as the admin user, with the opendistro_admin_password password in ``$KAYOBE_CONFIG_PATH/wazuh-secrets.yml``. This will create the necessary indices. @@ -378,9 +392,9 @@ Logs are in ``/var/log/wazuh-indexer/wazuh.log``. There are also logs in the jou Wazuh agents ============ -Wazuh agent playbook is located in ``etc/kayobe/ansible/wazuh-agent.yml``. +Wazuh agent playbook is located in ``$KAYOBE_CONFIG_PATH/ansible/wazuh-agent.yml``. -Wazuh agent variables file is located in ``etc/kayobe/inventory/group_vars/wazuh-agent/wazuh-agent``. +Wazuh agent variables file is located in ``$KAYOBE_CONFIG_PATH/inventory/group_vars/wazuh-agent/wazuh-agent``. You may need to modify some variables, including: @@ -390,6 +404,25 @@ Deploy the Wazuh agents: ``kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/wazuh-agent.yml`` +The Wazuh Agent is deployed to all hosts in the ``wazuh-agent`` +inventory group, comprising the ``seed`` group +plus the ``overcloud`` group (containing all hosts in the +OpenStack control plane). + +.. code-block:: ini + + [wazuh-agent:children] + seed + overcloud + +The hosts running Wazuh Agent should automatically be registered +and visible within the Wazuh Manager dashboard. + +.. note:: It is good practice to use a `Kayobe deploy hook + `_ + to automate deployment and configuration of the Wazuh Agent + following a run of ``kayobe overcloud host configure``. + Verification ------------ diff --git a/doc/source/contributor/pre-commit.rst b/doc/source/contributor/pre-commit.rst index 3afffc11b..dc9f691bf 100644 --- a/doc/source/contributor/pre-commit.rst +++ b/doc/source/contributor/pre-commit.rst @@ -29,12 +29,12 @@ Once done you should find `pre-commit` is available within the `kayobe` virtuale To run the playbook using the following command -- ``kayobe playbook run ${KAYOBE_CONFIG_PATH}/ansible/install-pre-commit-hooks.yml`` +- ``kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/install-pre-commit-hooks.yml`` Whereas to run the playbook when control host bootstrap runs ensure it registered as symlink using the following command -- ``mkdir -p ${KAYOBE_CONFIG_PATH}/hooks/control-host-bootstrap/post.d`` -- ``ln -s ${KAYOBE_CONFIG_PATH}/ansible/install-pre-commit-hooks.yml ${KAYOBE_CONFIG_PATH}/hooks/control-host-bootstrap/post.d/install-pre-commit-hooks.yml`` +- ``mkdir -p $KAYOBE_CONFIG_PATH/hooks/control-host-bootstrap/post.d`` +- ``ln -s $KAYOBE_CONFIG_PATH/ansible/install-pre-commit-hooks.yml $KAYOBE_CONFIG_PATH/hooks/control-host-bootstrap/post.d/install-pre-commit-hooks.yml`` All that remains is the installation of the hooks themselves which can be accomplished either by running `pre-commit run` or using `git commit` when you have changes that need to be committed. diff --git a/doc/source/operations/bifrost-hardware-inventory-management.rst b/doc/source/operations/bifrost-hardware-inventory-management.rst new file mode 100644 index 000000000..673041819 --- /dev/null +++ b/doc/source/operations/bifrost-hardware-inventory-management.rst @@ -0,0 +1,282 @@ +===================================== +Bifrost Hardware Inventory Management +===================================== + +In most deployments, hardware inventory is managed by the Bifrost service. + +Reconfiguring Control Plane Hardware +==================================== + +If a server's hardware or firmware configuration is changed, it should be +re-inspected in Bifrost before it is redeployed into service. A single server +can be reinspected like this: + +.. code-block:: console + + kayobe overcloud hardware inspect --limit + +.. _enrolling-new-hypervisors: + +Enrolling New Hypervisors +------------------------- + +New hypervisors can be added to the Bifrost inventory by using its discovery +capabilities. Assuming that new hypervisors have IPMI enabled and are +configured to network boot on the provisioning network, the following commands +will instruct them to PXE boot. The nodes will boot on the Ironic Python Agent +kernel and ramdisk, which is configured to extract hardware information and +send it to Bifrost. Note that IPMI credentials can be found in the encrypted +file located at ``$KAYOBE_CONFIG_PATH/secrets.yml``. + +.. code-block:: console + + ipmitool -I lanplus -U -H -ipmi chassis bootdev pxe + +If node is are off, power them on: + +.. code-block:: console + + ipmitool -I lanplus -U -H -ipmi power on + +If nodes is on, reset them: + +.. code-block:: console + + ipmitool -I lanplus -U -H -ipmi power reset + +Once node have booted and have completed introspection, they should be visible +in Bifrost: + +.. code-block:: console + + baremetal node list --provision-state enroll + +--------------------------------------+-----------------------+---------------+-------------+--------------------+-------------+ + | UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance | + +--------------------------------------+-----------------------+---------------+-------------+--------------------+-------------+ + | da0c61af-b411-41b9-8909-df2509f2059b | example-hypervisor-01 | None | power off | enroll | False | + +--------------------------------------+-----------------------+---------------+-------------+--------------------+-------------+ + +After editing ``$KAYOBE_CONFIG_PATH/overcloud.yml`` (or +``$KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/overcloud.yml`` +if Kayobe environment is used) to add these new hosts to +the correct groups, import them in Kayobe's inventory with: + +.. code-block:: console + + kayobe overcloud inventory discover + +We can then provision and configure them: + +.. code-block:: console + + kayobe overcloud provision --limit + kayobe overcloud host configure --limit + kayobe overcloud service deploy --limit --kolla-limit + +.. note:: + + Reconfiguring monitoring services on controllers is required after provisioning them. + Otherwise, they will not show up. + +Replacing a Failing Hypervisor +------------------------------ + +To replace a failing hypervisor, proceed as follows: + +* :ref:`Disable the hypervisor to avoid scheduling any new instance on it ` +* :ref:`Evacuate all instances ` +* :ref:`Set the node to maintenance mode in Bifrost ` +* Physically fix or replace the node +* It may be necessary to reinspect the node if hardware was changed (this will require deprovisioning and reprovisioning) +* If the node was replaced or reprovisioned, follow :ref:`enrolling-new-hypervisors` + +To deprovision an existing hypervisor, run: + +.. code-block:: console + + kayobe overcloud deprovision --limit + +.. warning:: + + Always use ``--limit`` with ``kayobe overcloud deprovision`` on a production + system. Running this command without a limit will deprovision all overcloud + hosts. + +.. _evacuating-all-instances: + +Evacuating all instances +------------------------ + +.. code-block:: console + + openstack server evacuate $(openstack server list --host --format value --column ID) + +You should now check the status of all the instances that were running on that +hypervisor. They should all show the status ACTIVE. This can be verified with: + +.. code-block:: console + + openstack server show + +Troubleshooting +=============== + +Servers that have been shut down +-------------------------------- + +If there are any instances that are SHUTOFF they won’t be migrated, but you can +use ``openstack server migrate`` for them once the live migration is finished. + +Also if a VM does heavy memory access, it may take ages to migrate (Nova tries +to incrementally increase the expected downtime, but is quite conservative). +You can use ``openstack server migration force complete --os-compute-api-version 2.22 +`` to trigger the final move. + +You get the migration ID via ``openstack server migration list --server ``. + +For more details see: +http://www.danplanet.com/blog/2016/03/03/evacuate-in-nova-one-command-to-confuse-us-all/ + +Flavors have changed +-------------------- + +If the size of the flavors has changed, some instances will also fail to +migrate as the process needs manual confirmation. You can do this with: + +.. code-block:: console + + openstack server resize confirm + +The symptom to look out for is that the server is showing a status of ``VERIFY +RESIZE`` as shown in this snippet of ``openstack server show ``: + +.. code-block:: console + + | status | VERIFY_RESIZE | + +.. _set-bifrost-maintenance-mode: + +Set maintenance mode on a node in Bifrost +----------------------------------------- + +.. code-block:: console + + docker exec -it bifrost_deploy /bin/bash + (bifrost-deploy)[root@seed bifrost-base]# export OS_CLOUD=bifrost + (bifrost-deploy)[root@seed bifrost-base]# baremetal node maintenance set + +.. _unset-bifrost-maintenance-mode: + +Unset maintenance mode on a node in Bifrost +------------------------------------------- + +.. code-block:: console + + docker exec -it bifrost_deploy /bin/bash + (bifrost-deploy)[root@seed bifrost-base]# export OS_CLOUD=bifrost + (bifrost-deploy)[root@seed bifrost-base]# baremetal node maintenance unset + +Detect hardware differences with ADVise +======================================= + +Hardware information captured during the Ironic introspection process can be +analysed to detect hardware differences, such as mismatches in firmware +versions or missing storage devices. The `ADVise `__ +tool can be used for this purpose. + +Extract Bifrost introspection data +---------------------------------- + +The ADVise tool assumes that hardware introspection data has already been gathered in JSON format. +The ``extra-hardware`` disk builder element enabled when building the IPA image for the required data to be available. + +To build ipa image with extra-hardware you need to edit ``ipa.yml`` and add this: +.. code-block:: console + + # Whether to build IPA images from source. + ipa_build_images: true + + # List of additional Diskimage Builder (DIB) elements to use when building IPA + images. Default is none. + ipa_build_dib_elements_extra: + - "extra-hardware" + + # List of additional inspection collectors to run. + ipa_collectors_extra: + - "extra-hardware" + +Extract introspection data from Bifrost with Kayobe. JSON files will be created +into ``$KAYOBE_CONFIG_PATH/overcloud-introspection-data``: + +.. code-block:: console + + kayobe overcloud introspection data save + +Using ADVise +------------ + +The Ansible playbook ``advise-run.yml`` can be found at ``$KAYOBE_CONFIG_PATH/ansible/advise-run.yml``. + +The playbook will: + +1. Install ADVise and dependencies +2. Run the mungetout utility for extracting the required information from the introspection data ready for use with ADVise. +3. Run ADVise on the data. + +.. code-block:: console + + cd $KAYOBE_CONFIG_PATH + ansible-playbook $KAYOBE_CONFIG_PATH/ansible/advise-run.yml + +The playbook has the following optional parameters: + +- venv : path to the virtual environment to use. Default: ``"~/venvs/advise-review"`` +- input_dir: path to the hardware introspection data. Default: ``"{{ lookup('env', 'PWD') }}/overcloud-introspection-data"`` +- output_dir: path to where results should be saved. Default: ``"{{ lookup('env', 'PWD') }}/review"`` +- advise-pattern: regular expression to specify what introspection data should be analysed. Default: ``".*.eval"`` + +You can override them by provide new values with ``-e =`` + +Example command to run the tool on data about the compute nodes in a system, where compute nodes are named cpt01, cpt02, cpt03…: + +.. code-block:: console + + ansible-playbook advise-run.yml -e advise_pattern=’(cpt)(.*)(.eval)’ + + +.. note:: + The mungetout utility will always use the file extension .eval + +Using the results +----------------- + +The ADVise tool will output a selection of results found under output_dir/results these include: + +- ``.html`` files to display network visualisations of any hardware differences. +- The folder ``Paired_Comparisons`` which contains information on the shared and differing fields found between the systems. + This is a reflection of the network visualisation webpage, with more detail as to what the differences are. +- ``_summary``, a listing of how the systems can be grouped into sets of identical hardware. +- ``_performance``, the results of analysing the benchmarking data gathered. +- ``_perf_summary``, a subset of the performance metrics, just showing any potentially anomalous data such as where variance + is too high, or individual nodes have been found to over/underperform. + +The ADVise tool will also launch an interactive Dash webpage, which displays the network visualisations, +tables with information on the differing hardware attributes, the performance metrics as a range of box-plots, +and specifies which individual nodes may be anomalous via box-plot outliers. This can be accessed at ``localhost:8050``. +To close this service, simply ``Ctrl+C`` in the terminal where you ran the playbook. + +To get visuallised result, It is recommanded to copy instrospection data to your local machine then run ADVise playbook locally. + +Recommanded Workflow +-------------------- + +1. Run the playbook as outlined above. +2. Open the Dash webpage at ``localhost:8050``. +3. Review the hardware differences. Note that hovering over a group will display the nodes it contains. +4. Identify any unexpected differences in the systems. If multiple differing fields exist they will be graphed separately. + As an example, here we expected all compute nodes to be identical. +5. Use the dropdown menu beneath each graph to show a table of the differences found between two sets of groups. + If required, information on shared fields can be found under ``output_dir/results/Paired_Comparisons``. +6. Scroll down the webpage to the performance review. Identify if any of the discovered performance results could be + indicative of a larger issue. +7. Examine the ``_performance`` and ``_perf_summary`` files if you require any more information. diff --git a/doc/source/operations/ceph-management.rst b/doc/source/operations/ceph-management.rst new file mode 100644 index 000000000..98988959b --- /dev/null +++ b/doc/source/operations/ceph-management.rst @@ -0,0 +1,294 @@ +=========================== +Managing and Operating Ceph +=========================== + +Working with Cephadm +==================== + +This documentation provides guide for Ceph operations. For deploying Ceph, +please refer to :ref:`cephadm-kayobe` documentation. + +Cephadm configuration location +------------------------------ + +In kayobe-config repository, under ``$KAYOBE_CONFIG_PATH/cephadm.yml`` (or in a specific +Kayobe environment when using multiple environment, e.g. +``$KAYOBE_CONFIG_PATH/environments//cephadm.yml``) + +StackHPC's Cephadm Ansible collection relies on multiple inventory groups: + +- ``mons`` +- ``mgrs`` +- ``osds`` +- ``rgws`` (optional) + +Those groups are usually defined in ``$KAYOBE_CONFIG_PATH/inventory/groups``. + +Running Cephadm playbooks +------------------------- + +In kayobe-config repository, under ``$KAYOBE_CONFIG_PATH/ansible`` there is a set of +Cephadm based playbooks utilising stackhpc.cephadm Ansible Galaxy collection. + +``cephadm.yml`` runs the end to end process of Cephadm deployment and +configuration. It is composed with following list of other Cephadm playbooks +and they can be run separately. + +- ``cephadm-deploy.yml`` - Runs the bootstrap/deploy playbook without the + additional playbooks +- ``cephadm-commands-pre.yml`` - Runs Ceph commands before post-deployment + configuration (You can set a list of commands at ``cephadm_commands_pre_extra`` + variable in ``$KAYOBE_CONFIG_PATH/cephadm.yml``) +- ``cephadm-ec-profiles.yml`` - Defines Ceph EC profiles +- ``cephadm-crush-rules.yml`` - Defines Ceph crush rules according +- ``cephadm-pools.yml`` - Defines Ceph pools +- ``cephadm-keys.yml`` - Defines Ceph users/keys +- ``cephadm-commands-post.yml`` - Runs Ceph commands after post-deployment + configuration (You can set a list of commands at ``cephadm_commands_post_extra`` + variable in ``$KAYOBE_CONFIG_PATH/cephadm.yml``) + +There are also other Ceph playbooks that are not part of ``cephadm.yml`` + +- ``cephadm-gather-keys.yml`` - Populate ``ceph.conf`` and keyrings in kayobe-config by + gathering Ceph configuration and keys +- ``ceph-enter-maintenance.yml`` - Set Ceph to maintenance mode for storage + hosts (Can limit the hosts with ``-l ``) +- ``ceph-exit-maintenance.yml`` - Unset Ceph to maintenance mode for storage + hosts (Can limit the hosts with ``-l ``) + +Running Ceph commands +--------------------- + +Ceph commands are usually run inside a ``cephadm shell`` utility container: + +.. code-block:: console + + # From storage host + sudo cephadm shell + +Operating a cluster requires a keyring with an admin access to be available for Ceph +commands. Cephadm will copy such keyring to the nodes carrying +`_admin `__ +label - present on MON servers by default when using +`StackHPC Cephadm collection `__. + +Adding a new storage node +------------------------- + +Add a node to a respective group (e.g. osds) and run ``cephadm-deploy.yml`` +playbook. + +.. note:: + To add other node types than osds (mons, mgrs, etc) you need to specify + ``-e cephadm_bootstrap=True`` on playbook run. + +Removing a storage node +----------------------- + +First drain the node + +.. code-block:: console + + # From storage host + sudo cephadm shell + ceph orch host drain + +Once all daemons are removed - you can remove the host: + +.. code-block:: console + + # From storage host + sudo cephadm shell + ceph orch host rm + +And then remove the host from inventory (usually in +``$KAYOBE_CONFIG_PATH/inventory/overcloud``) + +Additional options/commands may be found in +`Host management `_ + +Replacing failing drive +----------------------- + +A failing drive in a Ceph cluster will cause OSD daemon to crash. +In this case Ceph will go into `HEALTH_WARN` state. +Ceph can report details about failed OSDs by running: + +.. code-block:: console + + # From storage host + sudo cephadm shell + ceph health detail + +.. note:: + + Remember to run ceph/rbd commands from within ``cephadm shell`` + (preferred method) or after installing Ceph client. Details in the + official `documentation `__. + It is also required that the host where commands are executed has admin + Ceph keyring present - easiest to achieve by applying + `_admin `__ + label (Ceph MON servers have it by default when using + `StackHPC Cephadm collection `__). + +A failed OSD will also be reported as down by running: + +.. code-block:: console + + ceph osd tree + +Note the ID of the failed OSD. + +The failed disk is usually logged by the Linux kernel too: + +.. code-block:: console + + # From storage host + dmesg -T + +Cross-reference the hardware device and OSD ID to ensure they match. +(Using `pvs` and `lvs` may help make this connection). + +See upstream documentation: +https://docs.ceph.com/en/latest/cephadm/services/osd/#replacing-an-osd + +In case where disk holding DB and/or WAL fails, it is necessary to recreate +all OSDs that are associated with this disk - usually NVMe drive. The +following single command is sufficient to identify which OSDs are tied to +which physical disks: + +.. code-block:: console + + ceph device ls + +Once OSDs on failed disks are identified, follow procedure below. + +If rebooting a Ceph node, first set ``noout`` to prevent excess data +movement: + +.. code-block:: console + + # From storage host + sudo cephadm shell + ceph osd set noout + +Reboot the node and replace the drive + +Unset noout after the node is back online + +.. code-block:: console + + # From storage host + sudo cephadm shell + ceph osd unset noout + +Remove the OSD using Ceph orchestrator command: + +.. code-block:: console + + # From storage host + sudo cephadm shell + ceph orch osd rm --replace + +After removing OSDs, if the drives the OSDs were deployed on once again become +available, Cephadm may automatically try to deploy more OSDs on these drives if +they match an existing drivegroup spec. +If this is not your desired action plan - it's best to modify the drivegroup +spec before (``cephadm_osd_spec`` variable in ``$KAYOBE_CONFIG_PATH/cephadm.yml``). +Either set ``unmanaged: true`` to stop Cephadm from picking up new disks or +modify it in some way that it no longer matches the drives you want to remove. + +Host maintenance +---------------- + +https://docs.ceph.com/en/latest/cephadm/host-management/#maintenance-mode + +Upgrading +--------- + +https://docs.ceph.com/en/latest/cephadm/upgrade/ + + +Troubleshooting +=============== + +Inspecting a Ceph Block Device for a VM +--------------------------------------- + +To find out what block devices are attached to a VM, go to the hypervisor that +it is running on (an admin-level user can see this from ``openstack server +show``). + +On this hypervisor, enter the libvirt container: + +.. code-block:: console + + # From hypervisor host + docker exec -it nova_libvirt /bin/bash + +Find the VM name using libvirt: + +.. code-block:: console + + (nova-libvirt)[root@compute-01 /]# virsh list + Id Name State + ------------------------------------ + 1 instance-00000001 running + +Now inspect the properties of the VM using ``virsh dumpxml``: + +.. code-block:: console + + (nova-libvirt)[root@compute-01 /]# virsh dumpxml instance-00000001 | grep rbd + + +On a Ceph node, the RBD pool can be inspected and the volume extracted as a RAW +block image: + +.. code-block:: console + + # From storage host + sudo cephadm shell + rbd ls + rbd export /51206278-e797-4153-b720-8255381228da_disk blob.raw + +The raw block device (blob.raw above) can be mounted using the loopback device. + +Inspecting a QCOW Image using LibGuestFS +---------------------------------------- + +The virtual machine's root image can be inspected by installing +libguestfs-tools and using the guestfish command: + +.. code-block:: console + + # From storage host + export LIBGUESTFS_BACKEND=direct + guestfish -a blob.qcow + > run + 100% [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] 00:00 + > list-filesystems + /dev/sda1: ext4 + > mount /dev/sda1 / + > ls / + bin + boot + dev + etc + home + lib + lib64 + lost+found + media + mnt + opt + proc + root + run + sbin + srv + sys + tmp + usr + var + > quit diff --git a/doc/source/operations/control-plane-operation.rst b/doc/source/operations/control-plane-operation.rst new file mode 100644 index 000000000..3dfd1ec44 --- /dev/null +++ b/doc/source/operations/control-plane-operation.rst @@ -0,0 +1,386 @@ +======================= +Operating Control Plane +======================= + +Backup of the OpenStack Control Plane +===================================== + +As the backup procedure is constantly changing, it is normally best to check +the upstream documentation for an up to date procedure. Here is a high level +overview of the key things you need to backup: + +Controllers +----------- + +* `Back up SQL databases `__ +* `Back up configuration in /etc/kolla `__ + +Compute +------- + +The compute nodes can largely be thought of as ephemeral, but you do need to +make sure you have migrated any instances and disabled the hypervisor before +rebooting, decommissioning or making any disruptive configuration change. + +Monitoring +---------- + +* `Back up InfluxDB `__ +* `Back up OpenSearch `__ +* `Back up Prometheus `__ + +Seed +---- + +* `Back up bifrost `__ + +Ansible control host +-------------------- + +* Back up service VMs such as the seed VM + +Control Plane Monitoring +======================== + +This section shows user guide of monitoring control plane. To see how to +configure monitoring services, read :ref:`monitoring-service-configuration`. + +The control plane has been configured to collect logs centrally using Fluentd, +OpenSearch and OpenSearch Dashboards. + +Telemetry monitoring of the control plane is performed by Prometheus. Metrics +are collected by Prometheus exporters, which are either running on all hosts +(e.g. node exporter), on specific hosts (e.g. controllers for the memcached +exporter or monitoring hosts for the OpenStack exporter). These exporters are +scraped by the Prometheus server. + +Configuring Prometheus Alerts +----------------------------- + +Alerts are defined in code and stored in Kayobe configuration. See ``*.rules`` +files in ``$KAYOBE_CONFIG_PATH/kolla/config/prometheus`` as a model to add +custom rules. + +Silencing Prometheus Alerts +--------------------------- + +Sometimes alerts must be silenced because the root cause cannot be resolved +right away, such as when hardware is faulty. For example, an unreachable +hypervisor will produce several alerts: + +* ``InstanceDown`` from Node Exporter +* ``OpenStackServiceDown`` from the OpenStack exporter, which reports status of + the ``nova-compute`` agent on the host +* ``PrometheusTargetMissing`` from several Prometheus exporters + +Rather than silencing each alert one by one for a specific host, a silence can +apply to multiple alerts using a reduced list of labels. Log into Alertmanager, +click on the ``Silence`` button next to an alert and adjust the matcher list +to keep only ``instance=`` label. +Then, create another silence to match ``hostname=`` (this is +required because, for the OpenStack exporter, the instance is the host running +the monitoring service rather than the host being monitored). + +Control Plane Shutdown Procedure +================================ + +Overview +-------- + +* Verify integrity of clustered components (RabbitMQ, Galera, Keepalived). They + should all report a healthy status. +* Put node into maintenance mode in bifrost to prevent it from automatically + powering back on +* Shutdown down nodes one at a time gracefully using systemctl poweroff + +Controllers +----------- + +If you are restarting the controllers, it is best to do this one controller at +a time to avoid the clustered components losing quorum. + +Checking Galera state ++++++++++++++++++++++ + +On each controller perform the following: + +.. code-block:: console + + [stack@controller0 ~]$ docker exec -i mariadb mysql -u root -p -e "SHOW STATUS LIKE 'wsrep_local_state_comment'" + Variable_name Value + wsrep_local_state_comment Synced + +The password can be found using: + +.. code-block:: console + + ansible-vault view $KAYOBE_CONFIG_PATH/kolla/passwords.yml \ + --vault-password-file | grep ^database + +Checking RabbitMQ ++++++++++++++++++ + +RabbitMQ health is determined using the command ``rabbitmqctl cluster_status``: + +.. code-block:: console + + [stack@controller0 ~]$ docker exec rabbitmq rabbitmqctl cluster_status + + Cluster status of node rabbit@controller0 ... + [{nodes,[{disc,['rabbit@controller0','rabbit@controller1', + 'rabbit@controller2']}]}, + {running_nodes,['rabbit@controller1','rabbit@controller2', + 'rabbit@controller0']}, + {cluster_name,<<"rabbit@controller2">>}, + {partitions,[]}, + {alarms,[{'rabbit@controller1',[]}, + {'rabbit@controller2',[]}, + {'rabbit@controller0',[]}]}] + +Checking Keepalived ++++++++++++++++++++ + +On (for example) three controllers: + +.. code-block:: console + + [stack@controller0 ~]$ docker logs keepalived + +Two instances should show: + +.. code-block:: console + + VRRP_Instance(kolla_internal_vip_51) Entering BACKUP STATE + +and the other: + +.. code-block:: console + + VRRP_Instance(kolla_internal_vip_51) Entering MASTER STATE + +Ansible Control Host +-------------------- + +The Ansible control host is not enrolled in bifrost. This node may run services +such as the seed virtual machine which will need to be gracefully powered down. + +Compute +------- + +If you are shutting down a single hypervisor, to avoid down time to tenants it +is advisable to migrate all of the instances to another machine. See +:ref:`evacuating-all-instances`. + +Ceph +---- + +The following guide provides a good overview: +https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/8/html/director_installation_and_usage/sect-rebooting-ceph + +Shutting down the seed VM +------------------------- + +.. code-block:: console + + virsh shutdown + +.. _full-shutdown: + +Full shutdown +------------- + +In case a full shutdown of the system is required, we advise to use the +following order: + +* Perform a graceful shutdown of all virtual machine instances +* Shut down compute nodes +* Shut down monitoring node (if separate from controllers) +* Shut down network nodes (if separate from controllers) +* Shut down controllers +* Shut down Ceph nodes (if applicable) +* Shut down seed VM +* Shut down Ansible control host + +Rebooting a node +---------------- + +Use ``reboot.yml`` playbook to reboot nodes +Example: Reboot all compute hosts apart from compute0: + +.. code-block:: console + + kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/reboot.yml --limit 'compute:!compute0' + +References +---------- + +* https://galeracluster.com/library/training/tutorials/restarting-cluster.html + +Control Plane Power on Procedure +================================ + +Overview +-------- + +* Remove the node from maintenance mode in bifrost +* Bifrost should automatically power on the node via IPMI +* Check that all docker containers are running +* Check OpenSearch Dashboards for any messages with log level ERROR or equivalent + +Controllers +----------- + +If all of the servers were shut down at the same time, it is necessary to run a +script to recover the database once they have all started up. This can be done +with the following command: + +.. code-block:: console + + kayobe overcloud database recover + +Ansible Control Host +-------------------- + +The Ansible control host is not enrolled in Bifrost and will have to be powered +on manually. + +Seed VM +------- + +The seed VM (and any other service VM) should start automatically when the seed +hypervisor is powered on. If it does not, it can be started with: + +.. code-block:: console + + virsh start + +Full power on +------------- + +Follow the order in :ref:`full-shutdown`, but in reverse order. + +Shutting Down / Restarting Monitoring Services +---------------------------------------------- + +Shutting down ++++++++++++++ + +Log into the monitoring host(s): + +.. code-block:: console + + ssh stack@monitoring0 + +Stop all Docker containers: + +.. code-block:: console + + monitoring0# for i in `docker ps -a`; do systemctl stop kolla-$i-container; done + +Shut down the node: + +.. code-block:: console + + monitoring0# sudo shutdown -h + +Starting up ++++++++++++ + +The monitoring services containers will automatically start when the monitoring +node is powered back on. + +Software Updates +================ + +Sync local Pulp server with StackHPC Release Train +-------------------------------------------------- + +The host packages and Kolla container images are distributed from `StackHPC Release Train +`__ to ensure tested and reliable +software releases are provided. + +Syncing new StackHPC Release Train contents to local Pulp server is needed before updating +host packages and/or Kolla services. + +To sync host packages: + +.. code-block:: console + + kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/pulp-repo-sync.yml + kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/pulp-repo-publish.yml + +If the system is production environment and want to use packages tested in test/staging +environment, you can promote them by: + +.. code-block:: console + + kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/pulp-repo-promote-production.yml + +To sync container images: + +.. code-block:: console + + kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/pulp-container-sync.yml + kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/pulp-container-publish.yml + +For more information about StackHPC Release Train, see :ref:`stackhpc-release-train` documentation. + +Once sync with StackHPC Release Train is done, new contents will be accessible from local +Pulp server. + +Update Host Packages on Control Plane +------------------------------------- + +Host packages can be updated with: + +.. code-block:: console + + kayobe overcloud host package update --limit --packages '*' + kayobe seed host package update --packages '*' + +See https://docs.openstack.org/kayobe/latest/administration/overcloud.html#updating-packages + +Troubleshooting +=============== + +Deploying to a Specific Hypervisor +---------------------------------- + +To test creating an instance on a specific hypervisor, *as an admin-level user* +you can specify the hypervisor name. + +To see the list of hypervisor names: + +.. code-block:: console + + # From host that can reach Openstack + openstack hypervisor list + +To boot an instance on a specific hypervisor + +.. code-block:: console + + openstack server create --flavor --network --key-name --image --os-compute-api-version 2.74 --host + +OpenSearch indexes retention +============================= + +To alter default rotation values for OpenSearch, edit + +``$KAYOBE_CONFIG_PATH/kolla/globals.yml``: + +.. code-block:: console + + # Duration after which index is closed (default 30) + opensearch_soft_retention_period_days: 90 + # Duration after which index is deleted (default 60) + opensearch_hard_retention_period_days: 180 + +Reconfigure Opensearch with new values: + +.. code-block:: console + + kayobe overcloud service reconfigure --kolla-tags opensearch + +For more information see the `upstream documentation +`__. diff --git a/doc/source/operations/customising-horizon.rst b/doc/source/operations/customising-horizon.rst new file mode 100644 index 000000000..586bb242c --- /dev/null +++ b/doc/source/operations/customising-horizon.rst @@ -0,0 +1,118 @@ +=================== +Customising Horizon +=================== + +Horizon is the most frequent site-specific container customisation required: +other customisations tend to be common across deployments, but personalisation +of Horizon is unique to each institution. + +This describes a simple process for customising the Horizon theme. + +Creating a custom Horizon theme +------------------------------- + +A simple custom theme for Horizon can be implemented as small modifications of +an existing theme, such as the `Default +`__ +one. + +A theme contains at least two files: ``static/_styles.scss``, which can be empty, and +``static/_variables.scss``, which can reference another theme like this: + +.. code-block:: scss + + @import "/themes/default/variables"; + @import "/themes/default/styles"; + +Some resources such as logos can be overridden by dropping SVG image files into +``static/img`` (since the Ocata release, files must be SVG instead of PNG). See +`the Horizon documentation +`__ +for more details. + +Content on some pages such as the splash (login) screen can be updated using +templates. + +See `our example horizon-theme `__ +which inherits from the default theme and includes: + +* a custom splash screen logo +* a custom top-left logo +* a custom message on the splash screen + +Further reading: + +* https://docs.openstack.org/horizon/latest/configuration/customizing.html +* https://docs.openstack.org/horizon/latest/configuration/themes.html +* https://docs.openstack.org/horizon/latest/configuration/branding.html + +Adding the custom theme +----------------------- + +Create a directory and transfer custom theme files to it ``$KAYOBE_CONFIG_PATH/kolla/config/horizon/themes/``. + +Define the custom theme in ``$KAYOBE_CONFIG_PATH/kolla/globals.yml`` + +.. code-block:: yaml + + horizon_custom_themes: + - name: + label: # This will be the visible name to users + +Deploy and use the custom theme +------------------------------- + +Configure Horizon to include the custom theme and use it by default: + +.. code-block:: console + + mkdir -p $KAYOBE_CONFIG_PATH/kolla/config/horizon/ + +Create file ``$KAYOBE_CONFIG_PATH/kolla/config/horizon/custom_local_settings`` and add followings + +.. code-block:: console + + AVAILABLE_THEMES = [ + ('default', 'Default', 'themes/default'), + ('material', 'Material', 'themes/material'), + ('', '', 'themes/'), + ] + DEFAULT_THEME = '' + +You can also set other customisations in this file, such as the HTML title of the page: + +.. code-block:: console + + SITE_BRANDING = "" + +Deploy with: + +.. code-block:: console + + kayobe overcloud service reconfigure --kolla-tags horizon + +Troubleshooting +--------------- + +If the theme is selected but the logo doesn’t load, try running these commands +inside the ``horizon`` container: + +.. code-block:: console + + /var/lib/kolla/venv/bin/python /var/lib/kolla/venv/bin/manage.py collectstatic --noinput --clear + /var/lib/kolla/venv/bin/python /var/lib/kolla/venv/bin/manage.py compress --force + settings_bundle | md5sum > /var/lib/kolla/.settings.md5sum.txt + +Alternatively, try changing anything in ``custom_local_settings`` and restarting +the ``horizon`` container. + +If the ``horizon`` container is restarting with the following error: + +.. code-block:: console + + /var/lib/kolla/venv/bin/python /var/lib/kolla/venv/bin/manage.py compress --force + CommandError: An error occurred during rendering /var/lib/kolla/venv/lib/python3.6/site-packages/openstack_dashboard/templates/horizon/_scripts.html: Couldn't find any precompiler in COMPRESS_PRECOMPILERS setting for mimetype '\'text/javascript\''. + +It can be resolved by dropping cached content with ``systemctl restart +kolla-memcached-container``. Note this will log out users from Horizon, as Django +sessions are stored in Memcached. diff --git a/doc/source/operations/gpu-in-openstack.rst b/doc/source/operations/gpu-in-openstack.rst new file mode 100644 index 000000000..1fd99d30d --- /dev/null +++ b/doc/source/operations/gpu-in-openstack.rst @@ -0,0 +1,807 @@ +============================= +Support for GPUs in OpenStack +============================= + +Virtual GPUs +############ + +BIOS configuration +------------------ + +See upstream documentation: `BIOS configuration `__ + +Obtain driver from NVIDIA licensing portal +------------------------------------------ + +See upstream documentation: `Obtain driver from NVIDIA licencing portal `__ + +.. _NVIDIA Pulp: + +Uploading the GRID driver to pulp +--------------------------------- + +Uploading the driver to pulp will make it possible to run kayobe from any host. This can be useful when +running in a CI environment. + +.. code:: shell + + pulp artifact upload --file ~/NVIDIA-GRID-Linux-KVM-525.105.14-525.105.17-528.89.zip + pulp file content create --relative-path "NVIDIA-GRID-Linux-KVM-525.105.14-525.105.17-528.89.zip" --sha256 c8e12c15b881df35e618bdee1f141cbfcc7e112358f0139ceaa95b48e20761e0 + pulp file repository create --name nvidia + pulp file repository content add --repository nvidia --sha256 c8e12c15b881df35e618bdee1f141cbfcc7e112358f0139ceaa95b48e20761e0 --relative-path "NVIDIA-GRID-Linux-KVM-525.105.14-525.105.17-528.89.zip" + pulp file publication create --repository nvidia + pulp file distribution create --name nvidia --base-path nvidia --repository nvidia + +The file will then be available at ``/pulp/content/nvidia/NVIDIA-GRID-Linux-KVM-525.105.14-525.105.17-528.89.zip``. You +will need to set the ``vgpu_driver_url`` configuration option to this value: + +.. code-block:: yaml + :caption: $KAYOBE_CONFIG_PATH/vgpu.yml + + # URL of GRID driver in pulp + vgpu_driver_url: "{{ pulp_url }}/pulp/content/nvidia/NVIDIA-GRID-Linux-KVM-525.105.14-525.105.17-528.89.zip" + +See :ref:`NVIDIA Role Configuration`. + +.. _NVIDIA control host: + +Placing the GRID driver on the ansible control host +--------------------------------------------------- + +Copy the driver bundle to a known location on the ansible control host. Set the ``vgpu_driver_url`` configuration variable to reference this +path using ``file`` as the url scheme e.g: + +.. code-block:: yaml + :caption: $KAYOBE_CONFIG_PATH/vgpu.yml + + # Location of NVIDIA GRID driver on localhost + vgpu_driver_url: "file://{{ lookup('env', 'HOME') }}/NVIDIA-GRID-Linux-KVM-525.105.14-525.105.17-528.89.zip" + +See :ref:`NVIDIA Role Configuration`. + +.. _NVIDIA OS Configuration: + +OS Configuration +---------------- + +Host OS configuration is done by using roles in the `stackhpc.linux `_ ansible collection. + +Create a new playbook or update an existing on to apply the roles: + +.. code-block:: yaml + :caption: $KAYOBE_CONFIG_PATH/ansible/host-configure.yml + + --- + - hosts: iommu + tags: + - iommu + tasks: + - import_role: + name: stackhpc.linux.iommu + handlers: + - name: reboot + set_fact: + kayobe_needs_reboot: true + + - hosts: vgpu + tags: + - vgpu + tasks: + - import_role: + name: stackhpc.linux.vgpu + handlers: + - name: reboot + set_fact: + kayobe_needs_reboot: true + + - name: Reboot when required + hosts: iommu:vgpu + tags: + - reboot + tasks: + - name: Reboot + reboot: + reboot_timeout: 3600 + become: true + when: kayobe_needs_reboot | default(false) | bool + +Ansible Inventory Configuration +------------------------------- + +Add some hosts into the ``vgpu`` group. The example below maps two custom +compute groups, ``compute_multi_instance_gpu`` and ``compute_vgpu``, +into the ``vgpu`` group: + +.. code-block:: yaml + :caption: $KAYOBE_CONFIG_PATH/inventory/custom + + [compute] + [compute_multi_instance_gpu] + [compute_vgpu] + + [vgpu:children] + compute_multi_instance_gpu + compute_vgpu + + [iommu:children] + vgpu + +Having multiple groups is useful if you want to be able to do conditional +templating in ``nova.conf`` (see :ref:`NVIDIA Kolla Ansible +Configuration`). Since the vgpu role requires iommu to be enabled, all of the +hosts in the ``vgpu`` group are also added to the ``iommu`` group. + +If using bifrost and the ``kayobe overcloud inventory discover`` mechanism, +hosts can automatically be mapped to these groups by configuring +``overcloud_group_hosts_map``: + +.. code-block:: yaml + :caption: ``$KAYOBE_CONFIG_PATH/overcloud.yml`` + + overcloud_group_hosts_map: + compute_vgpu: + - "computegpu000" + compute_mutli_instance_gpu: + - "computegpu001" + +.. _NVIDIA Role Configuration: + +Role Configuration +^^^^^^^^^^^^^^^^^^ + +Configure the VGPU devices: + +.. code-block:: yaml + :caption: $KAYOBE_CONFIG_PATH/inventory/group_vars/compute_vgpu/vgpu + + #nvidia-692 GRID A100D-4C + #nvidia-693 GRID A100D-8C + #nvidia-694 GRID A100D-10C + #nvidia-695 GRID A100D-16C + #nvidia-696 GRID A100D-20C + #nvidia-697 GRID A100D-40C + #nvidia-698 GRID A100D-80C + #nvidia-699 GRID A100D-1-10C + #nvidia-700 GRID A100D-2-20C + #nvidia-701 GRID A100D-3-40C + #nvidia-702 GRID A100D-4-40C + #nvidia-703 GRID A100D-7-80C + #nvidia-707 GRID A100D-1-10CME + vgpu_definitions: + # Configuring a MIG backed VGPU + - pci_address: "0000:17:00.0" + virtual_functions: + - mdev_type: nvidia-700 + index: 0 + - mdev_type: nvidia-700 + index: 1 + - mdev_type: nvidia-700 + index: 2 + - mdev_type: nvidia-699 + index: 3 + mig_devices: + "1g.10gb": 1 + "2g.20gb": 3 + # Configuring a card in a time-sliced configuration (non-MIG backed) + - pci_address: "0000:65:00.0" + virtual_functions: + - mdev_type: nvidia-697 + index: 0 + - mdev_type: nvidia-697 + index: 1 + +.. _NVIDIA Kolla Ansible Configuration: + +Kolla-Ansible configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +See upstream documentation: `Kolla Ansible configuration `__ +then follow the rest. + +Map through the kayobe inventory groups into kolla: + +.. code-block:: yaml + :caption: $KAYOBE_CONFIG_PATH/kolla.yml + + kolla_overcloud_inventory_top_level_group_map: + control: + groups: + - controllers + network: + groups: + - network + compute_cpu: + groups: + - compute_cpu + compute_gpu: + groups: + - compute_gpu + compute_multi_instance_gpu: + groups: + - compute_multi_instance_gpu + compute_vgpu: + groups: + - compute_vgpu + compute: + groups: + - compute + monitoring: + groups: + - monitoring + storage: + groups: + "{{ kolla_overcloud_inventory_storage_groups }}" + +Where the ``compute_`` groups have been added to the kayobe defaults. + +You will need to reconfigure nova for this change to be applied: + +.. code-block:: shell + + kayobe overcloud service deploy -kt nova --kolla-limit compute_vgpu + +Openstack flavors +^^^^^^^^^^^^^^^^^ + +See upstream documentation: `OpenStack flavors `__ + +NVIDIA License Server +^^^^^^^^^^^^^^^^^^^^^ + +The Nvidia delegated license server is a virtual machine based appliance. You simply need to boot an instance +using the image supplied on the NVIDIA Licensing portal. This can be done on the OpenStack cloud itself. The +requirements are: + +* All tenants wishing to use GPU based instances must have network connectivity to this machine. (network licensing) + - It is possible to configure node locked licensing where tenants do not need access to the license server +* Satisfy minimum requirements detailed `here `__. + +The official documentation for configuring the instance +can be found `here `__. + +Below is a snippet of openstack-config for defining a project, and a security group that can be used for a non-HA deployment: + +.. code-block:: yaml + + secgroup_rules_nvidia_dls: + # Allow ICMP (for ping, etc.). + - ethertype: IPv4 + protocol: icmp + # Allow SSH. + - ethertype: IPv4 + protocol: tcp + port_range_min: 22 + port_range_max: 22 + # https://docs.nvidia.com/license-system/latest/nvidia-license-system-user-guide/index.html + - ethertype: IPv4 + protocol: tcp + port_range_min: 443 + port_range_max: 443 + - ethertype: IPv4 + protocol: tcp + port_range_min: 80 + port_range_max: 80 + - ethertype: IPv4 + protocol: tcp + port_range_min: 7070 + port_range_max: 7070 + + secgroup_nvidia_dls: + name: nvidia-dls + project: "{{ project_cloud_services.name }}" + rules: "{{ secgroup_rules_nvidia_dls }}" + + openstack_security_groups: + - "{{ secgroup_nvidia_dls }}" + + project_cloud_services: + name: "cloud-services" + description: "Internal Cloud services" + project_domain: default + user_domain: default + users: [] + quotas: "{{ quotas_project }}" + +Booting the VM: + +.. code-block:: shell + + # Uploading the image and making it available in the cloud services project + $ openstack image create --file nls-3.0.0-bios.qcow2 nls-3.0.0-bios --disk-format qcow2 + $ openstack image add project nls-3.0.0-bios cloud-services + $ openstack image set --accept nls-3.0.0-bios --project cloud-services + $ openstack image member list nls-3.0.0-bios + + # Booting a server as the admin user in the cloud-services project. We pre-create the port so that + # we can recreate it without changing the MAC address. + $ openstack port create --mac-address fa:16:3e:a3:fd:19 --network external nvidia-dls-1 --project cloud-services + $ openstack role add member --project cloud-services --user admin + $ export OS_PROJECT_NAME=cloud-services + $ openstack server group create nvidia-dls --policy anti-affinity + $ openstack server create --flavor 8cpu-8gbmem-30gbdisk --image nls-3.0.0-bios --port nvidia-dls-1 --hint group=179dfa59-0947-4925-a0ff-b803bc0e58b2 nvidia-dls-cci1-1 --security-group nvidia-dls + $ openstack server add security group nvidia-dls-1 nvidia-dls + + +Manual VM driver and licence configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +vGPU client VMs need to be configured with Nvidia drivers to run GPU workloads. +The host drivers should already be applied to the hypervisor. + +GCP hosts compatible client drivers `here +`__. + +Find the correct version (when in doubt, use the same version as the host) and +download it to the VM. The exact dependencies will depend on the base image you +are using but at a minimum, you will need GCC installed. + +Ubuntu Jammy example: + +.. code-block:: bash + + sudo apt update + sudo apt install -y make gcc wget + wget https://storage.googleapis.com/nvidia-drivers-us-public/GRID/vGPU17.1/NVIDIA-Linux-x86_64-550.54.15-grid.run + sudo sh NVIDIA-Linux-x86_64-550.54.15-grid.run + +Check the ``nvidia-smi`` client is available: + +.. code-block:: bash + + nvidia-smi + +Generate a token from the licence server, and copy the token file to the client +VM. + +On the client, create an Nvidia grid config file from the template: + +.. code-block:: bash + + sudo cp /etc/nvidia/gridd.conf.template /etc/nvidia/gridd.conf + +Edit it to set ``FeatureType=1`` and leave the rest of the settings as default. + +Copy the client configuration token into the ``/etc/nvidia/ClientConfigToken`` +directory. + +Ensure the correct permissions are set: + +.. code-block:: bash + + sudo chmod 744 /etc/nvidia/ClientConfigToken/client_configuration_token_.tok + +Restart the ``nvidia-gridd`` service: + +.. code-block:: bash + + sudo systemctl restart nvidia-gridd + +Check that the token has been recognised: + +.. code-block:: bash + + nvidia-smi -q | grep 'License Status' + +If not, an error should appear in the journal: + +.. code-block:: bash + + sudo journalctl -xeu nvidia-gridd + +A successfully licenced VM can be snapshotted to create an image in Glance that +includes the drivers and licencing token. Alternatively, an image can be +created using Diskimage Builder. + +Disk image builder recipe to automatically license VGPU on boot +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +`stackhpc-image-elements `__ provides a ``nvidia-vgpu`` +element to configure the nvidia-gridd service in VGPU mode. This allows you to boot VMs that automatically license themselves. +Snippets of ``openstack-config`` that allow you to do this are shown below: + +.. code-block:: shell + + image_rocky9_nvidia: + name: "Rocky9-NVIDIA" + type: raw + elements: + - "rocky-container" + - "rpm" + - "nvidia-vgpu" + - "cloud-init" + - "epel" + - "cloud-init-growpart" + - "selinux-permissive" + - "dhcp-all-interfaces" + - "vm" + - "extra-repos" + - "grub2" + - "stable-interface-names" + - "openssh-server" + is_public: True + packages: + - "dkms" + - "git" + - "tmux" + - "cuda-minimal-build-12-1" + - "cuda-demo-suite-12-1" + - "cuda-libraries-12-1" + - "cuda-toolkit" + - "vim-enhanced" + env: + DIB_CONTAINERFILE_NETWORK_DRIVER: host + DIB_CONTAINERFILE_RUNTIME: docker + DIB_RPMS: "http://192.168.1.2:80/pulp/content/nvidia/nvidia-linux-grid-525-525.105.17-1.x86_64.rpm" + YUM: dnf + DIB_EXTRA_REPOS: "https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo" + DIB_NVIDIA_VGPU_CLIENT_TOKEN: "{{ lookup('file' , 'secrets/client_configuration_token_05-30-2023-12-41-40.tok') }}" + DIB_CLOUD_INIT_GROWPART_DEVICES: + - "/" + DIB_RELEASE: "9" + properties: + os_type: "linux" + os_distro: "rocky" + os_version: "9" + + openstack_images: + - "{{ image_rocky9_nvidia }}" + + openstack_image_git_elements: + - repo: "https://github.com/stackhpc/stackhpc-image-elements" + local: "{{ playbook_dir }}/stackhpc-image-elements" + version: master + elements_path: elements + +The gridd driver was uploaded pulp using the following procedure: + +.. code-block:: shell + + $ unzip NVIDIA-GRID-Linux-KVM-525.105.14-525.105.17-528.89.zip + $ pulp artifact upload --file ~/nvidia-linux-grid-525-525.105.17-1.x86_64.rpm + $ pulp file content create --relative-path "nvidia-linux-grid-525-525.105.17-1.x86_64.rpm" --sha256 58fda68d01f00ea76586c9fd5f161c9fbb907f627b7e4f4059a309d8112ec5f5 + $ pulp file repository add --name nvidia --sha256 58fda68d01f00ea76586c9fd5f161c9fbb907f627b7e4f4059a309d8112ec5f5 --relative-path "nvidia-linux-grid-525-525.105.17-1.x86_64.rpm" + $ pulp file publication create --repository nvidia + $ pulp file distribution update --name nvidia --base-path nvidia --repository nvidia + +This is the file we reference in ``DIB_RPMS``. It is important to keep the driver versions aligned between hypervisor and guest VM. + +The client token can be downloaded from the web interface of the licensing portal. Care should be taken +when copying the contents as it can contain invisible characters. It is best to copy the file directly +into your openstack-config repository and vault encrypt it. The ``file`` lookup plugin can be used to decrypt +the file (as shown in the example above). + +Testing vGPU VMs +^^^^^^^^^^^^^^^^ + +vGPU VMs can be validated using the following test workload. The test should +succeed if the VM is correctly licenced and drivers are correctly installed for +both the host and client VM. + +Install ``cuda-toolkit`` using the instructions `here +`__. + +Ubuntu Jammy example: + +.. code-block:: bash + + wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb + sudo dpkg -i cuda-keyring_1.1-1_all.deb + sudo apt update -y + sudo apt install -y cuda-toolkit make + +The VM may require a reboot at this point. + +Clone the ``cuda-samples`` repo: + +.. code-block:: bash + + git clone https://github.com/NVIDIA/cuda-samples.git + +Build and run a test workload: + +.. code-block:: bash + + cd cuda-samples/Samples/6_Performance/transpose + make + ./transpose + +Example output: + +.. code-block:: + + Transpose Starting... + + GPU Device 0: "Ampere" with compute capability 8.0 + + > Device 0: "GRID A100D-1-10C MIG 1g.10gb" + > SM Capability 8.0 detected: + > [GRID A100D-1-10C MIG 1g.10gb] has 14 MP(s) x 64 (Cores/MP) = 896 (Cores) + > Compute performance scaling factor = 1.00 + + Matrix size: 1024x1024 (64x64 tiles), tile size: 16x16, block size: 16x16 + + transpose simple copy , Throughput = 159.1779 GB/s, Time = 0.04908 ms, Size = 1048576 fp32 elements, NumDevsUsed = 1, Workgroup = 256 + transpose shared memory copy, Throughput = 152.1922 GB/s, Time = 0.05133 ms, Size = 1048576 fp32 elements, NumDevsUsed = 1, Workgroup = 256 + transpose naive , Throughput = 117.2670 GB/s, Time = 0.06662 ms, Size = 1048576 fp32 elements, NumDevsUsed = 1, Workgroup = 256 + transpose coalesced , Throughput = 135.0813 GB/s, Time = 0.05784 ms, Size = 1048576 fp32 elements, NumDevsUsed = 1, Workgroup = 256 + transpose optimized , Throughput = 145.4326 GB/s, Time = 0.05372 ms, Size = 1048576 fp32 elements, NumDevsUsed = 1, Workgroup = 256 + transpose coarse-grained , Throughput = 145.2941 GB/s, Time = 0.05377 ms, Size = 1048576 fp32 elements, NumDevsUsed = 1, Workgroup = 256 + transpose fine-grained , Throughput = 150.5703 GB/s, Time = 0.05189 ms, Size = 1048576 fp32 elements, NumDevsUsed = 1, Workgroup = 256 + transpose diagonal , Throughput = 117.6831 GB/s, Time = 0.06639 ms, Size = 1048576 fp32 elements, NumDevsUsed = 1, Workgroup = 256 + Test passed + +Changing VGPU device types +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +See upstream documentation: `Changing VGPU device types `__ + +PCI Passthrough +############### + +This guide has been developed for Nvidia GPUs and CentOS 8. + +See `Kayobe Ops `_ for +a playbook implementation of host setup for GPU. + +BIOS Configuration Requirements +------------------------------- + +On an Intel system: + +* Enable `VT-x` in the BIOS for virtualisation support. +* Enable `VT-d` in the BIOS for IOMMU support. + +Hypervisor Configuration Requirements +------------------------------------- + +Find the GPU device IDs +^^^^^^^^^^^^^^^^^^^^^^^ + +From the host OS, use ``lspci -nn`` to find the PCI vendor ID and +device ID for the GPU device and supporting components. These are +4-digit hex numbers. + +For example: + +.. code-block:: text + + 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204M [GeForce GTX 980M] [10de:13d7] (rev a1) (prog-if 00 [VGA controller]) + 01:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1) + +In this case the vendor ID is ``10de``, display ID is ``13d7`` and audio ID is ``0fbb``. + +Alternatively, for an Nvidia Quadro RTX 6000: + +.. code-block:: yaml + + # NVIDIA Quadro RTX 6000/8000 PCI device IDs + vendor_id: "10de" + display_id: "1e30" + audio_id: "10f7" + usba_id: "1ad6" + usba_class: "0c0330" + usbc_id: "1ad7" + usbc_class: "0c8000" + +These parameters will be used for device-specific configuration. + +Kernel Ramdisk Reconfiguration +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The ramdisk loaded during kernel boot can be extended to include the +vfio PCI drivers and ensure they are loaded early in system boot. + +.. code-block:: yaml + + - name: Template dracut config + blockinfile: + path: /etc/dracut.conf.d/gpu-vfio.conf + block: | + add_drivers+="vfio vfio_iommu_type1 vfio_pci vfio_virqfd" + owner: root + group: root + mode: 0660 + create: true + become: true + notify: + - Regenerate initramfs + - reboot + +The handler for regenerating the Dracut initramfs is: + +.. code-block:: yaml + + - name: Regenerate initramfs + shell: |- + #!/bin/bash + set -eux + dracut -v -f /boot/initramfs-$(uname -r).img $(uname -r) + become: true + +Kernel Boot Parameters +^^^^^^^^^^^^^^^^^^^^^^ + +Set the following kernel parameters by adding to +``GRUB_CMDLINE_LINUX_DEFAULT`` or ``GRUB_CMDLINE_LINUX`` in +``/etc/default/grub.conf``. We can use the +`stackhpc.grubcmdline `_ +role from Ansible Galaxy: + +.. code-block:: yaml + + - name: Add vfio-pci.ids kernel args + include_role: + name: stackhpc.grubcmdline + vars: + kernel_cmdline: + - intel_iommu=on + - iommu=pt + - "vfio-pci.ids={{ vendor_id }}:{{ display_id }},{{ vendor_id }}:{{ audio_id }}" + kernel_cmdline_remove: + - iommu + - intel_iommu + - vfio-pci.ids + +Kernel Device Management +^^^^^^^^^^^^^^^^^^^^^^^^ + +In the hypervisor, we must prevent kernel device initialisation of +the GPU and prevent drivers from loading for binding the GPU in the +host OS. We do this using ``udev`` rules: + +.. code-block:: yaml + + - name: Template udev rules to blacklist GPU usb controllers + blockinfile: + # We want this to execute as soon as possible + path: /etc/udev/rules.d/99-gpu.rules + block: | + #Remove NVIDIA USB xHCI Host Controller Devices, if present + ACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x{{ vendor_id }}", ATTR{class}=="0x{{ usba_class }}", ATTR{remove}="1" + #Remove NVIDIA USB Type-C UCSI devices, if present + ACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x{{ vendor_id }}", ATTR{class}=="0x{{ usbc_class }}", ATTR{remove}="1" + owner: root + group: root + mode: 0644 + create: true + become: true + +Kernel Drivers +^^^^^^^^^^^^^^ + +Prevent the ``nouveau`` kernel driver from loading by +blacklisting the module: + +.. code-block:: yaml + + - name: Blacklist nouveau + blockinfile: + path: /etc/modprobe.d/blacklist-nouveau.conf + block: | + blacklist nouveau + options nouveau modeset=0 + mode: 0664 + owner: root + group: root + create: true + become: true + notify: + - reboot + - Regenerate initramfs + +Ensure that the ``vfio`` drivers are loaded into the kernel on boot: + +.. code-block:: yaml + + - name: Add vfio to modules-load.d + blockinfile: + path: /etc/modules-load.d/vfio.conf + block: | + vfio + vfio_iommu_type1 + vfio_pci + vfio_virqfd + owner: root + group: root + mode: 0664 + create: true + become: true + notify: reboot + +Once this code has taken effect (after a reboot), the VFIO kernel drivers should be loaded on boot: + +.. code-block:: text + + # lsmod | grep vfio + vfio_pci 49152 0 + vfio_virqfd 16384 1 vfio_pci + vfio_iommu_type1 28672 0 + vfio 32768 2 vfio_iommu_type1,vfio_pci + irqbypass 16384 5 vfio_pci,kvm + + # lspci -nnk -s 3d:00.0 + 3d:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM107GL [Tesla M10] [10de:13bd] (rev a2) + Subsystem: NVIDIA Corporation Tesla M10 [10de:1160] + Kernel driver in use: vfio-pci + Kernel modules: nouveau + +IOMMU should be enabled at kernel level as well - we can verify that on the compute host: + +.. code-block:: text + + # docker exec -it nova_libvirt virt-host-validate | grep IOMMU + QEMU: Checking for device assignment IOMMU support : PASS + QEMU: Checking if IOMMU is enabled by kernel : PASS + +OpenStack Nova configuration +---------------------------- + +See upsteram Nova documentation: `Attaching physical PCI devices to guests `__ + +Configure a flavor +^^^^^^^^^^^^^^^^^^ + +For example, to request two of the GPUs with alias **a1** + +.. code-block:: text + + openstack flavor set m1.medium --property "pci_passthrough:alias"="a1:2" + + +This can be also defined in the openstack-config repository + +add extra_specs to flavor in etc/openstack-config/openstack-config.yml: + +.. code-block:: console + + cd src/openstack-config + vim etc/openstack-config/openstack-config.yml + + name: "m1.medium-gpu" + ram: 4096 + disk: 40 + vcpus: 2 + extra_specs: + "pci_passthrough:alias": "a1:2" + +Invoke configuration playbooks afterwards: + +.. code-block:: console + + source src/kayobe-config/etc/kolla/public-openrc.sh + source venvs/openstack/bin/activate + tools/openstack-config --vault-password-file + +Create instance with GPU passthrough +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. code-block:: text + + openstack server create --flavor m1.medium-gpu --image ubuntu22.04 --wait test-pci + +Testing GPU in a Guest VM +------------------------- + +The Nvidia drivers must be installed first. For example, on an Ubuntu guest: + +.. code-block:: text + + sudo apt install nvidia-headless-440 nvidia-utils-440 nvidia-compute-utils-440 + +The ``nvidia-smi`` command will generate detailed output if the driver has loaded +successfully. + +Further Reference +----------------- + +For PCI Passthrough and GPUs in OpenStack: + +* Consumer-grade GPUs: https://gist.github.com/claudiok/890ab6dfe76fa45b30081e58038a9215 +* https://www.jimmdenton.com/gpu-offloading-openstack/ +* https://docs.openstack.org/nova/latest/admin/pci-passthrough.html +* https://docs.openstack.org/nova/latest/admin/virtual-gpu.html (vGPU only) +* Tesla models in OpenStack: https://egallen.com/openstack-nvidia-tesla-gpu-passthrough/ +* https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF +* https://www.kernel.org/doc/Documentation/Intel-IOMMU.txt +* https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.1/html/installation_guide/appe-configuring_a_hypervisor_host_for_pci_passthrough +* https://www.gresearch.co.uk/article/utilising-the-openstack-placement-service-to-schedule-gpu-and-nvme-workloads-alongside-general-purpose-instances/ diff --git a/doc/source/operations/hotfix-playbook.rst b/doc/source/operations/hotfix-playbook.rst index ee4d9df01..8f7c6145e 100644 --- a/doc/source/operations/hotfix-playbook.rst +++ b/doc/source/operations/hotfix-playbook.rst @@ -20,7 +20,7 @@ The playbook can be invoked with: .. code-block:: console - kayobe playbook run ${KAYOBE_CONFIG_PATH}/ansible/hotfix-containers.yml + kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/hotfix-containers.yml Playbook variables: ------------------- @@ -49,7 +49,7 @@ to a file, then add them as an extra var. e.g: .. code-block:: console - kayobe playbook run ${KAYOBE_CONFIG_PATH}/ansible/hotfix-containers.yml -e "@~/vars.yml" + kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/hotfix-containers.yml -e "@~/vars.yml" Example Variables file diff --git a/doc/source/operations/index.rst b/doc/source/operations/index.rst index 825384c4b..fd21c8c69 100644 --- a/doc/source/operations/index.rst +++ b/doc/source/operations/index.rst @@ -1,15 +1,23 @@ -================= +============== Operator Guide -================= +============== This guide is for operators of the StackHPC Kayobe configuration project. .. toctree:: :maxdepth: 1 + ceph-management + control-plane-operation + customising-horizon + gpu-in-openstack + bifrost-hardware-inventory-management hotfix-playbook + migrating-vm nova-compute-ironic octavia + openstack-projects-and-users-management + openstack-reconfiguration rabbitmq secret-rotation tempest diff --git a/doc/source/operations/migrating-vm.rst b/doc/source/operations/migrating-vm.rst new file mode 100644 index 000000000..031df1b60 --- /dev/null +++ b/doc/source/operations/migrating-vm.rst @@ -0,0 +1,22 @@ +========================== +Migrating virtual machines +========================== + +To see where all virtual machines are running on the hypervisors: + +.. code-block:: console + + openstack server list --all-projects --long + +To move a virtual machine with shared storage or booted from volume from one hypervisor to another, for example to +hypervisor-01: + +.. code-block:: console + + openstack server migrate --live-migration --host hypervisor-01 + +To move a virtual machine with local disks: + +.. code-block:: console + + openstack server migrate --live-migration --block-migration --host hypervisor-01 diff --git a/doc/source/operations/nova-compute-ironic.rst b/doc/source/operations/nova-compute-ironic.rst index 6cbe00550..9b9af76ff 100644 --- a/doc/source/operations/nova-compute-ironic.rst +++ b/doc/source/operations/nova-compute-ironic.rst @@ -68,7 +68,7 @@ Moving from multiple Nova Compute Instances to a single instance 1. Decide where the single instance should run. This should normally be one of the three OpenStack control plane hosts. For convention, pick the first one, unless you can think of a good reason not to. Once you - have chosen, set the following variable in ``etc/kayobe/nova.yml``. + have chosen, set the following variable in ``$KAYOBE_CONFIG_PATH/nova.yml``. Here we have picked ``controller1``. .. code-block:: yaml @@ -196,7 +196,7 @@ constant, such that the new Nova Compute Ironic instance comes up with the same name as the one it replaces. For example, if the original instance resides on ``controller1``, then set the -following in ``etc/kayobe/nova.yml``: +following in ``$KAYOBE_CONFIG_PATH/nova.yml``: .. code-block:: yaml @@ -270,8 +270,8 @@ Current host is not accessible ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In this case you will need to remove the inaccessible host from the inventory. -For example, in ``etc/kayobe/inventory/hosts``, remove ``controller1`` from -the ``controllers`` group. +For example, in ``$KAYOBE_CONFIG_PATH/inventory/hosts``, remove ``controller1`` +from the ``controllers`` group. Adjust the ``kolla_nova_compute_ironic_host`` variable to point to the new host, eg. diff --git a/doc/source/operations/octavia.rst b/doc/source/operations/octavia.rst index f884d130f..e13b0a1b3 100644 --- a/doc/source/operations/octavia.rst +++ b/doc/source/operations/octavia.rst @@ -12,7 +12,7 @@ With your kayobe environment activated, you can build a new amphora image with: .. code-block:: console - kayobe playbook run ${KAYOBE_CONFIG_PATH}/ansible/octavia-amphora-image-build.yml + kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/octavia-amphora-image-build.yml The resultant image is based on Ubuntu. By default the image will be built on the seed, but it is possible to change the group in the ansible inventory using the @@ -29,7 +29,7 @@ You can then run the playbook to upload the image: .. code-block:: console - kayobe playbook run ${KAYOBE_CONFIG_PATH}/ansible/octavia-amphora-image-register.yml + kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/octavia-amphora-image-register.yml This will rename the old image by adding a timestamp suffix, before uploading a new image with the name, ``amphora-x64-haproxy``. Octavia should be configured diff --git a/doc/source/operations/openstack-projects-and-users-management.rst b/doc/source/operations/openstack-projects-and-users-management.rst new file mode 100644 index 000000000..676db20ca --- /dev/null +++ b/doc/source/operations/openstack-projects-and-users-management.rst @@ -0,0 +1,38 @@ +======================================= +Openstack Projects and Users Management +======================================= + +Projects (in OpenStack) can be defined in the ``openstack-config`` repository + +To initialise the working environment for ``openstack-config``: + +.. code-block:: console + + git clone ~/src/openstack-config + python3 -m venv ~/venvs/openstack-config-venv + source ~/venvs/openstack-config-venv/bin/activate + cd ~/src/openstack-config + pip install -U pip + pip install -r requirements.txt + ansible-galaxy collection install \ + -p ansible/collections \ + -r requirements.yml + +To define a new project, add a new project to +``etc/openstack-config/openstack-config.yml``: + +Example invocation: + +.. code-block:: console + + source ~/src/kayobe-config/etc/kolla/public-openrc.sh + source ~/venvs/openstack-config-venv/bin/activate + cd ~/src/openstack-config + tools/openstack-config -- --vault-password-file + +Deleting Users and Projects +--------------------------- + +Ansible is designed for adding configuration that is not present; removing +state is less easy. To remove a project or user, the configuration should be +manually removed. diff --git a/doc/source/operations/openstack-reconfiguration.rst b/doc/source/operations/openstack-reconfiguration.rst new file mode 100644 index 000000000..392b92421 --- /dev/null +++ b/doc/source/operations/openstack-reconfiguration.rst @@ -0,0 +1,150 @@ +========================= +OpenStack Reconfiguration +========================= + +Disabling a Service +=================== + +Ansible is oriented towards adding or reconfiguring services, but removing a +service is handled less well, because of Ansible's imperative style. + +To remove a service, it is disabled in Kayobe's Kolla config, which prevents +other services from communicating with it. For example, to disable +``cinder-backup``, edit ``$KAYOBE_CONFIG_PATH/kolla.yml``: + +.. code-block:: diff + + -enable_cinder_backup: true + +enable_cinder_backup: false + +Then, reconfigure Cinder services with Kayobe: + +.. code-block:: console + + kayobe overcloud service reconfigure --kolla-tags cinder + +However, the service itself, no longer in Ansible's manifest of managed state, +must be manually stopped and prevented from restarting. + +On each controller: + +.. code-block:: console + + docker rm -f cinder_backup + +Some services may store data in a dedicated Docker volume, which can be removed +with ``docker volume rm``. + +.. _installing-external-tls-certificates: + +Installing External TLS Certificates +==================================== + +This section explains the process of deploying external TLS. +For internal and backend TLS, see `Hashicorp Vault for internal PKI +`__. + +To configure TLS for the first time, we write the contents of a PEM +file to the ``secrets.yml`` file as ``secrets_kolla_external_tls_cert``. +Use a command of this form: + +.. code-block:: console + + ansible-vault edit $KAYOBE_CONFIG_PATH/secrets.yml --vault-password-file= + +Concatenate the contents of the certificate and key files to create +``secrets_kolla_external_tls_cert``. The certificates should be installed in +this order: + +* TLS certificate for the public endpoint FQDN +* Any intermediate certificates +* The TLS certificate private key + +In ``$KAYOBE_CONFIG_PATH/kolla.yml``, set the following: + +.. code-block:: yaml + + kolla_enable_tls_external: True + kolla_external_tls_cert: "{{ secrets_kolla_external_tls_cert }}" + +To apply TLS configuration, we need to reconfigure all services, as endpoint URLs need to +be updated in Keystone: + +.. code-block:: console + + kayobe overcloud service reconfigure + +Alternative Configuration +------------------------- + +As an alternative to writing the certificates as a variable to +``secrets.yml``, it is also possible to write the same data to a file, +``$KAYOBE_CONFIG_PATH/kolla/certificates/haproxy.pem``. The file should be +vault-encrypted in the same manner as secrets.yml. In this instance, +variable ``kolla_external_tls_cert`` does not need to be defined. + +See `Kolla-Ansible TLS guide +`__ for +further details. + +Updating External TLS Certificates +---------------------------------- + +Check the expiry date on an installed TLS certificate from a host that can +reach the OpenStack APIs: + +.. code-block:: console + + openssl s_client -connect :443 2> /dev/null | openssl x509 -noout -dates + +.. note:: + + Prometheus Blackbox monitoring can check certificates automatically + and alert when expiry is approaching. + +To update an existing certificate, for example when it has reached expiration, +change the value of ``secrets_kolla_external_tls_cert``, in the same order as +above. Run the following command: + +.. warning:: + + Services can be briefly unavailable during reconfiguring HAProxy. + +.. code-block:: console + + kayobe overcloud service reconfigure --kolla-tags haproxy + +.. _taking-a-hypervisor-out-of-service: + +Taking a Hypervisor out of Service +================================== + +To take a hypervisor out of Nova scheduling: + +.. code-block:: console + + openstack compute service set --disable nova-compute + +Running instances on the hypervisor will not be affected, but new instances +will not be deployed on it. + +A reason for disabling a hypervisor can be documented with the +``--disable-reason`` flag: + +.. code-block:: console + + openstack compute service set --disable \ + --disable-reason "Broken drive" nova-compute + +Details about all hypervisors and the reasons they are disabled can be +displayed with: + +.. code-block:: console + + openstack compute service list --long + +And then to enable a hypervisor again: + +.. code-block:: console + + openstack compute service set --enable nova-compute diff --git a/doc/source/operations/secret-rotation.rst b/doc/source/operations/secret-rotation.rst index a01f66fa9..127635ab4 100644 --- a/doc/source/operations/secret-rotation.rst +++ b/doc/source/operations/secret-rotation.rst @@ -104,8 +104,8 @@ Full method 3. Navigate to the directory containing your ``passwords.yml`` file - (``kayobe-config/etc/kolla/passwords.yml`` OR - ``kayobe-config/etc/kayobe/environments/envname/kolla/passwords.yml``) + (``$KOLLA_CONFIG_PATH/passwords.yml`` OR + ``$KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/kolla/passwords.yml``) 4. Create a file called ``deletelist.txt`` and populate it with this content (including all whitespace): diff --git a/doc/source/operations/upgrading-openstack.rst b/doc/source/operations/upgrading-openstack.rst index 0b0df5056..647ac2702 100644 --- a/doc/source/operations/upgrading-openstack.rst +++ b/doc/source/operations/upgrading-openstack.rst @@ -459,7 +459,7 @@ To upgrade the Ansible control host: Syncing Release Train artifacts ------------------------------- -New :ref:`stackhpc_release_train` content should be synced to the local Pulp +New :ref:`stackhpc-release-train` content should be synced to the local Pulp server. This includes host packages (Deb/RPM) and container images. .. _sync-rt-package-repos: