-
Notifications
You must be signed in to change notification settings - Fork 35
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
package_tests: Invoke tests according to the specification #383
Open
stefwalter
wants to merge
1
commit into
CentOS-PaaS-SIG:master
Choose a base branch
from
stefwalter:invoke-tests-specification
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
92 changes: 48 additions & 44 deletions
92
config/Dockerfiles/package_tests/ansible/ansible_package_test.sh
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,50 +1,54 @@ | ||
| #!/bin/bash | ||
| set -eu | ||
|
|
||
| # This script requires that the docker run mounts the artifacts | ||
| # dir to /tmp/artifacts, the ansible inventory file to /tmp/inventory, | ||
| # and the ssh private key to /tmp/ssh_key, so mount some dir with | ||
| # the inventory and ssh_key to /tmp and expect artifacts there | ||
| # Invoke tests according to section 1.7.2 here: | ||
| # https://fedoraproject.org/wiki/Changes/InvokingTests | ||
|
|
||
| # Check if there is an upstream first repo for this package | ||
| curl -s --head https://upstreamfirst.fedorainfracloud.org/${package} | head -n 1 | grep "HTTP/1.[01] [23].." > /dev/null | ||
| if [ $? -ne 0 ]; then | ||
| echo "No upstream repo for this package! Exiting..." | ||
| exit 0 | ||
| if [ -z "${package:-}" ]; then | ||
| if [ $# -lt 1 ]; then | ||
| echo "No package defined" | ||
| exit 2 | ||
| else | ||
| package="$1" | ||
| fi | ||
| fi | ||
| git clone https://upstreamfirst.fedorainfracloud.org/${package} | ||
| if [[ $(grep "standard-test-beakerlib" ${package}/*.yml) == "" ]]; then | ||
| echo "No beakerlib tests in this repo! Exiting.." | ||
| exit 0 | ||
|
|
||
| # Make sure we have or have downloaded the test subject | ||
| if [ -z "${TEST_SUBJECTS:-}" ]; then | ||
| echo "No subject defined" | ||
| exit 2 | ||
| elif ! file ${TEST_SUBJECTS:-}; then | ||
| wget -q -O testimage.qcow2 ${TEST_SUBJECTS} | ||
| export TEST_SUBJECTS=${PWD}/testimage.qcow2 | ||
| fi | ||
| if [ -f ${package}/tests.yml ]; then | ||
| if [[ $(ansible-playbook --list-tags ${package}/tests.yml) != *"atomic"* ]]; then | ||
| echo "No atomic tagged tests for this package!" | ||
| exit 0 | ||
| fi | ||
| sed 's/hosts: localhost/hosts: all/g' ${package}/tests.yml > ${package}/test_atomic.yml | ||
| ansible-playbook -i /tmp/inventory --private-key=/tmp/ssh_key --tags=atomic --start-at-task='Define remote_artifacts if it is not already defined' ${package}/test_atomic.yml | ||
| exit $? | ||
| # All code from here down is for legacy purposes and does not | ||
| # guarantee the tests running are meant to pass on atomic host | ||
| elif [ -f ${package}/test_local.yml ]; then | ||
| sed 's/hosts: localhost/hosts: all/g' ${package}/test_local.yml > ${package}/test_atomic.yml | ||
| else | ||
| # Write test_atomic.yml header | ||
| cat << EOF > ${package}/test_atomic.yml | ||
| --- | ||
| - hosts: all | ||
| roles: | ||
| - role: standard-test-beakerlib | ||
| tests: | ||
| EOF | ||
| # Find the tests | ||
| if [ $(find ${package} -name "runtest.sh" | wc -l) -eq 0 ]; then | ||
| echo "No runtest.sh files found in package's repo. Exiting..." | ||
| exit 1 | ||
| fi | ||
| for test in $(find ${package} -name "runtest.sh"); do | ||
| echo " - $test" >> ${package}/test_atomic.yml | ||
| done | ||
|
|
||
| # Check out the dist-git repository for this package | ||
| if ! git clone https://src.fedoraproject.org/rpms/${package}; then | ||
| echo "No dist-git repo for this package! Exiting..." | ||
| exit 0 | ||
| fi | ||
| ansible-playbook -i /tmp/inventory --private-key=/tmp/ssh_key --start-at-task='Define remote_artifacts if it is not already defined' ${package}/test_atomic.yml | ||
| exit $? | ||
|
|
||
| # The specification requires us to invoke the tests in the checkout directory | ||
| cd ${package} | ||
|
|
||
| # Check out the appropriate branch | ||
| # TODO: Where does this branch come from, currently f26? | ||
| git checkout f26 | ||
|
|
||
| # The test artifacts must be an empty directory | ||
| TEST_ARTIFACTS=${TEST_ARTIFACTS:-$PWD/artifacts} | ||
| rm -rf $TEST_ARTIFACTS | ||
| export TEST_ARTIFACTS | ||
|
|
||
| # The inventory must be from the test if present (file or directory) or defaults | ||
| ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /usr/share/ansible/inventory) | ||
| export ANSIBLE_INVENTORY | ||
|
|
||
| # Invoke each playbook according to the specification | ||
| for playbook in tests/tests*.yml; do | ||
| if [ -f ${playbook} ]; then | ||
| ansible-playbook --inventory=$ANSIBLE_INVENTORY \ | ||
| --extra-vars "subjects=$TEST_SUBJECTS" --extra-vars "artifacts=$TEST_ARTIFACTS" \ | ||
| --tags atomic ${playbook} | ||
| fi | ||
| done | ||
106 changes: 46 additions & 60 deletions
106
config/Dockerfiles/package_tests/image/image_package_test.sh
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,68 +1,54 @@ | ||
| #!/bin/bash | ||
| set -eu | ||
|
|
||
| # Check if there is an upstream first repo for this package | ||
| curl -s --head https://upstreamfirst.fedorainfracloud.org/${package} | head -n 1 | grep "HTTP/1.[01] [23].." > /dev/null | ||
| if [ $? -ne 0 ]; then | ||
| echo "No upstream repo for this package! Exiting..." | ||
| exit 0 | ||
| fi | ||
| git clone https://upstreamfirst.fedorainfracloud.org/${package} | ||
| if [[ $(grep "standard-test-beakerlib" ${package}/*.yml) == "" ]]; then | ||
| echo "No beakerlib tests in this repo! Exiting.." | ||
| exit 0 | ||
| # Invoke tests according to section 1.7.2 here: | ||
| # https://fedoraproject.org/wiki/Changes/InvokingTests | ||
|
|
||
| if [ -z "${package:-}" ]; then | ||
| if [ $# -lt 1 ]; then | ||
| echo "No package defined" | ||
| exit 2 | ||
| else | ||
| package="$1" | ||
| fi | ||
| fi | ||
| if [[ $(file ${TEST_SUBJECTS}) == *"No such file or directory"* ]]; then | ||
|
|
||
| # Make sure we have or have downloaded the test subject | ||
| if [ -z "${TEST_SUBJECTS:-}" ]; then | ||
| echo "No subject defined" | ||
| exit 2 | ||
| elif ! file ${TEST_SUBJECTS:-}; then | ||
| wget -q -O testimage.qcow2 ${TEST_SUBJECTS} | ||
| export TEST_SUBJECTS=${PWD}/testimage.qcow2 | ||
| fi | ||
| if [ -f ${package}/tests.yml ]; then | ||
| if [[ $(ansible-playbook --list-tags ${package}/tests.yml) != *"atomic"* ]]; then | ||
| echo "No atomic tagged tests for this package!" | ||
| exit 0 | ||
| fi | ||
| # Execute the tests | ||
| ansible-playbook --tags atomic ${package}/tests.yml | ||
| exit $? | ||
|
|
||
| # Check out the dist-git repository for this package | ||
| if ! git clone https://src.fedoraproject.org/rpms/${package}; then | ||
| echo "No dist-git repo for this package! Exiting..." | ||
| exit 0 | ||
| fi | ||
| # Note: The below code should work, but we are not calling it. | ||
| # The reason for this is that if repos do not have a tests.yml file, | ||
| # then they have not been modified since test tagging came out, | ||
| # which means we have no idea if the tests are meant to run/will pass | ||
| # on atomic hosts. | ||
|
|
||
| #else | ||
| # # Write test_cloud.yml file | ||
| # cat << EOF > test_cloud.yml | ||
| #--- | ||
| #- hosts: localhost | ||
| # vars: | ||
| # artifacts: ./ | ||
| # playbooks: ./${package}/test_local.yml | ||
| # vars_prompt: | ||
| # - name: subjects | ||
| # prompt: "A QCow2/raw test subject file" | ||
| # private: no | ||
| # | ||
| # roles: | ||
| # - standard-test-cloud | ||
| #EOF | ||
| # # Write test_local.yml header | ||
| # cat << EOF > ${package}/test_local.yml | ||
| #--- | ||
| #- hosts: all | ||
| # roles: | ||
| # - role: standard-test-beakerlib | ||
| # tests: | ||
| #EOF | ||
| # # Find the tests | ||
| # if [ $(find ${package} -name "runtest.sh" | wc -l) -eq 0 ]; then | ||
| # echo "No runtest.sh files found in package's repo. Exiting..." | ||
| # exit 1 | ||
| # fi | ||
| # for test in $(find ${package} -name "runtest.sh"); do | ||
| # echo " - $test" >> ${package}/test_local.yml | ||
| # done | ||
| # # Execute the tests legacy method | ||
| # ansible-playbook test_cloud.yml -e subjects=$TEST_SUBJECTS -e artifacts=$TEST_ARTIFACTS | ||
| # exit $? | ||
| #fi | ||
| # The specification requires us to invoke the tests in the checkout directory | ||
| cd ${package} | ||
|
|
||
| # Check out the appropriate branch | ||
| # TODO: Where does this branch come from, currently f26? | ||
| git checkout f26 | ||
|
|
||
| # The test artifacts must be an empty directory | ||
| TEST_ARTIFACTS=${TEST_ARTIFACTS:-$PWD/artifacts} | ||
| rm -rf $TEST_ARTIFACTS | ||
| export TEST_ARTIFACTS | ||
|
|
||
| # The inventory must be from the test if present (file or directory) or defaults | ||
| ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /usr/share/ansible/inventory) | ||
| export ANSIBLE_INVENTORY | ||
|
|
||
| # Invoke each playbook according to the specification | ||
| for playbook in tests/tests*.yml; do | ||
| if [ -f ${playbook} ]; then | ||
| ansible-playbook --inventory=$ANSIBLE_INVENTORY \ | ||
| --extra-vars "subjects=$TEST_SUBJECTS" --extra-vars "artifacts=$TEST_ARTIFACTS" \ | ||
| --tags atomic ${playbook} | ||
| fi | ||
| done |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My point with having an ansible one and an image one is that the image one will take a qcow2 as a TEST_SUBJECT. I also created the ansible one to fit the following use case: We already have an atomic host up and running and rebased to our newly created ostree, and we want to target that exact host. Because we create ostrees for every pipeline run, and we do not create qcow2s for every package run, we need to be able to run the testing on a certain ostree, not just the base qcow2. So, for example, the workflow here would be, we create the ostree, we provision an atomic host with our latest qcow2, we rebase to the brand new ostree, we make sure it boots, we then run the upstream tests, then we tear it down. Is there any standard way to do this with standard-test-roles now, that you know of? Last I knew there was not, so I was changing the hosts: localhost to hosts: all in the test playbooks to accomplish this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's good to know.
One of the intents of the test specification is that the test launches up the host (ie: inventory). This was done for two reasons:
The test may choose to launch two of them, may choose to reboot the host, destroy it in some way, and so on.
Many of the tests do things to the host that change the host or destroy it. This is just part of routine testing. The order of tests run should not matter.
We should be able to broadly scale out invocation of testing based on available resources.
But I think we can match the pipeline flow here. Can you:
qemu-img create -q -f qcow2 -o backing_file=/path/to/base.qcow2,backing_fmt=qcow2 /path/to/overlay.qcow2Do you have a pointer to the code that performs these tasks so I could take a look?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here is the code that provisions the atomic host then rebases to the ostree and make sure it boots: https://github.com/CentOS-PaaS-SIG/ci-pipeline/blob/master/config/Dockerfiles/ostree_boot_image/ostree-boot-image.sh
The workflow you described sounds sane, I just need to check on the specifics of it and make sure it can all happen starting from a container, not bare metal.
The package level tests were part of the ostree-boot-sanity stage. When that stage became a container, the package tests were taken out of it, so I have been working this week on making the package tests its own stage in the pipeline. So, I will most likely get it added in (ideally today) the way I described, and then attempt to improve it to the workflow you described.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Early googling makes the workflow you describe seem as easy as well, as you described, but I will have to test it out of course first.