Individual changes to keylime are introduced using the pull-request workflow and for each pull-request update a set of test is executed.
These are the tests stored in test directory. Tests are run on the ubuntu-latest image through GitHub Actions.
E2E tests from keylime-tests repository are run through Packit-as-a-Service on currently supported Fedora releases, Fedora Rawhide and CentOS Stream releases. Test plan is stored in packit-ci.fmf file.
Tests are run for every pull-request update. One can also re-run all tests by adding the comment /packit test.
During test execution we can measure Python code coverage, in particular code coverage for changes introduced in pull-request. Such a measurement is currently enabled only on Fedora 35 release and executed using Packit service. We do measure code coverage for keylime unittests, keylime testsuite and functional tests from keylime-tests repository.
There are two ways how to review code coverage details.
codecov.io service should provide code coverage related statistics for every pull request. These statistics can be accessed either from a comment added by the codecov.io service or via respective pull request checks. Be aware that GitHub/Gitlab/Bitbucket authentication is necessary for accessing statistics at codecov.io.
From an implementation perspective, for proper codecov.io functioning we need to submit code coverage data for every HEAD commit in a pull request and for every new HEAD commit in the main keylime branch. This comes from the fact that on GitHub merged commits are different from commits in PR. For differential coverage metrics codecov.io needs coverage data both for PR base commit (typically originates in a main branch) and the PR HEAD commit so a diff can be made. Therefore we need to submit coverage data on two occasions. First, when a commit gets merged into the keylime main branch - this way we provide coverage data for a baseline of a future pull request. Second, when a new commit lands in a pull request - this way we provide data for the PR HEAD commit so that codecov.io can update pull request with computed coverage metrics. Therefore we have two GitHub workflows configured. The first workflow is triggered by a pull request update. It waits until tests scheduled by Packit service are finished and then downloads and submits code coverage data to codecov.io. The second workflow is triggered by a commit merged to a main keylime branch and the main difference here is that this workflow doesn't wait for tests to be finished as there are none - we can't run tests again on a merged commit. Instead, we find the related pull request and download existing coverage data from the respective (and already finished) test in a pull request. The advantage is that we do not have to run tests again and the workflow is fast. The disadvantage is that such data won't be correct if any other commit has been merged to the main branch in the meantime. Therefore, we need to verify that all commits between PR BASE commit and main branch HEAD comes from a single pull request and if not, we won't submit coverage data. This results in a gap in our data sequence and we won't be able to get differential statistics for the next pull request.
In order to access code coverage report one has to open Packit-as-a-Service logs for Fedora 35 and display test logs of /setup/generate_coverage_report task.
In the log one can see indivudual code coverage report summaries for unittests, keylime testsuite and keylime-tests. Also, there is an overall report. One can also download HTML report from transfer.sh, specific URL is listed in the test log too.
Name Stmts Miss Cover
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
/usr/local/bin/keylime_agent 18 5 72%
/usr/local/bin/keylime_ima_emulator 18 5 72%
/usr/local/bin/keylime_registrar 18 5 72%
/usr/local/bin/keylime_tenant 18 5 72%
/usr/local/bin/keylime_verifier 18 5 72%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/__init__.py 0 0 100%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/agentstates.py 106 54 49%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/api_version.py 42 12 71%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/ca_impl_openssl.py 53 0 100%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/ca_util.py 348 175 50%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/cloud_verifier_common.py 154 129 16%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/cloud_verifier_tornado.py 747 632 15%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/cmd/__init__.py 0 0 100%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/cmd/agent.py 11 4 64%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/cmd/ima_emulator_adapter.py 68 10 85%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/cmd/migrations_apply.py 20 6 70%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/cmd/registrar.py 15 4 73%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/cmd/tenant.py 13 6 54%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/cmd/user_data_encrypt.py 45 26 42%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/cmd/verifier.py 15 4 73%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/cmd_exec.py 32 4 88%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/common/__init__.py 0 0 100%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/common/algorithms.py 55 12 78%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/common/exception.py 8 3 62%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/common/retry.py 7 6 14%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/common/states.py 30 1 97%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/common/validators.py 35 18 49%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/config.py 113 33 71%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/crypto.py 106 23 78%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/db/__init__.py 0 0 100%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/db/keylime_db.py 60 9 85%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/db/registrar_db.py 20 0 100%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/db/verifier_db.py 49 0 100%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/failure.py 112 42 62%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/fs_util.py 5 1 80%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/gpg.py 17 13 24%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/ima.py 320 188 41%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/ima_ast.py 228 105 54%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/ima_file_signatures.py 279 180 35%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/json.py 45 15 67%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/keylime_agent.py 520 189 64%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/keylime_logging.py 52 16 69%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/measured_boot.py 60 41 32%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/migrations/env.py 56 19 66%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/migrations/versions/1ac1513ef2a1_fix_mb_and_ima_column_types.py 22 5 77%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/migrations/versions/257fe0f0c039_add_fields_for_revocation_context_to_.py 20 4 80%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/migrations/versions/63c30820fdc1_add_mtls_cert_and_ak_to_verifier_db.py 20 4 80%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/migrations/versions/7d5db1a6ffb0_add_agentstates_columns.py 25 6 76%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/migrations/versions/8a44a4364f5a_initial_database_migration.py 21 3 86%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/migrations/versions/8da20383f6e1_extend_ip_field.py 19 3 84%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/migrations/versions/9169f80345ed_add_supported_version_to_verifiermain_.py 20 3 85%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/migrations/versions/a79c27ec1054_add_mtls_cert_field_to_registrar.py 18 3 83%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/migrations/versions/a7a64155ab3a_add_ima_filesigning_keys_column.py 18 3 83%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/migrations/versions/ae898986c6e9_add_mb_refstate_column.py 18 3 83%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/migrations/versions/b4d024197413_add_verfier_id_to_verifiermain_table.py 22 5 77%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/migrations/versions/c3842cc9ee69_store_keyrings_learned_from_ima_log.py 19 3 84%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/migrations/versions/cc2630851a1f_receive_the_aik_tpm_from_the_agent.py 28 8 71%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/migrations/versions/eb869a77abd1_create_allowlist_table.py 18 3 83%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/migrations/versions/eeb702f77d7d_allowlist_rename.py 20 4 80%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/migrations/versions/f35cdd35eb83_move_v_tpm_policy_to_jsonpickletype.py 23 5 78%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/migrations/versions/f82c4252bc4f_add_ip_and_port_to_registrar.py 22 5 77%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/registrar_client.py 143 50 65%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/registrar_common.py 341 145 57%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/requests_client.py 55 10 82%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/revocation_notifier.py 144 114 21%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/secure_mount.py 59 16 73%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/tenant.py 898 376 58%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/tornado_requests.py 37 30 19%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/tpm/__init__.py 0 0 100%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/tpm/tpm2_objects.py 199 81 59%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/tpm/tpm_abstract.py 243 75 69%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/tpm/tpm_main.py 979 386 61%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/tpm_ek_ca.py 24 14 42%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/user_utils.py 54 45 17%
/usr/local/lib/python3.10/site-packages/keylime-6.3.1-py3.10.egg/keylime/web_util.py 137 35 74%
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL 7602 3452 55%
There is also code coverage report specifically for code changes presented in a form of a patch. This way one can see which code is executed by particular E2E tests. E.g. for PR#916 the output looks like this.
diff --git a/keylime/db/keylime_db.py b/keylime/db/keylime_db.py
index 1cd4a07..91885a7 100644
--- a/keylime/db/keylime_db.py
+++ b/keylime/db/keylime_db.py
@@ -39,8 +39,9 @@ class DBEngineManager:
url = config.get(service, 'database_url')
if url:
logger.info('database_url is set, using it to establish database connection')
- engine_args['pool_size'] = int(p_sz)
- engine_args['max_overflow'] = int(m_ovfl)
EF + if not url.count('sqlite:') :
EF + engine_args['pool_size'] = int(p_sz)
EF + engine_args['max_overflow'] = int(m_ovfl)
else :
logger.info('database_url is not set, using multi-parameter database configuration options')
--------------------------------------------------------------------------------
Overall patch coverage: 100 %, 3 out of 3 lines are covered by a test
Legend:
+ there are additional tests executing this line
! line not covered by a test
~ line is not being measured
E /functional/db-postgresql-sanity-on-localhost
F /functional/db-mariadb-sanity-on-localhost
The Keylime test suite can be run locally by executing .ci/run_local.sh (requires Docker).
Tests can be easily run on Fedora and CentOS Stream using the tmt tool.
For more details on local E2E test execution and troubleshooting please see TESTING.md document from keylime-tests repository, especially section Running CI tests from the upstream keylime project.