[skip-ci] Packit/TMT: Run gating tests #1960

lsm5 · 2023-04-03T14:45:54Z

This commit adds enables unit, validate and integration testing via TMT.

Tests have only been added for the x86_64 arch on Fedora and CentOS Stream as the tests depend on quay.io/libpod/skopeo_cidev which isn't multi-arch currently.

Items for followup:

enable system tests
enable aarch64 tests
sync test files from upstream to downstream fedora

plans/main.fmf

github-actions · 2023-05-04T00:00:44Z

A friendly reminder that this PR had no activity for 30 days.

github-actions · 2023-06-04T00:00:49Z

A friendly reminder that this PR had no activity for 30 days.

github-actions · 2023-07-06T00:01:06Z

A friendly reminder that this PR had no activity for 30 days.

mtrmac · 2023-07-07T02:24:28Z

@lsm5 are you still working on this?

(If so, this is not urgent from my POV, no need to push it to the front of your queue — I’d just like to close the PR if there is absolutely no chance this work will continue.)

lsm5 · 2023-07-07T13:16:47Z

@mtrmac I do plan to work on this. But may take a while. I have bookmarked it for myself so I don't lose track. So please feel free to close it.

mtrmac · 2023-07-07T15:02:52Z

Thanks, if it is still relevant, let’s keep it open. It might help, or inspire, other contributors.

github-actions · 2023-08-07T00:00:55Z

A friendly reminder that this PR had no activity for 30 days.

github-actions · 2023-09-07T00:00:47Z

A friendly reminder that this PR had no activity for 30 days.

lsm5 · 2023-10-11T14:33:17Z

@mtrmac @vrothberg @cevich RE: system tests in CI, is it critical to run them in a container or would we be ok with running test-system-local ? Asking mainly RE: packit + tmt tests.

mtrmac · 2023-10-11T14:43:31Z

@edsantiago is authoritative for system tests (question above).

edsantiago · 2023-10-11T14:50:08Z

Forgive me please, I don't understand the question. It's been a while since I've looked at skopeo gating tests, but my recollection is that they run on a plain system, via dnf install skopeo-tests. What's the context for "in a container"?

mtrmac · 2023-10-11T14:51:46Z

@mtrmac @vrothberg @cevich RE: system tests in CI, is it critical to run them in a container or would we be ok with running test-system-local ?

IIRC the system tests run a registry server (container). So… running -local would be much simpler in that there would be no need for nested Podman; OTOH the tests would be much less isolated from the surrounding environment, which might be a difficulty.

I’m not immediately sure how much custom tooling exists in the CI container, and would need to be reproduced; https://github.com/containers/automation_images/blob/main/skopeo_cidev/setup.sh is relevant to that, but IIRC almost all of that exists for integration, not system, tests.

thrix · 2023-10-11T15:36:12Z

/packit test

thrix · 2023-10-11T15:57:16Z

So, unfortunately, the issue we are hitting is a bug in Packit:

packit/packit-service#2028

It was not considered a bug before, as people were not hitting it and we did not know it had this consequence.
They are working on a fix.

lsm5 · 2023-10-11T17:52:38Z

So, unfortunately, the issue we are hitting is a bug in Packit:

packit/packit-service#2028

It was not considered a bug before, as people were not hitting it and we did not know it had this consequence. They are working on a fix.

Thanks @thrix . I'll disable the rhel tests for now.

lsm5 · 2023-10-11T17:53:53Z

@mtrmac @vrothberg @cevich RE: system tests in CI, is it critical to run them in a container or would we be ok with running test-system-local ?

IIRC the system tests run a registry server (container). So… running -local would be much simpler in that there would be no need for nested Podman; OTOH the tests would be much less isolated from the surrounding environment, which might be a difficulty.

I’m not immediately sure how much custom tooling exists in the CI container, and would need to be reproduced; https://github.com/containers/automation_images/blob/main/skopeo_cidev/setup.sh is relevant to that, but IIRC almost all of that exists for integration, not system, tests.

ack, the test-system target seems to spawn some container too. Anyway, I'll try to mimic the cirrus setup in tmt. Thanks

lsm5 · 2024-08-29T11:01:59Z

/packit retest

packit-as-a-service · 2024-09-24T12:00:59Z

Tests failed. @containers/packit-build please check.

This commit adds enables unit, validate and integration testing via TMT. Tests have only been added for the x86_64 arch on Fedora and CentOS Stream as the tests depend on `quay.io/libpod/skopeo_cidev` which isn't multi-arch currently. Items for followup: - enable system tests - enable aarch64 tests - sync test files from upstream to downstream fedora Signed-off-by: Lokesh Mandvekar <[email protected]>

lsm5 · 2024-09-25T12:59:57Z

@mtrmac PTAL. Only added x86_64 integration, validate and unit tests for now.

lsm5 · 2024-09-25T14:01:52Z

@mtrmac PTAL. Only added x86_64 integration, validate and unit tests for now.

If -local targets in CI tests are acceptable, that'd save us the pain of quay.io/libpod/skopeo_cidev image maintenance. Else, I'll look at making that image multiarch.

lsm5 · 2024-09-25T14:18:19Z

Hmm, and I guess running the tests in the skopeo_cidev container renders the multiple testing-farm- distro tests pointless.

mtrmac

I really don’t know what I’m doing here and how the things fit together — so a bunch of what I’m sure are stupid questions.

What is the longer-term plan? Is this going intended to be in addition to .cirrus.yml, or eventually replace it?

mtrmac · 2024-09-25T17:45:36Z

.packit.yaml

@@ -43,7 +43,7 @@ jobs:
    enable_net: true

  - job: copr_build
-    trigger: pull_request
+    trigger: ignore


Is this an intentional part of the PR? It’s not obvious to me that changes to tests should affect whether we test building.

mtrmac · 2024-09-25T17:51:33Z

hack/test-integration.sh


-make PREFIX=/usr install
+if [[ ! -f /usr/bin/skopeo ]]; then


AFAICS this introduces a risk that we are going to test some pre-existing system binary, not the one we just built (or, see elsewhere about SKOPEO_BINARY), and that it is not caller-determined for certain, but it depends on what happens to be in the environment. That ambiguity seems undesirable to me.

mtrmac · 2024-09-25T17:53:27Z

integration/tmt/main.fmf

+    - golang
+    - make
+    - podman
+    - skopeo


Is this requiring some other Skopeo binary than the one we are testing? Or does that guarantee to use the one from copr_build?

… and if this is the one we should be testing, how does that /usr/bin/skopeo get injected into the test-integration container? That doesn’t happen AFAICS.

mtrmac · 2024-09-25T17:54:07Z

rpm/skopeo.spec

@@ -71,7 +71,7 @@ BuildRequires: ostree-devel
 BuildRequires: glib2-devel
 BuildRequires: make
 BuildRequires: shadow-utils-subid-devel
-Requires: containers-common >= 4:1-21
+Requires: containers-common


How is this related to the rest of the PR?

mtrmac · 2024-09-25T17:55:03Z

.packit.yaml

@@ -31,7 +31,7 @@ jobs:
    notifications: &copr_build_failure_notification
      failure_comment:
        message: "Ephemeral COPR build failed. @containers/packit-build please check."
-    targets:
+    targets: &fedora_copr_targets


This, and centos_copr_targets, seem to be unused. Is that intentional?

mtrmac · 2024-09-25T17:57:09Z

.packit.yaml

+      failure_comment:
+        message: "Tests failed. @containers/packit-build please check."
+    tmt_plan: /plans/upstream
+    targets:


Note to self: Per https://packit.dev/docs/configuration/upstream/tests#required-parameters , targets which don’t specify an architecture imply x86_64 when used in tests.

Maybe worth a comment in the file? Especially when it differs from the build job semantics.

mtrmac · 2024-09-25T18:07:20Z

Makefile

+ifdef SKOPEO_BINARY
+$(info Skipping build as SKOPEO_BINARY is specified)
+test-integration-local:
+else
 test-integration-local: bin/skopeo
+endif


This suggests that SKOPEO_BINARY might be set. (I can’t see anything setting it. for integration tests.)

But the Go test code invoked by go test just assumes Skopeo is found in PATH, and noting sets up SKOPEO_BINARY to be handled.

How is this intended to work?

mtrmac · 2024-09-25T18:21:18Z

If -local targets in CI tests are acceptable, that'd save us the pain of quay.io/libpod/skopeo_cidev image maintenance. Else, I'll look at making that image multiarch.

From an extremely brief look at the Packit / TMT documentation, the tests seem to run in an isolated VM (chroot? container?) and install RPM test subjects into that. Running an extra level of isolation in a container seems unnecessary in that case. (Also, to my knowledge no-one has been using the $CONTAINER_RUN targets for some time, they might have been broken for months without anyone noticing.) So, using the -local targets does seem better to me.

Also, testing the binary, if it comes from a RPM, in its expected distribution, seems clearly more valuable than running it in some other frozen / differently-built container.

OTOH, at least one reason why the container exists, and is regularly rebuilt, is so that individual Skopeo CI runs don’t need to install and compile all of those tools, which rarely change, on every single CI RUN. Hence, in Cirrus, runner.sh uses the -local targets but fetches pre-built binaries from that container.

If the Packit tests are ever intended to be merge-blocking, something like that to shorten the CI time would be very valuable.

And, also, I’d prefer not to maintain the test tool setup code ( https://github.com/containers/automation_images/blob/main/skopeo_cidev/setup.sh and make tools ) in two duplicated copies, if that could be helped.

lsm5 · 2024-09-26T10:23:23Z

@mtrmac thanks a lot for all the comments. Setting back to draft while I work through them.

mtrmac reviewed Apr 3, 2023

View reviewed changes

plans/main.fmf Outdated Show resolved Hide resolved

github-actions bot added the stale-pr label May 4, 2023

lsm5 removed the stale-pr label May 4, 2023

github-actions bot added the stale-pr label Jun 4, 2023

lsm5 removed the stale-pr label Jun 5, 2023

github-actions bot added the stale-pr label Jul 6, 2023

github-actions bot removed the stale-pr label Jul 8, 2023

github-actions bot added the stale-pr label Aug 7, 2023

lsm5 removed the stale-pr label Aug 7, 2023

github-actions bot added the stale-pr label Sep 7, 2023

lsm5 removed the stale-pr label Oct 10, 2023

lsm5 force-pushed the packit-gating-tests branch from 7bfbd1c to 2e9afc9 Compare October 11, 2023 13:17

lsm5 mentioned this pull request Oct 12, 2023

ELN builds failing on copr rpm-software-management/mock#1238

Closed

lsm5 force-pushed the packit-gating-tests branch from 2e9afc9 to 49952d4 Compare October 13, 2023 19:21

lsm5 force-pushed the packit-gating-tests branch 7 times, most recently from 9c3a8a6 to 3607cd8 Compare August 19, 2024 13:13

lsm5 force-pushed the packit-gating-tests branch 2 times, most recently from 662a159 to 83405bc Compare August 28, 2024 14:57

lsm5 force-pushed the packit-gating-tests branch from 83405bc to 3451bff Compare August 29, 2024 11:06

lsm5 force-pushed the packit-gating-tests branch 2 times, most recently from 5bc34cb to 9abdfca Compare September 13, 2024 11:41

lsm5 mentioned this pull request Sep 16, 2024

Moving default-policy.json and default.yaml to containers/image #2173

Open

lsm5 force-pushed the packit-gating-tests branch 3 times, most recently from 8790b41 to 135c7ec Compare September 24, 2024 11:52

lsm5 force-pushed the packit-gating-tests branch 2 times, most recently from 95fbd34 to 70fb2e5 Compare September 25, 2024 12:07

lsm5 force-pushed the packit-gating-tests branch from 70fb2e5 to 5b3bb2c Compare September 25, 2024 12:16

lsm5 marked this pull request as ready for review September 25, 2024 12:59

mtrmac reviewed Sep 25, 2024

View reviewed changes

lsm5 marked this pull request as draft September 26, 2024 10:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[skip-ci] Packit/TMT: Run gating tests #1960

[skip-ci] Packit/TMT: Run gating tests #1960

lsm5 commented Apr 3, 2023 •

edited

Loading

github-actions bot commented May 4, 2023

github-actions bot commented Jun 4, 2023

github-actions bot commented Jul 6, 2023

mtrmac commented Jul 7, 2023

lsm5 commented Jul 7, 2023

mtrmac commented Jul 7, 2023

github-actions bot commented Aug 7, 2023

github-actions bot commented Sep 7, 2023

lsm5 commented Oct 11, 2023

mtrmac commented Oct 11, 2023

edsantiago commented Oct 11, 2023

mtrmac commented Oct 11, 2023

thrix commented Oct 11, 2023

thrix commented Oct 11, 2023

lsm5 commented Oct 11, 2023

lsm5 commented Oct 11, 2023

lsm5 commented Aug 29, 2024

packit-as-a-service bot commented Sep 24, 2024

lsm5 commented Sep 25, 2024

lsm5 commented Sep 25, 2024 •

edited

Loading

lsm5 commented Sep 25, 2024

mtrmac left a comment

mtrmac Sep 25, 2024

mtrmac Sep 25, 2024

mtrmac Sep 25, 2024

mtrmac Sep 25, 2024

mtrmac Sep 25, 2024

mtrmac Sep 25, 2024

mtrmac Sep 25, 2024

mtrmac Sep 25, 2024

mtrmac commented Sep 25, 2024

lsm5 commented Sep 26, 2024

[skip-ci] Packit/TMT: Run gating tests #1960

Are you sure you want to change the base?

[skip-ci] Packit/TMT: Run gating tests #1960

Conversation

lsm5 commented Apr 3, 2023 • edited Loading

github-actions bot commented May 4, 2023

github-actions bot commented Jun 4, 2023

github-actions bot commented Jul 6, 2023

mtrmac commented Jul 7, 2023

lsm5 commented Jul 7, 2023

mtrmac commented Jul 7, 2023

github-actions bot commented Aug 7, 2023

github-actions bot commented Sep 7, 2023

lsm5 commented Oct 11, 2023

mtrmac commented Oct 11, 2023

edsantiago commented Oct 11, 2023

mtrmac commented Oct 11, 2023

thrix commented Oct 11, 2023

thrix commented Oct 11, 2023

lsm5 commented Oct 11, 2023

lsm5 commented Oct 11, 2023

lsm5 commented Aug 29, 2024

packit-as-a-service bot commented Sep 24, 2024

lsm5 commented Sep 25, 2024

lsm5 commented Sep 25, 2024 • edited Loading

lsm5 commented Sep 25, 2024

mtrmac left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtrmac commented Sep 25, 2024

lsm5 commented Sep 26, 2024

lsm5 commented Apr 3, 2023 •

edited

Loading

lsm5 commented Sep 25, 2024 •

edited

Loading