Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hack/build-and-push.sh: fix digest existence check #904

Merged
merged 2 commits into from
Apr 3, 2024

Conversation

chmeliik
Copy link
Contributor

The current check queries Quay's custom API and expects that the tags in the response always exist. This is not the case for deleted/expired tags. See for example:

curl 'https://quay.io/api/v1/repository/redhat-appstudio-tekton-catalog/pull-request-builds/tag/?specificTag=buildah-0.1-2f4e137b5ed645c6d5279c52eaf1d20b071729f1'

{
  "tags": [
    {
      "name": "buildah-0.1-2f4e137b5ed645c6d5279c52eaf1d20b071729f1",
      "reversion": false,
      "start_ts": 1709804197,
      "end_ts": 1711038680,
      "manifest_digest": "sha256:ebb2fe8fe89005229962167b9a31015a023a49a7169c52edfc34d18fcb1270dd",
      "is_manifest_list": false,
      "size": 5920,
      "last_modified": "Thu, 07 Mar 2024 09:36:37 -0000",
      "expiration": "Thu, 21 Mar 2024 16:31:20 -0000"
    }
  ],
  "page": 1,
  "has_additional": false
}

This tag has expired (and Quay reports it as such), but our script does not notice that.

Fix the problem by using skopeo inspect instead of querying Quay's custom API.

Before you complete this pull request ...

Look for any open pull requests in the repository with the title "e2e-tests update" and
see if there are recent e2e-tests updates that will be applicable to your change.

@chmeliik chmeliik force-pushed the fix-tag-existence-check branch from d33c1a4 to ae5a584 Compare March 22, 2024 12:49
@chmeliik chmeliik changed the title hach/build-and-push.sh: fix digest existence check hack/build-and-push.sh: fix digest existence check Mar 22, 2024
@chmeliik
Copy link
Contributor Author

Previous test run failed with a timeout on the EC pipelinerun

$ kubectl tekton -n build-templates-e2e get pr ec-integration-test-scenario-run-zq78g
NAME                                     STARTED      DURATION   STATUS             UID
ec-integration-test-scenario-run-zq78g   1 hour ago   ---        Running   4df9d745-46e0-40bd-b592-631df5bb3aab

Not sure what happened there. I can't even get the pipelineRun from Kubernetes, only from the Tekton results DB

@chmeliik
Copy link
Contributor Author

/retest

just to check that we don't re-push existing tags

Copy link
Collaborator

@mmorhun mmorhun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tkdchen
Copy link
Contributor

tkdchen commented Mar 26, 2024

/retest

@chmeliik chmeliik force-pushed the fix-tag-existence-check branch from cbe8cd1 to 9e4422d Compare March 27, 2024 11:33
@chmeliik
Copy link
Contributor Author

  [FAILED] Unexpected error:
      <*errors.errorString | 0xc00215d9b0>: 
      task with verify name doesn't exist in ec-integration-test-scenario-run-8vfhb pipelinerun
      {
          s: "task with verify name doesn't exist in ec-integration-test-scenario-run-8vfhb pipelinerun",
      }

the hell

@chmeliik
Copy link
Contributor Author

/retest

@chmeliik
Copy link
Contributor Author

Hmm, same failure again. Need to look into it

@chmeliik chmeliik force-pushed the fix-tag-existence-check branch from 9e4422d to 9800c2a Compare March 27, 2024 15:32
@chmeliik
Copy link
Contributor Author

chmeliik commented Mar 28, 2024

Timed out while running the build pipeline.

attempt 3/3: PipelineRun "devfile-sample-python-basic-lncu-f6lkl" failed: context deadline exceeded

It took 47 minutes, but can't really figure out why

$ kubectl tekton -n build-templates-e2e get pr devfile-sample-python-basic-lncu-f6lkl            
NAME                                     STARTED        DURATION   STATUS                          UID
devfile-sample-python-basic-lncu-f6lkl   17 hours ago   46m45s     Succeeded(Completed)   584f3d17-87e9-4400-8aae-1a00429cf1de

$ kubectl tekton -n build-templates-e2e logs pr devfile-sample-python-basic-lncu-f6lkl         
error: rpc error: code = NotFound desc = Log doesn't exist

@chmeliik chmeliik force-pushed the fix-tag-existence-check branch from 9800c2a to e4b3e6a Compare March 28, 2024 09:13
@lcarva
Copy link
Contributor

lcarva commented Mar 28, 2024

Some of the Tasks from the devfile-sample-python-basic-lncu-f6lkl PipelineRun:

🐚 kubectl tekton get tr devfile-sample-python-basic-lncu-f6lkl-sast-snyk-check -n build-templates-e2e -o yaml | yq '.status.startTime + " - " + .status.completionTime'
2024-03-27T16:01:12Z - 2024-03-27T16:13:02Z

🐚 kubectl tekton get tr devfile-sample-python-basic-lncu-f6lkl-clamav-scan -n build-templates-e2e -o yaml | yq '.status.startTime + " - " + .status.completionTime'
2024-03-27T16:13:21Z - 2024-03-27T16:25:20Z

These seem particularly long. Not sure what would cause those tasks to take over 10 minutes each and to not run in parallel. Maybe hitting some quota in the build-templates-e2e repo?

The current check queries Quay's custom API and expects that the tags
in the response always exist. This is not the case for deleted/expired
tags. See for example:

    curl 'https://quay.io/api/v1/repository/redhat-appstudio-tekton-catalog/pull-request-builds/tag/?specificTag=buildah-0.1-2f4e137b5ed645c6d5279c52eaf1d20b071729f1'

    {
      "tags": [
        {
          "name": "buildah-0.1-2f4e137b5ed645c6d5279c52eaf1d20b071729f1",
          "reversion": false,
          "start_ts": 1709804197,
          "end_ts": 1711038680,
          "manifest_digest": "sha256:ebb2fe8fe89005229962167b9a31015a023a49a7169c52edfc34d18fcb1270dd",
          "is_manifest_list": false,
          "size": 5920,
          "last_modified": "Thu, 07 Mar 2024 09:36:37 -0000",
          "expiration": "Thu, 21 Mar 2024 16:31:20 -0000"
        }
      ],
      "page": 1,
      "has_additional": false
    }

This tag has expired (and Quay reports it as such), but our script does
not notice that.

Fix the problem by using `skopeo inspect` instead of querying Quay's
custom API.

Signed-off-by: Adam Cmiel <[email protected]>
@chmeliik chmeliik force-pushed the fix-tag-existence-check branch from e4b3e6a to a508a79 Compare March 28, 2024 13:16
Copy link

sonarqubecloud bot commented Apr 3, 2024

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

@mkosiarc
Copy link
Contributor

mkosiarc commented Apr 3, 2024

Seems like an intermittent issue
Error: initializing source docker://registry.access.redhat.com/ubi8/go-toolset:1.17: reading manifest 1.17 in registry.access.redhat.com/ubi8/go-toolset: received unexpected HTTP status: 502 Bad Gateway

@mkosiarc
Copy link
Contributor

mkosiarc commented Apr 3, 2024

/rerun

@mkosiarc
Copy link
Contributor

mkosiarc commented Apr 3, 2024

/retest

@mkosiarc mkosiarc merged commit 2e36124 into konflux-ci:main Apr 3, 2024
6 checks passed
@chmeliik chmeliik deleted the fix-tag-existence-check branch April 4, 2024 07:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants