Allow configuring timeout for external sources #1812

pouyan021 · 2024-04-18T09:31:49Z

This pull request closes #1624. It adds and enforces the ability to set a new property abort-after to external sources. As discussed in the issue, it supports both a global prop and a maven property that overrides the global if it is set.

README.md

spiffcs · 2024-04-22T17:35:34Z

@pouyan021 Approved and running checks now - Thank you so much for the contribution!

spiffcs · 2024-04-22T17:36:29Z

@pouyan021 looks like there is a small change needed where go mod tidy is run

pouyan021 · 2024-04-22T17:49:12Z

@spiffcs Appreciate your support, thanks a lot! The problem with go.mod is addressed but I forgot to sign-off my commit! I fixed that and pushed again. Could you kindly approve the checks once more?

spiffcs

Everything else LGTM - Just waiting for @wagoodman and his final say on the config direction he wants to go here

spiffcs · 2024-04-25T18:31:48Z

README.md

    base-url: https://search.maven.org/solrsearch/select
+    abort-after: 5m #override the global config


cc @wagoodman - I know he's pretty sensitive to duplicate fields that override each other so I'd like him to chime in on where he sees this going or what his preference would be

stale review and waiting for IC input

wagoodman

Overall the functionality looks good, but have some comments on testing and configuration. I'll push up some changes shortly to help out.

I do think that RequestTimeout is a better name for this config item -- what do you think @pouyan021 ?

cmd/grype/cli/options/datasources.go

grype/matcher/java/matcher_mocks_test.go

wagoodman · 2024-05-16T14:20:45Z

README.md

@@ -278,9 +278,11 @@ feature is currently disabled by default. To enable this feature add the followi
 ```yaml
 external-sources:
  enable: true
+  abort-after: 10m


I think we can clarify what this means by changing the name some. This could be interpreted as either:
a. aborting looking up from external sources in general after the duration elapses
b. aborting a single request to an external source after the duration elapses

From the functionality implemented b is implied.

Regarding naming and the above context, request-timeout feels like a more descriptive name.

cmd/grype/cli/options/datasources.go

Signed-off-by: Pouyan Khodabakhsh <[email protected]>

… functionality Signed-off-by: Pouyan Khodabakhsh <[email protected]>

…-after functionality Signed-off-by: Pouyan Khodabakhsh <[email protected]>

Signed-off-by: Pouyan Khodabakhsh <[email protected]>

Signed-off-by: Alex Goodman <[email protected]>

wagoodman · 2024-05-16T15:30:05Z

note: I force pushed to get this branch rebased onto the latest commit on main

pouyan021 · 2024-05-17T03:31:22Z

Overall the functionality looks good, but have some comments on testing and configuration. I'll push up some changes shortly to help out.

I do think that RequestTimeout is a better name for this config item -- what do you think @pouyan021 ?

Hey @wagoodman thanks a lot for the thorough review and your additional changes. The name was chosen as abort-after based on the suggestion by one of the contributors here. I agree with you that RequestTimeout is more self-explanatory.

pouyan021 · 2024-05-24T11:41:27Z

Hey @wagoodman should I go for the requestTimeout as per your suggestion?

…sources

pouyan021 · 2024-06-13T11:37:52Z

Hey @wagoodman @spiffcs I resolved the conflict on this branch recently, any plans for this to move forward?

spiffcs · 2024-07-02T16:10:49Z

@pouyan021 I'm so sorry here - I've approved and have run the final checks and will get this merged - I did not see the notification for this and apologize for letting it sit

I'll get the commits in the wrap this up 😄

…#1871) Bumps [anchore/sbom-action](https://github.com/anchore/sbom-action) from 0.15.11 to 0.16.0. - [Release notes](https://github.com/anchore/sbom-action/releases) - [Commits](anchore/sbom-action@7ccf588...e8d2a69) --- updated-dependencies: - dependency-name: anchore/sbom-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…#1870) Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2.13.4 to 3.25.6. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@cdcdbb5...9fdb3e4) --- updated-dependencies: - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: westonsteimel <[email protected]>

Bumps [actions/checkout](https://github.com/actions/checkout) from 4.1.5 to 4.1.6. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@44c2b7a...a5ac7e5) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…4.4 (anchore#1889) Bumps [github.com/gabriel-vasile/mimetype](https://github.com/gabriel-vasile/mimetype) from 1.4.3 to 1.4.4. - [Release notes](https://github.com/gabriel-vasile/mimetype/releases) - [Commits](gabriel-vasile/mimetype@v1.4.3...v1.4.4) --- updated-dependencies: - dependency-name: github.com/gabriel-vasile/mimetype dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [github.com/charmbracelet/bubbletea](https://github.com/charmbracelet/bubbletea) from 0.26.2 to 0.26.3. - [Release notes](https://github.com/charmbracelet/bubbletea/releases) - [Commits](charmbracelet/bubbletea@v0.26.2...v0.26.3) --- updated-dependencies: - dependency-name: github.com/charmbracelet/bubbletea dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: westonsteimel <[email protected]>

…anchore#1887) Bumps [github.com/hashicorp/go-version](https://github.com/hashicorp/go-version) from 1.6.0 to 1.7.0. - [Release notes](https://github.com/hashicorp/go-version/releases) - [Changelog](https://github.com/hashicorp/go-version/blob/main/CHANGELOG.md) - [Commits](hashicorp/go-version@v1.6.0...v1.7.0) --- updated-dependencies: - dependency-name: github.com/hashicorp/go-version dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [github.com/charmbracelet/lipgloss](https://github.com/charmbracelet/lipgloss) from 0.10.0 to 0.11.0. - [Release notes](https://github.com/charmbracelet/lipgloss/releases) - [Commits](charmbracelet/lipgloss@v0.10.0...v0.11.0) --- updated-dependencies: - dependency-name: github.com/charmbracelet/lipgloss dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* update to latest syft Signed-off-by: Alex Goodman <[email protected]> * fix tests related to syft bump Signed-off-by: Alex Goodman <[email protected]> --------- Signed-off-by: Alex Goodman <[email protected]>

Bumps [docker/login-action](https://github.com/docker/login-action) from 3.1.0 to 3.2.0. - [Release notes](https://github.com/docker/login-action/releases) - [Commits](docker/login-action@e92390c...0d4c9c5) --- updated-dependencies: - dependency-name: docker/login-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Signed-off-by: Alex Goodman <[email protected]>

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: westonsteimel <[email protected]>

Signed-off-by: Dan Luhring <[email protected]>

Signed-off-by: Keith Zantow <[email protected]>

Signed-off-by: Avtar Gill <[email protected]>

Bumps [github.com/charmbracelet/bubbletea](https://github.com/charmbracelet/bubbletea) from 0.26.3 to 0.26.4. - [Release notes](https://github.com/charmbracelet/bubbletea/releases) - [Commits](charmbracelet/bubbletea@v0.26.3...v0.26.4) --- updated-dependencies: - dependency-name: github.com/charmbracelet/bubbletea dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…#1901) Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3.25.6 to 3.25.7. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@9fdb3e4...f079b84) --- updated-dependencies: - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Signed-off-by: Alex Goodman <[email protected]>

Bumps [github.com/docker/docker](https://github.com/docker/docker) from 26.1.3+incompatible to 26.1.4+incompatible. - [Release notes](https://github.com/docker/docker/releases) - [Commits](moby/moby@v26.1.3...v26.1.4) --- updated-dependencies: - dependency-name: github.com/docker/docker dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…#1909) Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3.25.7 to 3.25.8. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@f079b84...2e230e8) --- updated-dependencies: - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Signed-off-by: Alex Goodman <[email protected]>

* feat(signature): Checksum signature verification Signed-off-by: Shubham Hibare <[email protected]> Signed-off-by: Alex Goodman <[email protected]> * Update message Signed-off-by: Shubham Hibare <[email protected]> Signed-off-by: Alex Goodman <[email protected]> * address comments Signed-off-by: Shubham Hibare <[email protected]> Signed-off-by: Alex Goodman <[email protected]> * consider -v flag across supported releases Signed-off-by: Alex Goodman <[email protected]> * add tests for install.sh signature verification Signed-off-by: Alex Goodman <[email protected]> * check that release is run from main Signed-off-by: Alex Goodman <[email protected]> * summarize install.sh flags and recommendations Signed-off-by: Alex Goodman <[email protected]> * remove regex use on cosign verify-blob Co-authored-by: Dominique Martinet <[email protected]> Signed-off-by: Alex Goodman <[email protected]> * simplify the compare_semver install function Co-authored-by: Dominique Martinet <[email protected]> Signed-off-by: Alex Goodman <[email protected]> * add more tests to compare_semver Signed-off-by: Alex Goodman <[email protected]> * nit copy change for install help Signed-off-by: Alex Goodman <[email protected]> * keep original compare_semver implementation Signed-off-by: Alex Goodman <[email protected]> * update copy to include default install path Signed-off-by: Alex Goodman <[email protected]> --------- Signed-off-by: Shubham Hibare <[email protected]> Signed-off-by: Alex Goodman <[email protected]> Co-authored-by: Alex Goodman <[email protected]> Co-authored-by: Dominique Martinet <[email protected]>

Bumps [actions/checkout](https://github.com/actions/checkout) from 4.1.1 to 4.1.6. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v4.1.1...a5ac7e5) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): update tools to latest versions Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * chore: update releaser file to compatible fields Signed-off-by: Christopher Phillips <[email protected]> --------- Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Christopher Phillips <[email protected]> Co-authored-by: westonsteimel <[email protected]> Co-authored-by: Christopher Phillips <[email protected]>

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: westonsteimel <[email protected]>

Signed-off-by: Alex Goodman <[email protected]>

…or-external-sources' into feat/allow-configuring-timeout-for-external-sources

pouyan021 · 2024-09-17T07:21:59Z

Hey @spiffcs, I rebased this branch again, could you please trigger the checks so we can hopefully merge this?

pouyan021 · 2024-09-19T13:14:16Z

cc @wagoodman

spiffcs previously approved these changes Apr 22, 2024

View reviewed changes

README.md Show resolved Hide resolved

pouyan021 force-pushed the feat/allow-configuring-timeout-for-external-sources branch 2 times, most recently from 5cc6f4b to e5ed88d Compare April 22, 2024 17:46

pouyan021 requested a review from spiffcs April 25, 2024 06:19

spiffcs reviewed Apr 25, 2024

View reviewed changes

spiffcs self-requested a review April 25, 2024 18:34

wagoodman changed the title ~~Feat/allow configuring timeout for external sources~~ Allow configuring timeout for external sources May 16, 2024

wagoodman reviewed May 16, 2024

View reviewed changes

pouyan021 and others added 7 commits May 16, 2024 11:27

adding abort-after property to external-sources

4cbb4f0

Signed-off-by: Pouyan Khodabakhsh <[email protected]>

considering the timeout in java matcher

b8fe080

Signed-off-by: Pouyan Khodabakhsh <[email protected]>

make the maven search method respect the context time out

49ae4c9

Signed-off-by: Pouyan Khodabakhsh <[email protected]>

adding a number of additional tests for the external-sources time-out…

7043535

… functionality Signed-off-by: Pouyan Khodabakhsh <[email protected]>

facilitating java matcher mocks to imitate the external-sources abort…

db426d2

…-after functionality Signed-off-by: Pouyan Khodabakhsh <[email protected]>

updating readme to reflect the new changes in external-sources

1e947b6

Signed-off-by: Pouyan Khodabakhsh <[email protected]>

adjust multi level configuration + faster tests

0bf64dd

Signed-off-by: Alex Goodman <[email protected]>

wagoodman force-pushed the feat/allow-configuring-timeout-for-external-sources branch from ca5d50c to 0bf64dd Compare May 16, 2024 15:29

Merge branch 'main' into feat/allow-configuring-timeout-for-external-…

2c79d3a

…sources

dependabot bot and others added 4 commits September 17, 2024 11:15

chore(deps): update tools to latest versions (anchore#1864)

c50a72a

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: westonsteimel <[email protected]>

anchore-actions-token-generator bot and others added 28 commits September 17, 2024 11:16

chore(deps): update tools to latest versions (anchore#1883)

991513f

chore(deps): update tools to latest versions (anchore#1891)

48fc783

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: westonsteimel <[email protected]>

Update syft to 1.4.2-0.20240528141306-ac34808b9c55 (anchore#1895)

ecee300

* update to latest syft Signed-off-by: Alex Goodman <[email protected]> * fix tests related to syft bump Signed-off-by: Alex Goodman <[email protected]> --------- Signed-off-by: Alex Goodman <[email protected]>

update syft to v1.5.0 (anchore#1897)

b3d1bcf

Signed-off-by: Alex Goodman <[email protected]>

chore(deps): update tools to latest versions (anchore#1898)

20f16c4

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: westonsteimel <[email protected]>

fix: main mod pseudo version default off (anchore#1894)

0a2fdb4

Signed-off-by: Dan Luhring <[email protected]>

fix: uppercased package in json (anchore#1900)

f043899

Signed-off-by: Keith Zantow <[email protected]>

fix: add note about TMPDIR env var (anchore#1880)

c9ec282

Signed-off-by: Avtar Gill <[email protected]>

use dco tool during gh app outage (anchore#1910)

5d1324f

Signed-off-by: Alex Goodman <[email protected]>

remove dco workflow (anchore#1914)

22ec108

Signed-off-by: Alex Goodman <[email protected]>

add skopeo to managed utilities (anchore#1915)

92a889f

Signed-off-by: Alex Goodman <[email protected]>

chore(deps): update tools to latest versions (anchore#1921)

e561e7e

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: westonsteimel <[email protected]>

chore(deps): update tools to latest versions (anchore#1925)

3433baf

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: westonsteimel <[email protected]>

sort order for matches should consider fix info (anchore#1933)

8d9e74e

Signed-off-by: Alex Goodman <[email protected]>

Updating maven URLs in README.md (anchore#1934)

ccd8c3e

Merge remote-tracking branch 'origin/feat/allow-configuring-timeout-f…

75a337c

…or-external-sources' into feat/allow-configuring-timeout-for-external-sources

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow configuring timeout for external sources #1812

Allow configuring timeout for external sources #1812

pouyan021 commented Apr 18, 2024 •

edited

Loading

spiffcs commented Apr 22, 2024

spiffcs commented Apr 22, 2024

pouyan021 commented Apr 22, 2024 •

edited

Loading

spiffcs left a comment

spiffcs Apr 25, 2024

wagoodman left a comment

wagoodman May 16, 2024

wagoodman commented May 16, 2024

pouyan021 commented May 17, 2024 •

edited

Loading

pouyan021 commented May 24, 2024

pouyan021 commented Jun 13, 2024

spiffcs commented Jul 2, 2024 •

edited

Loading

pouyan021 commented Sep 17, 2024

pouyan021 commented Sep 19, 2024

		base-url: https://search.maven.org/solrsearch/select
		abort-after: 5m #override the global config

Allow configuring timeout for external sources #1812

Are you sure you want to change the base?

Allow configuring timeout for external sources #1812

Conversation

pouyan021 commented Apr 18, 2024 • edited Loading

spiffcs commented Apr 22, 2024

spiffcs commented Apr 22, 2024

pouyan021 commented Apr 22, 2024 • edited Loading

spiffcs left a comment

Choose a reason for hiding this comment

spiffcs Apr 25, 2024

Choose a reason for hiding this comment

wagoodman left a comment

Choose a reason for hiding this comment

wagoodman May 16, 2024

Choose a reason for hiding this comment

wagoodman commented May 16, 2024

pouyan021 commented May 17, 2024 • edited Loading

pouyan021 commented May 24, 2024

pouyan021 commented Jun 13, 2024

spiffcs commented Jul 2, 2024 • edited Loading

pouyan021 commented Sep 17, 2024

pouyan021 commented Sep 19, 2024

pouyan021 commented Apr 18, 2024 •

edited

Loading

pouyan021 commented Apr 22, 2024 •

edited

Loading

pouyan021 commented May 17, 2024 •

edited

Loading

spiffcs commented Jul 2, 2024 •

edited

Loading