Allow configuring timeout for external sources #1812

pouyan021 · 2024-04-18T09:31:49Z

This pull request closes #1624. It adds and enforces the ability to set a new property abort-after to external sources. As discussed in the issue, it supports both a global prop and a maven property that overrides the global if it is set.

README.md

spiffcs · 2024-04-22T17:35:34Z

@pouyan021 Approved and running checks now - Thank you so much for the contribution!

spiffcs · 2024-04-22T17:36:29Z

@pouyan021 looks like there is a small change needed where go mod tidy is run

pouyan021 · 2024-04-22T17:49:12Z

@spiffcs Appreciate your support, thanks a lot! The problem with go.mod is addressed but I forgot to sign-off my commit! I fixed that and pushed again. Could you kindly approve the checks once more?

spiffcs

Everything else LGTM - Just waiting for @wagoodman and his final say on the config direction he wants to go here

spiffcs · 2024-04-25T18:31:48Z

README.md

    base-url: https://search.maven.org/solrsearch/select
+    abort-after: 5m #override the global config


cc @wagoodman - I know he's pretty sensitive to duplicate fields that override each other so I'd like him to chime in on where he sees this going or what his preference would be

stale review and waiting for IC input

wagoodman

Overall the functionality looks good, but have some comments on testing and configuration. I'll push up some changes shortly to help out.

I do think that RequestTimeout is a better name for this config item -- what do you think @pouyan021 ?

cmd/grype/cli/options/datasources.go

grype/matcher/java/matcher_mocks_test.go

wagoodman · 2024-05-16T14:20:45Z

README.md

@@ -278,9 +278,11 @@ feature is currently disabled by default. To enable this feature add the followi
 ```yaml
 external-sources:
  enable: true
+  abort-after: 10m


I think we can clarify what this means by changing the name some. This could be interpreted as either:
a. aborting looking up from external sources in general after the duration elapses
b. aborting a single request to an external source after the duration elapses

From the functionality implemented b is implied.

Regarding naming and the above context, request-timeout feels like a more descriptive name.

cmd/grype/cli/options/datasources.go

Signed-off-by: Pouyan Khodabakhsh <pouyan021@gmail.com>

… functionality Signed-off-by: Pouyan Khodabakhsh <pouyan021@gmail.com>

…-after functionality Signed-off-by: Pouyan Khodabakhsh <pouyan021@gmail.com>

Signed-off-by: Pouyan Khodabakhsh <pouyan021@gmail.com>

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

wagoodman · 2024-05-16T15:30:05Z

note: I force pushed to get this branch rebased onto the latest commit on main

pouyan021 · 2024-05-17T03:31:22Z

Overall the functionality looks good, but have some comments on testing and configuration. I'll push up some changes shortly to help out.

I do think that RequestTimeout is a better name for this config item -- what do you think @pouyan021 ?

Hey @wagoodman thanks a lot for the thorough review and your additional changes. The name was chosen as abort-after based on the suggestion by one of the contributors here. I agree with you that RequestTimeout is more self-explanatory.

pouyan021 · 2024-05-24T11:41:27Z

Hey @wagoodman should I go for the requestTimeout as per your suggestion?

…sources

pouyan021 · 2024-06-13T11:37:52Z

Hey @wagoodman @spiffcs I resolved the conflict on this branch recently, any plans for this to move forward?

spiffcs · 2024-07-02T16:10:49Z

@pouyan021 I'm so sorry here - I've approved and have run the final checks and will get this merged - I did not see the notification for this and apologize for letting it sit

I'll get the commits in the wrap this up 😄

…#1871) Bumps [anchore/sbom-action](https://github.com/anchore/sbom-action) from 0.15.11 to 0.16.0. - [Release notes](https://github.com/anchore/sbom-action/releases) - [Commits](anchore/sbom-action@7ccf588...e8d2a69) --- updated-dependencies: - dependency-name: anchore/sbom-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…#1870) Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2.13.4 to 3.25.6. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@cdcdbb5...9fdb3e4) --- updated-dependencies: - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: westonsteimel <1593939+westonsteimel@users.noreply.github.com>

Bumps [actions/checkout](https://github.com/actions/checkout) from 4.1.5 to 4.1.6. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@44c2b7a...a5ac7e5) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…4.4 (anchore#1889) Bumps [github.com/gabriel-vasile/mimetype](https://github.com/gabriel-vasile/mimetype) from 1.4.3 to 1.4.4. - [Release notes](https://github.com/gabriel-vasile/mimetype/releases) - [Commits](gabriel-vasile/mimetype@v1.4.3...v1.4.4) --- updated-dependencies: - dependency-name: github.com/gabriel-vasile/mimetype dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [github.com/charmbracelet/bubbletea](https://github.com/charmbracelet/bubbletea) from 0.26.2 to 0.26.3. - [Release notes](https://github.com/charmbracelet/bubbletea/releases) - [Commits](charmbracelet/bubbletea@v0.26.2...v0.26.3) --- updated-dependencies: - dependency-name: github.com/charmbracelet/bubbletea dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: westonsteimel <1593939+westonsteimel@users.noreply.github.com>

…anchore#1887) Bumps [github.com/hashicorp/go-version](https://github.com/hashicorp/go-version) from 1.6.0 to 1.7.0. - [Release notes](https://github.com/hashicorp/go-version/releases) - [Changelog](https://github.com/hashicorp/go-version/blob/main/CHANGELOG.md) - [Commits](hashicorp/go-version@v1.6.0...v1.7.0) --- updated-dependencies: - dependency-name: github.com/hashicorp/go-version dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [github.com/charmbracelet/lipgloss](https://github.com/charmbracelet/lipgloss) from 0.10.0 to 0.11.0. - [Release notes](https://github.com/charmbracelet/lipgloss/releases) - [Commits](charmbracelet/lipgloss@v0.10.0...v0.11.0) --- updated-dependencies: - dependency-name: github.com/charmbracelet/lipgloss dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* update to latest syft Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * fix tests related to syft bump Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> --------- Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

Bumps [docker/login-action](https://github.com/docker/login-action) from 3.1.0 to 3.2.0. - [Release notes](https://github.com/docker/login-action/releases) - [Commits](docker/login-action@e92390c...0d4c9c5) --- updated-dependencies: - dependency-name: docker/login-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: westonsteimel <1593939+westonsteimel@users.noreply.github.com>

Signed-off-by: Dan Luhring <dluhring@chainguard.dev>

Signed-off-by: Keith Zantow <kzantow@gmail.com>

Signed-off-by: Avtar Gill <avtargill@gmail.com>

Bumps [github.com/charmbracelet/bubbletea](https://github.com/charmbracelet/bubbletea) from 0.26.3 to 0.26.4. - [Release notes](https://github.com/charmbracelet/bubbletea/releases) - [Commits](charmbracelet/bubbletea@v0.26.3...v0.26.4) --- updated-dependencies: - dependency-name: github.com/charmbracelet/bubbletea dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…#1901) Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3.25.6 to 3.25.7. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@9fdb3e4...f079b84) --- updated-dependencies: - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

Bumps [github.com/docker/docker](https://github.com/docker/docker) from 26.1.3+incompatible to 26.1.4+incompatible. - [Release notes](https://github.com/docker/docker/releases) - [Commits](moby/moby@v26.1.3...v26.1.4) --- updated-dependencies: - dependency-name: github.com/docker/docker dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…#1909) Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3.25.7 to 3.25.8. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@f079b84...2e230e8) --- updated-dependencies: - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* feat(signature): Checksum signature verification Signed-off-by: Shubham Hibare <shubham@hibare.in> Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * Update message Signed-off-by: Shubham Hibare <shubham@hibare.in> Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * address comments Signed-off-by: Shubham Hibare <shubham@hibare.in> Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * consider -v flag across supported releases Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * add tests for install.sh signature verification Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * check that release is run from main Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * summarize install.sh flags and recommendations Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * remove regex use on cosign verify-blob Co-authored-by: Dominique Martinet <asmadeus@codewreck.org> Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * simplify the compare_semver install function Co-authored-by: Dominique Martinet <asmadeus@codewreck.org> Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * add more tests to compare_semver Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * nit copy change for install help Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * keep original compare_semver implementation Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> * update copy to include default install path Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> --------- Signed-off-by: Shubham Hibare <shubham@hibare.in> Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com> Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com> Co-authored-by: Dominique Martinet <asmadeus@codewreck.org>

Bumps [actions/checkout](https://github.com/actions/checkout) from 4.1.1 to 4.1.6. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v4.1.1...a5ac7e5) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): update tools to latest versions Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * chore: update releaser file to compatible fields Signed-off-by: Christopher Phillips <32073428+spiffcs@users.noreply.github.com> --------- Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Christopher Phillips <32073428+spiffcs@users.noreply.github.com> Co-authored-by: westonsteimel <1593939+westonsteimel@users.noreply.github.com> Co-authored-by: Christopher Phillips <32073428+spiffcs@users.noreply.github.com>

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: westonsteimel <1593939+westonsteimel@users.noreply.github.com>

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

…or-external-sources' into feat/allow-configuring-timeout-for-external-sources

pouyan021 · 2024-09-17T07:21:59Z

Hey @spiffcs, I rebased this branch again, could you please trigger the checks so we can hopefully merge this?

pouyan021 · 2024-09-19T13:14:16Z

cc @wagoodman

spiffcs previously approved these changes Apr 22, 2024

View reviewed changes

README.md Show resolved Hide resolved

pouyan021 force-pushed the feat/allow-configuring-timeout-for-external-sources branch 2 times, most recently from 5cc6f4b to e5ed88d Compare April 22, 2024 17:46

pouyan021 requested a review from spiffcs April 25, 2024 06:19

spiffcs reviewed Apr 25, 2024

View reviewed changes

spiffcs self-requested a review April 25, 2024 18:34

wagoodman changed the title ~~Feat/allow configuring timeout for external sources~~ Allow configuring timeout for external sources May 16, 2024

wagoodman reviewed May 16, 2024

View reviewed changes

pouyan021 and others added 7 commits May 16, 2024 11:27

adding abort-after property to external-sources

4cbb4f0

Signed-off-by: Pouyan Khodabakhsh <pouyan021@gmail.com>

considering the timeout in java matcher

b8fe080

Signed-off-by: Pouyan Khodabakhsh <pouyan021@gmail.com>

make the maven search method respect the context time out

49ae4c9

Signed-off-by: Pouyan Khodabakhsh <pouyan021@gmail.com>

adding a number of additional tests for the external-sources time-out…

7043535

… functionality Signed-off-by: Pouyan Khodabakhsh <pouyan021@gmail.com>

facilitating java matcher mocks to imitate the external-sources abort…

db426d2

…-after functionality Signed-off-by: Pouyan Khodabakhsh <pouyan021@gmail.com>

updating readme to reflect the new changes in external-sources

1e947b6

Signed-off-by: Pouyan Khodabakhsh <pouyan021@gmail.com>

adjust multi level configuration + faster tests

0bf64dd

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

wagoodman force-pushed the feat/allow-configuring-timeout-for-external-sources branch from ca5d50c to 0bf64dd Compare May 16, 2024 15:29

Merge branch 'main' into feat/allow-configuring-timeout-for-external-…

2c79d3a

…sources

dependabot bot and others added 4 commits September 17, 2024 11:15

chore(deps): update tools to latest versions (anchore#1864)

c50a72a

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: westonsteimel <1593939+westonsteimel@users.noreply.github.com>

anchore-actions-token-generator bot and others added 28 commits September 17, 2024 11:16

chore(deps): update tools to latest versions (anchore#1883)

991513f

chore(deps): update tools to latest versions (anchore#1891)

48fc783

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: westonsteimel <1593939+westonsteimel@users.noreply.github.com>

update syft to v1.5.0 (anchore#1897)

b3d1bcf

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

chore(deps): update tools to latest versions (anchore#1898)

20f16c4

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: westonsteimel <1593939+westonsteimel@users.noreply.github.com>

fix: main mod pseudo version default off (anchore#1894)

0a2fdb4

Signed-off-by: Dan Luhring <dluhring@chainguard.dev>

fix: uppercased package in json (anchore#1900)

f043899

Signed-off-by: Keith Zantow <kzantow@gmail.com>

fix: add note about TMPDIR env var (anchore#1880)

c9ec282

Signed-off-by: Avtar Gill <avtargill@gmail.com>

use dco tool during gh app outage (anchore#1910)

5d1324f

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

remove dco workflow (anchore#1914)

22ec108

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

add skopeo to managed utilities (anchore#1915)

92a889f

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

chore(deps): update tools to latest versions (anchore#1921)

e561e7e

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: westonsteimel <1593939+westonsteimel@users.noreply.github.com>

chore(deps): update tools to latest versions (anchore#1925)

3433baf

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: westonsteimel <1593939+westonsteimel@users.noreply.github.com>

sort order for matches should consider fix info (anchore#1933)

8d9e74e

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

Updating maven URLs in README.md (anchore#1934)

ccd8c3e

Merge remote-tracking branch 'origin/feat/allow-configuring-timeout-f…

75a337c

…or-external-sources' into feat/allow-configuring-timeout-for-external-sources

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow configuring timeout for external sources #1812

Allow configuring timeout for external sources #1812

pouyan021 commented Apr 18, 2024 •

edited

Loading

spiffcs commented Apr 22, 2024

spiffcs commented Apr 22, 2024

pouyan021 commented Apr 22, 2024 •

edited

Loading

spiffcs left a comment

spiffcs Apr 25, 2024

wagoodman left a comment

wagoodman May 16, 2024

wagoodman commented May 16, 2024

pouyan021 commented May 17, 2024 •

edited

Loading

pouyan021 commented May 24, 2024

pouyan021 commented Jun 13, 2024

spiffcs commented Jul 2, 2024 •

edited

Loading

pouyan021 commented Sep 17, 2024

pouyan021 commented Sep 19, 2024

		base-url: https://search.maven.org/solrsearch/select
		abort-after: 5m #override the global config

Allow configuring timeout for external sources #1812

Are you sure you want to change the base?

Allow configuring timeout for external sources #1812

Conversation

pouyan021 commented Apr 18, 2024 • edited Loading

spiffcs commented Apr 22, 2024

spiffcs commented Apr 22, 2024

pouyan021 commented Apr 22, 2024 • edited Loading

spiffcs left a comment

Choose a reason for hiding this comment

spiffcs Apr 25, 2024

Choose a reason for hiding this comment

wagoodman left a comment

Choose a reason for hiding this comment

wagoodman May 16, 2024

Choose a reason for hiding this comment

wagoodman commented May 16, 2024

pouyan021 commented May 17, 2024 • edited Loading

pouyan021 commented May 24, 2024

pouyan021 commented Jun 13, 2024

spiffcs commented Jul 2, 2024 • edited Loading

pouyan021 commented Sep 17, 2024

pouyan021 commented Sep 19, 2024

pouyan021 commented Apr 18, 2024 •

edited

Loading

pouyan021 commented Apr 22, 2024 •

edited

Loading

pouyan021 commented May 17, 2024 •

edited

Loading

spiffcs commented Jul 2, 2024 •

edited

Loading