Prioritize snapshot builds in distribution downloader

[peternied]

Description

In the Security plugin's BWC tests there have been an issue with stale artifacts, when troubleshooting we noticed that a release candidate for OpenSearch 3.0.0 was being picked up in priority before the latest snapshot build. This was causing build failures preventing our tests from running.

Related Issues

• Related bwcTests failing with Caused by: java.lang.ClassNotFoundException: org.opensearch.core.common.settings.SecureString security#3020

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[peternied]

@cwperks I know you are working on the source of these failures / stale artifacts, this might resolve some of those issues.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/20311/
• CommitID: d0d3f33

[codecov]

Codecov Report

Merging #8731 (d0d3f33) into main (688c53f) will decrease coverage by 0.21%.
The diff coverage is 50.00%.

@@             Coverage Diff              @@
##               main    #8731      +/-   ##
============================================
- Coverage     71.02%   70.81%   -0.21%     
+ Complexity    57229    57140      -89     
============================================
  Files          4771     4771              
  Lines        270323   270323              
  Branches      39509    39509              
============================================
- Hits         192007   191442     -565     
- Misses        62140    62773     +633     
+ Partials      16176    16108      -68     

Impacted Files	Coverage Δ
.../opensearch/gradle/DistributionDownloadPlugin.java	87.50% <50.00%> (ø)

... and 455 files with indirect coverage changes

[dblock]

Isn't that by design because a released version is better than the latest snapshot?

[peternied]

Isn't that by design because a released version is better than the latest snapshot?

@dblock I don't think its so easily clarified, we've got 'release versions' of 3.0.0 already according to the logic of this script. I'd rather have an snapshot of a patch release than the latest official release, as that likely has resolutions to bugs / CVEs in it.

Is there use in letting the consumer decide between releases, latest minor, latest patched, and snapshots?

[saratvemulapalli]

@peternied do we have a release candidate for 3.0? There shouldn't be one. [1]
Distribution download plugin always downloads the latest version depending on version you specify. If it has a qualifier of SNAPSHOT in the version it would never download a release artifact.

Trying to understand the failures you were seeing, was it downloading an older snapshot of OpenSearch 3.0.0?
I guess your problem could have been that a latest 3.0.0-SNAPSHOT was not published, mostly due to failures on opensearch-build workflows.

Is there use in letting the consumer decide between releases, latest minor, latest patched, and snapshots?

SNAPSHOT qualifier will always download the latest in the repo. Same with releases, we dont have a feature that supports latest minor/patched as the consumers are anyway tightly coupled. We could definitely make this a feature request.

[1] https://aws.oss.sonatype.org/content/repositories/releases/org/opensearch/opensearch/

[peternied]

Trying to understand the failures you were seeing, was it downloading an older snapshot of OpenSearch 3.0.0?
I guess your problem could have been that a latest 3.0.0-SNAPSHOT was not published, mostly due to failures on opensearch-build workflows.

It looks like its unblocked, see [1]. This change doesn't seem worth iterating on based on the feedback - we will need to add more logging to help figure out what version we did get and from where.

• [1] [Build Break] Update imports for files refactored in core PR #8157 security#3003

[cwperks]

@saratvemulapalli Thank you for the context. The problem was that the bwc tests in the security plugin relied on release artifacts and needed to be updated to use SNAPSHOT artifacts so that the tests ran with the most recent changes from core.