[MNG-7758] Report dependency problems for all repository #1563

slawekjaranowski · 2024-06-05T21:00:50Z

The build result will be as:

[ERROR] Failed to execute goal on project test-artifact: Could not resolve dependencies for project test-group:test-artifact:jar:1
[ERROR] dependency: org.apache.test:test1:jar:1.0 (compile)
[ERROR] 	Could not find artifact org.apache.test:test1:jar:1.0 in central (https://repo.maven.apache.org/maven2)
[ERROR] 	Could not find artifact org.apache.test:test1:jar:1.0 in repo.apache-stage (https://repository.apache.org/content/groups/staging/)
[ERROR] dependency: org.apache.test:test2:jar:1.0 (compile)
[ERROR] 	Could not find artifact org.apache.test:test2:jar:1.0 in central (https://repo.maven.apache.org/maven2)
[ERROR] 	Could not find artifact org.apache.test:test2:jar:1.0 in repo.apache-stage (https://repository.apache.org/content/groups/staging/)

[ERROR] Failed to execute goal on project test-artifact: Could not resolve dependencies for project test-group:test-artifact:jar:1
[ERROR] dependency: org.apache.test:test1:jar:1.0 (compile)
[ERROR] 	Prefix org/apache/test NOT allowed from mirror (https://artifactory.office.../artifactory/remote-repos/, default, releases)
[ERROR] 	Could not find artifact org.apache.test:test1:jar:1.0 in payu (https://artifactory.office.../artifactory/group-payu/)
[ERROR] dependency: org.apache.test:test2:jar:1.0 (compile)
[ERROR] 	Prefix org/apache/test NOT allowed from mirror (https://artifactory.office.../artifactory/remote-repos/, default, releases)
[ERROR] 	Could not find artifact org.apache.test:test2:jar:1.0 in payu (https://artifactory.office..../artifactory/group-payu/)

michael-o · 2024-06-05T21:12:36Z

How does it look now?

slawekjaranowski · 2024-06-05T21:36:29Z

Now looks:

[ERROR] Failed to execute goal on project test-artifact: Could not resolve dependencies for project test-group:test-artifact:jar:1: The following artifacts could not be resolved: org.apache.test:test1:jar:1.0 (absent), org.apache.test:test2:jar:1.0 (absent): Could not find artifact org.apache.test:test1:jar:1.0 in repo.apache-stage (https://repository.apache.org/content/groups/staging/) -> [Help 1]

[ERROR] Failed to execute goal on project test-artifact: Could not resolve dependencies for project test-group:test-artifact:jar:1: The following artifacts could not be resolved: org.apache.test:test1:jar:1.0 (absent), org.apache.test:test2:jar:1.0 (absent): Could not find artifact org.apache.test:test1:jar:1.0 in payu (https://artifactory.office..../artifactory/group-payu/) -> [Help 1]

only problem with last repository is reported - everything in one line ...

slawekjaranowski · 2024-06-05T21:38:47Z

IT MavenITmng3477DependencyResolutionErrorMessageTest to fix

slawekjaranowski · 2024-06-06T06:15:24Z

I also would like to cherry pick it to 3.9.x

michael-o · 2024-06-06T07:37:11Z

maven-core/src/main/java/org/apache/maven/project/DefaultProjectDependenciesResolver.java

-                    result, "Could not resolve dependencies for project " + project.getId() + ": " + e.getMessage(), e);
+                    result,
+                    "Could not collect dependencies for project " + project.getId(),
+                    logger.isDebugEnabled() ? e : null);


Why omitting the exception?

When we have exception in stack all message from all exception will be added to output in normal run we will have some information duplicate now

But isn't that a problem of the catcher, not the thrower?

If we have more detailed info at debug level, especially in case of an error, it may make sense to indicate so in the non detailed error message ?

detailed contains more info which is not available in o.e.aether.resolution.ArtifactResolutionException
ArtifactResolutionException.getMessage returns error for last repository

Without it some of message will be duplicated

without passing exception (current proposition) we will have:

[ERROR] Failed to execute goal on project test: Could not resolve dependencies for project org.apache.maven.its.mng3477:test:jar:1 [ERROR] dependency: org.apache.maven.its.mng3477:dep:jar:1.0 (compile) [ERROR] Could not find artifact org.apache.maven.its.mng3477:dep:jar:1.0 in maven-core-it (http://localhost:64468/repo) [ERROR] Could not find artifact org.apache.maven.its.mng3477:dep:jar:1.0 in central (http://localhost:64468/repo) [ERROR] -> [Help 1]

with exception will be as:

[ERROR] Failed to execute goal on project test: Could not resolve dependencies for project org.apache.maven.its.mng3477:test:jar:1 [ERROR] dependency: org.apache.maven.its.mng3477:dep:jar:1.0 (compile) [ERROR] Could not find artifact org.apache.maven.its.mng3477:dep:jar:1.0 in central (http://localhost:64398/repo) [ERROR] Could not find artifact org.apache.maven.its.mng3477:dep:jar:1.0 in maven-core-it (http://localhost:64398/repo): The following artifacts could not be resolved: org.apache.maven.its.mng3477:dep:jar:1.0 (absent): Could not find artifact org.apache.maven.its.mng3477:dep:jar:1.0 in central (http://localhost:64398/repo) [ERROR] -> [Help 1]

so it is a reason

I hope that simply message will be more readable for user, if user need more details for debug or investigation can execute in verbose/debug mode

VERY UGLY. I guess this needs a long term solution.

we can think about it ... how to reporting a problems to user but it is another issue for me, now I did as simple as possible
we also have many other UGLY code ... 😸

maven-core/src/main/java/org/apache/maven/project/DependencyResolutionException.java

michael-o · 2024-06-06T11:29:54Z

Give me a day or two, I try to try this out in a realworld env.

slawekjaranowski · 2024-06-06T14:40:31Z

Give me a day or two, I try to try this out in a realworld env.

I would be happy to have it in 3.9.x also

So I fix ITs first 😄

slawekjaranowski · 2024-06-06T17:35:44Z

Fixed IT - apache/maven-integration-testing#346

slawekjaranowski · 2024-06-09T13:11:49Z

Give me a day or two, I try to try this out in a realworld env.

@michael-o any progress on it

michael-o · 2024-06-09T13:17:59Z

Give me a day or two, I try to try this out in a realworld env.

@michael-o any progress on it

No, I can do that only at work were I have the setup. Will happen tomorrow. Maven Release kept me busy. I will update you.

michael-o · 2024-06-10T06:46:47Z

Trying...

michael-o · 2024-06-10T07:21:43Z

I have done some further testing. Good news: DOXIASITETOOLS-303 seems to be solved by this. Bad news: It does not work the way advertised. I have in my settings.xml:

central-mirror
lda-public (serves only com.example, rest is blocked with 403)
dynamowerk-public

tried:

osipovmi@deblndw011x:~/var/Projekte/maven-site (master *=)
$ git diff -U0
diff --git a/pom.xml b/pom.xml
index 66e7e5bc..556fb363 100644
--- a/pom.xml
+++ b/pom.xml
@@ -77 +77 @@
-    <fluidoVersion>2.0.0-M6</fluidoVersion>
+    <fluidoVersion>1.12.0</fluidoVersion>

ran:

osipovmi@deblndw011x:~/var/Projekte/maven-site (master *=)
$ JAVA_VERSION=17 /tmp/apache-maven-4.0.0-beta-4-SNAPSHOT/bin/mvn clean site -Dmaven.repo.local=/var/tmp/repo-site2
...
[INFO] --------------------------------------------------------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] --------------------------------------------------------------------------------------------------------------------------
[INFO] Total time:  1.580 s
[INFO] Finished at: 2024-06-10T09:18:14+02:00
[INFO] --------------------------------------------------------------------------------------------------------------------------
[ERROR] Failed to execute goal on project maven-site: Could not collect dependencies for project org.apache.maven:maven-site:jar:1.0
[ERROR] Failed to read artifact descriptor for org.jsoup:jsoup:jar:1.17.3
[ERROR]         Caused by: The following artifacts could not be resolved: org.jsoup:jsoup:pom:1.17.3 (absent): Could not transfer artifact org.jsoup:jsoup:pom:1.17.3 from/to lda-public (https://.../lda-public/): HTTP Status: 403
[ERROR]

Is it what you expect? I would expect to try all four repos.

It is trying:

Downloading from lda-public: https://.../nexus/content/groups/lda-public/org/jsoup/jsoup/1.17.3/jsoup-1.17.3.pom
Downloading from dynamowerk-public: https://.../nexus/content/groups/dynamowerk-public/org/jsoup/jsoup/1.17.3/jsoup-1.17.3.pom
Downloading from apache.staging: https://repository.apache.org/content/repositories/maven-1906/org/jsoup/jsoup/1.17.3/jsoup-1.17.3.pom
Downloading from central-mirror: https://.../nexus/content/repositories/central/org/jsoup/jsoup/1.17.3/jsoup-1.17.3.pom

so does jsoup-1.17.3.pom.lastUpdated:

#NOTE: This is a Maven Resolver internal implementation file, its format can be changed without prior notice.
#Mon Jun 10 09:19:48 CEST 2024
https\://.../nexus/content/groups/dynamowerk-public/.lastUpdated=1718003988205
https\://.../nexus/content/repositories/central/.error=
https\://repository.apache.org/content/repositories/maven-1906/.error=
https\://.../nexus/content/groups/lda-public/.error=Could not transfer artifact org.jsoup\:jsoup\:pom\:1.17.3 from/to lda-public (https\://.../nexus/content/gr
oups/lda-public/)\: HTTP Status\: 403
https\://.../nexus/content/groups/dynamowerk-public/.error=
https\://repository.apache.org/content/repositories/maven-1906/.lastUpdated=1718003988246
02b43e8beafaf18c207565d70bf0dc68eea2ac76@default-lda-public-https\://.../nexus/content/groups/lda-public/.lastUpdated=1718003988184
https\://.../nexus/content/repositories/central/.lastUpdated=1718003988342

Thoughts?

The issue can be easily recreated by setting up web sever denying all requests with 403.

cstamas · 2024-06-10T18:05:22Z

Resolver handles differently 403, as it assumes "user error" in those cases (and user would fix/update access credentials, or remove repo).

cstamas · 2024-06-10T18:06:08Z

See https://github.com/apache/maven-resolver/blob/master/maven-resolver-spi/src/main/java/org/eclipse/aether/spi/connector/transport/Transporter.java#L37-L51

Resolver "knows" 404 and "everything else" (like 403)

michael-o · 2024-06-10T18:08:52Z

See https://github.com/apache/maven-resolver/blob/master/maven-resolver-spi/src/main/java/org/eclipse/aether/spi/connector/transport/Transporter.java#L37-L51

Resolver "knows" 404 and "everything else" (like 403)

403 is not an authentication problem, but solely authorization: https://www.rfc-editor.org/rfc/rfc9110#status.403

Unfornately, I cannot configure Nexus to send 404 in this case which would be better for Nexus.

slawekjaranowski · 2024-06-10T19:55:00Z

We have a different behavior for: DependencyCollectionException and
for DependencyResolutionException

DependencyResolutionException has an list errors for all repositories but DependencyCollectionException has only one exception on list

slawekjaranowski · 2024-06-10T19:55:43Z

Please look at apache/maven-integration-testing#348

slawekjaranowski · 2024-06-10T20:00:36Z

For me it is not a big problem, like connection exception is not common, more important is to have a list of not found artifact per repository

slawekjaranowski · 2024-06-10T21:46:36Z

and similar for plugins:

[ERROR] Plugin org.apache.maven.its.plugins:maven-it-plugin-not-exists:1.2.3 or one of its dependencies could not be resolved:
[ERROR] 	Could not find artifact org.apache.maven.its.plugins:maven-it-plugin-not-exists:jar:1.2.3 in central (http://localhost:64391/repo)
[ERROR] 	Could not find artifact org.apache.maven.its.plugins:maven-it-plugin-not-exists:jar:1.2.3 in maven-core-it (http://localhost:64391/repo)

michael-o · 2024-06-11T10:22:48Z

@slawekjaranowski Should I try again?

slawekjaranowski · 2024-06-11T10:29:31Z

@slawekjaranowski Should I try again?

you can - but result for connection problems will still show last / one repo
I also changed reporting for missing plugins

https://issues.apache.org/jira/browse/MRESOLVER-569

michael-o · 2024-06-11T10:32:11Z

@slawekjaranowski Should I try again?

you can - but result for connection problems will still show last / one repo I also changed reporting for missing plugins

https://issues.apache.org/jira/browse/MRESOLVER-569

@cstamas Is this a huge issue to fix in Resolver?

cstamas · 2024-06-11T10:38:52Z

@cstamas Is this a huge issue to fix in Resolver?

Unsure, did not look yet, but IMHO is certainly something that would NOT happen in 1.9.x resolver but 2.0.0...

slawekjaranowski · 2024-06-11T11:30:01Z

@michael-o is it any objections from you to merge without resolver fix ...
resolver can be fixed later

michael-o · 2024-06-11T11:49:13Z

@michael-o is it any objections from you to merge without resolver fix ... resolver can be fixed later

Let me please review/test again today.

cstamas · 2024-06-11T12:16:02Z

As I looked into, this is a problem that probably looks like this:
Collector (CO, in resolver) -> ArtifactDescriptorReader (ADR, in maven) -> ArtifactResolver (AR, in resolver)

What happens, is that CO during collecting, issues series of requests to ADR to build the graph, but ADR "forwards" these to AR (to resolve POM), and it fails.

If you "rollback", you will see that AR throws ArtifactResolutionException that carries a list (repo, cause). So far all nice and dandy. But ADR loses them. Instead, it will create ArtifactDescriptorException w/o cause (so AR thrown ArtifactResolutionEx is lost), instead it uses result, and gets 1st exception from result as "cause". Finally, CO receives this exception and sets it as one single CollectionResult ex and throws DependencyCollectionException w/ result.

The problem stems from here:
https://github.com/apache/maven-resolver/blob/master/maven-resolver-api/src/main/java/org/eclipse/aether/resolution/ArtifactDescriptorException.java#L35

As ADR wants to reuse this message, but to do so, it simply leaves out cause (ArtifactResolutionEx), instead it does this: https://github.com/apache/maven-resolver/blob/master/maven-resolver-api/src/main/java/org/eclipse/aether/resolution/ArtifactDescriptorException.java#L76

cstamas · 2024-06-11T12:26:18Z

Went thru resolver cobase, and we have 5 copies of this method:

    private static Throwable getCause(SomeResultType result) {
        Throwable cause = null;
        if (result != null && !result.getExceptions().isEmpty()) {
            cause = result.getExceptions().get(0);
        }
        return cause;
    }

And these methods are bad, as they blindly choose 1st exception as "cause" in various scenarios. They are in classes:

VersionResolutionException
DependencyCollectionException
ArtifactDescriptorException
VersionRangeResolutionException
(fifth is in ArtifactResolutionException but it is modified, is not copy-pasta of this)

cstamas · 2024-06-11T12:38:40Z

My proposal: let's split the issues, have two: one for Maven 3.9 and one for master. Let's merge whatever improvement we have for 3.9 to make 3.9.8 rollin' and see about master branches of Resolver and Maven later down the road.

slawekjaranowski · 2024-06-11T14:35:21Z

My proposal: let's split the issues, have two: one for Maven 3.9 and one for master. Let's merge whatever improvement we have for 3.9 to make 3.9.8 rollin' and see about master branches of Resolver and Maven later down the road.

if we want it in 3.9 it is not necessary to split - after merge to master I simply cherry-pick it to 3.9.x branch
when we have two PR we will have the same discussion twice

michael-o

Given the issues in Resolver and good spirit in @cstamas knowning how to fix, I am fine with the change when my open question is resolved.

michael-o · 2024-06-12T07:42:46Z

I am fine with the merge.

slawekjaranowski requested review from cstamas, gnodet and michael-o June 5, 2024 21:01

slawekjaranowski marked this pull request as draft June 5, 2024 21:38

michael-o reviewed Jun 6, 2024

View reviewed changes

maven-core/src/main/java/org/apache/maven/project/DependencyResolutionException.java Show resolved Hide resolved

cstamas approved these changes Jun 6, 2024

View reviewed changes

slawekjaranowski force-pushed the MNG-7758-next-try branch from 26695ff to ea7309b Compare June 6, 2024 17:36

slawekjaranowski added this to the 4.0.0-beta-4 milestone Jun 6, 2024

slawekjaranowski self-assigned this Jun 6, 2024

slawekjaranowski marked this pull request as ready for review June 6, 2024 18:59

slawekjaranowski requested a review from michael-o June 6, 2024 19:28

slawekjaranowski force-pushed the MNG-7758-next-try branch from ea7309b to ed7db20 Compare June 10, 2024 19:42

slawekjaranowski force-pushed the MNG-7758-next-try branch from ed7db20 to 0ff0d74 Compare June 10, 2024 21:45

slawekjaranowski requested a review from cstamas June 11, 2024 05:56

slawekjaranowski and others added 2 commits June 11, 2024 10:16

[MNG-7758] Report dependency problems for all repository

8483603

[MNG-7758] Report dependency problems for all repository - plugins

ca4c13a

slawekjaranowski force-pushed the MNG-7758-next-try branch from 0ff0d74 to ca4c13a Compare June 11, 2024 08:16

cstamas approved these changes Jun 11, 2024

View reviewed changes

michael-o approved these changes Jun 12, 2024

View reviewed changes

slawekjaranowski merged commit 768ebbc into master Jun 12, 2024
28 checks passed

slawekjaranowski deleted the MNG-7758-next-try branch June 12, 2024 07:49

gnodet modified the milestones: 4.0.0-beta-4, 4.0.0 Jul 6, 2024

This pull request was closed.

[MNG-7758] Report dependency problems for all repository #1563

[MNG-7758] Report dependency problems for all repository #1563

Conversation

slawekjaranowski commented Jun 5, 2024

michael-o commented Jun 5, 2024

slawekjaranowski commented Jun 5, 2024

slawekjaranowski commented Jun 5, 2024

slawekjaranowski commented Jun 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michael-o commented Jun 6, 2024

slawekjaranowski commented Jun 6, 2024

slawekjaranowski commented Jun 6, 2024

slawekjaranowski commented Jun 9, 2024

michael-o commented Jun 9, 2024

michael-o commented Jun 10, 2024

michael-o commented Jun 10, 2024

cstamas commented Jun 10, 2024

cstamas commented Jun 10, 2024

michael-o commented Jun 10, 2024

slawekjaranowski commented Jun 10, 2024

slawekjaranowski commented Jun 10, 2024

slawekjaranowski commented Jun 10, 2024

slawekjaranowski commented Jun 10, 2024

michael-o commented Jun 11, 2024

slawekjaranowski commented Jun 11, 2024

michael-o commented Jun 11, 2024

cstamas commented Jun 11, 2024

slawekjaranowski commented Jun 11, 2024

michael-o commented Jun 11, 2024

cstamas commented Jun 11, 2024

cstamas commented Jun 11, 2024 • edited Loading

cstamas commented Jun 11, 2024

slawekjaranowski commented Jun 11, 2024

michael-o left a comment

Choose a reason for hiding this comment

michael-o commented Jun 12, 2024

cstamas commented Jun 11, 2024 •

edited

Loading