spotlessXmlApply task is never completing #308

JLLeitschuh · 2018-10-05T17:26:50Z

Running ./gradlew spotlessXmlApply the :spotlessXml task is never completing.

spotless {
    xml {
        eclipse()
    }
}

This is a little bit of the output running with the gradle --info flag:

All projects evaluated.
Selected primary task ':spotlessXmlApply' from project :
Tasks to be executed: [task ':spotlessXml', task ':spotlessXmlApply']
:spotlessXml (Thread[Task worker for ':',5,main]) started.

> Task :spotlessXml
Caching disabled for task ':spotlessXml': Caching has not been enabled for the task
Task ':spotlessXml' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task ':spotlessXml'.
<-------------> 0% EXECUTING [1m 17s]
> :spotlessXml

After this, the process gets hung up.

Spotless version: 3.15.0.

------------------------------------------------------------
Gradle 4.10.1
------------------------------------------------------------

Build time:   2018-09-12 11:33:27 UTC
Revision:     76c9179ea9bddc32810f9125ad97c3315c544919

Kotlin DSL:   1.0-rc-6
Kotlin:       1.2.61
Groovy:       2.4.15
Ant:          Apache Ant(TM) version 1.9.11 compiled on March 23 2018
JVM:          1.8.0_92 (Oracle Corporation 25.92-b14)
OS:           Mac OS X 10.13.6 x86_64

The text was updated successfully, but these errors were encountered:

fvgh · 2018-10-05T20:00:01Z

@JLLeitschuh Do you have a little bit more input. I applied the formatter in spotlessSelf.gradle and also in a small sample.

JLLeitschuh · 2018-10-08T17:20:04Z

This might be caused by spotless trying to format a file that is opened by another process? Is there any way I could tell that?

fvgh · 2018-10-08T18:22:03Z

I am only aware that Windows uses mandatory file locking. What OS do you use?

JLLeitschuh · 2018-10-08T19:48:21Z

It's listed in the original issue, but anyways: OS: Mac OS X 10.13.6 x86_64.

fvgh · 2018-10-08T19:58:24Z

Ah, sorry, have no web at the moment. OS X behaves to my understanding like any other UNIX. No mandatory file locking. Has the download from Maven Central been completed? Can you check the open TCP connections? Unfortunately I have not finished the logging implementation, so there is not much feedback from the Eclipse formatters yet.

fvgh · 2018-10-09T08:00:57Z

Just realized that you can see the download of the Eclipse dependencies also in the Gradle debug output. If that is also okay, I can only think of the DTD/XSD resolver. You can check by removing the references in your XML files.

JLLeitschuh · 2018-10-09T16:04:22Z

Why would the formatter/linter be downloading the XSD or DTD's? I want my formatter to format, not ensure that the XML is valid.

fvgh · 2018-10-09T17:11:28Z

XSD and DTD can contain instructions how to treat whitespace. Eclipse just ignores (got a unit test) missing XSD and DTD. However, theoretically there can be network setups where a timeout takes a while. I also once had an issue where the XSD was badly written (generated) and was in total so big, that the parser required more memory.

fvgh · 2018-10-20T10:59:22Z

@JLLeitschuh I was quite busy the last two weeks. With my small sample, I was not able to reproduce the problem, but I found that the debug output from gradle is quite good. In my example I get the following debug output:

> Task :spotlessXml 
12:48:54.667 [DEBUG] [sun.net.www.protocol.http.HttpURLConnection] sun.net.www.MessageHeader@58202e3d5 pairs: {GET http://java.sun.com/xml/ns/javaee/javaee_5.xsd HTTP/1.1: null}{User-Agent: Java/1.8.0_181}{Host: java.sun.com}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Proxy-Connection: keep-alive}
12:48:55.012 [DEBUG] [sun.net.www.protocol.http.HttpURLConnection] sun.net.www.MessageHeader@7ed3c1b89 pairs: {null: HTTP/1.1 301 Moved Permanently}{Location: http://www.oracle.com/webfolder/technetwork/jsc/xml/ns/javaee/javaee_5.xsd}{Server: BigIP}{Content-Length: 0}{Date: Sat, 20 Oct 2018 10:48:55 GMT}{X-Cache: MISS from ###)}{Connection: keep-alive}
12:48:55.013 [DEBUG] [sun.net.www.protocol.http.HttpURLConnection] Redirected from http://java.sun.com/xml/ns/javaee/javaee_5.xsd to http://www.oracle.com/webfolder/technetwork/jsc/xml/ns/javaee/javaee_5.xsd
12:48:55.014 [DEBUG] [sun.net.www.protocol.http.HttpURLConnection] sun.net.www.MessageHeader@58202e3d5 pairs: {GET http://www.oracle.com/webfolder/technetwork/jsc/xml/ns/javaee/javaee_5.xsd HTTP/1.1: null}{User-Agent: Java/1.8.0_181}{Host: www.oracle.com}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Proxy-Connection: keep-alive}
...

Can you provide your output? Maybe it gives me a better idea, where the problem might be...

JLLeitschuh · 2018-10-20T14:48:28Z

I'll see what I can do on monday, sure.

DJViking · 2019-02-13T10:58:25Z

I experienced the same problem using spotless-gradle-plugin 3.15. It would not do any formatting.

spotless {
    xml {
        indentWithSpaces(2)
    }
}

Moving to a newer spotless 3.18 that now has changed the XML configuration to eclipseWtp I now can format my XML again.

spotless {
    format 'xml', {
        target fileTree('.') {
            include '**/*.xml', '**/*.xsd'
            exclude '**/build/**'
        }

        eclipseWtp('xml').configFile 'config/spotless/spotless.xml.prefs'
    }
}

Sadly though it does more than I want, when using the eclipse configuration. In comparison with xmllint, it breaks lines after the lineWidth, and it formats comment sections making them "ugly" and unreadable.

The lineWidth problem I can circumvent by setting it to a high value 500.
However I have not yet found a way for it to avoid formatting comments.

fvgh · 2019-02-13T17:00:47Z

@DJViking I am surprised to hear that the formatting did not work with spotless-gradle-plugin 3.15. As stated before, I was never able to reproduce the problem. For spotless-gradle-plugin 3.18 nothing has changed on the XML processor, except the scripting syntax. But the old syntax is still supported. Both syntax using internally the same functions. I would really appreciate some trace logs from gradle demonstrating the problem.

About the "ugly and unreadable" formatting:
I am afraid, I cannot do anything about it. I would recommend that you use Eclipse to find configuration settings that suites you. Anyhow, you can disable comment formatting by:
formatCommentText=false

The properties you configure in the Eclipse GUI are stored in
.metadata/.plugins/org.eclipse.core.runtime/.settings/org.eclipse.wst.xml.core.prefs

JLLeitschuh · 2019-02-13T17:50:04Z

12:48:55.013 [DEBUG] [sun.net.www.protocol.http.HttpURLConnection] Redirected from http://java.sun.com/xml/ns/javaee/javaee_5.xsd to http://www.oracle.com/webfolder/technetwork/jsc/xml/ns/javaee/javaee_5.xsd

Hold on. @fvgh. If you're seeing HTTP get requests inside of your XML parser that means that the parser is vulnerable to XXE.

We need to fix this so that the spotless XML formatter is not making external entity requests.

We can't have our linting infrastructure making web requests. Especially web requests over HTTP as those can be maliciously intercepted by a MITM.

Here's an example where this has been a serious problem in the past.

https://research.checkpoint.com/parsedroid-targeting-android-development-research-community/

CC: @nedtwigg

DJViking · 2019-02-13T19:03:05Z

About the "ugly and unreadable" formatting:
I am afraid, I cannot do anything about it. I would recommend that you use Eclipse to find configuration settings that suites you. Anyhow, you can disable comment formatting by:
formatCommentText=false

Thanks for the heads up. It was there all along in Eclipse, staring me right in the face, but I did not see it.

fvgh · 2019-02-13T19:56:41Z

@JLLeitschuh I admit that the WTP should respect the resolveExternalEntities configuration, which is per default false. I will investigate.

Anyhow, please open a new issue for this topic. To be hones, it is not urgent for me. Every formatter can contain holes in its parsing step. With external URIs it is easier to get something into the parser unnoticed, that is true.

nedtwigg · 2019-06-25T04:56:43Z

Can we close this one @JLLeitschuh?

JLLeitschuh · 2019-06-26T19:36:25Z

I no longer have access to the repository that had this problem.

New job. I now work for Gradle! 😄

nedtwigg · 2019-06-26T20:13:48Z

Congrats! Gradle is very lucky, and so are the rest of us! Since this issue is unlikely to see more action, I'll close it for now.

nedtwigg added the bug label Oct 16, 2018

JLLeitschuh mentioned this issue Feb 13, 2019

CVE-2019-9843: The XML parser isn't respecting resolveExternalEntities as false #358

Closed

nedtwigg closed this as completed Jun 26, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

spotlessXmlApply task is never completing #308

spotlessXmlApply task is never completing #308

JLLeitschuh commented Oct 5, 2018

fvgh commented Oct 5, 2018

JLLeitschuh commented Oct 8, 2018

fvgh commented Oct 8, 2018 via email

JLLeitschuh commented Oct 8, 2018

fvgh commented Oct 8, 2018 via email

fvgh commented Oct 9, 2018 via email

JLLeitschuh commented Oct 9, 2018

fvgh commented Oct 9, 2018 via email

fvgh commented Oct 20, 2018 •

edited

Loading

JLLeitschuh commented Oct 20, 2018

DJViking commented Feb 13, 2019

fvgh commented Feb 13, 2019

JLLeitschuh commented Feb 13, 2019 •

edited

Loading

DJViking commented Feb 13, 2019 •

edited

Loading

fvgh commented Feb 13, 2019

nedtwigg commented Jun 25, 2019

JLLeitschuh commented Jun 26, 2019

nedtwigg commented Jun 26, 2019

spotlessXmlApply task is never completing #308

spotlessXmlApply task is never completing #308

Comments

JLLeitschuh commented Oct 5, 2018

fvgh commented Oct 5, 2018

JLLeitschuh commented Oct 8, 2018

fvgh commented Oct 8, 2018 via email

JLLeitschuh commented Oct 8, 2018

fvgh commented Oct 8, 2018 via email

fvgh commented Oct 9, 2018 via email

JLLeitschuh commented Oct 9, 2018

fvgh commented Oct 9, 2018 via email

fvgh commented Oct 20, 2018 • edited Loading

JLLeitschuh commented Oct 20, 2018

DJViking commented Feb 13, 2019

fvgh commented Feb 13, 2019

JLLeitschuh commented Feb 13, 2019 • edited Loading

DJViking commented Feb 13, 2019 • edited Loading

fvgh commented Feb 13, 2019

nedtwigg commented Jun 25, 2019

JLLeitschuh commented Jun 26, 2019

nedtwigg commented Jun 26, 2019

fvgh commented Oct 20, 2018 •

edited

Loading

JLLeitschuh commented Feb 13, 2019 •

edited

Loading

DJViking commented Feb 13, 2019 •

edited

Loading