Cannot use Wire within a spring boot executable jar #2209

martypitt · 2022-02-25T13:17:08Z

martypitt
Feb 25, 2022

Spring boot has the ability to create executable jars, which it does by rewriting the first few bytes of the jar file

At runtime, wire tries to load proto files from the classpath, treating the classpath as a zip file.

In CoreLoader:

  override fun load(path: String): ProtoFile {
    if (isWireRuntimeProto(path)) {
      FileSystem.RESOURCES.read("/".toPath() / path) {
        val data = readUtf8()
        val location = Location.get(path)
        val element = ProtoParser.parse(location, data)
        return ProtoFile.get(element)
      }
    }

    throw error("unexpected load: $path")
  }

This ultimately ends up in the following stack trace:

Caused by: java.io.IOException: not a zip: expected 0x4034b50 but was 0x622f2123
	at okio.internal.ZipKt.openZip(zip.kt:75) ~[okio-jvm-3.0.0.jar!/:na]
	at okio.internal.ResourceFileSystem$Companion.toJarRoot(ResourceFileSystem.kt:201) ~[okio-jvm-3.0.0.jar!/:na]
	at okio.internal.ResourceFileSystem$Companion.toClasspathRoots(ResourceFileSystem.kt:184) ~[okio-jvm-3.0.0.jar!/:na]
	at okio.internal.ResourceFileSystem$roots$2.invoke(ResourceFileSystem.kt:47) ~[okio-jvm-3.0.0.jar!/:na]
	at okio.internal.ResourceFileSystem$roots$2.invoke(ResourceFileSystem.kt:47) ~[okio-jvm-3.0.0.jar!/:na]
	at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:74) ~[kotlin-stdlib-1.4.30.jar!/:1.4.30-release-302 (1.4.30)]
	at okio.internal.ResourceFileSystem.getRoots(ResourceFileSystem.kt:47) ~[okio-jvm-3.0.0.jar!/:na]
	at okio.internal.ResourceFileSystem.source(ResourceFileSystem.kt:126) ~[okio-jvm-3.0.0.jar!/:na]
	at com.squareup.wire.schema.CoreLoader.load(CoreLoader.kt:79) ~[wire-schema-jvm-4.1.0.jar!/:na]

This code all works fine when running from IntelliJ, or in tests. It's not until runtime, within the booted jar that it becomes problematic.

This snippet from the spring docs suggest that this should still be possible, but I'm not sure how to change things to make it work.

Spring Boot Loader strives to remain compatible with existing code and libraries. org.springframework.boot.loader.jar.JarFile extends from java.util.jar.JarFile and should work as a drop-in replacement. The getURL() method returns a URL that opens a connection compatible with java.net.JarURLConnection and can be used with Java’s URLClassLoader.

I'm fairly stuck / blocked by this.

Does anyone have any suggestions on how to push this forward?

Answered by martypitt

Mar 1, 2022

FWIW, I ended up working around this by removing spring boot's maven plugin, and using a classic appassembler-maven-plugin and maven-assembly-plugin instead.

Not ideal, but works.

If anyone is interested in the final config, you can see it here

View full answer

JakeWharton · 2022-02-25T13:23:20Z

JakeWharton
Feb 25, 2022
Maintainer

This probably needs an Okio issue filed.

0 replies

oldergod · 2022-02-25T16:26:57Z

oldergod
Feb 25, 2022
Maintainer

I created an issue in Okio here square/okio#1094

0 replies

martypitt · 2022-02-25T18:08:26Z

martypitt
Feb 25, 2022
Author

Thanks for creating the issue. I agree that's probably the right place.

However, right now, I seem to be hitting brick walls with every workaround I try. Any advice here would be greatly apprecaited.

Here's what I've tried:

Inject a different implementation

I couldn't find a way to wire in a different implementation of the CoreLoader, or of the FileSystem.RESOURCES.

I noticed this comment in CoreLoader

 * If the user has provided their own version of these protos, those are preferred.

However, I can't find a way to provide my own implementation that will prevent CoreLoader calling FileSystem.RESOURCES.read(...)

Turn off Spring Boot executable jars, and export the dependencies explicitly

Turns out that the Spring Boot plugin will allow you to make specific jars explicitly available in the fat jar. This kinda worked, but the CoreLoader is iterating over every jar in the classpath, and throws NullPointer exceptions when the file isn't available.

(Again, this is more for the okio team, but listing it here for completeness).

Try to implement my own SchemaLoader / CoreLoader implementations

Since my entry point is to a SchemaLoader(), I figured I could just take the existing one, and modify it, to provide avoid it calling out to the FileSystem.RESOURCES.

However, there's quite a few internal scoped classes involved (Like Root or .FileSystem.toOkioFileSystem() ) that this quickly started feeling like I was rebuilding the whole library.

Monkey patching the reference to `FileSystem.RESOURCES`

Yep, I'll admit that this was a bad idea, but I gave it a go anyway, and failed :)

I feel like there has to be a way around this, but I can't seem to find it.
Any advice would be greatly appreciated!

0 replies

martypitt · 2022-03-01T14:21:11Z

martypitt
Mar 1, 2022
Author

FWIW, I ended up working around this by removing spring boot's maven plugin, and using a classic appassembler-maven-plugin and maven-assembly-plugin instead.

Not ideal, but works.

If anyone is interested in the final config, you can see it here

0 replies

gigi · 2022-04-18T19:43:16Z

gigi
Apr 18, 2022

I have issues deploying the Vaadin app using appassembler (

I see that something related has been fixed in 4.3.0

But still receive exception when running app as spring jar with the wire lib inside

 private Schema loadFromFilesystem(List<String> locations) throws IOException {
    SchemaLoader schemaLoader = new SchemaLoader(FileSystems.getDefault());

    List<Location> wireLocations =
        locations.stream().map(Location::get).collect(Collectors.toList());

    schemaLoader.initRoots(wireLocations, List.of());

    return schemaLoader.loadSchema();   // exception here
  }

java.io.FileNotFoundException: /app.jar!/BOOT-INF/lib/my-service-sdk-1.2.0.jar (No such file or directory)
        at java.base/java.io.RandomAccessFile.open0(Native Method) ~[na:na]
        at java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:344) ~[na:na]
        at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:259) ~[na:na]
        at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:213) ~[na:na]
        at okio.JvmSystemFileSystem.openReadOnly(JvmSystemFileSystem.kt:82) ~[okio-jvm-3.0.0.jar!/:na]
        at okio.internal.ZipKt.openZip(zip.kt:67) ~[okio-jvm-3.0.0.jar!/:na]
        at okio.internal.ResourceFileSystem$Companion.toJarRoot(ResourceFileSystem.kt:201) ~[okio-jvm-3.0.0.jar!/:na]
        at okio.internal.ResourceFileSystem$Companion.toClasspathRoots(ResourceFileSystem.kt:184) ~[okio-jvm-3.0.0.jar!/:na]
        at okio.internal.ResourceFileSystem$roots$2.invoke(ResourceFileSystem.kt:47) ~[okio-jvm-3.0.0.jar!/:na]
        at okio.internal.ResourceFileSystem$roots$2.invoke(ResourceFileSystem.kt:47) ~[okio-jvm-3.0.0.jar!/:na]
        at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:74) ~[kotlin-stdlib-1.6.10.jar!/:1.6.10-release-923(1.6.10)]
        at okio.internal.ResourceFileSystem.getRoots(ResourceFileSystem.kt:47) ~[okio-jvm-3.0.0.jar!/:na]
        at okio.internal.ResourceFileSystem.<init>(ResourceFileSystem.kt:51) ~[okio-jvm-3.0.0.jar!/:na]
        at okio.Okio__JvmOkioKt.asResourceFileSystem(JvmOkio.kt:233) ~[okio-jvm-3.0.0.jar!/:na]
        at okio.Okio.asResourceFileSystem(Unknown Source) ~[okio-jvm-3.0.0.jar!/:na]
        at com.squareup.wire.schema.CoreLoader$resourceFileSystem$2.invoke(CoreLoader.kt:46) ~[wire-schema-jvm-4.3.0.jar!/:na]
        at com.squareup.wire.schema.CoreLoader$resourceFileSystem$2.invoke(CoreLoader.kt:45) ~[wire-schema-jvm-4.3.0.jar!/:na]
        at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:74) ~[kotlin-stdlib-1.6.10.jar!/:1.6.10-release-923(1.6.10)]
        at com.squareup.wire.schema.CoreLoader.getResourceFileSystem(CoreLoader.kt:45) ~[wire-schema-jvm-4.3.0.jar!/:na]
        at com.squareup.wire.schema.CoreLoader.load(CoreLoader.kt:51) ~[wire-schema-jvm-4.3.0.jar!/:na]
        at com.squareup.wire.schema.SchemaLoader.load(SchemaLoader.kt:140) ~[wire-compiler-4.3.0.jar!/:na]
        at com.squareup.wire.schema.Linker.getFileLinker$wire_schema(Linker.kt:74) ~[wire-schema-jvm-4.3.0.jar!/:na]
        at com.squareup.wire.schema.Linker.link(Linker.kt:87) ~[wire-schema-jvm-4.3.0.jar!/:na]
        at com.squareup.wire.schema.SchemaLoader.loadSchema(SchemaLoader.kt:95) ~[wire-compiler-4.3.0.jar!/:na]

Is it something wrong with my code or CoreLoader.load still has unsolvable problem?

6 replies

gigi Apr 18, 2022

What do you mean? Discussion branch here or git branch? I don't have

oldergod Apr 18, 2022
Maintainer

Sorry, I was asking if that's something you could share so that I can try debugging locally. I don't have much clue right now unfortunately.

gigi Apr 18, 2022

I see. I'll prepare small example for debug tomorrow. Thank you

JakeWharton Apr 18, 2022
Maintainer

Paths with exclamation marks in their name represent files nested inside a jar and at least with built-in Java IO or NIO require the use of the jar:file URL scheme for this to load properly. Okio does not support them out-of-the-box and it would likely require a special FileSystem implementation as ! is a valid character to appear in a normal file name so we cannot blindly assume its present means nesting.

gigi Apr 19, 2022

@JakeWharton
So there is no way to quickly fix this right now, right?

@oldergod
I've added the demo project to illustrate the problem
https://github.com/gigi/square-wire-issue

swankjesse · 2022-04-20T11:08:29Z

swankjesse
Apr 20, 2022
Maintainer

What it looks like is ResourceFileSystem is trying (and failing) to load a .jar file from inside another .jar file, and failing because it doesn't recognize the convention that a ! in a path means ‘look inside a .zip.’

Some fixes to try:

confirm what the regular JVM classpath loader does here. Is nesting a thing? Can we do it reliably?
recover if a file on the classpath is missing.

1 reply

swankjesse Apr 20, 2022
Maintainer

Tracking here: square/okio#1112

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot use Wire within a spring boot executable jar #2209

{{title}}

Replies: 6 comments 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Cannot use Wire within a spring boot executable jar #2209

martypitt Feb 25, 2022

Replies: 6 comments · 7 replies

JakeWharton Feb 25, 2022 Maintainer

oldergod Feb 25, 2022 Maintainer

martypitt Feb 25, 2022 Author

Inject a different implementation

Turn off Spring Boot executable jars, and export the dependencies explicitly

Try to implement my own SchemaLoader / CoreLoader implementations

Monkey patching the reference to FileSystem.RESOURCES

martypitt Mar 1, 2022 Author

gigi Apr 18, 2022

gigi Apr 18, 2022

oldergod Apr 18, 2022 Maintainer

gigi Apr 18, 2022

JakeWharton Apr 18, 2022 Maintainer

gigi Apr 19, 2022

swankjesse Apr 20, 2022 Maintainer

swankjesse Apr 20, 2022 Maintainer

martypitt
Feb 25, 2022

Replies: 6 comments 7 replies

JakeWharton
Feb 25, 2022
Maintainer

oldergod
Feb 25, 2022
Maintainer

martypitt
Feb 25, 2022
Author

Monkey patching the reference to `FileSystem.RESOURCES`

martypitt
Mar 1, 2022
Author

gigi
Apr 18, 2022

oldergod Apr 18, 2022
Maintainer

JakeWharton Apr 18, 2022
Maintainer

swankjesse
Apr 20, 2022
Maintainer

swankjesse Apr 20, 2022
Maintainer