Packaging Plan

[jasonrclark]

Packaging is the last big outstanding thing between us and a Shoes 4.0 rc. Shoes without packaging just doesn't feel like Shoes.

That said, our JVM-based plans mean Shoes 4's packaging story is going to be different than Shoes 3. Some of that's nice (warbler easily provides us neat, self-contained jar files), some less so (Java dependencies, difficulty in getting single executables)

The Quest

The single executable thing has been a big hangup for me. I was sure "something" must exist to turn a jar into an exe or an equivalent Linux-y single-file-executable. I've hunted far and wide, but the many options are all either 1) ancient and unsupported, 2) GUI-only 3) Windows-only or 4) a combination of all the above.

The one exception in my journey was packr which at first seemed perfect. While it had a few rough edges, it gave me hope... then I looked at the actual output, and found that it didn't output a singular artifact--it's gave you a directory with a native executable alongside the packaged jar and a JRE.

So if folks who know what they're doing around Java packaging land on this layout... I guess I'm inclined to accept that and move on from there

A Proposal

While a single binary executable would be ideal, I feel like that goal's getting in the way of actually shipping something that would be acceptable. As a "good enough" solution, I'd like to build out like this:

• Mac apps like they are (just a directory with specific stuff crammed in it)
• Windows, an executable, alongside the jar and JRE
• Linux, an executable, along the jar and JRE

If the "easy to pass around bit" matters a lot, these could be zipped/tar'd into a single artifact.

What do you think @PragTob? For the sake of reaching a release, could we live with that?

Steps

In a bit more detail, I see this going something like the following:

• Reconsider the command-line interface for packaging (swt:app for Mac apps doesn't seem quite right... I'd love something like `shoes package [jar|shy|mac|windows|linux] my_app.rb (Revisions to packaging arguments #1417)
• Windows packaging with batch file (for simplicity to start off), alongside jar and an appropriate JRE
  • furoshiki changes (Windows "app" support furoshiki#24)
  • shoes changes (Wire up packaging for Windows #1420)
• Revise Mac packaging to use same approach as new templates
  • furoshiki changes (Revise mac app furoshiki#27)
  • shoes changes (Revise to use new MacApp class #1430)
• Linux packaging with shell executable, alongside jar and an appropriate JRE
  • furoshiki changes (Linux app support furoshiki#29)
  • shoes changes (Support Linux app #1436)
• Lock down our solution for hosting the JRE/template downloads (thanks to @wasnotrice for holding the Mac stuff in his S3 buckets!) GitHub releases look like just the thing!
  • Windows templates
  • Mac templates
  • Linux templates
• Look at slimming down the JRE's we ship there. Inspired by packr, I'm betting we could drop a lot of stuff from those. FWIW, if we get away from pulling JRE with every single package, I'm less concerned about this
• Consider making the JRE inclusion in the app bundles optional, for folks who want to let their users provide their own Java. Probably not necessary now if all templates move to just downloading a JRE if necessary?
• Create more realistic packaging samples, maybe scripts to make running them a bit easier Tracked as More realistic packaging samples #1437
• Replace Windows batch file with a native executable that we can control icon and naming on... but that can wait Tracked as Native executable to bootstrap on Windows #1438
• How important is it to handle shy files (ala Implement packager for Shoes "apps" (.shy named zip-installs) #124)? I'm guessing it'd be nice, and we can poach the code from Shoes 3, but I've never encountered them before. Tracked as Implement packager for Shoes "apps" (.shy named zip-installs) #124

[ccoupe]

FWIW, Shoes 3.3.3 is likely to move to the QT IFW installer - you don't need a full QT to run it.
See shoes/shoes3#330

[PragTob]

Great stuff @jasonrclark - thank you! 👍

I'm ok with the solution, a single file would be ideal but JAR + JRE + launcher that uses these two seems just as fine to me.

I'd consider maybe asking folks at jruby-user if they know anything else, I trust your research of course just sometimes these Java/JRuby folks have an additional magic word/experience or whatever :)

Although, seeing that eclipse even comes with only a zip/tar.gz give me little hope that there is such a thing :(

Regarding .shy files... I don't even have an example of those. It's fair to leave out for either a next version or unless actual users actually demand them 😁

Thanks for all the work Jason 💚

[jasonrclark]

Glad you agree @PragTob, and hitting up the jruby-user list is a great idea. I won't get too far along before doing that one last bit of research.

Will see where time gets to with .shy files. I can see how to generate them from the Shoes 3 app, so I can get one to test against that way, but don't know how wide-spread they are.

[PragTob]

Not sure if I just misremember, but I thought there was something here about seeing if we can slim down the JRE/what we use in libraries.

My local Java/Android guru uses proguard as a tool so that might be interesting. Maybe deserves its own issue though... and definitely something that can wait until waaaayyy after a release or 2 ;)

[jasonrclark]

All right, so one of the question marks I still had was hosting and where to put the templates for packaging to find. And I have an answer!

Turns out GitHub has this lovely releases feature that lets you put arbitrary binaries up. I've made an example with an updated JRE Mac app template at https://github.com/jasonrclark/mac-app-templates/releases

So keeping all the loose binaries in the git repo would work for a while, but might blow out our cap on the org over the medium-term. I'm inclined to just write up instructions for making a new app image (with some supporting scripts perhaps?), and just host the release binaries for download. Sound cool?

Other notes:

• Keep each app type in separate repo so we can version them independently
• Can update furoshiki to read public Github API and grab latest version without embedding it in the gem allowing easier app updates

[PragTob]

Sounds good to me, I just wish binaries wouldn't have to be in the repo and github would allow to add them seperately, which is what I thought it does but never used it

[KCErb]

Sorry to jump into this randomly, been crazy busy for many months now and probably poking my nose in where it doesn't belong, but on one project I'm on we use google drive to store a dev database and our initialization scripts use the command line tool https://github.com/prasmussen/gdrive

Don't know if that's a fit or if I've spent enough time to understand the issue properly but it has been pretty seamless for us and kept our repo nice and small.

[pjfitzgibbons]

HI Folks,
Long time listener, first time caller. (well, not first time :)) . So first, @jasonrclark 👍. Excellent plan.
This email represents an intent to step my toe back into the Shoes water.
I think the "executable" alongside Jre/files is a fine solution, and demonstrated as canonically good by examples like Eclipse like @PragTob noted. I think the long-winded projects like eclipse get by doing their thing for decades this way, so it seems like a stable example to follow.

My note/questions :

1. Shy : "originally" (this is a fuzzy history), the .shy files were the "packaging of a Shoes app" [https://github.com/shoes/shoes/wiki/A-Developer's-Tour-Through-Shoes]. Although that writeup conflates the packaging of multiple *.rb and the packaging of the whole shoes runtime (what we in this issue are targeting as an *.{exe,sh,whatever}.
   So, is there still a place in the distribution plan to switch between building a .shy and building a .exe ?

2. If we build a stripped-down JRE, what will be the "advanced" plan for starting a shoes on the elsewhere-installed full-JRE ? Seems like with the stripped one we'll trip up some advanced developer who's suddenly looking for the Java Concurrency library and their code is just flat failing because it's running the stripped JRE. Thoughts?

Clearly I've stuck my nose waay more in than @KCErb, as I can easily claim at least 1/2 a decade since I've actively written anything in the Shoes repo.

Hopefully this is the renewal of a great friendship. :)

[jasonrclark]

@PragTob I might have been unclear, but the files we upload in releases don't have to be part of the repo--they stand off to the side, and don't count toward the size budget.

@KCErb Google drive is an interesting idea if do find that not having the loose files around in version control is an issue. I think since the results are just the single zip file that we can upload in releases, we should be ok.

@pjfitzgibbons Welcome back! Have seen you around on many of the early Shoes issues, but glad to hear from you in real-time.

1. Appreciate the link there... I hadn't seen that one around. Especially like the quote "That's the promise and it really did work, once. Some of it still does." 😆 Interesting how much it talks about the packaging as an installer rather than a fully self-contained app.

That said, I'm open to the possibilities around .shy or other flavors of self-contained stuff, so if you have suggestions I'd love to hear them!

2. My thoughts around the "advanced" mode here would just be to have an OS-native app that would try to run your jar with the installed system Java (or maybe detect if there wasn't one and warn you). So someone wanting their own fancy jars from elsewhere would just need to set up their JVM environment.

A alternative for folks wanting other Java libraries is to wrap them up into gems like we do with swt and pack those along with you, but that's clearly quite a bit more work.

This weekend I'm working through making an updated Mac OS X template with a more recent JVM (1.8.0_121), and then will start playing with making a files-alongside template for another OS (probably Windows)

[pjfitzgibbons]

Yeah, "some of it still does"... was relevant for WindowsXP. Seriously, that old. Recall that bloopsaphone and the video wrapper were the first two modules to go down, because they refused to compile in 32-bit only. Yeah, back in the transition from 16->32 bit.

I think it would make sense to users to have possibly two different switches for packaging. One builds the .shy (zipped .rb), which can be run with shoes myapp.shy as in the example.
The other builds a wholesale installed app .exe/whatever that is downloaded/installed by .zip and runs wholly independently on the target OS. I'm thinking that once you've packaged an app w/ shoes it should be named by your app (MyApp) and the startup file should also be named MyApp.exe within the .zip. It seems at he point of packaging a zip exe that we "hide" the Shoes name behind the name the user has desired for their App.
These are my thoughts, integration to the plan I leave completely up to you all. Just my €.02.

As for packaging, I was interpreting your description as meaning we "re-compile" (?) a new JRE that has less of the "standard-lib" in it than the normal defacto JRE-install? If I mis-understood, then my repsonse is probably moot. As for adding your own .jar to the package, I thought we handled that because requiring the jar with app-root context is how jruby require works anyway, no?

[KCErb]

OK cool. If I remember right you get a unique id for the file which you can track in vc but I could be wrong. At any rate looks like we don't need that here 😎

Thanks for everything you guys are doing to take care of Shoes!

[jasonrclark]

Last couple items logged as separate issues that we'll handle later. Bulk of this is ready to 🤘 with pre10.

[PragTob]

🎉