Add support for "package extensions" to code loading

[KristofferC]

This is an implementation of the proposal in #43119 (with some tweaks) that aims to replace Requires.jl for loading code based on the presence of other packages. It is a different implementation to #47040 and should work better in for example big environments where you expect to only load a small subset of packages.

In spirit, it is very similar to Requires.jl in that given a set of packages and a file, you load the file when those packages have been loaded in the julia session. The differences to Requires.jl are:

• The file with the conditional code gets precompiled in the same way as a normal package would. Requires.jl instead directly evaluates the code into the package.
• It is possible to use Pkg to set compatibility on the packages that "activates" the conditional code. This is a new feature in Pkg and could in theory also be used together with Requires.jl.
• It is "declarative" in the sense that all information requires id stored in the Project.toml and Manifest.toml file so it avoids having to run any arbitrary Julia code in e.g. __init__ (which is done when using Requries.jl).

The added docs in code loading as well as the added docs in the accompanying Pkg PR (JuliaLang/Pkg.jl#3264) can be used to get a more detail description of its usage.

A lot of the diff in this PR is due to the code loading being pretty awkward and would have been easier to be built on top of #46690 but I don't want to couple those PRs.

For reviewers: going through all the uuid lookups etc in the code loading is probably fairly uninteresting since it is mostly mechanical, the interesting parts are those around line 1080 - 1200 in loading.jl which deals with the insertion of gluepkgs "callbacks" and the loading of them.

An example of a real package moving from Requires.jl to using this system can be seen in https://github.com/KristofferC/PGFPlotsX.jl/compare/kc/glue. It should be pretty easy to be backwards compatible with Requires. You just put the @require inside a version check and include the file for the conditional file in the @require block.

One TODO that is left is that Pkg should precompile these "glue packages" during Pkg.precompile (which is often run automatically) in the case of them having a chance to be loaded in the current environment.

[KristofferC]

Pushed a bump to Pkg here to include JuliaLang/Pkg.jl#3264 to run CI of these together.

[Seelengrab]

Does this have downsides/a tradeoff compared to #47040?

[KristofferC]

Some are discussed in #47040 (comment)

[Seelengrab]

Gotcha, I just thought there may be some other ones. Thinking about the description of this in the Pkg PR, what happens when there's glue needed between glue, e.g. when you have weak dependencies A and B and want different behavior for when only A, only B or both A and B are loaded? Would this need GlueA, GlueB and GlueAB modules..?

[KristofferC]

when you have weak dependencies A and B and want different behavior for when only A, only B or both A and B are loaded? Would this need GlueA, GlueB and GlueAB modules..?

Yes.

Another question is, what if you want to activate extra functionality inside a glue package when some other package is loaded. Basically glue^2. The reason I think about this is that I saw this line:

https://github.com/JuliaPlots/Plots.jl/blob/7bb8f6aee1c93d4dd478cc0d596a804abec8a723/src/backends/pgfplotsx.jl#L1042-L1046

in Plots, where a @requires block is used inside another requires block. In theory, this could be specified like:

[gluedeps]
PGFPlotsX = "..."
Measurements = "..."

[gluepkgs]
PGFPlotsGlue = "PGFPlotsX"
PGFPlotsMeasurementGlue = ["PGFPlotsGlue", "Measurements"]

And PGFPlotsMeasurementGlue would load when both PGFPlotsGlue and Measurements are loaded.

[Seelengrab]

Another question is, what if you want to activate extra functionality inside a glue package when some other package is loaded. Basically glue^2.

And that's exactly the reason why I'm more in favor of the other proposal :D No risk of accidentally going quadratic in terms of complexity, at the cost of penalizing larger environments a bit (which probably should just use a large sysimage anyway?).

[KristofferC]

@nanosoldier runtests()

[KristofferC]

And that's exactly the reason why I'm more in favor of the other proposal :D No risk of accidentally going quadratic in terms of complexity, at the cost of penalizing larger environments a bit (which probably should just use a large sysimage anyway?).

The weak deps idea has some advantages and is more "powerful" since you have access to all information during precompile time, yes. But after thinking about this for quite some time I do think that the suggestion here is what should be tried first. Some reasons:

• It is conceptually similar to Requires which people seem happy to use (with the exception of its problems which are more side effects of the implementation). This also makes it quite easy to be backward compatible with Requires (which the Pkg PR has docs for how to be).
• When the "conditional features" (or glue packages) are separate modules that are loaded, the extra stuff is more or less "pure" additions to what the base package offers. I think that would mean that it would be easier to test each thing separately.
• I think the amount of recompilation that would be required with the weak deps thing is pretty bad. If you remove a package, you might have to recompile your whole environment. With glue deps, adding a package will at most cause you to compile new glue packages (which are small), and removing a package will never require new compilation.
• I think this will on average give users a smaller load time. Weak dependencies really punish you if you don't have super clean environment usage and a lot of people just don't use Julia like that. We can say "oh but people should make sysimages or have smaller envs etc" but if that is just not how the majority of people use Julia, that should be taken into account.'

[nanosoldier]

Your package evaluation job has completed - possible new issues were detected. A full report can be found here.

[KristofferC]

PkgEval looks good.

[IanButterworth]

Obviously there are more changes to like this, but just to mark it as I think this is where the discussion settled

Suggested change

	### "Glue" packages and dependencies
	### "Weak" dependencies and "Glue" packages

[KristofferC]

For those following along, it is likely that "glue packages" will be renamed to "package extensions".

[IanButterworth]

Awesome.

[johnnychen94]

Not sure if this is a deliberate design so wanna ask it here. Does it sound like a good idea to make Plotting.ContourExt a "normal" submodule? And if not loaded, Plotting.ContourExt keeps to be nothing.

This makes it easy and intuitive to get objects inside the glue module, for instance:

Base.get_extension(Plotting, :ContourExt).Contour

becomes

Plotting.ContourExt.Contour

One hack/workaround is to insert

module ContourExt

using Plotting

function __init__()
    setproperty!(Plotting, :ContourExt, Base.get_extension(Plotting, :ContourExt))
end
end

[KristofferC]

The extensions should in my opinion be "private" and their behavior should almost only be visible to users by new dispatch rules being defined. You shouldn't really need to reach into the extension module itself. The get_extension is thus mostly for debugging.