flake: add private inputs & defer locking

[nrdxp]

Includes our fork of the official nix registry:
https://github.com/input-output-hk/flake-registry/commits/iog

Here is a PoC of a scheme that allows for defining private flake inputs that do not propagate to the consuming flake, for easily management of dependencies without the overhead of cluttering everybodies flake.lock. Inputs that are defined in the top-level flake.nix should be expected to be defered to the consumer for locking, and preferably the consuming cluster should have our registry set in the nixConfig to keep values consistent and bumps explicit.

This is only the first step, but if we do this consistently throughout our projects, it will solve a majority of the lock related problems we've had so far while giving us a much easier path for updating dependencies on the cluster level. For example, it's been a pain in the past to update ssh keys from ops-lib, because we first had to bump the bitte dep and then again bump bitte on the cluster. Now that locking is deferred to the cluster flake will simply update ops-lib whenever a user requests an update to it, without having to channel through bitte at or define any special semantics.

This also gives us a reasonable path to avoid massive lock files that impose heavy costs: NixOS/nix#6626

For the sake of not breaking too many things, I have kept the API the same, that is, public inputs and private inputs are combined at the top of the flake.nix so minimal code changes to the rest of the codebase are needed. Once we stdize we may be able to break this up a bit if desired.

[manveru]

Generally in favor of this. I see problems with not committing lock files, unless we rely on direnv with --no-write-lock-file (we may already do that?), and even then figuring out exactly which version of what input you use might become tedious.
The registry would have to be way more specific than it is now for this to work.

[nrdxp]

Well we would commit a lock file on the consuming cluster flake, just not on the upstreams to make them easier to update from the registry. if NixOS/nix#6550 were to be merged at some point, it wouldn't be necessary, but without that functionality, not commiting a lock on bitte is the only way for the consuming flake to easily control and update bitte's inputs (assuming they are all registry inputs, and not direct references).

But yeah, some fine tuning of the registry will definitely be necessary. Thankfully it's easy to modify the json registry with the nix cli if you just pass nix registry --registry path/to/our-registry.json.

[nrdxp]

Added some addtional, more specific pins to the registry, and gave this idea a testrun in a smaller project as a poc here:
input-output-hk/devshell-capsules#7

[nrdxp]

ping @manveru, @blaggacao when you have time to look at this, I've implemented the majority of the idea, that can be safetly consumed by clusters, so long as they define our registry.

In order to get the most out of it, we'll have to do something similar for most of our other projects, but on the plus side, it's really not too difficult to do so.

[gytis-ivaskevicius]

Unfortunately, this does not do much. nix by default force adds the lock file

[blaggacao]

@nrdxp can you create separate PRs; one per motivation?

These are unfamiliar patterns, we should scrutinize and field one at a time...

... probably starting with private inputs that won't then propagate to downstream consumers' lock files, anymore. This is relatively straight forward and will help bring lock file sizes down, but also needs for full effect its std correspondence in Cardano World via: divnix/std#111

As for the registry, I'd like to better understand what abstract root problem a registry would solve and what undesired side-effects this might have. But we can discuss this on the respective PR.

There is also a third consideration: bitte iteslf is a private input to the bitteProfiles organelles, so why would we care at all, in here? (And not stop propagation in function of wether the bitteProfiles-organelle is of interest via divnix/blank)