Runtime extensibility

[tjkirch]

Bottlerocket is built to host containers, and most user needs can be addressed by running user-level containers, for example through an orchestrator.

Sometimes you want to alter the host system itself, the components not accessible to user-level containers. This Bottlerocket core can be extended at build-time by adding or changing a variant, or at runtime using host containers. Build-time changes are costly in terms of build time and distribution. Host containers have limitations around update process - they're too manual for components that change frequently, can't maintain state, and can't tie to other components through the API.

We'd like to have a more thorough system for runtime extensibility that combines the advantages of variants and host containers without these disadvantages.

[tjkirch]

One idea that's been discussed is having a more advanced form of host containers:

• they could be built in to an image or pulled at runtime; if the idea is successful, we could move more of the core system to this format.
• they would be bundled together with a namespaced API model, templates, and migrations, allowing for safe stateful upgrades.
• they would be tied in to the the update API so it would not allow a system update until installed bundles are compatible, according to specified semver constraints.
• they could be in the form of an OCI image-index to reuse common technologies, like separate layers for the base code and the API integrations.

Something like this could remove the need for an increasing number of variants; Bottlerocket could use different orchestrators without having different builds, for example.

[stmcginnis]

@bcressey can you please take a look and see if this is something we might want to do some day. If not, please close.

[bcressey]

Let's see if this is needed after #2669 is complete and the dust settles. If so, we can reopen.