Provide a high level tool to backup and restore the cluster

[klizhentas]

Description

It will be helpful to have a tool to backup and restore the cluster, here are the steps that have to be made:

• On the old cluster, execute tctl backup that will capture the old cluster name, find and capture config file, capture certificate authorities state

• Start new cluster with the backup path, e.g. tctl start --backup=backup-file that will verify that cluster name is the same and init the certificate authorities to the right values.

[kontsevoy]

@fspmarshall can you please CC me when user experience becomes more clear?

[fspmarshall]

@kontsevoy Its not 100% clear yet, but here's the state of my thinking thinking thus far:

The usage pattern I'm targeting is basically what @klizhentas describes above: export a backup file with tctl backup, and then later import the backed up state via a --backup option supplied to teleport start.

I think the most generic & powerful backup implementation would be one that could be implemented as a generic collection of "resources", making tctl backup basically just shorthand for a set of tctl get operations. This approach is appealing because it minimizes the overhead of adding new state elements down the road, operates on mechanics the user is already familiar with, and (in theory) doesn't mandate much in the way of custom logic.

In practice, I do see a few challenges around actually starting a new auth server from the produced backup. As far as I can tell, the teleport program doesn't ever actually import resources in a generic manner (the tctl create subcommand takes in generic resources, but resolves their concrete type before passing them to teleport via type-specific endpoints), which means such a mechanism would need to be added. Furthermore, such a mechanism would need to be able to correctly partition resources between process-storage and the primary backend, and should probably have some kind of safety mechanism which prevents overwriting of newer resources with older resources. Finally, some elements of auth server state (HostUUID at the very least) are resolved before the process storage or primary backend are initialized, so simply adding the ability to initialize the backend(s) with a set of pre-defined resources isn't sufficient to keep state consistent (though I believe the scope of changes needed to fix this should be fairly limited).

The above solution doesn't cover backing up the config file, but bundling the config file alongside the exported resources would be fairly simple.