RFE: Optionally Encrypt Certs as Secrets on init use token on join

[timothysc]

edit(neolit123): google doc link for the proposals:
https://docs.google.com/document/d/1XkByzeehah20CgNY47NUMr2VAz-D_ok-GqRX5YRQH0o/edit?usp=sharing

---

The general UX for standing up an HA control plane is a bit manual for most consumers. There has been a desire for folks to be able to store control plane certs as secrets but there is a broader security concern that these secrets are not encrypted, or any token could be obtained through the cluster itself.

• Proposal:
  • ControlPlane 1:
    • kubeadm init --upload-certs
    • ...
    • Encrypts the certs locally (using pick your favorite method) and uploads ~= sealed-secret
    • Output a token/key (file or command line)
    • ....
  • ControlPlane 2:
    • kubeadm join --control-plane --download-certs=key

Certs are stored as secrets but the key is only created on init and used on join. The key is never to be stored on cluster.

/cc @kubernetes/sig-cluster-lifecycle

[timothysc]

/assign @fabriziopandini @timothysc
/cc @mattmoyer

[inercia]

@timothysc When you say "Output a token/key (file or command line)" you mean another token? couldn't we use the same token used for joining the cluster? otherwise we would be moving from a "how to distribute certs in the cluster" problem to a "how to distribute a token" problem... :-/

[fabriziopandini]

@timothysc
This is a possible solution but what concerns me is that encrypted secrets are another bit of cluster settings that we are duplicating from its "source of through" and that we should manage acróss upgrades.

what if kubeadm join

• creates a temporary encryption key
• runs a job/pod on one existing master that grabs secrets from file system and save to an encryped config map
• Reads from config map and save certs to the local filesystem
• delete the encrypted config map

Pros: We are duplicating certs only for a short time during the join workflow and the encryption key will be managed by kubeadm
Cons: We need a container image

edited 22/12 - this cannot be implemented to a RBAC problem. see gdoc

[timothysc]

That's a lot of extra jiggery, but it would work too.

[neolit123]

/assign

[neolit123]

my proposal here:

• we add a new flag for kubeadm init called e.g. --serve.
• once kubeadm init has finished we start listening for connections on a TCP port but with a TTL (e.,g. 10 minutes)
• nodes calling kubeadm join --experimental-control-plane will probe for that connection at the controlplane endpoint IP.
• if a connection is established a secure handshake will be performed using the boostrap token bytes as a secret.
• the joining node will them download certs + admin.conf
  (can happen for more than one joining node)
• TTL will expire and the first CP node will stop the server.
• to server again a separate kubeadm command can be added.

pros:

• one source of truth for the certs, no intermediate copying medium - certs go from one disk to another over TCP.
• easy to implement.

cons:

• turns kubeadm into a server / client.
• needs risk evaluation for MITM attacks, although i'm pretty sure this needs super computers already.

working POC project - try it:
https://github.com/neolit123/kubeadm-getter

i can explain how the security implementation and protocol works.

[inercia]

In my view, the idea of serving files for a limited period of time would mean that new masters could not be added later on, and that would import important restrictions in the topoloogy in the cluster. I guess that the solution in this case would be that users would have to restart the --server side and then join the new master, but this effectively would be no better than a regular scp.

Besides this, I think that the whole idea of having a fileserver embeded in the seeder is not the best approach. Not only because of the MitM problems you mentioned (that would require the distribution of certifciates to the nodes joining as master) but because of this mechanism for distributing files out-of-band.

I think it would be better to distribute these files as part of the bootstrapping process, in the response from the seeder, I don't know the internals of the bootstrapping process, but maybe when the client gets the reposnse to the CSR... So the client could ask "I want to join, but as a new master" and the seeder could perform a regular bootstrap but attaching in some way the new master certificates. Maybe clients should provide some additional token (so not every node could join as a new master), but the whole certificates distribution would be part of the protocol.

What do you guys think?

[neolit123]

In my view, the idea of serving files for a limited period of time would mean that new masters could not be added later on, and that would import important restrictions in the topoloogy in the cluster. I guess that the solution in this case would be that users would have to restart the --server side and then join the new master, but this effectively would be no better than a regular scp.

that's a limitation yes.
of course if someone wants to they could leave the file server open indefinitely.

Besides this, I think that the whole idea of having a fileserver embeded in the seeder is not the best approach. Not only because of the MitM problems you mentioned (that would require the distribution of certifciates to the nodes joining as master) but because of this mechanism for distributing files out-of-band.

MITM problems are removed by the encryption of the file transfer. there is no need to distribute public/private keys because the bootstrap token itself is the secret key.

I think it would be better to distribute these files as part of the bootstrapping process, in the response from the seeder, I don't know the internals of the bootstrapping process, but maybe when the client gets the reposnse to the CSR... So the client could ask "I want to join, but as a new master" and the seeder could perform a regular bootstrap but attaching in some way the new master certificates. Maybe clients should provide some additional token (so not every node could join as a new master), but the whole certificates distribution would be part of the protocol.

@fabriziopandini proposed something similar. having the certs as part of the bootstrap token.
i don't know how it works completely too, but i think we might end up abusing it a little if we start embedding certs in the transport.

[neolit123]

after the discussion of 28.11 office hours, we decided that we should collect proposals in a gdoc for this and eventually do a KEP.
TODO

[neolit123]

@timothysc @fabriziopandini @luxas

proposals added in a gdoc here:
https://docs.google.com/document/d/1XkByzeehah20CgNY47NUMr2VAz-D_ok-GqRX5YRQH0o/edit?usp=sharing

[fabriziopandini]

drafts of the KEP for this feature https://docs.google.com/document/d/1hSghREAoM8f_usJjGuQS1bWEtCC2wBYFNoBC7FyISwA/edit?usp=sharing
I will quickly iterate on google doc for the first round of feedbacks until friday and the move on a PR

[yagonobre]

will we allow the user to create a cluster without create the kubeadm-certs secret and the encryption-key?

[neolit123]

will we allow the user to create a cluster without create the kubeadm-certs secret and the encryption-key?

we should, as it is now, and this will allow the user to copy certs manually if they want HA.

[neolit123]

closing in favor of #1373
please re-open if needed.