[Question] Upcoming expiration of our current sub public key (expire on 20220511)

[peterzhuamazon]

Our current sub private key that we use for signing detached signature (.sig), and the sub public key that we attach to website for community to verify, will expire soon.
https://opensearch.org/verify-signatures.html

pub   4096R/39D319879310D3FC 2021-05-11
uid                          OpenSearch project <opensearch@amazon.com>
sub   2048R/C2EE2AF6542C03B4 2021-05-11 [expires: 2022-05-11]

pub   rsa4096/39D319879310D3FC 2021-05-11 [SC]
      C5B7498965EFD1C2924BA9D539D319879310D3FC
uid                 [ unknown] OpenSearch project <opensearch@amazon.com>
sub   rsa2048/C2EE2AF6542C03B4 2021-05-11 [S] [expired: 2022-05-11]

Once the key expired we need to do a few things to resolve this.
Questions are:

1. Do we still want to add new subkey based on master private key, or do we want to expose the master public key directly. This was the strategy that we used to do in ODFE.
2. Once we add the new pair, do we still want to expire the subkey after a year. Or we can just set the subkey to not expire at all.
3. Once we add the new pair, do we still want to keep a link for the old subkey. If I understand correctly the new subkey can verify both the old/new signatures signed by old/new sub secret key. If not, then we can just override the artifact opensearch.pgp with the new sub public key.
4. Do we still want to use old key (not expired yet but very soon) to sign 2.0.0-rc1, or do we want to create a new subkey now and use that? 1.3.2 after rc1 is even more of a stretch.

Note: Do we also want to just extend the existing key so it wont expire?

Thanks.

---

Options:

1. Extend the existing subkey to be not expired.

• Pros:
  • Straight forward, no need to changes the key and rotate every year
  • All of our current existing standalone artifacts, tar, rpm, etc. are signed with this already
  • We can always revoke the subkey if we know the sub secret key leaked
• Cons:
  • No annual rotation
  • Everything is signed with the same key

2. Get a new pair of subkey from the master, no expiration.

• Pros:
  • No difference as 1
• Cons:
  • Even worse than 1 as we need to announce a new key but never expire anymore

3. Get a new pair of subkey from the master, set it to expire in 1 year.

• Pros:
  • Annual rotation
• Cons:
  • Have to announce to the public every year that we have a new pub key, and have to have documentations recording which artifact is signed with old vs new key
  • Every year we need to rotate the key and the key can expire at undesired time
  • Current signing workflow does not support rotation, everything manual

4. Get a new pair of subkey from the master for every product, or every major release of all products.

• Pros:
  • A lot of keys then less risk of leaking one affecting all artifacts
• Cons:
  • Too many keys to manage and will get very messy
  • Every year we need to rotate all the keys, if set expired in one year, for all products
  • Current signing workflow does not support more than 1 key
  • Need to have very detailed documentation record all the keys and all the products and all the versions, on which product/version/time match which key, and which one has expired or not, etc.

---

• Update 20220428:
  • Huan, Tao, David attend the meeting and our decision is to expire the current subkey for a year. This probably requires upload the updated key to S3 bucket. Not sure if we want to officially announce that we renew the key. We will write an issue to track the signing workflow to support rotation and automation soon. The security guardians think that having the subkey unexpired is not good practice, but we can still keep the expiration period longer than a year.

[peterzhuamazon]

@spotrh @dblock @bbarani would love to have your thoughts on this.

Thanks.

[bbarani]

@anirudha Do you have any inputs since your team helped create the initial signing keys? The current sub key is set to expire in a year from the creation date. We are wondering if its better to create a new sub key from master key without expiration (we feel its risky) OR create sub key with 1 year expiration. Do you have any security recommendation?

[dblock]

@peterzhuamazon What are the pros/cons of each of these proposals?

[peterzhuamazon]

@peterzhuamazon What are the pros/cons of each of these proposals?

Added in description.

Thanks.

[dblock]

What do other projects do? We also have the option to defer this decision and extend the key for a limited period of time (e.g. another year)?

[peterzhuamazon]

Our subkey officially expired right now:

pub   4096R/9310D3FC 2021-05-11
uid                  OpenSearch project <opensearch@amazon.com>

# Note you cannot see any `sub` key anymore

% gpg --verify opensearch-2.0.0-rc1-linux-x64.tar.gz.sig
gpg: Signature made Tue 03 May 2022 05:30:55 PM UTC using RSA key ID 542C03B4
gpg: Good signature from "OpenSearch project <opensearch@amazon.com>"
gpg: Note: This key has expired!
Primary key fingerprint: C5B7 4989 65EF D1C2 924B  A9D5 39D3 1987 9310 D3FC
     Subkey fingerprint: 2187 3199 B103 0FCD 49DA  83F8 C2EE 2AF6 542C 03B4

[peterzhuamazon]

Just work with @prudhvigodithi on extending to a new public key, the private key will not expire but include the old sub public key.

Notes:

1. You can only extend sub public key, not sub private key, as the latter will not expire.
2. Tho sub private key not expire, it will include the old expired sub public key in the chain.
3. Previously we only need to import sub private key for signing, now we need to import both sub private key and the new sub public key for signing.
4. The new sub public key can verify both the old signature and the new signature.

Steps:

Additional steps after RHEL9 introduce strict verification and requires SHA2:

• SHA-1 GPG keys are deprecated, request to update GPG-keys and RPM repository packages #3124

Run gpg command one in your system.
Add these to the gpg.conf file:

personal-digest-preferences SHA512
digest-algo SHA512
cert-digest-algo SHA512
s2k-digest-algo SHA512
s2k-mode 3
s2k-count 65011712

1. Clear all the existing keys related to our project in GPG.
2. Import sub private key
3. Import master private key
4. import sub public key that about to expire (Optional) The theory is if you dont import this it wont include algo 2 so rpm import would be fine; if import this it will have both algo 2 and algo 10, yum runs fine on higher version of rhel.
5. gpg --edit-key C2EE2AF6542C03B4
6. use key <number> such as key 1 to select the key that is not 39D319879310D3FC or C2EE2AF6542C03B4 (see a * next to the selected keys), then delkey to delete it as it is not needed in the signing key chain.
7. Key Usages Charts:
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
   Constant Character
   ───────────────────────────────
   PUBKEY_USAGE_SIG S
   PUBKEY_USAGE_CERT C
   PUBKEY_USAGE_ENC E
   PUBKEY_USAGE_AUTH A
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
8. use key number again to select the key C2EE2AF6542C03B4 then type expire.
9. choose either unexpire or 1 year or any time (suggest to use days to get the exact same expire date for next year), use the passphrase to confirm the prompt.
10. type save to save all the progress.
11. export new C2EE2AF6542C03B4 with gpg --export --armor C2EE2AF6542C03B4 > new_sub_public. This key should be about 8.0K in size instead of 4.0K.
12. remove master private key 39D319879310D3FC as well as master public.
13. Next time when we need to sign, import the sub private key and new_sub_public to sign.
14. This new sub public key can be uploaded to S3 to replace the old sub public key.

Next Step:

1. Replace S3 key
2. Publish the new public key to key server
3. Edit the existing signing workflow to use the new import process for signing
4. Either documentation or post or update the page here https://opensearch.org/verify-signatures.html to inform users the old key expired the new key needs to be used to verify.

Thanks.

[peterzhuamazon]

We will make some changes to the public key page and replace the extended key for another year until 2023/05/12.

[krisfreedain]

@peterzhuamazon as discussed, we'll need to add a note to https://opensearch.org/verify-signatures.html. I propose the following - under the line
Our current PGP key fingerprint is C5B7 4989 65EF D1C2 924B A9D5 39D3 1987 9310 D3FC

"*Note: On 2022-05-11, the existing public key expired. If used, you will see "gpg: Note: This key has expired!" as noted in Issue 2040. Please download the new key which we have extended to 2023-05-12."

[krisfreedain]

pushing out the edit for that page in project-website PR 823

[peterzhuamazon]

We have updated the key on the same url with the new sub public key that extended from the original sub public key that was expired on 20220511.

This extended new sub public key will expire on 20230512, and can be used to verify all previous and later signatures.

pub   4096R/39D319879310D3FC 2021-05-11
      Key fingerprint = C5B7 4989 65EF D1C2 924B  A9D5 39D3 1987 9310 D3FC
uid                          OpenSearch project <opensearch@amazon.com>
sub   2048R/C2EE2AF6542C03B4 2021-05-11 [expires: 2023-05-12]

% gpg --verify opensearch-2.0.0-rc1-linux-x64.tar.gz.sig
gpg: Signature made Tue 03 May 2022 05:30:55 PM UTC using RSA key ID 542C03B4
gpg: Good signature from "OpenSearch project <opensearch@amazon.com>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: C5B7 4989 65EF D1C2 924B  A9D5 39D3 1987 9310 D3FC
     Subkey fingerprint: 2187 3199 B103 0FCD 49DA  83F8 C2EE 2AF6 542C 03B4

Thanks.

[peterzhuamazon]

Add follow up issue for next year:

• [Renew Key] Upcoming expiration of our current sub public key (expire on 20230512)  #2136

[peterzhuamazon]

Upload to these key servers to replace the existing ones, manually upload do not use --send-key command (reason):

• https://keys.openpgp.org/search?q=opensearch%40amazon.com
• https://keyserver.ubuntu.com/pks/lookup?op=get&search=0xc5b7498965efd1c2924ba9d539d319879310d3fc

{"inserted":null,"updated":["rsa4096/c5b7498965efd1c2924ba9d539d319879310d3fc"],"ignored":null}

• https://pgp.mit.edu/pks/lookup?op=get&search=0x39D319879310D3FC (This one updated slow and upload extremely slow so will wait for some time to renew on their website)

Key block added to key server database. New public keys added:
1 key(s) added successfully.

Bare in mind the keyserver is trying to chain the old key with new key and trusted with their server so the copy on their server is not exactly the copy you upload. But when download then import it shows the same behavior in PGP and same fingerprint.

[peterzhuamazon]

Thanks @krisfreedain the website is now updated:
https://opensearch.org/verify-signatures.html

[peterzhuamazon]

We will close this issue for now.
Thanks.