Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Forward requests for rotate-root on Performance secondary / standbys #9606

Merged
merged 1 commit into from
Jul 28, 2020

Conversation

catsby
Copy link
Contributor

@catsby catsby commented Jul 27, 2020

In the AWS secret backend, rotating root credentials first creates a new key pair before attempting to store the updated key.

If the rotation request is sent to a performance standby, the new key gets created but saving to Vault state will fail due to the read-only access to state. The request is forwarded to the primary, and at that point the AWS user has 2 keys created; the original it was configured with, and the new key that failed to save.

When the primary attempts to complete the request, it also tries to create a new key before saving it to state, triggering an AWS error LimitExceeded: Cannot exceed quota for AccessKeysPerUser: 2. This then bubbles up in a confusing error message:

Errors:
2 errors occurred:
* errors from both primary and secondary; primary error was error calling CreateAccessKey: LimitExceeded: Cannot exceed quota for AccessKeysPerUser: 2
status code: 409, request id: [Request 1234]; secondary errors follow
* error calling CreateAccessKey: LimitExceeded: Cannot exceed quota for AccessKeysPerUser: 2
status code: 409, request id: [Request 1234]

In this PR we update the config/rotate-root path to use the update OperationHandler attribute, and designate this request should be forwarded before even attempting it in the secondary.

An alternate solution would be to delete the key that was created but which failed to save to state. A PR that does just that will likely follow this one, however due to the eventual consistency nature of IAM in AWS I feel it's better to immediately forward the request instead.

Similar PRs:

@catsby catsby added bug Used to indicate a potential bug secret/aws labels Jul 27, 2020
Copy link
Member

@calvn calvn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for the detailed context in the description!

@catsby
Copy link
Contributor Author

catsby commented Jul 30, 2020

catsby added a commit that referenced this pull request Aug 17, 2020
Moving the CL entry for #9606 to the `v1.4.4` section. It's implied that anything in 1.n is also in 1.n+1
catsby added a commit that referenced this pull request Aug 18, 2020
Moving the CL entry for #9606 to the `v1.4.4` section. It's implied that anything in 1.n is also in 1.n+1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport bug Used to indicate a potential bug secret/aws
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants