Kibana user can't change own password in cloud deployments

[pheyos]

Found in version

• 7.11.0-bc3 cloud deployment

Browser

• Chrome

Steps to reproduce

• Create a user and login with that user
• Go to the user profile page and change the password

Expected result

• The password can be changed without errors

Actual result

• The password is not changed
• An error taost is displayed
• The network tab shows that the request returns with a 404

Additional information

• Related to this is an API test failure on cloud: [test-failed]: X-Pack API Integration Tests1.x-pack/test/api_integration/apis/security/change_password·ts - apis security Change password should reject password change if current password is wrong #86520
• Doesn't happen in 7.10.2 GA cloud deployments, so it's a regression
• Doesn't happen in 7.11.0 local deployments
• A first discussion with @AlexP-Elastic on slack showed that's the problem is likely to be on the Kibana side

[rashmivkulkarni]

cc @elastic/kibana-security

[rashmivkulkarni]

able to reproduce it consistently on 7.11 cloud deployment BC3

[jportner]

Interestingly enough, when using Stack Management > Users to change a password:

• Changing your own password yields the same 404 error
• Changing another user's password works fine

[rashmivkulkarni]

This has been caught by the cloud test failure : #86520

[jportner]

This appears to be a problem with the Cloud proxy. We cannot reproduce this locally in Kibana, only in the Cloud environment.

We tested the following scenarios within Kibana:

• Account Management : Change your password, results in 404 error
• Stack Management > Users : Change your own password, results in 404 error
• Stack Management > Users : Change someone else's password, results in success
• Dev Tools : Change your own password, results in success
• Dev Tools : Change another user's password, results in success

We enabled verbose logging and audit logging in Elasticsearch, and we also enabled verbose logging in Kibana.

In the failure cases, Kibana logs showed the 40, but there are no logs on Elasticsearch indicating that the request was received. We also checked the Elasticsearch proxy logs (go to Discover and search for _password, which is the HTTP route in question). We verified that successful requests are logged by the Elasticsearch proxy, but failed requests are not.

From the Kibana perspective, the only difference between the two failure cases and the three success cases is that the failure cases include a request authorization header. See the code snippet here:

kibana/x-pack/plugins/security/server/routes/users/change_password.ts

Lines 33 to 72 in 9eb993a

	createLicensedRouteHandler(async (context, request, response) => {
	const { username } = request.params;
	const { password: currentPassword, newPassword } = request.body;
	const currentUser = getAuthenticationService().getCurrentUser(request);
	const isUserChangingOwnPassword =
	currentUser && currentUser.username === username && canUserChangePassword(currentUser);
	const currentSession = isUserChangingOwnPassword ? await session.get(request) : null;
	// If user is changing their own password they should provide a proof of knowledge their
	// current password via sending it in `Authorization: Basic base64(username:current password)`
	// HTTP header no matter how they logged in to Kibana.
	const options = isUserChangingOwnPassword
	? {
	headers: {
	...request.headers,
	authorization: new HTTPAuthorizationHeader(
	'Basic',
	new BasicHTTPAuthorizationHeaderCredentials(
	username,
	currentPassword || ''
	).toString()
	).toString(),
	},
	}
	: undefined;
	try {
	await context.core.elasticsearch.client.asCurrentUser.security.changePassword(
	{ username, body: { password: newPassword } },
	options
	);
	} catch (error) {
	// This may happen only if user's credentials are rejected meaning that current password
	// isn't correct.
	if (isUserChangingOwnPassword && getErrorStatusCode(error) === 401) {
	return response.forbidden({ body: error });
	}
	return response.customError(wrapIntoCustomErrorResponse(error));

[AlexP-Elastic]

This appears to be a problem with the Cloud proxy.

Is there any evidence (eg in the Kibana logs) that Kibana issued the request that is never received by ES? (Eg if the proxy was the issue presumably the request errors out in Kibana - is the error logged?)

The proxy definitely doesn't parse the header so it would be very unexpected if the presence or not of an auth header changes the HTTP-level behavior within Cloud.

Also the proxy doesn't know what version its routing to, so if a given request works with 7.10 it will always work with 7.11 (unless of course the request has changed from 7.10 to 7.11,)

[jportner]

Is there any evidence (eg in the Kibana logs) that Kibana issued the request that is never received by ES? (Eg if the proxy was the issue presumably the request errors out in Kibana - is the error logged?)

Yes, Kibana makes a call to the Elasticsearch client which returns a 404 error. This is present in the Kibana logs, but the same request does not show up in the Elasticsearch proxy logs.

    "message": "404\nPUT /_security/user/foo/_password\n{\"password\":\"changeme2\"} [undefined]: Response Error",

The proxy definitely doesn't parse the header so it would be very unexpected if the presence or not of an auth header changes the HTTP-level behavior within Cloud.

Also the proxy doesn't know what version its routing to, so if a given request works with 7.10 it will always work with 7.11 (unless of course the request has changed from 7.10 to 7.11,)

We did start using the new ES client in 7.11 (PR #84528), but it is making the same request. And it works locally, without a proxy in front of ES.

[AlexP-Elastic]

Can you get the exact body and curl it directly at the ES deployment? (ideally for 7.10 which works and 7.11 which doesn't)

(I'm struggling to come up with an explanation that: given an identical request X, if you send X to a 7.10 ES cluster it works, but if you send X to a 7.11 cluster then the proxy fails. It seems way more likely X is different.)

[jportner]

Can you get the exact body and curl it directly at the ES deployment? (ideally for 7.10 which works and 7.11 which doesn't)

We did this in the test case above "Dev Tools : Change your own password, results in success", it's a short request:

PUT /_security/user/foo/_password
{ "password": "changeme" }

It does work in 7.11 through Kibana dev tools (and we see that request in the proxy logs). It doesn't work when changing your own password via Kibana's ES client (we don't see the request in the proxy logs).

Edit: I recreated the same command to my staging deployment using curl:

curl -XPUT "https://ELASTICSEARCH-ENDPOINT:9243/_security/user/foo/_password" -H 'Content-Type: application/json' -H 'Authorization: Basic BASE64-ENCODED-CREDENTIALS' -d'{ "password": "changeme" }'

It results in a 200 success.

[AlexP-Elastic]

I looked in the proxy application error logs and couldn't find anything that lined up with the counts/times for 404s occurring. There were a few "content-length is wrong" errors and a few "connection reset by peer" errors at similar, but I don't think they are related.

It might be necessary to run a local Kibana hitting a Cloud ES so you can see exactly what the ES client is doing?

[jportner]

It might be necessary to run a local Kibana hitting a Cloud ES so you can see exactly what the ES client is doing?

OK, good idea, I'll try that and report back here, thanks.

[jportner]

It appears that the source of our problems is this line:

kibana/x-pack/plugins/security/server/routes/users/change_password.ts

Line 48 in 9eb993a

...request.headers,

It is intended to ensure that the password is provided in the Authorization header, regardless of what authentication provider is used (e.g., Token provider). However, it has the unintended side effect of including all of the end user's request headers, and overwriting some of the ES client's request headers.

I tested and verified this by running Kibana locally, connecting to a cloud ES instance, and intercepting the HTTPS traffic. Here is a comparison of testuser attempting to change his own password (left, failure) vs attempting to change someone else's password (right, success):

I believe the correct fix is to remove the specified line. The authorization header is the only one we need to include in the ES client options. Will open a PR shortly.

[azasypkin]

Good find @jportner!

@AlexP-Elastic is there any way to track down issues like this via proxy or any other available type of logs (ES API endpoint in the Cloud is being accessed with an invalid/not-allowed Host HTTP header)? Or such requests are just silently discarded by the proxy?

[AlexP-Elastic]

@azasypkin this issue did make it clear that we need to log the "no matching id", we discussed a bit in cloud slack - @dtuck9 when you get a moment could you create an issue?