Add telemetry for LRU cache

[clly]

Vault creates an LRU cache that is used when interacting with the
physical backend. Add telemetry when the cache is hit, missed, written
to and deleted from.

[hashicorp-cla]

All committers have signed the CLA.

[mgritter]

The new counters should be documented in vault/website/pages/docs/internals/telemetry.mdx as part of the same PR.

[mgritter]

These build failures:

vault/core_util.go:37:46: too many arguments in call to physical.NewTransactionalCache
	have (physical.Backend, int, hclog.Logger, *metricsutil.ClusterMetricSink)
	want (physical.Backend, int, hclog.Logger)

are probably the result of the changes in SDK not making it to the vendor directory (this is a difference between local builds and CI.) Try "go mod vendor" and see if CI is happier.

[mgritter]

Looks like we are good to go; you will need to fix the CI failures before committing (like I said above, this is probably because of changes in sdk)..

[clly]

Thanks for the tip :) It seemed to build locally. But also I didn't realize Go doesn't use the vendor directory by default. I should have the fix in tonight.

[clly]

I ran go mod vendor and added what got put in there. One of the things that changed was the go.mod and go.sum for vault/api I wasn't 100% sure if it should get added but since it came from the command I figured yes?

I also fixed a metric -> metricSink change I missed and passed the correct MetricSink to the caches

[mgritter]

I ran go mod vendor and added what got put in there. One of the things that changed was the go.mod and go.sum for vault/api I wasn't 100% sure if it should get added but since it came from the command I figured yes?

Yes, that's good.

I also fixed a metric -> metricSink change I missed and passed the correct MetricSink to the caches

Looks like cache_test.go needs to be fixed too, based on the latest CI run?

[clly]

Makes sense to me. I forgot to update the tests after the latest change. I satisfied the requirement for a MetricSink by providing a BlackholeSink. If prefered I could attempt to mock the MetricSink interface to ensure that the expected methods are called.

[mgritter]

Looks good! You will need to merge in the master branch and resolve any conflicts, but if CI passes then you should be able to squash and merge (and if not, I can.) We're doing a feature freeze tomorrow so it'd be good to get these changes in so they can hit the next release.

[clly]

I ended up doing the rebase and squash first and then the merge. I tried a rebase on the branch instead which didn't do what I expected. I had to force push to try to fix up the changes, but I did the merge to master. I think it's right. If I messed it please let me know how but feel free to fixup the issues. Thank you so much for working through this with me.

[mgritter]

All set! I will add a Changelong entry for this.