Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replaced ClusterMetricSink's cluster name with an atomic.Value. #9252

Merged
merged 4 commits into from
Jun 18, 2020

Conversation

mgritter
Copy link
Contributor

Replaced ClusterMetricSink's cluster name with an atomic.Value. This should permit go-race tests to pass which seal and unseal the core.

Alternatives considered:

  • refer back to Core.cluster every time. This would create a circular dependency if we did it directly.
  • call a function every time. This would break the circular dependency, but seems cumbersome.
  • put a mutex around the cluster name

The approach here seems like the least heavyweight solution to the potential race.

mgritter added 2 commits June 17, 2020 15:31
This should permit go-race tests to pass which seal and unseal
the core.
@mgritter mgritter added this to the 1.5 milestone Jun 17, 2020
@mgritter
Copy link
Contributor Author

Another failure, in an odd place, this is @sgmiller's new code:

				c.metricSink.SetGaugeWithLabels([]string{"core", "unsealed"}, 1, nil)

Does metricSink get modified by seal or unseal?

==================
 WARNING: DATA RACE
 Read at 0x00c002e38278 by goroutine 613: 
  github.com/hashicorp/vault/vault.(*Core).emitMetrics()
      /go/src/github.com/hashicorp/vault/vault/core.go:2107 +0x794
 
Previous write at 0x00c002e38278 by goroutine 738:
  [failed to restore the stack]

Goroutine 613 (running) created at:
  github.com/hashicorp/vault/vault.(*Core).postUnseal()
      /go/src/github.com/hashicorp/vault/vault/core.go:1993 +0x574
  github.com/hashicorp/vault/vault.(*Core).unsealInternal()
      /go/src/github.com/hashicorp/vault/vault/core.go:1485 +0xd75
  github.com/hashicorp/vault/vault.(*Core).unseal()
      /go/src/github.com/hashicorp/vault/vault/core.go:1109 +0x9cf
  github.com/hashicorp/vault/vault.TestCoreUnseal()
      /go/src/github.com/hashicorp/vault/vault/core.go:1022 +0x62
  github.com/hashicorp/vault/vault.testCoreUnsealed()
      /go/src/github.com/hashicorp/vault/vault/testing.go:330 +0x1a1
  github.com/hashicorp/vault/vault.TestCoreUnsealed()
      /go/src/github.com/hashicorp/vault/vault/testing.go:307 +0x9f
  github.com/hashicorp/vault/vault.TestRequestHandling_LoginMetric()
       /go/src/github.com/hashicorp/vault/vault/request_handling_test.go:209 +0x5f
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:909 +0x199
 

Goroutine 738 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:960 +0x651
  testing.runTests.func1()
      /usr/local/go/src/testing/testing.go:1202 +0xa6
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:909 +0x199
  testing.runTests()
      /usr/local/go/src/testing/testing.go:1200 +0x521
  testing.(*M).Run()
      /usr/local/go/src/testing/testing.go:1117 +0x2ff
  main.main()
      _testmain.go:924 +0x223
 
==================

@mgritter
Copy link
Contributor Author

Ah, that test swaps in a different metricSink after the core is unsealed. Need to think about how to do this in a race-free manner.

@@ -59,17 +63,30 @@ func (m *ClusterMetricSink) MeasureSinceWithLabels(key []string, start time.Time
func BlackholeSink() *ClusterMetricSink {
sink, _ := metrics.New(metrics.DefaultConfig(""),
Copy link
Member

@calvn calvn Jun 18, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we do return NewClusterMetricSink("", sink) over here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, good point, I added the second factory later and then forgot to refactor this one. I'll fix that and commit.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On second thought, I'll commit now and fix later.

@mgritter mgritter merged commit 02326b9 into master Jun 18, 2020
@mgritter mgritter deleted the new_metrics_5 branch June 18, 2020 17:56
catsby added a commit that referenced this pull request Jun 19, 2020
* master: (31 commits)
  changelog++
  changelog++
  Ui/replication status discoverability (#8705)
  Update CHANGELOG.md
  Counter that increments on every secret engine lease creation. (#9244)
  Add password_policy field to Azure docs (#9249)
  Replaced ClusterMetricSink's cluster name with an atomic.Value. (#9252)
  Fix database creds rotation panic for nil resp (#9258)
  changelog++
  changelog++
  Move sdk/helper/random -> helper/random (#9226)
  UI: Disallow kv2 with too large 'max versions' value (#9242)
  Allow mTLS for mysql secrets engine (#9181)
  docs: add sample revocation for mongodb (#9245)
  Add new Telemetry config options (#9238)
  Add a simple sealed gauge, updated when seal status changes (#9177)
  Test Shamir-to-Transit and Transit-to-Shamir Seal Migration for post-1.4 Vault. (#9214)
  Configure metrics wrapper with the "global" object, not just the fanout. (#9099)
  changelog++
  Add backend type to audit logs (#9167)
  ...
andaley pushed a commit that referenced this pull request Jul 17, 2020
* Replaced ClusterMetricSink's cluster name with an atomic.Value.
This should permit go-race tests to pass which seal and unseal
the core.

* Replace metric sink before unseal to avoid data races.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants