Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

splunkhecexporter panic #34357

Closed
PaulBernier opened this issue Jul 31, 2024 · 2 comments · Fixed by #34507
Closed

splunkhecexporter panic #34357

PaulBernier opened this issue Jul 31, 2024 · 2 comments · Fixed by #34507
Labels
bug Something isn't working exporter/splunkhec

Comments

@PaulBernier
Copy link
Contributor

PaulBernier commented Jul 31, 2024

Component(s)

exporter/splunkhec

What happened?

Description

Our otel-collector panicked with the following stacktrace:

	/usr/local/go/src/net/http/transport.go:1800 +0x1585
created by net/http.(*Transport).dialConn in goroutine 176037
	/usr/local/go/src/net/http/transport.go:2447 +0x18e
net/http.(*persistConn).writeLoop(0xc0012d9e60)
	/usr/local/go/src/net/http/request.go:755 +0xb0d
net/http.(*Request).write(0xc0015c3c20, {0x1962520, 0xc0003ded00}, 0x0, 0xc001877ce0, 0x0)
	/usr/local/go/src/net/http/transfer.go:385 +0x426
net/http.(*transferWriter).writeBody(0xc0005c60a0, {0x1962520?, 0xc0003ded00?})
	/usr/local/go/src/net/http/transport.go:662 +0x22
net/http.(*readTrackingBody).Close(0xc0005c60a0?)
	/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.52.0/wrap.go:37 +0x1b
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*bodyWrapper).Close(0xc0017b4be0?)
	/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter@v0.104.0/buffer.go:117 +0x17
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter.(*cancellableGzipWriter).Close(0xc000918000?)
	/usr/local/go/src/compress/gzip/gzip.go:242 +0x7a
compress/gzip.(*Writer).Close(0xc001298000)
	/usr/local/go/src/compress/flate/deflate.go:730
compress/flate.(*Writer).Close(...)
	/usr/local/go/src/compress/flate/deflate.go:638 +0x6a
compress/flate.(*compressor).close(0xc001f2c000)
	/usr/local/go/src/compress/flate/deflate.go:408 +0x618
compress/flate.(*compressor).deflate(0xc001f2c000)
	/usr/local/go/src/compress/flate/deflate.go:170 +0x9d
compress/flate.(*compressor).writeBlock(0xc001f2c000, {0xc001552000?, 0xc001918f01?, 0xc000473bc0?}, 0xc0017b4b50?)
	/usr/local/go/src/compress/flate/huffman_bit_writer.go:437 +0xbe
compress/flate.(*huffmanBitWriter).writeBlock(0xc0005e32c0, {0xc001552000?, 0x80?, 0x80?}, 0x0, {0xc00140b1d7, 0x0, 0xee29})
	/usr/local/go/src/compress/flate/huffman_bit_writer.go:548 +0x2a5
compress/flate.(*huffmanBitWriter).indexTokens(0xc0005e32c0, {0xc001552000, 0x4e0, 0x47ecbd?})
goroutine 175868 [running]:
panic: runtime error: index out of range [43] with length 30

I suspect a race condition (it rarely occurs), and possibly a variation of the issue previously reported here #34255 But here the symptom is different/more sever.

Collector version

v0.104.0

Environment information

Environment

alpine linux
Go 1.22

OpenTelemetry Collector configuration

exporters:
  splunk_hec/1:
    token: "{{.hecToken}}"
    endpoint: "{{.hecEndpoint}}/services/collector/raw?index={{.splunkIndex}}&sourcetype={{.splunkSourceType}}&source=eventhub://{{.eventHubFullyQualifiedNamespace}}/{{.eventHubName}}
    sourcetype: "{{.splunkSourceType}}"
    index: "{{.splunkIndex}}"
    export_raw: true
    max_content_length_logs: {{.exporterMaxContentLengthLogs}}
    retry_on_failure:
      enabled: false
    sending_queue:
      enabled: false
    hec_metadata_to_otel_attrs:
      source: source
      host: host
    tls:
      min_version: 1.2

Log output

No response

Additional context

No response

@PaulBernier PaulBernier added bug Something isn't working needs triage New item requiring triage labels Jul 31, 2024
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@atoulme atoulme removed the needs triage New item requiring triage label Aug 1, 2024
@atoulme
Copy link
Contributor

atoulme commented Aug 1, 2024

Looks like the same issue and the same fix is needed.

f7o pushed a commit to f7o/opentelemetry-collector-contrib that referenced this issue Sep 12, 2024
…n reuse (open-telemetry#34507)

**Description:**
This bug is a manifestation of
golang/go#51907.
Under high load, the pool of buffers used to send requests is reused
enough
that the same buffer is used concurrently to process data and be sent as
request body.
The fix is to copy the payload into a new byte array before sending it.

**Link to tracking Issue:**
Fixes open-telemetry#34357

This change is markedly bad for performance but I don't see any way to
avoid it.

```
goos: darwin
goarch: arm64
pkg: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter
                                                     │  before.txt  │               after.txt                │
                                                     │    sec/op    │    sec/op      vs base                 │
_pushLogData_10_10_1024-10                             85.52µ ± ∞ ¹   107.67µ ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushLogData_10_10_8K-10                               76.02µ ± ∞ ¹   149.42µ ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushLogData_10_10_2M-10                               74.12µ ± ∞ ¹   108.30µ ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushLogData_10_200_2M-10                              1.519m ± ∞ ¹    2.519m ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushLogData_100_200_2M-10                             14.23m ± ∞ ¹    18.18m ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushLogData_100_200_5M-10                             13.87m ± ∞ ¹    17.96m ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushLogData_compressed_10_10_1024-10                  578.1µ ± ∞ ¹    947.0µ ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushLogData_compressed_10_10_8K-10                    134.7µ ± ∞ ¹    229.3µ ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushLogData_compressed_10_10_2M-10                    134.3µ ± ∞ ¹    193.0µ ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushLogData_compressed_10_200_2M-10                   3.043m ± ∞ ¹    3.950m ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushLogData_compressed_100_200_2M-10                  31.01m ± ∞ ¹    35.64m ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushLogData_compressed_100_200_5M-10                  31.23m ± ∞ ¹    36.80m ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_10_10_1024-10                          159.8µ ± ∞ ¹    176.7µ ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_10_10_8K-10                            162.2µ ± ∞ ¹    188.0µ ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_10_10_2M-10                            328.0µ ± ∞ ¹    520.0µ ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_10_200_2M-10                           3.282m ± ∞ ¹    3.725m ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_100_200_2M-10                          31.02m ± ∞ ¹    37.29m ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_100_200_5M-10                          31.39m ± ∞ ¹    37.30m ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_10_1024-10               248.3µ ± ∞ ¹    321.8µ ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_10_8K-10                 246.9µ ± ∞ ¹    316.1µ ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_10_2M-10                 414.4µ ± ∞ ¹    609.7µ ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_200_2M-10                4.918m ± ∞ ¹    6.390m ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_100_200_2M-10               46.97m ± ∞ ¹    66.91m ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_100_200_5M-10               47.10m ± ∞ ¹    60.84m ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_10_10_1024_MultiMetric-10              298.3µ ± ∞ ¹    546.6µ ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_10_10_8K_MultiMetric-10                303.6µ ± ∞ ¹    567.6µ ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_10_10_2M_MultiMetric-10                299.4µ ± ∞ ¹    612.3µ ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_10_200_2M_MultiMetric-10               5.933m ± ∞ ¹    8.737m ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_100_200_2M_MultiMetric-10              57.02m ± ∞ ¹    81.24m ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_100_200_5M_MultiMetric-10              57.12m ± ∞ ¹    77.35m ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_10_1024_MultiMetric-10   383.8µ ± ∞ ¹    625.9µ ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_10_8K_MultiMetric-10     399.6µ ± ∞ ¹    570.0µ ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_10_2M_MultiMetric-10     392.9µ ± ∞ ¹    644.6µ ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_200_2M_MultiMetric-10    7.560m ± ∞ ¹   12.370m ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_100_200_2M_MultiMetric-10   72.72m ± ∞ ¹   100.91m ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_100_200_5M_MultiMetric-10   81.60m ± ∞ ¹    88.94m ± ∞ ¹        ~ (p=1.000 n=1) ²
ConsumeLogsRejected-10                                 772.2µ ± ∞ ¹   1041.0µ ± ∞ ¹        ~ (p=1.000 n=1) ²
geomean                                                1.930m          2.723m        +41.10%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05

                                                     │  before.txt   │               after.txt               │
                                                     │     B/op      │     B/op       vs base                │
_pushLogData_10_10_1024-10                             73.12Ki ± ∞ ¹   73.11Ki ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushLogData_10_10_8K-10                               63.79Ki ± ∞ ¹   63.75Ki ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushLogData_10_10_2M-10                               63.19Ki ± ∞ ¹   63.17Ki ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushLogData_10_200_2M-10                              1.254Mi ± ∞ ¹   1.254Mi ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushLogData_100_200_2M-10                             12.59Mi ± ∞ ¹   12.73Mi ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushLogData_100_200_5M-10                             12.65Mi ± ∞ ¹   12.71Mi ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushLogData_compressed_10_10_1024-10                  2.411Mi ± ∞ ¹   2.415Mi ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushLogData_compressed_10_10_8K-10                    65.22Ki ± ∞ ¹   63.94Ki ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushLogData_compressed_10_10_2M-10                    65.26Ki ± ∞ ¹   64.83Ki ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushLogData_compressed_10_200_2M-10                   1.253Mi ± ∞ ¹   1.259Mi ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushLogData_compressed_100_200_2M-10                  12.54Mi ± ∞ ¹   12.54Mi ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushLogData_compressed_100_200_5M-10                  12.54Mi ± ∞ ¹   12.55Mi ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_10_10_1024-10                          81.56Ki ± ∞ ¹   81.49Ki ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_10_10_8K-10                            88.50Ki ± ∞ ¹   88.49Ki ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_10_10_2M-10                            2.081Mi ± ∞ ¹   2.081Mi ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_10_200_2M-10                           3.563Mi ± ∞ ¹   3.564Mi ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_100_200_2M-10                          19.76Mi ± ∞ ¹   19.79Mi ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_100_200_5M-10                          25.76Mi ± ∞ ¹   25.78Mi ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_10_1024-10               83.96Ki ± ∞ ¹   82.30Ki ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_10_8K-10                 89.35Ki ± ∞ ¹   89.29Ki ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_10_2M-10                 2.081Mi ± ∞ ¹   2.083Mi ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_200_2M-10                3.564Mi ± ∞ ¹   3.564Mi ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_compressed_100_200_2M-10               17.69Mi ± ∞ ¹   17.71Mi ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_compressed_100_200_5M-10               20.70Mi ± ∞ ¹   20.70Mi ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_10_10_1024_MultiMetric-10              150.6Ki ± ∞ ¹   150.5Ki ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_10_10_8K_MultiMetric-10                150.5Ki ± ∞ ¹   150.5Ki ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_10_10_2M_MultiMetric-10                150.7Ki ± ∞ ¹   150.5Ki ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_10_200_2M_MultiMetric-10               2.936Mi ± ∞ ¹   2.938Mi ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_100_200_2M_MultiMetric-10              29.39Mi ± ∞ ¹   29.45Mi ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_100_200_5M_MultiMetric-10              29.38Mi ± ∞ ¹   29.47Mi ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_10_1024_MultiMetric-10   151.0Ki ± ∞ ¹   150.8Ki ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_10_8K_MultiMetric-10     151.0Ki ± ∞ ¹   151.2Ki ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_10_2M_MultiMetric-10     151.0Ki ± ∞ ¹   151.3Ki ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_200_2M_MultiMetric-10    2.936Mi ± ∞ ¹   2.939Mi ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_compressed_100_200_2M_MultiMetric-10   29.25Mi ± ∞ ¹   29.27Mi ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_compressed_100_200_5M_MultiMetric-10   29.26Mi ± ∞ ¹   29.27Mi ± ∞ ¹       ~ (p=1.000 n=1) ²
ConsumeLogsRejected-10                                 641.6Ki ± ∞ ¹   640.9Ki ± ∞ ¹       ~ (p=1.000 n=1) ²
geomean                                                1.231Mi         1.230Mi        -0.04%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05

                                                     │  before.txt  │              after.txt               │
                                                     │  allocs/op   │  allocs/op    vs base                │
_pushLogData_10_10_1024-10                              821.0 ± ∞ ¹    821.0 ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushLogData_10_10_8K-10                                716.0 ± ∞ ¹    716.0 ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushLogData_10_10_2M-10                                709.0 ± ∞ ¹    709.0 ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushLogData_10_200_2M-10                              14.02k ± ∞ ¹   14.02k ± ∞ ¹       ~ (p=1.000 n=1) ³
_pushLogData_100_200_2M-10                             140.0k ± ∞ ¹   140.0k ± ∞ ¹       ~ (p=1.000 n=1) ³
_pushLogData_100_200_5M-10                             140.0k ± ∞ ¹   140.0k ± ∞ ¹       ~ (p=1.000 n=1) ³
_pushLogData_compressed_10_10_1024-10                   782.0 ± ∞ ¹    783.0 ± ∞ ¹       ~ (p=1.000 n=1) ³
_pushLogData_compressed_10_10_8K-10                     709.0 ± ∞ ¹    709.0 ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushLogData_compressed_10_10_2M-10                     709.0 ± ∞ ¹    709.0 ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushLogData_compressed_10_200_2M-10                   14.02k ± ∞ ¹   14.02k ± ∞ ¹       ~ (p=1.000 n=1) ³
_pushLogData_compressed_100_200_2M-10                  140.0k ± ∞ ¹   140.0k ± ∞ ¹       ~ (p=1.000 n=1) ³
_pushLogData_compressed_100_200_5M-10                  140.0k ± ∞ ¹   140.0k ± ∞ ¹       ~ (p=1.000 n=1) ³
_pushMetricData_10_10_1024-10                          1.410k ± ∞ ¹   1.410k ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_10_10_8K-10                            1.410k ± ∞ ¹   1.410k ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_10_10_2M-10                            1.417k ± ∞ ¹   1.417k ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_10_200_2M-10                           28.02k ± ∞ ¹   28.02k ± ∞ ¹       ~ (p=1.000 n=1) ³
_pushMetricData_100_200_2M-10                          280.1k ± ∞ ¹   280.1k ± ∞ ¹       ~ (p=1.000 n=1) ³
_pushMetricData_100_200_5M-10                          280.1k ± ∞ ¹   280.0k ± ∞ ¹       ~ (p=1.000 n=1) ³
_pushMetricData_compressed_10_10_1024-10               1.410k ± ∞ ¹   1.410k ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_10_8K-10                 1.410k ± ∞ ¹   1.410k ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_10_2M-10                 1.418k ± ∞ ¹   1.418k ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_200_2M-10                28.02k ± ∞ ¹   28.02k ± ∞ ¹       ~ (p=1.000 n=1) ³
_pushMetricData_compressed_100_200_2M-10               280.0k ± ∞ ¹   280.0k ± ∞ ¹       ~ (p=1.000 n=1) ³
_pushMetricData_compressed_100_200_5M-10               280.0k ± ∞ ¹   280.0k ± ∞ ¹       ~ (p=1.000 n=1) ³
_pushMetricData_10_10_1024_MultiMetric-10              1.835k ± ∞ ¹   1.835k ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_10_10_8K_MultiMetric-10                1.835k ± ∞ ¹   1.835k ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_10_10_2M_MultiMetric-10                1.835k ± ∞ ¹   1.835k ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_10_200_2M_MultiMetric-10               36.07k ± ∞ ¹   36.07k ± ∞ ¹       ~ (p=1.000 n=1) ³
_pushMetricData_100_200_2M_MultiMetric-10              360.4k ± ∞ ¹   360.4k ± ∞ ¹       ~ (p=1.000 n=1) ³
_pushMetricData_100_200_5M_MultiMetric-10              360.4k ± ∞ ¹   360.4k ± ∞ ¹       ~ (p=1.000 n=1) ³
_pushMetricData_compressed_10_10_1024_MultiMetric-10   1.835k ± ∞ ¹   1.835k ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_10_8K_MultiMetric-10     1.835k ± ∞ ¹   1.835k ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_10_2M_MultiMetric-10     1.835k ± ∞ ¹   1.835k ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_200_2M_MultiMetric-10    36.07k ± ∞ ¹   36.07k ± ∞ ¹       ~ (p=1.000 n=1) ³
_pushMetricData_compressed_100_200_2M_MultiMetric-10   360.4k ± ∞ ¹   360.4k ± ∞ ¹       ~ (p=1.000 n=1) ³
_pushMetricData_compressed_100_200_5M_MultiMetric-10   360.4k ± ∞ ¹   360.4k ± ∞ ¹       ~ (p=1.000 n=1) ³
ConsumeLogsRejected-10                                 7.015k ± ∞ ¹   7.014k ± ∞ ¹       ~ (p=1.000 n=1) ³
geomean                                                11.64k         11.64k        +0.00%
¹ need >= 6 samples for confidence interval at level 0.95
² all samples are equal
³ need >= 4 samples to detect a difference at alpha level 0.05
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working exporter/splunkhec
Projects
None yet
2 participants