Error 403 forbidden

[ccoqueiro]

Component(s)

No response

What happened?

Description

When trying to configure sampexporter and signalfxexporter returning error:

{"level":"warn","ts":1703621842.8044937,"logger":"lifecycle.manager","msg":"Failed to start the extension","error":"unable to start, otelcol state is 3"}{"level":"info","ts":1703621842.8054059,"msg":"done","error":"unable to start, otelcol state is 3; request failed with status 403 Forbidden","errorCauses":[{"error":"unable to start, otelcol state is 3"},{"error":"request failed with status 403 Forbidden"}]}EXTENSION Name: collector State: Ready Events: [INVOKE, SHUTDOWN]

Steps to Reproduce

Setting the collector.yaml file is set to the environment variable: OPENTELEMETRY_COLLECTOR_CONFIG_FILE. The collector.yaml has the following configuration:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "localhost:4317"
      http:
        endpoint: "localhost:4318"
  
exporters:
  sapm:
    access_token: "access_token"
    endpoint: "https://ingest.us1.signalfx.com/v2/trace"
    max_connections: 100
    num_workers: 8
  signalfx:
    access_token: "<access_token>"
    realm: "us1"
    send_compatible_metrics: true
  logging:
    loglevel: "debug"

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [logging,sapm]
    metrics:
      receivers: [otlp]
      exporters: [logging,signalfx]

Expected Result

Export Traces and Metrics to Splunk Cloud Observability.

Actual Result

:[{"error":"unable to start, otelcol state is 3"},{"error":"request failed with status 403 Forbidden"}

Collector version

latest

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")
AWS Lambda NodeJS v18

OpenTelemetry Collector configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "localhost:4317"
      http:
        endpoint: "localhost:4318"
  
exporters:
  sapm:
    access_token: "access_token"
    endpoint: "https://ingest.us1.signalfx.com/v2/trace"
    max_connections: 100
    num_workers: 8
  signalfx:
    access_token: "<access_token>"
    realm: "us1"
    send_compatible_metrics: true
  logging:
    loglevel: "debug"

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [logging,sapm]
    metrics:
      receivers: [otlp]
      exporters: [logging,signalfx]

Log output

INIT_START Runtime Version: nodejs:18.v18	Runtime Version ARN: arn:aws:lambda:us-east-1::runtime:d949ec0248c1b8fbc8cbc6df1c986e05fb933506b51d53be82d46f9a37125bea
{
    "level": "info",
    "ts": 1703622337.6173074,
    "msg": "Launching OpenTelemetry Lambda extension",
    "version": "v0.35.0"
}

{
    "level": "info",
    "ts": 1703622337.632078,
    "logger": "telemetryAPI.Listener",
    "msg": "Listening for requests",
    "address": "sandbox:53612"
}

{
    "level": "info",
    "ts": 1703622337.633342,
    "logger": "telemetryAPI.Client",
    "msg": "Subscribing",
    "baseURL": "http://127.0.0.1:9001/2022-07-01/telemetry"
}

TELEMETRY	Name: collector	State: Subscribed	Types: [Platform]
2023/12/26 20:25:37 attn: users of the prometheusremotewrite exporter please refer to https://github.com/aws-observability/aws-otel-collector/issues/2043 in regards to an ADOT Collector v0.31.0 breaking change
{
    "level": "info",
    "ts": 1703622337.6362078,
    "logger": "telemetryAPI.Client",
    "msg": "Subscription success",
    "response": "\"OK\""
}

{
    "level": "info",
    "ts": 1703622337.6362562,
    "logger": "NewCollector",
    "msg": "Using config URI from environment",
    "uri": "/var/task/collector.yaml"
}

{
    "level": "warn",
    "ts": 1703622337.695783,
    "logger": "lifecycle.manager",
    "msg": "Failed to start the extension",
    "error": "unable to start, otelcol state is 3"
}

{
    "level": "info",
    "ts": 1703622337.6975195,
    "msg": "done",
    "error": "unable to start, otelcol state is 3; request failed with status 403 Forbidden",
    "errorCauses": [
        {
            "error": "unable to start, otelcol state is 3"
        },
        {
            "error": "request failed with status 403 Forbidden"
        }
    ]
}

EXTENSION	Name: collector	State: Ready	Events: [INVOKE, SHUTDOWN]
INIT_REPORT Init Duration: 305.47 ms	Phase: init	Status: error	Error Type: Extension.Crash
EXTENSION	Name: collector	State: Started	Events: []
INIT_REPORT Init Duration: 3009.24 ms	Phase: invoke	Status: error	Error Type: Runtime.Unknown
START RequestId: 471b6343-045a-41ec-aec1-0e113515d1f4 Version: $LATEST
2023-12-26T20:25:40.810Z 471b6343-045a-41ec-aec1-0e113515d1f4 Task timed out after 3.09 seconds

END RequestId: 471b6343-045a-41ec-aec1-0e113515d1f4
REPORT RequestId: 471b6343-045a-41ec-aec1-0e113515d1f4	Duration: 3089.69 ms	Billed Duration: 3000 ms	Memory Size: 128 MB	Max Memory Used: 37 MB	
XRAY TraceId: 1-658b36c1-0a0122fa67f5885d0c24e98d	SegmentId: 323024ea323df1eb	Sampled: true	
INIT_START Runtime Version: nodejs:18.v18	Runtime Version ARN: arn:aws:lambda:us-east-1::runtime:d949ec0248c1b8fbc8cbc6df1c986e05fb933506b51d53be82d46f9a37125bea
{
    "level": "info",
    "ts": 1703622341.0623188,
    "msg": "Launching OpenTelemetry Lambda extension",
    "version": "v0.35.0"
}

{
    "level": "info",
    "ts": 1703622341.083076,
    "logger": "telemetryAPI.Listener",
    "msg": "Listening for requests",
    "address": "sandbox:53612"
}

{
    "level": "info",
    "ts": 1703622341.0831897,
    "logger": "telemetryAPI.Client",
    "msg": "Subscribing",
    "baseURL": "http://127.0.0.1:9001/2022-07-01/telemetry"
}

TELEMETRY	Name: collector	State: Subscribed	Types: [Platform]
{
    "level": "info",
    "ts": 1703622341.0854735,
    "logger": "telemetryAPI.Client",
    "msg": "Subscription success",
    "response": "\"OK\""
}

{
    "level": "info",
    "ts": 1703622341.0855327,
    "logger": "NewCollector",
    "msg": "Using config URI from environment",
    "uri": "/var/task/collector.yaml"
}

2023/12/26 20:25:41 attn: users of the prometheusremotewrite exporter please refer to https://github.com/aws-observability/aws-otel-collector/issues/2043 in regards to an ADOT Collector v0.31.0 breaking change
{
    "level": "warn",
    "ts": 1703622341.1193042,
    "logger": "lifecycle.manager",
    "msg": "Failed to start the extension",
    "error": "unable to start, otelcol state is 3"
}

{
    "level": "info",
    "ts": 1703622341.1307032,
    "msg": "done",
    "error": "unable to start, otelcol state is 3; request failed with status 403 Forbidden",
    "errorCauses": [
        {
            "error": "unable to start, otelcol state is 3"
        },
        {
            "error": "request failed with status 403 Forbidden"
        }
    ]
}

EXTENSION	Name: collector	State: Ready	Events: [INVOKE, SHUTDOWN]
INIT_REPORT Init Duration: 283.23 ms	Phase: init	Status: error	Error Type: Extension.Crash

Additional context

No response

[bryan-aguilar]

I see you are using one of the ADOT Lambda Layers. Can you please move this issue to https://github.com/aws-observability/aws-otel-lambda/issues instead? Sorry, I cannot transfer this issue across GitHub organizations.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[atoulme]

If you need any help regarding a vendor ingest, please open a support case directly against the vendor. For Splunk, you can start here: https://splunk.my.site.com/customer/s/need-help/create-case

I am going to close this issue as it is best to follow with the AWS distribution or the vendor.