Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unquote operators of filelog receiver error #29325

Closed
freshchen opened this issue Nov 17, 2023 · 5 comments
Closed

Unquote operators of filelog receiver error #29325

freshchen opened this issue Nov 17, 2023 · 5 comments
Labels
bug Something isn't working receiver/filelog waiting for author

Comments

@freshchen
Copy link

Component(s)

No response

What happened?

Description

helm： 0.73.1
app version：0.88.0
CRI：docker

otel filelog receiver -> loki

I get below log in grafana

{"body":"2023-11-17 21:03:26,830 [appEnv=web-empty.app instance=web-empty-564cd5f7d9-f25dh tid=0d17ab91fb81cb1da5eead2866add518 sid=47a1447df7b57517] [http-nio-8080-exec-5][INFO] c.b.m.cloudnative.web.log.HttpLog.? - Resp httpCode=200 ; body={\"status\":\"UP\"} ; cost=total:2ms\n"}

and i want to remove \n and Escape，then i try Unquote

  • type: move
    from: attributes.log
    to: "body"
  • type: unquote
    field: body

I didn’t encounter this problem when using promtail before

Expected Result

{"body":"2023-11-17 21:03:26,830 [appEnv=web-empty.app instance=web-empty-564cd5f7d9-f25dh tid=0d17ab91fb81cb1da5eead2866add518 sid=47a1447df7b57517] [http-nio-8080-exec-5][INFO] c.b.m.cloudnative.web.log.HttpLog.? - Resp httpCode=200 ; body={"status":"UP"} ; cost=total:2ms"}

Actual Result

error

Collector version

0.88.0

Environment information

Environment

CRI：docker
k8s: 1.24

OpenTelemetry Collector configuration

No response

Log output

2023-11-17T13:19:06.102Z	error	helper/transformer.go:98	Failed to process entry	{"kind": "receiver", "name": "filelog", "data_type": "logs", "operator_id": "unquote", "operator_type": "unquote", "error": "invalid syntax", "action": "send", "entry": {"observed_timestamp":"2023-11-17T13:19:06.102295254Z","timestamp":"2023-11-17T13:19:05.729587074Z","body":"2023-11-17T13:19:05.729Z\tinfo\tservice@v0.88.0/telemetry.go:84\tSetting up own telemetry...\n","attributes":{"log.file.path":"/var/log/pods/observability_opentelemetry-collector-agent-7cth6_b16b3e67-213a-430f-844f-471a7f7ec526/opentelemetry-collector/0.log","restart_count":"0","stream":"stderr","time":"2023-11-17T13:19:05.729587074Z","uid":"b16b3e67-213a-430f-844f-471a7f7ec526"},"resource":{"app":"opentelemetry-collector","namespace":"observability","pod":"opentelemetry-collector-agent-7cth6"},"severity":0,"scope_name":""}}
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*TransformerOperator).HandleEntryError
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/transformer.go:98
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*TransformerOperator).ProcessWith
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/transformer.go:90
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/unquote.(*Transformer).Process
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/transformer/unquote/unquote.go:63
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*WriterOperator).Write
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/writer.go:53
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*TransformerOperator).ProcessWith
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/transformer.go:92
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/move.(*Transformer).Process
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/transformer/move/move.go:69
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*WriterOperator).Write
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/writer.go:53
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*TransformerOperator).ProcessWith
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/transformer.go:92
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/move.(*Transformer).Process
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/transformer/move/move.go:69
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*WriterOperator).Write
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/writer.go:53
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*TransformerOperator).ProcessWith
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/transformer.go:92
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/move.(*Transformer).Process
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/transformer/move/move.go:69
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*WriterOperator).Write
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/writer.go:53
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*TransformerOperator).ProcessWith
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/transformer.go:92
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/move.(*Transformer).Process
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/transformer/move/move.go:69
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*WriterOperator).Write
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/writer.go:53
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ProcessWithCallback
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/parser.go:122
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ProcessWith
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/parser.go:98
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/regex.(*Parser).Process
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/parser/regex/regex.go:99
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*WriterOperator).Write
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/writer.go:53
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ProcessWithCallback
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/parser.go:122
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ProcessWith
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/parser.go:98
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/json.(*Parser).Process
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/parser/json/json.go:62
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/router.(*Transformer).Process
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/transformer/router/router.go:130
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*WriterOperator).Write
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/writer.go:53
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/input/file.(*Input).emit
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/input/file/file.go:52
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer/internal/reader.(*Reader).ReadToEnd
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/fileconsumer/internal/reader/reader.go:106
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).consume.func1
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/fileconsumer/file.go:156

Additional context

No response
@freshchen freshchen added bug Something isn't working needs triage New item requiring triage labels Nov 17, 2023
@github-actions GitHub Actions
Copy link
Contributor

Pinging code owners for receiver/filelog: @djaglowski. See Adding Labels via Comments if you do not have permissions to add labels yourself.

@crobert-1
Copy link
Member

Hello @freshchen, can you provide the full configuration of the OpenTelemetry Collector that you're using, as well as an example log file that is hitting this error? This would help us give you more direction on what's going wrong here.

@github-actions github-actions bot mentioned this issue Nov 21, 2023
Closed
@djaglowski
Copy link
Member

body={\"status\":\"UP\"}

Is this the part you are trying to unquote? This appears to be a json string, so you would need to use json_parser.

@github-actions github-actions bot mentioned this issue Nov 28, 2023
Closed
98 tasks
@github-actions github-actions bot mentioned this issue Dec 5, 2023
Closed
@djaglowski djaglowski added waiting for author and removed needs triage New item requiring triage labels Dec 6, 2023
@djaglowski
Copy link
Member

Closing as there is no clear action here.

@freshchen
Copy link
Author

Thanks for the answer. Later, I found a way to skip this question.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working receiver/filelog waiting for author
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@djaglowski @freshchen @crobert-1