The ExtractPatterns function doesn't seem to be working

[stephenhong]

Component(s)

pkg/ottl

What happened?

Description

Hello team, I'm testing the transform processor to format my logs and facing some trouble. I have the following log record in a file
16:45:57.603 INFO trace_id=4072391c964eb98bdc913a4f901907ba span_id=7c7a53100483dfa1 trace_flags=01 - Hello World
I use the file log receiver to read this log from the file and export it to the console via debug exporter. Here is the resulting log record

LogRecord #0
ObservedTimestamp: 2023-11-21 15:29:24.861318 +0000 UTC
Timestamp: 1970-01-01 00:00:00 +0000 UTC
SeverityText: INFO
SeverityNumber: Info(9)
Body: Str(16:45:57.603 INFO trace_id=4072391c964eb98bdc913a4f901907ba span_id=7c7a53100483dfa1 trace_flags=01 - Hello World)
Attributes:
     -> log.file.name: Str(app.log)
Trace ID: 
Span ID: 
Flags: 0

Then I want to use the transform processor to extract the span id from the log record and fill in the empty field like this

processors:
  transform:
    error_mode: ignore
    log_statements:
      - context: log
        statements:
          - set(severity_number, SEVERITY_NUMBER_INFO) where IsString(body) and IsMatch(body, "\\sINFO\\s")
          - set(severity_text, "INFO") where IsString(body) and IsMatch(body, "\\sINFO\\s")
          - set(cache, ExtractPatterns(body, "span_id\=(?P<SpanId>[\w]*)")
          - set(span_id.string, cache["SpanId"])

but I get the following error

Error: invalid configuration: processors::transform: unable to parse OTTL statement "set(span_id.string, ExtractPatterns(body, \"span_id\\=(?P<SpanId>[\\w]*)\")": statement has invalid syntax: 1:43: invalid quoted string "\"span_id\\=(?P<SpanId>[\\w]*)\"": invalid syntax
2023/11/21 10:38:16 collector server run finished with error: invalid configuration: processors::transform: unable to parse OTTL statement "set(span_id.string, ExtractPatterns(body, \"span_id\\=(?P<SpanId>[\\w]*)\")": statement has invalid syntax: 1:43: invalid quoted string "\"span_id\\=(?P<SpanId>[\\w]*)\"": invalid syntax

I tried to use different variations of the regex pattern but they all returned invalid syntax error, so I started to wonder if the ExtractPatterns function is working at all...? I also tried
- merge_maps(cache, ExtractPatterns(body, "span_id\=(?P<SpanId>[\w]*)"), "upsert")
but still got the invalid syntax error.

Steps to Reproduce

Set up the transform processor like this

processors:
  transform:
    error_mode: ignore
    log_statements:
      - context: log
        statements:
          - set(severity_number, SEVERITY_NUMBER_INFO) where IsString(body) and IsMatch(body, "\\sINFO\\s")
          - set(severity_text, "INFO") where IsString(body) and IsMatch(body, "\\sINFO\\s")
          - set(cache, ExtractPatterns(body, "span_id\=(?P<SpanId>[\w]*)")
          - set(span_id.string, cache["SpanId"])

then run the collector

Expected Result

span_id field is populated

Actual Result

invalid syntax error

Collector version

0.89.0

Environment information

Environment

OS: Mac OS X 13.6.1
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

processors:
  transform:
    error_mode: ignore
    log_statements:
      - context: log
        statements:
          - set(severity_number, SEVERITY_NUMBER_INFO) where IsString(body) and IsMatch(body, "\\sINFO\\s")
          - set(severity_text, "INFO") where IsString(body) and IsMatch(body, "\\sINFO\\s")
          - set(cache, ExtractPatterns(body, "span_id\=(?P<SpanId>[\w]*)")
          - set(span_id.string, cache["SpanId"])

Log output

Error: invalid configuration: processors::transform: unable to parse OTTL statement "set(span_id.string, ExtractPatterns(body, \"span_id\\=(?P<SpanId>[\\w]*)\")": statement has invalid syntax: 1:43: invalid quoted string "\"span_id\\=(?P<SpanId>[\\w]*)\"": invalid syntax
2023/11/21 10:38:16 collector server run finished with error: invalid configuration: processors::transform: unable to parse OTTL statement "set(span_id.string, ExtractPatterns(body, \"span_id\\=(?P<SpanId>[\\w]*)\")": statement has invalid syntax: 1:43: invalid quoted string "\"span_id\\=(?P<SpanId>[\\w]*)\"": invalid syntax

Additional context

No response

[github-actions]

Pinging code owners:

• pkg/ottl: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[stephenhong]

I was able to resolve this. I was missing a ) and the \ must be escaped because of how the collector reads the config.
Using - set(cache, ExtractPatterns(body, "span_id\\=(?P<SpanId>[\\w]*)")) works, instead of - set(cache, ExtractPatterns(body, "span_id\=(?P<SpanId>[\w]*)")

[crobert-1]

Thanks for following up and providing the solution for your issue!