[extension/ecsobserver] Write discovered targets as Prometheus file sd #3785

pingleig · 2021-06-14T18:20:18Z

Description:

Glue all pieces using serviceDiscovery and support real AWS client
Write output in prometheus file sd format
Filter ECS service based on config to reduce API calls

Link to tracking Issue:

Feature request [Feature Request] [Extension] ECS Service Discovery for Prometheus #1395

Previous PRs

Fetch EC2, Service [extension/ecsobserver] Add task definition, ec2 and service fetcher #3503
Service matcher [extension/ecsobserver] Add service and task definition based matcher #3386
Exporter [extension/ecsobserver] Add exporter to convert task to target #3333
Fetch Task [extension/ecsobserver] Init service client and mock #3284
Docker Label Matcher [extension/ecsobserver] Add docker label based matcher #3276
Task IP and Port [extension/ecsobserver] Add get ip and port from task #3133
Target and Task definition [extension][ecsobserver] Init config and target definition #2939
Config PR [extension][ecsobserver] Init ECS Observer README #2377
Original PR [extension][ecsobserver] Init ECS service discovery using docker label #2734

Testing:

unit test
e2e test using pingleig/aws-otel-collector@d30c480

Documentation:

No new doc. See existing doc

pingleig · 2021-06-14T18:37:11Z

cc @anuraaga @Aneurysm9

pingleig · 2021-06-14T18:56:56Z

Load test failed because some UDP message is lost https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/3785/checks?check_run_id=2822779561

2021/06/14 18:45:44 Agent process stopped, exit code=0
    validator.go:42: 
        	Error Trace:	validator.go:42
        	            				test_case.go:279
        	            				scenarios.go:190
        	            				log_test.go:177
        	Error:      	Not equal: 
        	            	expected: 139900
        	            	actual  : 139867
        	Test:       	TestLog10kDPS/udp
        	Messages:   	Received and sent counters do not match.

extension/observer/ecsobserver/factory.go

extension/observer/ecsobserver/fetcher.go

extension/observer/ecsobserver/sd.go

mxiamxia

Thanks!

extension/observer/ecsobserver/factory.go

extension/observer/ecsobserver/sd.go

extension/observer/ecsobserver/target.go

pingleig · 2021-06-18T03:17:23Z

windows test failed https://app.circleci.com/pipelines/github/open-telemetry/opentelemetry-collector-contrib/15444/workflows/90c5815a-0b99-4ab5-ab15-d93e2aeb0575/jobs/134087

--- FAIL: Test_statsdreceiver_EndToEnd (0.00s)
    --- FAIL: Test_statsdreceiver_EndToEnd/default_config_with_9s_interval (0.00s)
        receiver_test.go:132: 
            	Error Trace:	receiver_test.go:132
            	Error:      	Received unexpected error:
            	            	listen udp 127.0.0.1:50835: bind: An attempt was made to access a socket in a way forbidden by its access permissions.
            	Test:       	Test_statsdreceiver_EndToEnd/default_config_with_9s_interval

pingleig · 2021-06-22T23:28:38Z

Did a rebase, unit test passed but load test failed https://app.circleci.com/pipelines/github/open-telemetry/opentelemetry-collector-contrib/15652/workflows/b630b1c3-7505-47a1-aade-878eb52a8f2a/jobs/136105

Failed
2021/06/22 23:11:09 Starting mock backend...
2021/06/22 23:11:09 Starting Agent (/home/circleci/project/bin/otelcontribcol_linux_amd64)
2021/06/22 23:11:09 Writing Agent log to /home/circleci/project/testbed/tests/results/TestLog10kDPS/OTLP/agent.log
2021/06/22 23:11:09 Agent running, pid=34225
2021/06/22 23:11:10 Starting load generator at 10000 items/sec.
2021/06/22 23:11:12 Agent RAM (RES):   0 MiB, CPU: 0.0% | Sent:     17109 items | Received:    16,100 items (5,364/sec)
2021/06/22 23:11:12 Performance error: CPU consumption is 37.3%, max expected is 35%
2021/06/22 23:11:12 Gracefully terminating Agent pid=34225, sending SIGTEM...
2021/06/22 23:11:12 Cannot send logs: failed to push log data via OTLP exporter: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:37133: connect: connection refused"
2021/06/22 23:11:12 Agent process stopped, exit code=0
2021/06/22 23:11:12 CPU consumption is 37.3%, max expected is 35%
    test_case.go:314: CPU consumption is 37.3%, max expected is 35%
2021/06/22 23:11:12 Stopped generator. Sent:     18500 items
2021/06/22 23:11:12 Stopping mock backend...
2021/06/22 23:11:12 Stopped backend. Received:    17,400 items (5,556/sec)

btw: it seems github action is still running, so we are testing on both github action and circle CI?

bogdandrutu · 2021-06-24T14:47:33Z

@alolita would be good to have a small design about this, not sure why we decided to write this in prometheus format.

pingleig · 2021-06-24T17:09:59Z

@bogdandrutu it's mentioned in README https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/extension/observer/ecsobserver/README.md#notify-prometheus-receiver-of-discovered-targets Main reason is if we use receiver creator to create new receiver (i.e. pass discovered targets in process and create a new one for each target) the performance is not good #1395 (comment) . Eventually it can switch away from writing file in file sd format and support other receivers (e.g. a batch mode to reconfigure receiver in receiver creator framework).

bogdandrutu · 2021-06-28T13:39:19Z

/cc @jrcamp I think the performance problem was discuss, is this something we should fix?

mxiamxia · 2021-06-28T17:27:30Z

Hi @bogdandrutu , @jrcamp , We had discussed this performance issue before and @kohrapha has explored a few options including implemented a simple native Prom receiver as Jay proposed. But writing ECS scrape targets in static file was still the better solution we had ATM that can unblock AWS to support ECS Prom metrics auto-discovery in OTel.

This PR is the last PR we need to merge to enable ECS Prom metrics auto-discover for OTel that we'll launch this feature as Preview in AWS in a short time. Could you please help to merge this PR so we can be unblocked? We'll continue to work on the possible better solution and fixes.

cc/ @alolita

jrcamp · 2021-06-28T20:01:18Z

Yeah I'm going to look into the performance issues of many scrapers with receivercreator. I think this is a decent workaround in the meantime. Will look into it in #3977.

alolita · 2021-06-28T21:02:25Z

@bogdandrutu i think it would be useful to do a design on alternatives w pros and cons esp re: perf issues.

@mxiamxia can you and the team put together a short design proposal outlining the top alternatives.

mxiamxia · 2021-06-28T21:11:30Z

@bogdandrutu i think it would be useful to do a design on alternatives w pros and cons esp re: perf issues.

@mxiamxia can you and the team put together a short design proposal outlining the top alternatives.

Hi @alolita, The current implementation is a good alternative for us from the previous discussion in #1395. And we have the high level design in README. I think Jay will take a first stab on perf issue.

We need to get this PR merged ASAP.

alolita · 2021-06-28T21:21:50Z

I've marked this PR ready-to-merge. @bogdandrutu please take a look and flag any other concerns you may have.

- Update README

- Stop the collector process from extension using `host.ReportFatalError` otherwiese the failure of extension just log.

prom receiver is expecting metric's job name same as the one in config. In order to keep similar behaviour as cloudwatch agent's discovery impl, we support getting job name from docker label, but it will break metric type. For long term solution, see open-telemetry/opentelemetry-collector#575 (comment)

Was using context.

pingleig · 2021-06-28T23:42:10Z

Circle CI unit test failed https://app.circleci.com/pipelines/github/open-telemetry/opentelemetry-collector-contrib/16013/workflows/41d3192a-5bfb-4867-b721-7d46f891d1da/jobs/139341

running go unit test ./... + coverage in /home/circleci/project/exporter/sumologicexporter
--- FAIL: TestPushFailedBatch (21.62s)
    exporter_test.go:292: 
        	Error Trace:	exporter_test.go:292
        	Error:      	Error message not equal:
        	            	expected: "error during sending data: 500 Internal Server Error"
        	            	actual  : "Post \"http://127.0.0.1:33123\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
        	Test:       	TestPushFailedBatch
FAIL
coverage: 95.0% of statements
FAIL	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/sumologicexporter	25.675s

pingleig requested a review from jrcamp as a code owner June 14, 2021 18:20

pingleig requested a review from a team June 14, 2021 18:20

github-actions bot assigned dmitryax Jun 14, 2021

pingleig changed the title ~~[extension/ecsobserver] Write discovered targets as file sd~~ [extension/ecsobserver] Write discovered targets as Prometheus file sd Jun 14, 2021

mxiamxia reviewed Jun 15, 2021

View reviewed changes

extension/observer/ecsobserver/factory.go Outdated Show resolved Hide resolved

mxiamxia reviewed Jun 15, 2021

View reviewed changes

extension/observer/ecsobserver/fetcher.go Outdated Show resolved Hide resolved

mxiamxia reviewed Jun 15, 2021

View reviewed changes

extension/observer/ecsobserver/sd.go Outdated Show resolved Hide resolved

mxiamxia approved these changes Jun 16, 2021

View reviewed changes

Aneurysm9 suggested changes Jun 17, 2021

View reviewed changes

extension/observer/ecsobserver/factory.go Outdated Show resolved Hide resolved

extension/observer/ecsobserver/sd.go Outdated Show resolved Hide resolved

extension/observer/ecsobserver/target.go Outdated Show resolved Hide resolved

Aneurysm9 approved these changes Jun 18, 2021

View reviewed changes

pingleig force-pushed the ecssd-sd branch from 87f107b to d2a20b2 Compare June 22, 2021 22:58

alolita added the ready to merge Code review completed; ready to merge by maintainers label Jun 28, 2021

pingleig force-pushed the ecssd-sd branch from 42a2919 to e326293 Compare June 28, 2021 22:55

pingleig added 2 commits June 28, 2021 16:10

ext: ecsobserver Add filter and export yaml

c6d4b1f

ext: ecsobserver Add unit test for overall discovery

1837e9b

- Update README

pingleig added 7 commits June 28, 2021 16:11

ext: ecsobserver Rename exported structs

225d60f

ext: ecsobserver Merge duplicated create test fetcher

531bfd4

- Stop the collector process from extension using `host.ReportFatalError` otherwiese the failure of extension just log.

ext: ecsoberver Inject test fetcher using config

31dd90c

Was using context.

ext: ecsobserver Test fatal error on component.Host

5213fef

ext: ecsobserver Move fetcher injection to its own func

005df91

ext: ecsobserver Add comment to example config in README

415509e

pingleig force-pushed the ecssd-sd branch from e326293 to 415509e Compare June 28, 2021 23:11

bogdandrutu merged commit 33cc515 into open-telemetry:main Jun 30, 2021

pingleig mentioned this pull request Jun 30, 2021

[dep] Update to latest contrib repo to include ECS Prometheus aws-observability/aws-otel-collector#569

Merged

pingleig mentioned this pull request Sep 23, 2021

ECS sidecar task container observer #5316

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[extension/ecsobserver] Write discovered targets as Prometheus file sd #3785

[extension/ecsobserver] Write discovered targets as Prometheus file sd #3785

pingleig commented Jun 14, 2021

pingleig commented Jun 14, 2021

pingleig commented Jun 14, 2021

mxiamxia left a comment

pingleig commented Jun 18, 2021

pingleig commented Jun 22, 2021

bogdandrutu commented Jun 24, 2021

pingleig commented Jun 24, 2021

bogdandrutu commented Jun 28, 2021

mxiamxia commented Jun 28, 2021 •

edited

Loading

jrcamp commented Jun 28, 2021

alolita commented Jun 28, 2021

mxiamxia commented Jun 28, 2021 •

edited

Loading

alolita commented Jun 28, 2021

pingleig commented Jun 28, 2021

[extension/ecsobserver] Write discovered targets as Prometheus file sd #3785

[extension/ecsobserver] Write discovered targets as Prometheus file sd #3785

Conversation

pingleig commented Jun 14, 2021

pingleig commented Jun 14, 2021

pingleig commented Jun 14, 2021

mxiamxia left a comment

Choose a reason for hiding this comment

pingleig commented Jun 18, 2021

pingleig commented Jun 22, 2021

bogdandrutu commented Jun 24, 2021

pingleig commented Jun 24, 2021

bogdandrutu commented Jun 28, 2021

mxiamxia commented Jun 28, 2021 • edited Loading

jrcamp commented Jun 28, 2021

alolita commented Jun 28, 2021

mxiamxia commented Jun 28, 2021 • edited Loading

alolita commented Jun 28, 2021

pingleig commented Jun 28, 2021

mxiamxia commented Jun 28, 2021 •

edited

Loading

mxiamxia commented Jun 28, 2021 •

edited

Loading