[agent] Make timeout for reporting configurable #1034

gouthamve · 2018-08-30T13:45:48Z

So, this is less a fix for #1026 but more to kick off the discussion around: #927

While the original proposal has:

reporters:
    - kind: tchannel
      # tchannel-specific parameters, e.g.
      collector:
        hostPort: ...
      discovery:
        minPeers: 10
    - kind: grps
      # grpc-specific parameters

I kind of like:

tchannel:
# tchannel-config

grpc:
# grpc-config

I took a stab at implementing my version and turns out it's super straight forward and is very little changed code. But if the preference is to have the originally proposed format, I'll make the changes. But now I'm curious, can we 2 different tchannel reporters?

Note: This will be a breaking change.

codecov · 2018-08-30T13:54:48Z

Codecov Report

Merging #1034 into master will not change coverage.
The diff coverage is 100%.

@@          Coverage Diff           @@
##           master   #1034   +/-   ##
======================================
  Coverage     100%    100%           
======================================
  Files         159     159           
  Lines        7126    7136   +10     
======================================
+ Hits         7126    7136   +10

Impacted Files	Coverage Δ
cmd/agent/app/reporter/tchannel/flags.go	`100% <100%> (ø)`	⬆️
cmd/agent/app/reporter/tchannel/builder.go	`100% <100%> (ø)`	⬆️
cmd/agent/app/reporter/tchannel/reporter.go	`100% <100%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9f8d324...34d298d. Read the comment docs.

yurishkuro · 2018-09-03T02:41:09Z

Thanks for looking into this one.

can we 2 different tchannel reporters

We have some deployments at Uber where we're using two reporters of the same type (well, only tchannel reporter is supported anyway today) to send the data to two locations. However, it is done using the otherReporters feature:

jaeger/cmd/agent/app/builder.go

Lines 103 to 106 in 0c86682

    
           func (b *Builder) WithReporter(r reporter.Reporter) *Builder { 
        
           	b.otherReporters = append(b.otherReporters, r) 
        
           	return b 
        
           }

I believe we have a custom config file format that we parse in a custom main. We debated that we could achieve that functionality by forwarding spans to a dedicated splitter, but that seems like more complicated deployment vs. a small code change in the agent. Ideally, we would like to preserve that forking functionality in the agent.

Have you looked into going with the list approach as proposed? Another alternative is to use nested union-like config, instead of kind: tchannel qualifier, e.g.

  reporters:
    - collectorHostPort: ...
      tchannel:
        # tchannel-specific parameters
        discovery:
          minPeers: 10
    - collectorHostPort: ...
      grpc:
        # grpc-specific parameters

gouthamve · 2018-09-03T12:54:31Z

I think this nested union-like config approach better :) I'll make the changes.

yurishkuro · 2018-09-03T15:53:13Z

The tricky part is to decide how cli flags will support a list. If that proves to be too ugly, we could draw a line and say we only support one reporter by default. At Uber we can continue using the custom config and main for multiple reporters.

gouthamve · 2018-09-04T05:15:17Z

Yeah, spent an hour+ trying to make the flags and config match up and not fail tests, couldn't get it to work.

Prefer if there is only reporter being supported.

yurishkuro · 2018-09-04T18:06:25Z

let's go with just one reporter, as long as programmatically we can add more (as we can today). Could you post how the CLI / config will look ?

gouthamve · 2018-09-06T07:09:09Z

➜  jaeger git:(add-agent-timeout) ✗ ./agent --help
Jaeger agent is a daemon program that runs on every host and receives tracing data submitted by Jaeger client libraries.

.....

Flags:
      --collector.host-port string                            comma-separated string representing host:ports of a static list of collectors to connect to directly (e.g. when not using service discovery)
      --collector.timeout duration                            sets the timeout used when establishing reporting spans (default 1s)
      --config-file string                                    Configuration file in JSON, TOML, YAML, HCL, or Java properties formats (default none). See spf13/viper for precedence.
      --discovery.conn-check-timeout duration                 sets the timeout used when establishing new connections (default 250ms)
      --discovery.min-peers int                               if using service discovery, the min number of connections to maintain to the backend (default 3)
  -h, --help                                                  help for jaeger-agent
      --http-server.host-port string                          host:port of the http server (e.g. for /sampling point and /baggage endpoint) (default ":5778")
.....

So, now this just adds a simple --collector.timeout flag. But I can make the flags to be: --collector.tchannel.* if you want.

This is how the config will look like:

jaeger/cmd/agent/app/builder_test.go

Lines 55 to 61 in 66ed190

    
           tchannel: 
        
               collectorHostPorts: 
        
                   - 127.0.0.1:14267 
        
                   - 127.0.0.1:14268 
        
                   - 127.0.0.1:14269 
        
               minPeers: 4 
        
               collectorServiceName: some-collector-service

pavolloffay · 2018-09-07T16:01:32Z

The next week I will have time to look at #927

gouthamve · 2018-09-14T11:29:15Z

I'm running this in prod for the past 10 days and everything works great, we used to drop a ton of batches due to timeouts, but right now those occurrences are super rare when running with 5s timeout.

pavolloffay · 2018-09-19T12:47:49Z

@gouthamve sorry for a longer delay. If i understand it correctly tchannel flags are prefixed with tchannel.

If we cannot make it work like it's in #1034 (comment). I think this could go in as there are no other options. cc @yurishkuro

gouthamve · 2018-10-01T05:25:52Z

They are not prefixed right now. Changing flags is a breaking change. I could do it if that is preferred though.

pavolloffay · 2018-10-02T14:50:50Z

@gouthamve #1092 addresses agent refactoring. Once that is in this will have to be rebased.

pavolloffay · 2018-10-31T13:51:28Z

@gouthamve could you please rebase this on the current master? We have refactored agent flags now it should be easy to add new ones.

gouthamve · 2018-11-14T15:09:30Z

@pavolloffay Rebased :) I'll run this internally and report on how it performs.

gouthamve · 2018-11-16T10:54:19Z

https://github.com/jaegertracing/jaeger/blob/master/cmd/agent/app/reporter/tchannel/flags.go#L82-L83

would always overwrite this:
https://github.com/jaegertracing/jaeger/blob/master/cmd/agent/app/reporter/tchannel/flags.go#L70-L77
which means people using the deprecated flags are essentially getting the wrong values.

Fix is in the last commit. It wasn't caught before as flag values from the previous iteration are carried forward to the next iteration.

gouthamve · 2018-11-16T11:10:25Z

Hmm, the test failure looks like a flake.

pavolloffay · 2018-11-16T11:31:21Z

@gouthamve are you also fixing #1192 in this PR?

gouthamve · 2018-11-16T11:36:24Z

Yes, I need to re-init the flags for my tests to pass

pavolloffay

LGTM, thanks also for fixing the broken flags. A bonus would be a test verifying the timeout.

@yurishkuro could you please also have a look? I might lack some tchannel exprience.

yurishkuro

lgtm overall, a couple minor nits

yurishkuro · 2018-11-16T18:31:44Z

cmd/agent/app/reporter/tchannel/flags.go

@@ -30,6 +30,8 @@ const (
 	hostPort                  = "host-port"
 	discoveryMinPeers         = "discovery.min-peers"
 	discoveryConnCheckTimeout = "discovery.conn-check-timeout"
+


nit: remove blank line

cmd/agent/app/reporter/tchannel/flags.go

yurishkuro · 2018-11-16T18:34:04Z

cmd/agent/app/reporter/tchannel/flags.go

+	}
+	if v.GetDuration(tchannelPrefix+discoveryConnCheckTimeout) != defaultConnCheckTimeout {
+		b.ConnCheckTimeout = v.GetDuration(tchannelPrefix + discoveryConnCheckTimeout)
+	}


Strictly speaking this does not solve the overriding problem if the manually provided value was the same as the default value, but it's still an improvement. The key thing is that we just need to remove the legacy flags soon.

cmd/agent/app/reporter/tchannel/flags.go

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

gouthamve · 2018-11-17T14:05:32Z

Rebased and feedback incorporated

yurishkuro

Thanks!

* agent: Make timeout for reporting configurable Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Fix tests and make sure deprecated flags work Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

gouthamve requested review from black-adder, jpkrohling, pavolloffay, vprithvi and yurishkuro as code owners August 30, 2018 13:45

gouthamve changed the title ~~agent: Make timeout for reporting flags configurable~~ agent: Make timeout for reporting configurable Aug 30, 2018

yurishkuro changed the title ~~agent: Make timeout for reporting configurable~~ [agent] Make timeout for reporting configurable Sep 3, 2018

gouthamve force-pushed the add-agent-timeout branch from 0940c1e to f0d11c9 Compare November 14, 2018 15:06

gouthamve requested review from objectiser and tiffon as code owners November 14, 2018 15:06

gouthamve force-pushed the add-agent-timeout branch from f0d11c9 to bbf34aa Compare November 14, 2018 15:23

gouthamve force-pushed the add-agent-timeout branch from 31260a2 to 15476e0 Compare November 16, 2018 10:55

gouthamve mentioned this pull request Nov 16, 2018

[agent] Deprecated flags are ignored totally #1192

Closed

pavolloffay approved these changes Nov 16, 2018

View reviewed changes

yurishkuro approved these changes Nov 16, 2018

View reviewed changes

gouthamve force-pushed the add-agent-timeout branch from 15476e0 to 7229f86 Compare November 17, 2018 13:58

gouthamve added 2 commits November 17, 2018 19:32

agent: Make timeout for reporting configurable

9d48c4e

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Fix tests and make sure deprecated flags work

34d298d

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

gouthamve force-pushed the add-agent-timeout branch from 7229f86 to 34d298d Compare November 17, 2018 14:04

yurishkuro approved these changes Nov 17, 2018

View reviewed changes

yurishkuro merged commit 7d664e6 into jaegertracing:master Nov 17, 2018

gouthamve mentioned this pull request Nov 22, 2018

agent: tchannel timeout should be configurable #1026

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[agent] Make timeout for reporting configurable #1034

[agent] Make timeout for reporting configurable #1034

gouthamve commented Aug 30, 2018 •

edited

Loading

codecov bot commented Aug 30, 2018 •

edited

Loading

yurishkuro commented Sep 3, 2018

gouthamve commented Sep 3, 2018

yurishkuro commented Sep 3, 2018

gouthamve commented Sep 4, 2018

yurishkuro commented Sep 4, 2018

gouthamve commented Sep 6, 2018

pavolloffay commented Sep 7, 2018

gouthamve commented Sep 14, 2018

pavolloffay commented Sep 19, 2018

gouthamve commented Oct 1, 2018

pavolloffay commented Oct 2, 2018

pavolloffay commented Oct 31, 2018

gouthamve commented Nov 14, 2018

gouthamve commented Nov 16, 2018 •

edited

Loading

gouthamve commented Nov 16, 2018

pavolloffay commented Nov 16, 2018

gouthamve commented Nov 16, 2018

pavolloffay left a comment

yurishkuro left a comment

yurishkuro Nov 16, 2018

yurishkuro Nov 16, 2018

gouthamve commented Nov 17, 2018

yurishkuro left a comment

[agent] Make timeout for reporting configurable #1034

[agent] Make timeout for reporting configurable #1034

Conversation

gouthamve commented Aug 30, 2018 • edited Loading

codecov bot commented Aug 30, 2018 • edited Loading

Codecov Report

yurishkuro commented Sep 3, 2018

gouthamve commented Sep 3, 2018

yurishkuro commented Sep 3, 2018

gouthamve commented Sep 4, 2018

yurishkuro commented Sep 4, 2018

gouthamve commented Sep 6, 2018

pavolloffay commented Sep 7, 2018

gouthamve commented Sep 14, 2018

pavolloffay commented Sep 19, 2018

gouthamve commented Oct 1, 2018

pavolloffay commented Oct 2, 2018

pavolloffay commented Oct 31, 2018

gouthamve commented Nov 14, 2018

gouthamve commented Nov 16, 2018 • edited Loading

gouthamve commented Nov 16, 2018

pavolloffay commented Nov 16, 2018

gouthamve commented Nov 16, 2018

pavolloffay left a comment

Choose a reason for hiding this comment

yurishkuro left a comment

Choose a reason for hiding this comment

yurishkuro Nov 16, 2018

Choose a reason for hiding this comment

yurishkuro Nov 16, 2018

Choose a reason for hiding this comment

gouthamve commented Nov 17, 2018

yurishkuro left a comment

Choose a reason for hiding this comment

gouthamve commented Aug 30, 2018 •

edited

Loading

codecov bot commented Aug 30, 2018 •

edited

Loading

gouthamve commented Nov 16, 2018 •

edited

Loading