all-in-one integration test is unstable

[yurishkuro]

Build failing https://travis-ci.org/jaegertracing/jaeger/jobs/540902708#L990

=== RUN   TestAllInOne
--- FAIL: TestAllInOne (1.01s)
    require.go:794: 
        	Error Trace:	all_in_one_test.go:86
        	            				all_in_one_test.go:60
        	Error:      	Received unexpected error:
        	            	Get http://0.0.0.0:16686/api/traces?service=jaeger-query&tag=jaeger-debug-id:debug: dial tcp 0.0.0.0:16686: connect: connection refused
        	Test:       	TestAllInOne
FAIL
FAIL	github.com/jaegertracing/jaeger/cmd/all-in-one	1.222s
make: *** [integration-test] Error 1

Looking at

jaeger/cmd/all-in-one/all_in_one_test.go

Lines 121 to 129 in 5b36e3b

	func healthCheck() error {
	for i := 0; i < 100; i++ {
	if _, err := http.Get(queryURL); err == nil {
	return nil
	}
	time.Sleep(100 * time.Millisecond)
	}
	return fmt.Errorf("query service is not ready")
	}

it seems that it may not wait long enough for health check to become green. I would extend the loop to 10k iteration (10seconds).

[guanw]

Not sure about this.. Since the error was thrown on L86 where it was tryhing to getAPITrace instead of healthCheck. But I will try that in a bit.

[guanw]

Increase the loop count & sleep time don't work. I'm going to try that script on a linux vm environment see if I could reprod there.

[guanw]

Turns out this to be an issue with prometheus panic as it requires all categories sharing same structure (in this case, the tag set should also be the same) I made the fix in the original story.

[yurishkuro]

are you sure? I thought I saw the same failure on master.

[guanw]

i think so. The test is passing now on #1576

[yurishkuro]

OK, in that case this ticket is still relevant, in the sense that the integration test doesn't check something. For example, if your issue in #1576 was causing all-in-one to crash upon receiving a request, we should've seen it in the test somehow, e.g. by dumping logs from Docker.

[yurishkuro]

have not seen this issue for a long time, closing