Can't find any spans... keep getting "Unexpected end of JSON input"

[cphoover]

Here is the error messaging I'm seeing... For context I'm using the jaegertracing/all-in-one docker image locally...

There was an error querying for traces:
Unexpected end of JSON input

[jpkrohling]

I've seen that when the UI is loaded via the browser's cache but the query service itself can't be reached. Could you try reloading bypassing the cache (Ctrl+Shift+R would usually do it)? If you can't reach the UI anymore, you might have missed exposing the container port to your host port (like, -p 16686:16686).

If you do get the UI loaded but still see the same error message, then it would be useful to get into the developer console, copy the curl command, run on the command line and report back the output. I would also ask what's the exact command used to start the Docker container (like, docker run --rm -it -p 16686:16686 jaegertracing/all-in-one)

[cphoover]

yea.... All I did was copy the docker jaeger-all-in-one run command:

docker run -d -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 -p5775:5775/udp -p6831:6831/udp -p6832:6832/udp \
  -p5778:5778 -p16686:16686 -p14268:14268 -p9411:9411 jaegertracing/all-in-one:latest

I'm using the node.js client.

I've tried refreshing no dice.

Here is the curl command:

curl 'http://localhost:16686/api/traces?end=1505996037353000&limit=20&lookback=24h&maxDuration&minDuration&service=test-jaeger&start=1505909637353000' \
-H 'Accept: */*' \
-H 'Connection: keep-alive' \
-H 'Accept-Encoding: gzip, deflate, br' \
-H 'Referer: http://localhost:16686/search?end=1505996027475000&limit=20&lookback=24h&maxDuration&minDuration&service=test-jaeger&start=1505909627475000' \
-H 'Accept-Language: en-US,en;q=0.8' \
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36' \
--compressed

It is returning no response body.

[jpkrohling]

I've tried refreshing no dice.

Does that mean that you still see the error, or does that mean you don't get the UI loaded at all? Would you please try in an incognito window, just to make sure?

In any case, I cannot reproduce what you are experiencing. I started Jaeger with your exact docker command and ran your exact curl command, which returns:

{"data":[],"total":0,"limit":0,"offset":0,"errors":null}

The JSON above is the appropriate response for this scenario. If you don't send any traces to the server with your nodejs client, do you see the same empty response, or do you see a JSON like I have?

[cphoover]

@jpkrohling It means I'm still getting the error. I've tried hard refresh.

[cphoover]

I can show you docker logs if that helps?

[cphoover]

It seems to be when I have multiple span types within a trace.

Nothing unusual looking in logs:

{"level":"info","ts":1505995964.8162143,"caller":"tchannel/bulider.go:89","msg":"Enabling service discovery","service":"jaeger-collector"}
{"level":"info","ts":1505995964.8165386,"caller":"peerlistmgr/peer_list_mgr.go:111","msg":"Registering active peer","peer":"127.0.0.1:14267"}
{"level":"info","ts":1505995964.8177192,"caller":"standalone/main.go:111","msg":"Starting agent"}
{"level":"info","ts":1505995964.818583,"caller":"standalone/main.go:150","msg":"Starting jaeger-collector TChannel server","port":14267}
{"level":"info","ts":1505995964.8186705,"caller":"standalone/main.go:160","msg":"Starting jaeger-collector HTTP server","http-port":14268}
{"level":"info","ts":1505995964.8192267,"caller":"standalone/main.go:178","msg":"Listening for Zipkin HTTP traffic","zipkin.http-port":9411}
{"level":"info","ts":1505995964.821657,"caller":"standalone/main.go:228","msg":"Starting jaeger-query HTTP server","port":16686}
{"level":"info","ts":1505995965.8176525,"caller":"peerlistmgr/peer_list_mgr.go:159","msg":"Not enough connected peers","connected":0,"required":1}
{"level":"info","ts":1505995965.8180492,"caller":"peerlistmgr/peer_list_mgr.go:166","msg":"Trying to connect to peer","host:port":"127.0.0.1:14267"}
{"level":"info","ts":1505995965.819574,"caller":"peerlistmgr/peer_list_mgr.go:176","msg":"Connected to peer","host:port":"[::]:14267"}
{"level":"info","ts":1505998204.3477128,"caller":"tchannel/bulider.go:89","msg":"Enabling service discovery","service":"jaeger-collector"}
{"level":"info","ts":1505998204.3478334,"caller":"peerlistmgr/peer_list_mgr.go:111","msg":"Registering active peer","peer":"127.0.0.1:14267"}
{"level":"info","ts":1505998204.3486717,"caller":"standalone/main.go:111","msg":"Starting agent"}
{"level":"info","ts":1505998204.3495948,"caller":"standalone/main.go:150","msg":"Starting jaeger-collector TChannel server","port":14267}
{"level":"info","ts":1505998204.34974,"caller":"standalone/main.go:160","msg":"Starting jaeger-collector HTTP server","http-port":14268}
{"level":"info","ts":1505998204.3499165,"caller":"standalone/main.go:178","msg":"Listening for Zipkin HTTP traffic","zipkin.http-port":9411}
{"level":"info","ts":1505998204.3515298,"caller":"standalone/main.go:228","msg":"Starting jaeger-query HTTP server","port":16686}
{"level":"info","ts":1505998205.349482,"caller":"peerlistmgr/peer_list_mgr.go:159","msg":"Not enough connected peers","connected":0,"required":1}
{"level":"info","ts":1505998205.3497262,"caller":"peerlistmgr/peer_list_mgr.go:166","msg":"Trying to connect to peer","host:port":"127.0.0.1:14267"}
{"level":"info","ts":1505998205.3513584,"caller":"peerlistmgr/peer_list_mgr.go:176","msg":"Connected to peer","host:port":"[::]:14267"}
{"level":"info","ts":1505998214.6816676,"caller":"tchannel/bulider.go:89","msg":"Enabling service discovery","service":"jaeger-collector"}
{"level":"info","ts":1505998214.6818602,"caller":"peerlistmgr/peer_list_mgr.go:111","msg":"Registering active peer","peer":"127.0.0.1:14267"}
{"level":"info","ts":1505998214.6826618,"caller":"standalone/main.go:111","msg":"Starting agent"}
{"level":"info","ts":1505998214.6835291,"caller":"standalone/main.go:150","msg":"Starting jaeger-collector TChannel server","port":14267}
{"level":"info","ts":1505998214.6836135,"caller":"standalone/main.go:160","msg":"Starting jaeger-collector HTTP server","http-port":14268}
{"level":"info","ts":1505998214.6838784,"caller":"standalone/main.go:178","msg":"Listening for Zipkin HTTP traffic","zipkin.http-port":9411}
{"level":"info","ts":1505998214.6855378,"caller":"standalone/main.go:228","msg":"Starting jaeger-query HTTP server","port":16686}
{"level":"info","ts":1505998215.6830575,"caller":"peerlistmgr/peer_list_mgr.go:159","msg":"Not enough connected peers","connected":0,"required":1}
{"level":"info","ts":1505998215.6831539,"caller":"peerlistmgr/peer_list_mgr.go:166","msg":"Trying to connect to peer","host:port":"127.0.0.1:14267"}
{"level":"info","ts":1505998215.6844003,"caller":"peerlistmgr/peer_list_mgr.go:176","msg":"Connected to peer","host:port":"[::]:14267"}

[tiffon]

In many cases the UI shows a misleading error message.

Seems like for an erred HTTP response with no body, the error message should show the status code and a message along the lines of "An unknown error has occurred." Not exactly super helpful, but better than trying and failing to parse JSON data that isn't there and showing an error message about the failed JSON.parse(...) call.

I've created jaegertracing/jaeger-ui#127 to address the misleading error messages.

[jpkrohling]

I'm closing this, as this was addressed by jaegertracing/jaeger-ui#127

[belabartha]

I'm still seeing this issue on Jaeger 1.16 for some scenarios.

[jpkrohling]

@belabartha are you able to reproduce it consistently?

[belabartha]

@jpkrohling yes. We have missing data probably, but I'd still expect to have a direction where to fix it. We're using Moleculer framework for our microservices, and using Jaeger for APM.

[jpkrohling]

@belabartha are you able to pin-point it to the request that returns the broken results? An example of the response would also be useful.

[belabartha]

@jpkrohling not yet. I'm trying to get to it; It's buried deeply into about 5th level of spans in a span of 16.

[belabartha]

For now I have this: jaeger.js - line 42:

• response.url is: http://localhost:16686/api/traces?end=1582300681327000&limit=2
• ...and there I get: {"data":null,"total":0,"limit":0,"offset":0,"errors":[{"code":400,"msg":"parameter 'service' is required"}]}.
  I still don't know what is that param. Continuing.

[jpkrohling]

In the first screenshot at this issue, you can see the service field in the left-hand side form, first field. Have you selected a service there? If not, the "Find Traces" button should be disabled.

[belabartha]

Yep, service is populated in the left-had side form. Most of the Operations display a properly, but I have a few that don't.

[belabartha]

In case I add the service (&service=whatever) to the response URL I posted about 3 posts ago, I get: {"data":[],"total":0,"limit":0,"offset":0,"errors":null}. That seems to be way better.

[jpkrohling]

Yep, service is populated in the left-had side form. Most of the Operations display a properly, but I have a few that don't.

If you have the field populated, the "Find Traces" button enabled, and still get a request being made without the service parameter, then there's definitely a bug in there...

[gautaz]

Hello, also seeing this issue here.

What happens is that the UI receive a 200 status code with no JSON body from the server part.
The strange thing is that it seems caused by some particular spans.
When querying for spans other than these particular spans, the UI displays them gracefully.

Even if some spans are corrupted, shouldn't the server part at least return the valid ones to the UI?

[jpkrohling]

Are you able to consistently reproduce it, @gautaz? If you can build a reproducer that can be used by us, we can try to take a look. Also, this seems to be somewhat different than the original report, so, I'd ask you to open a new issue.

[gautaz]

@jpkrohling See #2541.
Thanks for your quick answer :-).