Does mylg hping do something different every 100 pings?

[coffenbacher]

Every 100 pings we see ridiculously slow times, is mylg hping doing something different on those requests? Or is this really bizarre ISP behavior? (We're trying to diagnose some bad network problems which is why we're running this at all, so that could easily be the case).

HTTP Response seq=98, proto=HTTP/1.1, status=200, time=13.749 ms
HTTP Response seq=99, proto=HTTP/1.1, status=200, time=13.664 ms
HTTP Response seq=100, proto=HTTP/1.1, status=200, time=5026.282 ms
HTTP Response seq=101, proto=HTTP/1.1, status=200, time=13.309 ms
HTTP Response seq=102, proto=HTTP/1.1, status=200, time=13.260 ms
...
HTTP Response seq=198, proto=HTTP/1.1, status=200, time=13.488 ms
HTTP Response seq=199, proto=HTTP/1.1, status=200, time=11.810 ms
HTTP Response seq=200, proto=HTTP/1.1, status=200, time=11040.815 ms
HTTP Response seq=201, proto=HTTP/1.1, status=200, time=14.015 ms
HTTP Response seq=202, proto=HTTP/1.1, status=200, time=13.150 ms

[gummiboll]

hping doesn't do anything special on those requests and I have tried running a couple of tests myself and did not see this problem. However, it seems a bit.. weird that you got slow responses on request exactly 100 and 200 if its a network problem.

Could you try running hping against something that doesn't have network issues?

[coffenbacher]

From further experimenting:

• I see the effect on unaffected networks as well, although the effect size is 100x smaller (to the point it's no problem, 4x slower as opposed to 500x)
• I don't see the effect when pinging all domains, only some

Feels like maybe a periodic DNS lookup or something, maybe it is an OS setting rather than hping that is causing it to be every 100. After testing, I certainly don't think it's something wrong with the library, but possibly something to know about when evaluating results.

Example testing from an unaffected network:

docker run --rm presbrey/mylg mylg hping https://mws.amazonservices.com -c 1000

...
HTTP Response seq=197, proto=HTTP/1.1, status=400, time=83.166 ms
HTTP Response seq=198, proto=HTTP/1.1, status=400, time=83.243 ms
HTTP Response seq=199, proto=HTTP/1.1, status=400, time=83.625 ms
HTTP Response seq=200, proto=HTTP/1.1, status=400, time=324.535 ms
HTTP Response seq=201, proto=HTTP/1.1, status=400, time=85.197 ms
HTTP Response seq=202, proto=HTTP/1.1, status=400, time=123.147 ms
...

[gummiboll]

@coffenbacher Thank you for experimenting. I will investigate and try to reproduce!

Two questions tho: Did you only try it in docker and what are some domains that you saw this on and domains you didn't see it on?

[gummiboll]

@coffenbacher: Im suspecting you are hitting maxclients/max requests per ip or something similar on the other end. The thing is that hping by default uses a interval set to 0s which really means it runs all the requests after each other. I think we will change this to add a longer default interval.

Could you try doing the same thing and just add -i 1s and see if you get the same result?

[coffenbacher]

Hi @gummiboll, I have only tried in Docker, yes.

Example affected domain: amazon.com (this and subdomains were the only ones I checked that are affected)
Example unaffected domain: google.com, facebook.com, github.com

I tried with -i 1s and also -i 5s but seemingly nothing changed (not just the problem...but nothing at all, still seemed continuous), perhaps the version on Dockerhub is out of date?

It's bizarre because whatever amazon.com's network is doing with those requests happens connecting both from good networks and bad networks, but is for some reason 1000x worse on our bad network (our host told us the network apparently has a bad route to he.net at ISP level).

[gummiboll]

Ah, good catch about dockerhub! I rarely use docker myself so haven't tried mylg in it but now I saw that mylg on Dockerhub builds against @presbrey's fork of mylg (which isn't up to date) instead of directly to this repository which is the reason -i doesn't do anything at all I believe (-i wasn't implemented in the version that was forked).

I think we should resolve this first and see if running with a longer interval than 0s resolves this problem. So, @presbrey would you mind deleting your dockerhub version of mylg and maybe @mehrdadrad could register this repo in dockerhub so we don't run into this issue again? :)