Low throughput benchmark test compared to vert.x

[duke-cliff]

For better understanding, I put our test programs both in vert.x and webflux here: https://github.com/uken/webflux-vertx-benchmark.

I am doing some load test comparison on webflux(spring boot2) vs vert.x.

The test case is a very simple text/json echo service. So far from the test result, webflux's throughput is much lower than vert.x(10x with annotation, 8x with functional). One interesting thing I found is with the same load, vert.x is only using basically one thread(eventloop) with a very low CPU usage.
But on webflux, by default it's using all of the NIO threads, but throughput is still not even comparable.

I thought both webflux and vert.x are based on netty(epoll on linux), hope someone could show me what I can tweak with in order to get the result closer.

The client I use is wrk/wrk2.
Options are:
wrk -d60s -t30 -c100 -R 500000 -s post.lua http://xxxx/text

$ more post.lua
wrk.method = "POST"
wrk.body = "{"key":"entity_1","point":1}"
wrk.headers["Content-Type"] = "application/json"

[violetagg]

I'm reproducing this only with Reactor Netty using

HttpServer.create(8080)
          .newRouter(r -> r.post("/text",
                     (req, res) -> res.addHeader("Content-Type", "text/plain")
                                      .sendString(req.receive()
                                                     .aggregate()
                                                     .asString())))
         .block();

[duke-cliff]

I wrote a spring 1.5+netty test project seems to be much faster but using

  @Override
  public void run(String... args) throws Exception {
    try{
    ServerBootstrap sb = new ServerBootstrap().group(selectGroup, workerGroup).channel(EpollServerSocketChannel.class).childHandler(httpChannelInitializer).option(ChannelOption.SO_BACKLOG, 1024).childOption(ChannelOption.SO_KEEPALIVE, true).childOption(ChannelOption.CONNECT_TIMEOUT_MILLIS, 3000);
    ChannelFuture channelFuture = sb.bind(8080);
    channelFuture.sync();
    channelFuture.channel().closeFuture().sync();
    }catch (InterruptedException e) {

    }finally{
      shutdown();
    }
  }

Do you know if there's a way to workaround it in spring boot 2 with webflux?

[violetagg]

@duke-cliff Can you try this https://github.com/violetagg/webflux-vertx-benchmark/tree/master/reactornetty
and tell us the results from your system? (this is pure Reactor Netty)

[violetagg]

@duke-cliff did you try Reactor Netty example? Thanks a lot.

[rstoyanchev]

We're currently looking into this in Spring WebFlux. Please follow https://jira.spring.io/browse/SPR-17250.

[duke-cliff]

@duke-cliff did you try Reactor Netty example? Thanks a lot.
@violetagg
Sorry, I did not notice your message(following too many threads).

I will also try the reactor+netty by the end of this week and will come back to you.
But the @dave-fl 's graphs might already reflect the problems well.
Thanks

[duke-cliff]

@violetagg
@rstoyanchev

I re-run all different implementations again with the same setup.
wrk -d60s -t30 -c100 -R 500000 -s post.lua ...
WRK Client runs on: AWS t2.xlarge. Server runs on: AWS c5.2xlarge. Both are in the same VPC.
reactor netty: 47747.90 req/sec
netty: 98912.20 req/sec
vertx: 97039.28 req/sec
webflux: 41141.73 req/sec

Reactor is 50% of native netty throughput. Reactor is 10x than webflux.

[violetagg]

@duke-cliff Did you run it several times or just once? On my side I do not see such big difference between Reactor Netty/Netty/Vert.x.

Reactor Netty

1923856 requests in 1.00m, 205.49MB read
2254641 requests in 1.00m, 240.82MB read
2322119 requests in 1.00m, 248.03MB read
2709667 requests in 1.00m, 289.42MB read
2719624 requests in 1.00m, 290.49MB read

Netty

2736768 requests in 1.00m, 281.88MB read
2688831 requests in 1.00m, 276.94MB read
3058321 requests in 1.00m, 315.00MB read
3064944 requests in 1.00m, 315.68MB read
3152539 requests in 1.00m, 324.70MB read

Vert.x

2118161 requests in 1.00m, 187.86MB read
1881575 requests in 1.00m, 166.88MB read
2185725 requests in 1.00m, 193.86MB read
2271255 requests in 1.00m, 201.44MB read
2295438 requests in 1.00m, 203.59MB read

[duke-cliff]

The server information was wrong above. It's actually running on c5.2xlarge(8CPU/ 16GB)

Maybe you hit some other bottlenecks on the (CPU/Memory/Network IO).
But I will be satisfied if the Webflux performance could be close to Reactor at this point(migrate from spring boot 1 with tomcat).

Reactor Netty

2903684 requests in 1.00m, 310.15MB read 
2878362 requests in 1.00m, 307.44MB read
2873312 requests in 1.00m, 306.90MB read
2898637 requests in 1.00m, 309.61MB read

Netty

5905323 requests in 1.00m, 608.23MB read
5972995 requests in 1.00m, 615.20MB read
5965454 requests in 1.00m, 614.42MB read

Vert.x

5841613 requests in 1.00m, 518.10MB read
5908461 requests in 1.00m, 524.03MB read
5868678 requests in 1.00m, 520.50MB read

[tkp1n]

@violetagg Have you considered setting up a benchmark over at TechEmpower (or would you support an effort to do so)?

[smaldini]

@duke-cliff what configuration are you using ? default ? 1 thread ? Epoll ?
@tkp1n we would love to have contributors here but that is very much in our radar to publish benchmarks when possible (specially we have been requested blocking vs non blocking).

[duke-cliff]

@smaldini On reactor project? I just run the default. On webflux I tried both 1 thread and 16 threads.
On linux, I guess netty will run on epoll with the default

[young891221]

I have the same problem. I want to use webflux, but I can't. This is because performance decreases as requests increase.

[gihad]

@violetagg Is this still being looked at? The low performance is really a barrier for our adoption. I think the comparison against Vertx is a really good point of reference.

Another way is to see Webflux rating high in these benchmarks: https://www.techempower.com/benchmarks/#section=data-r17&hw=ph&test=plaintext&l=zik0vz-1

[violetagg]

Hi All,

Can you test with the newest binaries for Reactor Netty and Spring Framework.
Reactor BOM Californium-SR8
Spring Framework v5.1.7.RELEASE

Thanks,
Violeta

[dave-fl]

@violetagg Spring Cloud Gateway (Greenwich SR1 and Snapshot) stops working under heavy load under this release.

[spencergibb]

What does "stops working" mean?

[dave-fl]

What I see is that TPS drops from 1200 to 200 to PCF rebooting the container because it runs out of memory. I do not see this behavior with SR6 (and all else the same).

[duke-cliff]

@dave-fl Did you get a chance to test against SR8 or Spring Boot 2.1.5 above? Haven't tested on my side. Wondering if there's any improvement from this version as @violetagg replied

[violetagg]

@duke-cliff @dave-fl I believe we fixed the issues from comment #392 (comment)
Please try the latest Spring Boot 2.1.6.RELEASE which is coming with Reactor Netty 0.8.9.RELEASE

[violetagg]

We did several improvements in the API and the internal implementation in 0.8.x, 0.9.x and 1.0.x
I'm closing this one.