High CPU and Low throughput

[akhare22sandeep]

Hi All,

We are using Crossbar version 20.7.1 in production as a WAMP protocol router. We have developed Python apps that make RPC calls to systems which are also connected to the router. Until now, everything was good, but we have encountered issues since introducing file uploads. The server application makes RPC calls to the system to upload files in 128 KB chunks. We have observed that with only 20 systems uploading files, Crossbar's CPU usage reaches 100% and chokes.

We also noticed a decrease in overall throughput due to the high CPU usage (uploading around total 7 to 8 GB files in 4 hours from each server). Recently, we set validate_utf8 to false, which improved performance by 20 to 30%, but the overall file upload process is still slow due to 100% CPU utilization.

Other RPC are simple JSON structure but file upload RPC uploads binary ,videos and other types of large files.

1. What would be the cause of high CPU , is there any way we can reduce the CPU by changing the configuration
2. How can we run the crossbar on multi CPU to take advantage of 8 cores on the server . Right now its using single core
3. Can we WSS and Compression settings increase the CPU ?

Below are our crossbar settings . Any Help would be appreciated

{
  "version": 2,
  "workers": [
    {
      "type": "router",
       "realms": [
        {
          "name": "realm1",
          "roles": [
            {
              "name": "system",
              "authorizer": "com.company.auth.authorize"
            },
            {
              "name": "user",
              "authorizer": "com.company.auth.authorize"
            },
            {
              "name": "service",
              "authorizer": "com.company.auth.authorize"
            }
          ]
        }
      ],
      "transports": [
        {
          "type": "web",
          "endpoint": {
            "type": "tcp",
            "port": 8080
          },
          "paths": {
            "status": {
              "type": "resource",
              "classname": "abcd_router_inspector.create_resource",
              "extra": {
                "services": [
                  {
                    "name": "abcd-realm-auditor",
                    "url": "http://localhost:9503/api/status"
                  },
                  {
                    "name": "abcd-realm-monitor",
                    "url": "http://localhost:9504/api/status"
                  }
                ],
                "platforms": [
                    {
                      "name": "abcd-heartbeat-consumer",
                      "url": "http://sl2ionapp302:9502/api/status"
                    }
                ],
                "status_healthy": "abcd up",
                "status_unhealthy": "abcd down",
                "db_connection_successfull": "DB connection is up",
                "db_connection_unsuccessfull": "DB connection is down"
              }
            },
            "info": {
              "type": "nodeinfo"
            },
            "ws": {
              "type": "websocket",
              "auth": {
                "anonymous": {
                  "type": "dynamic",
                  "authenticator": "com.company.auth.authenticate"
                },
                "ticket": {
                  "type": "dynamic",
                  "authenticator": "com.company.auth.authenticate"
                }
              },
              "options": {
                "auto_ping_interval": 60000,
                "auto_ping_timeout": 10000,
                "auto_ping_size": 4,
                "compression": {
                  "deflate": {
                    "request_no_context_takeover": false,
                    "request_max_window_bits": 11,
                    "no_context_takeover": false,
                    "max_window_bits": 11,
                    "memory_level": 4
                  }
                }
              }
            }
          }
        }
      ],
      "components": [
        {
          "id": "auth",
          "type": "class",
          "classname": "auth.F10MSSQLAuthSession",
          "realm": "realm1",
          "role": "trusted",
          "extra": {
            "authenticate_uri": "com.company.auth.authenticate",
            "authorize_uri": "com.company.auth.authorize",
            "database": {
              "module": "pymssql",
              "server": "DbServer.dv.local",
              "port": "1517",
              "database": "abcdIONAPP",
              "user": "user",
              "password": "sometoken==",
              "autocommit": true
            },
            "validate_token_url": "http://server.local:9000/validate_token/",
            "enforce_cert_match": false,
            "cert_sub_pattern": "[\\s_-]"
          }
        }
      ]
    }
  ]
}

[oberstet]

What would be the cause of high CPU , is there any way we can reduce the CPU by changing the configuration

You should use a recent Linux (or FreeBSD), use a CPU architecture supported as a PyPy JIT target (x86-64 or ARMv8, not RiscV yet .. sadly), use multiple cores (eg a 32-128 cores CPU), use a mainboard with sufficient main memory bandwidth, and capacity

How can we run the crossbar on multi CPU to take advantage of 8 cores on the server . Right now its using single core

Run a multi-core router worker setup in Crossbar.io to scale vertically to your number of CPU cores.

This will achieve ~50,000 routed RPC calls per second (each routed RPC call amounting to 4 WAMP messages) and GB/s aggregate RPC payload bandwidth - per CPU core used. I've tested vertical scalability at millions of routed RPCs/s and multiple GB/s, on up to ~100 CPU cores, but I'd expect it scales vertically to 256 cores (on Linux) at least on a single machine.

If you need more performance than a single machine can provide, you need a multi-node router cluster using Crossbar.io router-to-router links (r2r links).

Can we WSS and Compression settings increase the CPU ?

Both TLS and WebSocket compression (which means, deflate/zlib) will increase CPU load obviously. The CPU load of TLS/zlib in general, and from Python, specifically from PyPy, is what is pushing the limits here. Crossbar.io is simply using "what is there" in terms of hardware resources and libraries then.

If you need TLS, eg because you want to scale horizontally (over multiple machines), besides the raw TCP network performance, you might want a NIC with hardware TLS support from OpenSSL - and make sure such OpenSSL is then used from PyPy in Crossbar.io

The most efficient WAMP framing is WAMP RawSocket over Unix Domain Socket (without TLS), and the most efficient normal WAMP serialization is WAMP-CBOR, and the absolute most efficient is zero-serialization using WAMP-Flatbuffers

Hope this helps!

Cheers,

/Tobias

[akhare22sandeep]

Appreciate your response @oberstet !

Correction crossbar version is 20.7.1

We have 3 crossbar routers servers with 3 applications servers , the setup is 1 to 1 mapping and sessions are sticky.
Below is Crossbar Sever Information

`Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              8
On-line CPU(s) list: 0-7
Thread(s) per core:  1
Core(s) per socket:  2
Socket(s):           4
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               63
Model name:          Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz
Stepping:            2
CPU MHz:             1995.379
BogoMIPS:            3990.75
Hypervisor vendor:   VMware
Virtualization type: full
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            35840K
NUMA node0 CPU(s):   0-7
`

Regarding

multi-core router worker setup in Crossbar.io to scale vertically to your number of CPU cores

Our RPC are bidirectional can we achieve multi-core set up on same port ?
How can we use multi-core router setup without impacting sticky sessions ? Can you please show me the configuration , i have added my configuration as well. We tried multiple times but couldn't achieve multi-core architecture.

The most efficient WAMP framing is WAMP RawSocket over Unix Domain Socket (without TLS), and the most efficient normal WAMP serialization is WAMP-CBOR, and the absolute most efficient is zero-serialization using WAMP-Flatbuffers.

Can you please show me this settings as well , We well update the configuration and do some load testing.

With few client systems (12) we are seeing crossbar CPU 99%

[meejah]

Step zero is to run with PyPy, not CPython: https://www.pypy.org/

[akhare22sandeep]

our internal applications are using cpython , python command is for internal app realm monitor

[oberstet]

Yep! PyPy. Also: you seem to be running on on vSphere, and are using a 10 yrs old CPU which is EOL from Intel. Plus post the complete output from crossbar version ... this will reveal a lot of relevant system infos. Finally, who is "we" (as in, "We have 3 crossbar routers ...")?

[akhare22sandeep]

We means for my team . Crossbar 4.0.0.0 is our internal version
Below is the output
/current/bin/crossbar version

    :::::::::::::::::
          :::::          _____                      __
    :::::   :   :::::   / ___/____ ___   ___  ___  / /  ___ _ ____
    :::::::   :::::::  / /__ / __// _ \ (_-< (_-< / _ \/ _ `// __/
    :::::   :   :::::  \___//_/   \___//___//___//_.__/\_,_//_/
          :::::
    :::::::::::::::::   Crossbar v4.0.0.0

    Copyright (c) 2013-2024 Crossbar.io Technologies GmbH, licensed under AGPL 3.0.

 Crossbar.io        : 4.0.0.0
   txaio            : 20.4.1
   Autobahn         : 20.7.1
     UTF8 Validator : autobahn
     XOR Masker     : autobahn
     JSON Codec     : stdlib
     MsgPack Codec  : msgpack-1.0.0
     CBOR Codec     : cbor-1.0.0
     UBJSON Codec   : ubjson-0.16.1
     FlatBuffers    : flatbuffers-1.11
   Twisted          : 20.3.0-EPollReactor
   LMDB             : 0.98/lmdb-0.9.22
   Python           : 3.6.8/CPython
 Frozen executable  : no
 Operating system   : Linux-4.18.0-193.28.1.el8_2.x86_64-x86_64-with-redhat-8.2-Ootpa
 Host machine       : x86_64
 Release key        : RWT3yJ/q6XYhAf67c7VEgN/LBGCpjPOyoBsv6d32W/eyaKxQ4DR45diO

[oberstet]

We means for my team . Crossbar 4.0.0.0 is our internal version

Ok, I see! So may I ask:

• what's the name & country of your organization?
• where can I access the source code of your internal version?

[akhare22sandeep]

I will have to get lot of approvals for the same.

[oberstet]

No, you do not need any approval, because it is simply a requirement of the AGPL license you are using Crossbar.io under.

Where can I access the source code of your internal version?