Latency increased after upgrading to v18.16.0

[ojikeii]

Version

v18.16.0

Platform

Linux 971a67e84394 5.15.49-linuxkit-pr #1 SMP Thu May 25 07:17:40 UTC 2023 x86_64 Linux

Subsystem

No response

What steps will reproduce the bug?

Upgrade the node.js version of the docker image used by our application from v14.16.0 to v18.16.0 in production environment.

How often does it reproduce? Is there a required condition?

It is always reproduced in the production environment.

What is the expected behavior? Why is that the expected behavior?

The latency is not increased after upgrading.

What do you see instead?

The 99th percentile latency is increased (about 200ms to 1.0s).

Additional information

The source code of the application is not changed, just upgrade node.js version to v18.16.0 then it reproduce.

To investigate this problem, I performed the stress test of our application using some different versions of node.js with --trace-gc option.
Then, I found followings from the measurement result of latency and GC logs:

• GC via stack guard impacts the latency in both v14.16.0 and v18.16.0
• The impact in v18.16.0 is stronger than in v14.16.0
• The latency is not improved in v20.3.1
• The latency is improved to same level as in v14.16.0 when tested with a custom binary based on node.js v20.3.1 with V8 v11.7.107 embedded (which contains a bugfix related to stack guard GC).

I don’t understand the detail of the bugfix, but this seems to reduce the impact of GC to latency.
Would it be possible to make a release containing the bugfix for v18 and v20?

[abhashkmr]

@ojikeii Has there been any development on this . The response time for one of our services has increased after upgrading to node v18 . Were you able to fix this issue?

[ojikeii]

@abhashkmr Nothing as far as I know. As I wrote in above, in my case the V8 bug fix improved the performance. So Using Node v21 or later version which includes the bug fix probably resolve this issue but I have not checked it yet.