pummel/test-heapdump-shadow-realm.js is flaky due to OOM #49572

joyeecheung · 2023-09-09T11:43:38Z

Test

pummel/test-heapdump-shadow-realm.js

Platform

Linux x64

Console output

11:41:28 not ok 3651 pummel/test-heapdump-shadow-realm
11:41:28   ---
11:41:28   duration_ms: 72141.13100
11:41:28   severity: crashed
11:41:28   exitcode: -6
11:41:28   stack: |-
11:41:28     
11:41:28     <--- Last few GCs --->
11:41:28     0.[3846001:0x61f1c60]    46446 ms: Mark-Compact (reduce) 977.0 (993.2) -> 976.4 (993.7) MB, 1058.93 / 0.00 ms  (+ 78.7 ms in 13 steps since start of marking, biggest step 13.7 ms, walltime since start of marking 1165 ms) (average mu = 0.260, current mu = 0.[3846001:0x61f1c60]    47642 ms: Mark-Compact (reduce) 977.1 (993.7) -> 976.5 (993.9) MB, 892.02 / 0.00 ms  (+ 65.2 ms in 9 steps since start of marking, biggest step 16.2 ms, walltime since start of marking 978 ms) (average mu = 0.234, current mu = 0.200
11:41:28     
11:41:28     <--- JS stacktrace --->
11:41:28     
11:41:28     FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
11:41:28      1: 0xca2d40 node::Abort() [out/Release/node]
11:41:28      2: 0xb7f1ab  [out/Release/node]
11:41:28      3: 0xee4ff0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [out/Release/node]
11:41:28      4: 0xee53ac v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [out/Release/node]
11:41:28      5: 0x10fa1e5  [out/Release/node]
11:41:28      6: 0x111095d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [out/Release/node]
11:41:28      7: 0x10e875a v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [out/Release/node]
11:41:28      8: 0x10e92f5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [out/Release/node]
11:41:28      9: 0x10c72fe v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [out/Release/node]
11:41:28     10: 0x1516e42 v8::internal::Runtime_AllocateInOldGeneration(int, unsigned long*, v8::internal::Isolate*) [out/Release/node]
11:41:28     11: 0x1951ef6  [out/Release/node]
11:41:28   ...

Build links

nodejs/reliability#660

First CI: https://ci.nodejs.org/job/node-test-pull-request/53749/
Last CI: https://ci.nodejs.org/job/node-test-pull-request/53778/

Additional information

cc @legendecas At first glance I think we could lower the number of shadow realms created in the test. But then the realms are supposed to be GC-able anyway...perhaps we should switch to setImmediate() instead to give GC some time to kick in?

The text was updated successfully, but these errors were encountered:

With a tight loop the GC may not have enough time to kick in. Try setImmediate() instead. PR-URL: #49573 Refs: #49572 Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com>

legendecas · 2023-09-11T17:21:34Z

I'm not aware that weak callbacks for ShadowRealm are deferred to prevent it from being collected. Since this is related to the heapdump specifically (there is no reliability report on test/parallel/test-shadow-realm-gc.js), I'm wondering if this might be related to insufficient memory for the heapdump operation.

joyeecheung · 2023-09-11T17:45:25Z

Normally heapdump should not have more JS heap overhead than the size of a semi-space (because of promotion) + a little bit of margin for cached source positions (can't be that much for ShadowRealms that only have one tiny script). It could be though that because the weak callbacks are deferred, they are still recognized as reachable when the heap snapshot is taken and thus still in the heap snapshot which means the heap snapshot can get bloated a lot (and a huge heap snapshot in itself can eat a lot of memory because we serialize it into JS on the same thread for test verification). We can see if the setImmediate() trick deflakes it and if it doesn't, consider adding some logging to the test to see what's causing the flake.

joyeecheung · 2023-09-16T00:17:55Z

Looks like this disappeared from the CI. Closing for now. We can reopen if it reappears.

With a tight loop the GC may not have enough time to kick in. Try setImmediate() instead. PR-URL: #49573 Refs: #49572 Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com>

targos · 2023-09-30T06:18:45Z

I think it's back: https://ci.nodejs.org/job/node-test-pull-request/54396/

targos · 2023-10-02T06:40:08Z

Another one: https://ci.nodejs.org/job/node-test-pull-request/54404/

joyeecheung · 2023-10-02T13:54:55Z

Oh, my, this failed18 PRs across the last 100 CI runs. nodejs/reliability#681 looks like it started from 09-29

joyeecheung · 2023-10-02T13:56:19Z

I am guessing something landed around that time introduced a memory leak in the realms?

joyeecheung · 2023-10-03T11:26:59Z

Actually it seems this time around it's a different bug. It's STATUS_STACK_BUFFER_OVERRUN on Windows (or it could be other assertion failures, https://devblogs.microsoft.com/oldnewthing/20190108-00/?p=100655)

joyeecheung · 2023-10-03T14:14:29Z

First STATUS_STACK_BUFFER_OVERRUN failure goes back to https://ci.nodejs.org/job/node-test-pull-request/54325/ which is rebasing onto a4fdb1a

joyeecheung · 2023-10-03T14:27:45Z

Stress test on ubuntu1804-64 https://ci.nodejs.org/job/node-stress-single-test/455/ however the failures in the CI mostly come from win2016_vs2017-x64 and the stress tests do not run there..

targos · 2023-10-05T06:41:44Z

Stress test passed, but it looks like it ran other tests, not pummel/test-heapdump-shadow-realm

legendecas · 2023-10-05T13:44:43Z

https://ci.nodejs.org/job/node-stress-single-test/456/parameters/ Another run of stress test on pummel/test-heapdump-shadow-realm.

Edit: well, I do remember I checked win2016-vs2017 as one of the run labels, but it turned out that only ubuntu1804-64 was checked.

targos · 2023-10-07T09:00:14Z

No failures with ubuntu1804-64

targos · 2023-10-07T09:11:27Z

Some recent failures on Windows:

Note that they always happen with the x86 build!

ShadowRealm garbage-collection is covered in another test. Reduce the number of repetition in test-heapdump-shadowrealm.js trying to fix the flakiness of the test. PR-URL: #50104 Refs: #49572 Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>

With a tight loop the GC may not have enough time to kick in. Try setImmediate() instead. PR-URL: nodejs#49573 Refs: nodejs#49572 Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com>

ShadowRealm garbage-collection is covered in another test. Reduce the number of repetition in test-heapdump-shadowrealm.js trying to fix the flakiness of the test. PR-URL: nodejs#50104 Refs: nodejs#49572 Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>

ShadowRealm garbage-collection is covered in another test. Reduce the number of repetition in test-heapdump-shadowrealm.js trying to fix the flakiness of the test. PR-URL: #50104 Refs: #49572 Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>

lpinca · 2024-09-11T14:28:37Z

There have been no new failures since December 2023, closing.

joyeecheung added the flaky-test Issues and PRs related to the tests with unstable failures on the CI. label Sep 9, 2023

github-actions bot added the linux Issues and PRs related to the Linux platform. label Sep 9, 2023

joyeecheung mentioned this issue Sep 9, 2023

test: use setImmediate() in test-heapdump-shadowrealm.js #49573

Merged

joyeecheung closed this as completed Sep 16, 2023

targos reopened this Sep 30, 2023

legendecas mentioned this issue Oct 9, 2023

test: reduce number of repetition in test-heapdump-shadowrealm.js #50104

Merged

lpinca closed this as completed Sep 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pummel/test-heapdump-shadow-realm.js is flaky due to OOM #49572

pummel/test-heapdump-shadow-realm.js is flaky due to OOM #49572

joyeecheung commented Sep 9, 2023 •

edited

Loading

legendecas commented Sep 11, 2023 •

edited

Loading

joyeecheung commented Sep 11, 2023 •

edited

Loading

joyeecheung commented Sep 16, 2023

targos commented Sep 30, 2023

targos commented Oct 2, 2023

joyeecheung commented Oct 2, 2023

joyeecheung commented Oct 2, 2023 •

edited

Loading

joyeecheung commented Oct 3, 2023 •

edited

Loading

joyeecheung commented Oct 3, 2023

joyeecheung commented Oct 3, 2023 •

edited

Loading

targos commented Oct 5, 2023

legendecas commented Oct 5, 2023 •

edited

Loading

targos commented Oct 7, 2023

targos commented Oct 7, 2023

lpinca commented Sep 11, 2024

pummel/test-heapdump-shadow-realm.js is flaky due to OOM #49572

pummel/test-heapdump-shadow-realm.js is flaky due to OOM #49572

Comments

joyeecheung commented Sep 9, 2023 • edited Loading

Test

Platform

Console output

Build links

Additional information

legendecas commented Sep 11, 2023 • edited Loading

joyeecheung commented Sep 11, 2023 • edited Loading

joyeecheung commented Sep 16, 2023

targos commented Sep 30, 2023

targos commented Oct 2, 2023

joyeecheung commented Oct 2, 2023

joyeecheung commented Oct 2, 2023 • edited Loading

joyeecheung commented Oct 3, 2023 • edited Loading

joyeecheung commented Oct 3, 2023

joyeecheung commented Oct 3, 2023 • edited Loading

targos commented Oct 5, 2023

legendecas commented Oct 5, 2023 • edited Loading

targos commented Oct 7, 2023

targos commented Oct 7, 2023

lpinca commented Sep 11, 2024

joyeecheung commented Sep 9, 2023 •

edited

Loading

legendecas commented Sep 11, 2023 •

edited

Loading

joyeecheung commented Sep 11, 2023 •

edited

Loading

joyeecheung commented Oct 2, 2023 •

edited

Loading

joyeecheung commented Oct 3, 2023 •

edited

Loading

joyeecheung commented Oct 3, 2023 •

edited

Loading

legendecas commented Oct 5, 2023 •

edited

Loading