Lodestar FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

[bozoi3]

The validator start throwing:
<--- Last few GCs --->

[1:0x7f4a088226a0] 342525 ms: Scavenge (reduce) 4076.9 (4141.6) -> 4076.8 (4142.4) MB, 5.47 / 0.14 ms (average mu = 0.236, current mu = 0.091) allocation failure;
[1:0x7f4a088226a0] 342599 ms: Scavenge (reduce) 4077.8 (4142.4) -> 4077.9 (4142.6) MB, 6.17 / 0.14 ms (average mu = 0.236, current mu = 0.091) allocation failure;
[1:0x7f4a088226a0] 342605 ms: Scavenge (reduce) 4077.9 (4142.6) -> 4077.8 (4143.4) MB, 5.59 / 0.14 ms (average mu = 0.236, current mu = 0.091) allocation failure;

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace ----

Describe the bug
The validator stop validating

Expected behavior

Screenshots

DAppNode version:

• Package version: Version: 0.1.10 (v1.14.0 upstream)
• OS: DAppNode

Additional context

[nflaig]

Thanks for reporting this @bozoi3. Are you running Nethermind as execution layer and was this related to the incident from yesterday?

There are also ongoing discussions if we need to increase the default memory limit of Lodestar which we will likely do in the next release (this week).

If you are still seeing this issue on your end even with Nethermind issue being resolved you can also manually increase the memory limit by setting 6144 in the advanced configuration of the beacon node, the config param is called MEMORY_LIMIT.

[nflaig]

Default memory limit has been increased to better handle such cases and prevent Lodestar from crashing

• Increase default memory limit to 8192 #97