Limit system ram to 4GB

[freimair]

Might treat some symptoms of #3128, #3657, #3918, #3917, #3686, #3677

OpenJDK assumes total system RAM of 128GB. Setting it to 4GB manually
reduces the greediness of the JVM drastically.

Only after setting the MaxRAM to 1.5GB it finally went into a heap space error. 2GB work fine on my machine.

Proposed strategy: do a full release test with this, see how it goes.

This find is courtesy of @ghubstan #3918 (comment)

[wiz]

@ripcurlx @sqrrm please also consider cherry-pick for v1.2.8 - many users have RAM issues and it goes towards our goal of "improve reliability"

[ghubstan]

@freimair: "Only after setting the MaxRAM to 1.5GB it finally went into a heap space error. 2GB work fine on my machine". 2 GB was the minimum MaxRAM for my machine too; I got an OOM when setting MaxRAM=1.5GB.

[ripcurlx]

@ripcurlx @sqrrm please also consider cherry-pick for v1.2.8 - many users have RAM issues and it goes towards our goal of "improve reliability"

I am happy to cherry-pick this for v1.2.8 if someone with a linux system does the required testing for it.

[freimair]

please specify "required testing". It has been developed on Linux. It worked.

I would rather see it tested on Windows.

[ripcurlx]

This is just for the startup script when building from source, correct? So the created binaries won't have this change.

[ghubstan]

Setting this param value to "4GB" doesn't work on my machine; it needs to be "4G" or "4g".

$ java -XX:MaxRAM=4GB -version
Improperly specified VM option 'MaxRAM=4GB'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

$ java -XX:MaxRAM=4g -version
openjdk version "11" 2018-09-25
OpenJDK Runtime Environment 18.9 (build 11+28)
OpenJDK 64-Bit Server VM 18.9 (build 11+28, mixed mode)

[freimair]

This is just for the startup script when building from source, correct? So the created binaries won't have this change.

Yes, however, as discussed, I also changed the packager scripts.

[ripcurlx]

utACK

@freimair Let's give this a try as discussed pointing users in this issue to try out master (if they are able to build and run it). I'll create new snapshot Linux binaries after the release for testing.

[cbeams]

I just ran into the following heap space error on my always-on Bisq node. I upgraded that node from v1.3.2 to v1.3.4 on March 1st, and the heap space error occurred on March 4th. I haven't had any such errors before, or at least not for a very long time. Makes me think this change is probably the culprit. I just asked @bisq-network/support in Keybase whether they've been seeing similar reports from users, no response yet.

[cd2357]

I haven't had any such errors before, or at least not for a very long time. Makes me think this change is probably the culprit.

Probably the culprit, or probably just didn't fix the underlying issue.

I read somewhere that JavaFX also uses the system's video RAM, in addition to the normal RAM usually used by the Java VM.

This PR seems to restrict only the normal RAM usage -- we might have to investigate the VRAM side as well.

Btw, this means Bisq (or any JavaFX app) could behave differently on different systems, depending on that system's video settings (graphic card hardware? graphics drivers? amount of shared RAM between video and system? etc -- don't know what could play a role, but smth to keep in mind when looking at performance).

[ghubstan]

Btw, this means Bisq (or any JavaFX app) could behave differently on different systems, depending on that system's video settings (graphic card hardware? graphics drivers? amount of shared RAM between video and system? etc -- don't know what could play a role, but smth to keep in mind when looking at performance).

JavaFX does behave differently with different GPUs. It does use native GPU drivers, and there have been many leak problems in JFX related to GPU and mesa drivers.

Here is a list of some jfx leak related issues -- old and new. You have to hunt around for those related to GPUs, and for issues about leak related bugs found and fixed after JFX 11 was released.

Issue 8188094 shows how to make JFX leak from a simple app running on Linux. The unresolved issue reports the problem in JDK/JFX 8/9, and I reproduced it running JFX 11 on Ubuntu + NVIDIA GPU in Feb. 2020, when I started working on this problem.

JFX 14 has some leak bugfixes, but Bisq is holding back on upgrading to OpenJDK 14 until the new java packaging tool is released from incubation to GA.

I did a lot of profiling to find the leaks (maybe you saw some of my comments in issue 3918) using JMC/JFR, JProfiler, async-profiler, bcc and bpftrace tools, under the assumption the problem was/is in Bisq and JFX. I'm sure there are leaks to find and fix, but the only firm conclusion my analysis supported was that Bisq's JVM was reserving far more resident and virtual memory at startup than the Linux JVM needs. (I could not find problems with Bisq's on-heap management.) The problem was more serious when it starts with an empty data dir, syncs the entire chain, and the JVM never gives back any memory (to the OS).

[cd2357]

Thanks for the detailed reply @ghubstan, that clarifies a lot.

I'm sure there are leaks to find and fix, but the only firm conclusion my analysis supported was that Bisq's JVM was reserving far more resident and virtual memory at startup than the Linux JVM needs. (I could not find problems with Bisq's on-heap management.) The problem was more serious when it starts with an empty data dir, syncs the entire chain, and the JVM never gives back any memory (to the OS).

Have you tried using the -XX:+UseG1GC flag? According to some reports it causes JavaFX to use more aggressive garbage collection, which reduces memory consumption and effectively give it back to the system.

[freimair]

I have seen that too but only once for my always-on bisq. Seems like a legitimate Out-of-Heap error since Bisq is resource hungry beyond compare. Restart Bisq and everything is fine again.

There are multiple efforts and fixes in the pipeline, though.

• As @ghubstan already said, java10 is very old and outdated, yet, we refuse to upgrade, and that would fix a whole lot of issues (Upgrade JavaFX to v15 #4242).
• Plus, there are projects in line to help reduce RAM requirements (Migrate datastores to real database projects#29, Reduce bandwidth-requirement on Bisq app startup projects#25) which are either waiting to be reviewed and merged or waiting to start (with a clear 👍 from the community but no budget for over a month now).

@cd2357 we played around with the -XX:+UseG1GC flag way back and it did not change anything. Only when I hunted down and found the faulty line of code that produced the resource leak, thing changed. Bisq still holds the whole data stores in RAM (not at first, but just click the "trades" tab and there you go) and it will do so until we change things.

As long as we do not fix the underlying issues, we will only treat single failures and waste time and resources over and over again.