memory overflow using overlayfs with 256 MB rpi

[oscwah]

For a Raspberry Pi with 256 MB memory, enabling overlayfs causes memory to run out. This causes the boot log to get stuck at the following line (connecting via serial).

[ 19.198210] systemd[1]: Started Journal Service.

The sum of allocated mem is 183496 kB, see below, and the swap file size is 100 MB. When overlay fs is used, the 100 MB of swap will be allocated on the overlay tmpfs, causing the minimum ram requirement to be 184+100=284MB, which is more than what is available.

Filesystem      1K-blocks   Used    Available   Use%    Mounted on
udev            52964       0       52964       0.00%   /dev
tmpfs           18036       816     17220       5.00%   /run
/dev/mmcblk0p2  7359624     1343956 5684228     20.00%  /
tmpfs           90164       0       90164       0.00%   /dev/shm
tmpfs           5120        4       5116        1.00%   /run/lock
/dev/mmcblk0p1  261108      85320   175788      33.00%  /boot
tmpfs           18032       0       18032       0.00%   /run/user/1000
                                mmc 5860016     kB  
                                ram 183496      kB  

The issue can be resolved through the following steps (tested on a Raspberry Pi Model B from 2011).

1. Disable "boot=overlay" in /boot/cmdline.txt and boot up as normal
2. Edit the /etc/dphys-swapfile to only use 10 MB swapfile size

CONF_SWAPSIZE=10

3. run the following commands

sudo dphys-swapfile swapoff
sudo dphys-swapfile setup
sudo dphys-swapfile swapon

4. Add back "boot=overlay" to cmdline.txt and reboot

I would appreciate if this gets fixed because I have spent many hours debugging this. I can submit a pull request for the fix if that is wanted, but I cannot test it on any other hardware. I am attaching the boot log.

boot_initrd_raspi_overlay.log

[XECDesign]

This area of raspi-config is going to change a lot in bookworm. For now, I think the right thing to do is to disable dphys-swapfile altogether before enabling overlayfs.

One problem with adding that fix to bullseye is that we don't know whether the user wants dphys-swapfile enabled. You could end up re-enabling it in situations where the user has previously disabled it manually.

If you have a PR that deals with this without being too convoluted, I could take a look at merging it for bullseye, after bookworm is out.

[oscwah]

I agree that disabling dphys-swapfile upon enabling overlay fs is better than reducing swap file size as I did. I also agree that a potential problem is to know whether to enable swap file at the time overlay is disabled. There are two ways I see to deal with this.

1. Ask the user in the overlay dialog after disabling overlay fs if he/she wants to enable swap file.
2. Store information (perhaps in a file /var/lib/raspberrypi/raspi-config) after enabling overlay fs, indicating if swap file is in use or not. When disabling overlay fs, use this information to determine whether to enable swap file or not.

Alt 1 might be seen as convoluted, but I think it is the most upfront alternative and I think it could be integrated with the boot partition option as a checklist to reduce number of key presses for the user. The second option is ok from my perspective but I don't know how it is viewed upon to write pkg state information. There might be unforseen consequences with this, but shouldn't be, I think.

I also think a message should be added in the dialog for disabling overlay fs that the swap file will be disabled. This will make it less confusing in the enable overlay dialog that swap is mentioned. For alt 2, I think a message indicating enabling of swap is good.

[upsuper]

It seems to me that when overlayfs is enabled, dphys-swapfile service fails regardless, possibly because overlayroot disables swap by default. At least after enabling overlayfs, I observed that there is just no swap in free, and dphys-swapfile.service is marked failed in systemctl status, at the same time /var/swap is still there taking 100MB of space.

I think it would be good for the script to automatically disable dphys-swapfile before enabling overlayfs, because it's such a footgun that is almost always going to force people redoing the overlayfs if they do notice the memory usage for some reason.

[XECDesign]

It seems to me that when overlayfs is enabled, dphys-swapfile service fails regardless

Is this on bullseye, bookworm or both?

[upsuper]

Is this on bullseye, bookworm or both?

Mine is currently on bookworm. I haven't tested bullseye.

[XECDesign]

Since you have to boot the pi to enable overlayroot, the swap file gets generated on the actual underlying filesystem. It fails to activate, but the file's presence shouldn't be eating any actual RAM.

Yeah, it's not ideal and worth a re-visit next time that area of raspi-config is updated, but it doesn't seem like it causes any actual problems.

Or am I missing something?

[upsuper]

Since you have to boot the pi to enable overlayroot, the swap file gets generated on the actual underlying filesystem.

That is true.

It fails to activate, but the file's presence shouldn't be eating any actual RAM.

But this is not. The file definitely appears in the upper filesystem as well. It seems when dphys-swapfile calls swapon -f /var/swap, the command fails, but it nevertheless still brings the whole swap file into the upper filesystem.

This is what I see on my machine:

$ free -h
               total        used        free      shared  buff/cache   available
Mem:           418Mi       163Mi       133Mi        19Mi       195Mi       255Mi
Swap:             0B          0B          0B

$ sudo dphys-swapfile swapon
swapon: /var/swap: swapon failed: Invalid argument

$ free -h
               total        used        free      shared  buff/cache   available
Mem:           418Mi       259Mi        67Mi       119Mi       264Mi       158Mi
Swap:             0B          0B          0B

$ ls -lh /media/root-rw/overlay/var/swap 
-rw------- 1 root root 100M Dec 11 15:56 /media/root-rw/overlay/var/swap

At least according to free it does indeed take actual RAM.

[XECDesign]

But this is not. The file definitely appears in the upper filesystem as well.

Oh right, then yeah that's definitely not great. I'll update dphys-swapfile so that it doesn't try to do any of this nonsense.

Many thanks for checking

[XECDesign]

Just uploaded a new version which seems to work for me.

[upsuper]

@XECDesign Where do you upload the new version? There doesn't seem to be any new commit in this repo.

[XECDesign]

It's a change to the dphys-swapfile package.