[RFE] Add Support for lz4 and zstd Kernel Modules

[nemezo]

Current Situation

Flatcar OS does not include the lz4 and zstd kernel modules, which are necessary for enabling certain compression features, particularly in the case of zram (a compressed RAM disk). zram is frequently used for swap and caching due to its efficient in-memory compression, reducing the overall memory footprint and improving performance on systems with limited memory resources. Without these kernel modules, zram cannot use the zstd or lz4 compression algorithms, which are considered optimal for many scenarios.

For example, attempting to enable zram with zstd compression results in a fatal error because the necessary module is missing from the kernel.

Example:

modprobe zstd

Error Output:

modprobe: FATAL: Module zstd not found in directory /lib/modules/6.6.35-flatcar

Impact

• Reduced Memory Efficiency: zram cannot utilize lz4 or zstd compression, forcing users to rely on less efficient compression algorithms or forego compression entirely. (The default lzo algorithm is less efficient than lz4 or zstd.)
• Performance Degradation: Systems that could benefit from faster, more efficient memory compression are limited, which may lead to degraded performance, especially in resource-constrained environments.
• Workload Impact: Workloads that depend on zram for memory optimization may experience failures or suboptimal performance.

This lack of support can be especially problematic for cloud deployments or embedded systems, where memory and storage efficiency is critical.

Ideal Future Situation

In the ideal future situation, Flatcar OS would include the lz4 and zstd kernel modules by default. This would allow users to take full advantage of modern compression techniques for zram and other workloads requiring efficient memory and storage compression. Users would be able to configure zram with lz4 or zstd, resulting in better memory performance and overall system efficiency. The system would also align with other Linux distributions, where these modules are typically available and utilized for similar workloads.

Implementation Options

1. Reintroduce lz4 and zstd Modules: These modules are available in Fedora CoreOS, the base of Flatcar OS, and can be included in the default kernel build. This would enable full support for memory compression using zram and improve performance and memory efficiency for workloads requiring these compression algorithms.

2. Provide Optional Kernel Packages: Flatcar OS could offer kernel modules as optional packages, allowing users to install lz4 and zstd modules only if needed, maintaining minimal system size while allowing flexibility for those who require them.

Additional Information

These lz4 and zstd kernel modules are available in Fedora CoreOS. It appears the Flatcar OS maintainers have intentionally removed them during the development process. Including these modules in the Flatcar kernel would bring the system in line with other Linux distributions and allow users to benefit from enhanced memory and compression features without having to rely on custom kernel builds or workarounds. The decision to remove these modules likely stems from a desire to maintain minimalism, but their absence severely impacts system performance for workloads requiring advanced memory compression.

I hope this information is helpful and provides a clear understanding of the issue and potential solutions. Please let me know if you need further details.

[jepio]

hi @nemezo,
the default that zram uses on flatcar would be lzo. Does lzo perform much worse than zstd or lz4 in your scenario?

[nemezo]

Hi @jepio,

Yes, LZO does perform worse compared to LZ4 or ZSTD when used with zram.

While I don’t have specific benchmark numbers on hand, you can check out this (old) Reddit thread for a comparison: zram tuning benchmarks.

In summary:

• LZ4 offers faster compression and decompression than LZO, with a better compression ratio, making it ideal for both memory efficiency and system responsiveness.
• ZSTD has the highest compression ratio, which can significantly improve memory utilization, though it is slower than LZ4 in terms of compression speed.

I’ve mostly been using LZ4 in my deployments because of its excellent balance between performance and memory efficiency. This is particularly noticeable in IoT devices, where memory resources are limited, and the efficiency gains from LZ4 can make a substantial difference in overall performance and system stability.

[jepio]

Would you confirm if this build works for you: https://bincache.flatcar-linux.net/images/amd64/9999.0.0+kmod-zstd-lz4/ ?

[nemezo]

@jepio Thank you very much for the updated build. I'm happy to report that both the lz4 and zstd kernel modules are now loading and working as expected.

I have successfully tested zram with both lz4 and zstd compression algorithms, and everything appears to be functioning well.

However, I encountered an issue with the Ignition process: the provided URL for the image was unavailable, as shown in the screenshot. The URL https://developer.release.flatcar-linux.net/amd64-usr/9999.9.0/flatcar_production_image.bin.bz2 resulted in an error during installation. So I just replaced my base OS with the one given by your URL instead of doing a full ignition.

Overall, I appreciate the effort, and I hope this update can be merged upstream soon.

Thank you again for your continued work on this!