DietPi-Drive_Manager | Add native Paragon NTFS driver support

[GvY85]

Hi,

In kernel 5.15 support for NTFS is implemented so you dont have to rely on NTFS-3G via FUSE any longer.
So more or less the same as has been done for exFAT support in kernel 5.4.

Now that Bullseye-Backports probably will support kernel 5.15 in the next couple of days/weeks the question is if Dietpi needs to be adjusted in order to use NTFS kernel instead of NTFS-3G?

Perhaps still viable for 8.0 ;)?

[MichaIng]

Wow, how could I have missed that, this is totally awesome news. Here the details of the implementation: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f7464060f7ab9a2424428008f0ee9f1e267e410f

As far as I can see, UNIX permissions are natively supported, which is what I was first concerned about reading it. Though this should be tested. I even see a function "used to create symlinks", so this may not even make the CPU intense ntfs-3g driver obsolete but solve some limitations it still had. But again, should be tested carefully.

So yes we can implement it just the same way as we did for exFAT:

• Identify the related kernel driver/module to check for support
• Check for in case new userspace utilities (mkfs/fsck/...) to manipulate NTFS filesystems with the new driver
• Identify in case new mount options we may want/need to add. permissions is a ntfs-3g special, so would then need to be skipped.

[GvY85]

Good to see you are as enthousiastic as I am :).
5.15 is already in Bookworm/Testing so tests can probably be done in a VM that uses the Testing repo.
If 5.15 makes it to Backports before you release support I am happy to (beta) test as well but I don't want to upgrade my running servers (2) to Testing.

[MichaIng]

Perfect timing that I created and uploaded VirtualBox and RPi Bookworm images yesterday for testing reasons 💯, both now being used within the common testing schedule: https://dietpi.com/downloads/images/
If it helps, I can create further Bookworm images, like for PCs, other virtualizers or SBCs.

[Joulinar]

@MichaIng for RPI it's a ARMv6 image on Bookworm only, correct?

[MichaIng]

Ah yes that's true. I distro upgraded mine yesterday as debootstrap somehow was hanging at finding any package. Will retry to upload an ARMv7 image and ARMv8 consequently.

[Joulinar]

btw 5.15 is available on RPi OS as next LTS version https://forums.raspberrypi.com/viewtopic.php?t=322879

sudo rpi-update next

[GvY85]

Good news: 5.15 is imminent on Debian-Backports
https://tracker.debian.org/news/1292707/accepted-linux-5155-2bpo111-source-into-bullseye-backports-bullseye-backports/

[MichaIng]

Not quite yet, but yes it should arrive soon: https://packages.debian.org/bullseye-backports/linux-image-amd64

[GvY85]

Hmm, I wanted to try it out myself and upgraded a VM to Testing but apparently the Debian Kernel people don't trust NTFS3 enough (yet) so they have not enabled it in the Debian kernel:

linux: please enable the new NTFS3 driver in 5.15
linux-image-5.15.0-1-amd64: Linux 5.15.3 kernel is missing the NTFS3 driver
linux-image-amd64: NTFS3 module missing in new kernels

Good news is that when they do it is very simple to use:
kernel 5.15 - ntfs3 driver

[MichaIng]

The bigger issue than stability/trust is that there are no userland utilities yet to create and fsck an NTFS filesystem with the new kernel driver. So while you can enable it, you have no tools to use it (unless developed and compiled yourself). Paragon seems to plan a mkfs.ntfs but no word about an fsck.ntfs. And new tools are not gonna be added to Debian Bullseye even when available, so this is more a topic for Debian Bookworm.

However, it may at least be tested whether mkfs.ntfs and fsck.ntfs from ntfs-3g package work with the new kernel driver as well. I guess the fuse-based kernel module would need to be blacklisted or even fake-enabled to assure it is not loaded and takes over.

As of the many open questions and no kernel with this driver shipped available yet, I take this request off the milestone. But we will track development around it and ship support as fast as it is available, even when only for Bookworm and up.

[WolfganP]

The bigger issue than stability/trust is that there are no userland utilities yet to create and fsck an NTFS filesystem with the new kernel driver. So while you can enable it, you have no tools to use it (unless developed and compiled yourself). Paragon seems to plan a mkfs.ntfs but no word about an fsck.ntfs. And new tools are not gonna be added to Debian Bullseye even when available, so this is more a topic for Debian Bookworm.

However, it may at least be tested whether mkfs.ntfs and fsck.ntfs from ntfs-3g package work with the new kernel driver as well. I guess the fuse-based kernel module would need to be blacklisted or even fake-enabled to assure it is not loaded and takes over.

As of the many open questions and no kernel with this driver shipped available yet, I take this request off the milestone. But we will track development around it and ship support as fast as it is available, even when only for Bookworm and up.

Completely agree, it seems to be in the same thinking line as debian devs https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=998627

[MichaIng]

Good to see some updates on the bug report/request. Sad to read that the module maintainer has not shown any activity since November.

Let's keep an eye on that. As fast as the module has been enabled on Debian kernel, there will be userland tools available to make use of them, even if it's the existing ntfs-3g package, so then we can add it to dietpi-drive_manager as well.

[tundrablasen]

Is ntfs support now native in dietpi 8.12? @MichaIng

[MichaIng]

It has not been enabled in Debian kernel yet:

# modinfo ntfs3
modinfo: ERROR: Module ntfs3 not found.

Also see above linked discussion in Debian bug report.

Dedicated userspace tools btw seem to be not required, as the ones from ntfs-3g seem to work. But if it behaves similar to exFAT, the fuse driver shipped with ntfs-3g would be taken with priority even if the new ntfs3 driver was enabled. Hence the discussion includes splitting userspace tools off from the ntfs-3g package.

Also on Linux side the current status is unclear to me: https://lore.kernel.org/lkml/2433164a-4546-ca21-cdf2-441d3a492e0f@gmail.com/

• So Paragon states it is actively maintained but is waiting for a Linux Git account/GPG key being signed via personal meeting, which was difficult during COVID.
• Userland tools are/were planned as well.
• Last mail however was 7 months ago.

When I'm back home, I can check whether Armbian and Raspberry Pi have it enabled. You can check via:

modinfo ntfs3

[Thunderlab]

Is ntfs3 already implemented in DietPi?

[MichaIng]

It has still not been enabled for Debian kernel builds yet, but people keep asking: https://bugs.debian.org/998627

But the RPi kernel has it:

root@micha:~# modinfo ntfs3
filename:       /lib/modules/6.1.21-v7+/kernel/fs/ntfs3/ntfs3.ko.xz
alias:          fs-ntfs3
author:         Konstantin Komarov
description:    ntfs3 read/write filesystem
license:        GPL
srcversion:     F3E85C82A8376CFC31D58A7
depends:
intree:         Y
name:           ntfs3
vermagic:       6.1.21-v7+ SMP mod_unload modversions ARMv7 p2v8

There you have the command to check it. Not sure about Armbian kernel builds, can check this later. When loading this module, indeed a new filesystem type ntfs3 appears:

root@micha:~# cat /proc/filesystems | grep ntfs
root@micha:~# modprobe ntfs3
root@micha:~# cat /proc/filesystems | grep ntfs
        ntfs3

So generally this seems to work. Then we need to test userland tools. The ntfs-3g package comes together with the FUSE-based driver. We need to test whether it can mount NTFS drives using the kernel driver. There is no mount.ntfs3 and also no other dedicated executable wich refers to the kernel driver explicitly, so mount -t ntfs3 /mnt/foo might fail, and mount -t ntfs /mnt/foo uses FUSE, unless there is some flag or setting.

[Thunderlab]

I've just tested ntfs3, and it's working perfectly! I mounted the disk through fstab, and everything indicates that this new file system indeed uses fewer resources during transfers.

All additional mounting options are also working correctly!
https://docs.kernel.org/filesystems/ntfs3.html

[MichaIng]

Coincidentally I just found the ntfs3 Linux driver for native NTFS support pretty functional as well:

modinfo ntfs3
modprobe ntfs3
mount -t ntfs3 /dev/sda1 /mnt/test

The module must be of course present. The Debian and RPi kernels do not have it, but the Armbian kernel does. If you do not define -t ntfs3 explicitly (or ntfs3 as filesystem in /etc/fstab), it uses the FUSE driver by default. ntfs-3g is still required for fsck, mkfs etc, hence the FUSE driver is forcefully installed as well, but those tools to test/generate the filesystem are of course independent of the driver to mount it and access its content. Benchmark:

BENCH_CUSTOMFS_WRITE='39'
BENCH_CUSTOMFS_READ='89'

vs FUSE:

BENCH_CUSTOMFS_WRITE='5'
BENCH_CUSTOMFS_READ='89'

with the write speed varying very much between tests. I had 1 MiB/s once, often 3 MiB/s, 5 was the fastest. It is regularly hanging for a random time while writing. With the native Paragon NTFS driver, it just writes through. Read performance on the other hand is pretty much the same, at least with the USB stick I tested. With fast SSDs this might be different. The other difference and possible bottleneck is that the FUSE driver results in a process with ~70% CPU usage while writing (not sure while reading) on an RK3399 SoC. With the native driver, there is nearly no increased CPU usage. And with FUSE, I had a failure during one benchmark: the test file could not be removed anymore and the mount was not accessible with I/O errors. I needed to run unmount a few times before it finally unmounted. Not sure whether stability issues with FURE/ntfs-3g are known. EDIT: Was an issue with the USB dongle, not the driver.

Means we could add this driver with our own Linux builds and test via modprobe -n ntfs3 whether it is present. In case it is, we could add NTFS filesystems with ntfs3 mark to /etc/fstab, to have them automatically mounted using the native driver.

[mdr92]

Any updates on this? I've just mounted an NTFS drive and was surprised to see it still installing ntfs-3g and I'm getting really poor write speeds so an improvement would be nice. Pi 3 getting 12mb/s on a usb 3.0 drive (I know the pi 3 is only usb 2.0). Thanks

[MichaIng]

Makes sense to start implementing it once the kernel module is part of Debian and RPi kernels. Can someone check on current Debian Bullseye and RPi after firmware transition? I can otherwise do in some hours when back home.

[MichaIng]

Okay neither RPi kernel nor Debian (not even Trrixie/testing) have the module compiled. For Armbian-based kernels and own builds it does not seem too reasonable. I mean it is not much work either, but when Debian decides to not implement it yet, not even for next Debian version, there is probably a reason.

[mdr92]

Fair enough, that makes sense, hopefully one day it'll get picked up with the performance benefits that come with it.