Add the EROFS rootfs support #9486

btw616 · 2023-10-07T15:15:31Z

This PR adds the EROFS rootfs support. No gofer process will be created for the container, when there is no need to pass through host files into a container via the gofer process. Annotations for rootfs are introduced to provide extra information, including the mount source, mount type and overlay config. busybox-static is also added to the default image and will be used to build the EROFS rootfs images during the test. The overlay medium is refactored from string to a new type OverlayMedium. Now all overlay medium related methods, such as validation and extracting anon directory, are implemented on this type. The GoferMountConf is decoupled to two layers to allow us to configure all combinations of a gofer mount in a succinct way.

cc @ayushr2

ayushr2

Thanks for the patch! I really like the idea of getting rid of the gofer process when no gofer mounts exist. Before, this was not possible because rootfs was always a gofer mount. So gofer was always needed. But EROFS changes that.

Before I review further, could you rebase this on master. e77deec is a relevant change that was submitted yesterday. Specifically see changes around GoferMountConf.ShouldUseLisafs().

pkg/erofs/erofs.go

runsc/container/container.go

btw616 · 2023-10-20T11:37:44Z

Before I review further, could you rebase this on master. e77deec is a relevant change that was submitted yesterday. Specifically see changes around GoferMountConf.ShouldUseLisafs().

Sure, will do. And I like the idea in e77deec!

btw616 · 2023-10-23T15:01:31Z

@ayushr2 I just pushed a new version. In this version, I did some refactors to GoferMountConf. Apart from overlay, a new option backend is also introduced, and SelfTmpfs is treated as a backend instead of an overlay configuration now. Currently, there are three supported backends: lisafs, tmpfs, erofs. In the future, we can add host backend (by extending pkg/sentry/fsimpl/host) for single file bind mounts (e.g. /etc/resolv.conf) which will allow us to get rid of the dependency on gofer process in this case. Please have a look when you have time. Thanks!

ayushr2

396d0be and 78a667c look good to me. Regarding 95397d22973675bb235c0819b36b067d85d7ea30, did you consider adding a test in runsc/container/container_test.go instead? You can easily set annotations there. It is easier to maintain tests there. Separately, I think that the test should come with the functionality (hence squash the test into the feature commit).

I might take some more time to chew the third commit, in the meantime you could make a separate PR with the checkpoint restore work and we can submit that. The rootfs support work is orthogonal.

pkg/sentry/fsimpl/erofs/erofs.go

runsc/boot/gofer_conf.go

btw616 · 2023-10-24T13:28:24Z

Regarding 95397d2, did you consider adding a test in runsc/container/container_test.go instead? You can easily set annotations there. It is easier to maintain tests there. Separately, I think that the test should come with the functionality (hence squash the test into the feature commit).

Makes sense, I will see how to add a test in runsc/container/container_test.go instead and will squash the test into the feature commit.

I might take some more time to chew the third commit, in the meantime you could make a separate PR with the checkpoint restore work and we can submit that.

Sure. I will make a separate PR with the checkpoint/restore work. The third commit does require and deserve more time for review, and thanks for taking time on it!

PS. Currently the rootfs support patch has a minimal dependency on the erofs.InternalFilesystemOptions introduced by the C/R support patch. I will drop it for now and will rebase and add it back again once the C/R support patch is merged.

@ayushr2

This PR adds the checkpoint/restore support to EROFS. A test for checkpoint/restore will be included in the rootfs support PR (#9486). cc @ayushr2 FUTURE_COPYBARA_INTEGRATE_REVIEW=#9577 from btw616:erofs-checkpoint-restore-support 78a667c PiperOrigin-RevId: 576143292

ayushr2 · 2023-10-25T19:31:05Z

Can you rebase?

btw616 · 2023-10-26T00:44:22Z

Can you rebase?

Sure, will do. Thanks!

pkg/sentry/fsimpl/erofs/save_restore.go

runsc/boot/gofer_conf.go

runsc/boot/loader.go

runsc/boot/mount_hints.go

runsc/boot/vfs.go

runsc/cmd/gofer.go

runsc/boot/mount_hints.go

runsc/cmd/gofer.go

ayushr2

Nice work! This is taking good shape.

runsc/boot/gofer_conf.go

runsc/boot/mount_hints.go

runsc/cmd/gofer.go

runsc/container/container.go

runsc/boot/mount_hints.go

@ayushr2

This patch decouples GoferMountConf to two layers to allow us to configure all combinations of a gofer mount in a succinct way: - Upper layer config: none, memory, self, anon. The upper layer is always tmpfs. It describes the backend for tmpfs. - Lower layer config: none, lisafs. It describes the backend for the filesystem which actually holds the image contents. The old SelfTmpfs will be represented as "upper=self,lower=none", MemoryOverlay will be "upper=memory,lower=lisafs", SelfOverlay will be "upper=self,lower=lisafs", and so on. Thanks to @ayushr2 for the suggestion on how to better decouple this. This is a preparation for adding the EROFS rootfs support. There is no functional change intended. Signed-off-by: Tiwei Bie <tiwei.btw@antgroup.com>

runsc/cmd/boot.go

runsc/container/BUILD

runsc/container/container.go

ayushr2

Thanks for your work.

For the first time, it will be possible to create goferless runsc sandboxes.

runsc/container/container_test.go

btw616 · 2023-11-06T10:45:36Z

For the first time, it will be possible to create goferless runsc sandboxes.

It won't be possible without all your constructive comments, they are really helpful! :)

runsc/container/container.go

This patch refactors the overlay medium from string to a new type OverlayMedium. Now all overlay medium related methods, such as validation and extracting anon directory, are implemented on this type. This is a preparation for adding the EROFS rootfs support. There is no functional change intended. Co-authored-by: Ayush Ranjan <ayushranjan@google.com> Signed-off-by: Tiwei Bie <tiwei.btw@antgroup.com>

ayushr2 · 2023-11-06T19:37:33Z

Could you apply the following patch? Our internal build system is not happy with package tests inheriting dependencies implicitly.

diff --git a/runsc/boot/BUILD b/runsc/boot/BUILD
--- a/runsc/boot/BUILD
+++ b/runsc/boot/BUILD
@@ -152,6 +152,7 @@ go_test(
         "//pkg/cpuid",
         "//pkg/fspath",
         "//pkg/log",
+        "//pkg/sentry/fsimpl/erofs",
         "//pkg/sentry/kernel/auth",
         "//pkg/sentry/seccheck",
         "//pkg/sentry/vfs",
diff --git a/runsc/container/BUILD b/runsc/container/BUILD
--- a/runsc/container/BUILD
+++ b/runsc/container/BUILD
@@ -91,6 +91,7 @@ go_test(
         "//pkg/cleanup",
         "//pkg/log",
         "//pkg/sentry/control",
+        "//pkg/sentry/fsimpl/erofs",
         "//pkg/sentry/kernel",
         "//pkg/sentry/kernel/auth",
         "//pkg/sentry/limits",

This patch adds the EROFS rootfs support. No gofer process will be created for the container, when there is no need to pass through host files into a container via the gofer process. Annotations for rootfs are also introduced to provide extra information, including the mount source, mount type and overlay config. Additionally, busybox-static is added to the default image and will be used to build the EROFS rootfs images during the test. Updates google#8956 Signed-off-by: Tiwei Bie <tiwei.btw@antgroup.com>

btw616 · 2023-11-07T00:19:26Z

Could you apply the following patch? Our internal build system is not happy with package tests inheriting dependencies implicitly.

I see. Just fixed it. Thanks!

btw616 marked this pull request as draft October 7, 2023 15:15

btw616 force-pushed the erofs-CR-and-rootfs-support branch from 6692c76 to a11a914 Compare October 18, 2023 13:47

btw616 marked this pull request as ready for review October 18, 2023 13:53

ayushr2 self-requested a review October 18, 2023 16:15

ayushr2 reviewed Oct 19, 2023

View reviewed changes

pkg/erofs/erofs.go Outdated Show resolved Hide resolved

runsc/container/container.go Outdated Show resolved Hide resolved

btw616 force-pushed the erofs-CR-and-rootfs-support branch from a11a914 to 95397d2 Compare October 23, 2023 14:51

ayushr2 reviewed Oct 24, 2023

View reviewed changes

pkg/sentry/fsimpl/erofs/erofs.go Outdated Show resolved Hide resolved

ayushr2 reviewed Oct 24, 2023

View reviewed changes

runsc/boot/gofer_conf.go Outdated Show resolved Hide resolved

btw616 mentioned this pull request Oct 24, 2023

Add the checkpoint/restore support to EROFS #9577

Merged

btw616 force-pushed the erofs-CR-and-rootfs-support branch from 95397d2 to 51df621 Compare October 24, 2023 14:35

copybara-service bot mentioned this pull request Oct 24, 2023

Add the checkpoint/restore support to EROFS #9578

Closed

copybara-service bot mentioned this pull request Oct 24, 2023

Add the checkpoint/restore support to EROFS #9579

Closed

btw616 force-pushed the erofs-CR-and-rootfs-support branch from 51df621 to 35cdafd Compare October 25, 2023 10:51

btw616 force-pushed the erofs-CR-and-rootfs-support branch from 35cdafd to 05371ac Compare October 26, 2023 01:08

ayushr2 reviewed Oct 30, 2023

View reviewed changes

btw616 force-pushed the erofs-CR-and-rootfs-support branch 2 times, most recently from 8ad8854 to 59d61fa Compare October 31, 2023 14:48

ayushr2 reviewed Oct 31, 2023

View reviewed changes

runsc/cmd/gofer.go Show resolved Hide resolved

btw616 force-pushed the erofs-CR-and-rootfs-support branch 2 times, most recently from 86f1534 to e699051 Compare November 1, 2023 13:08

ayushr2 reviewed Nov 1, 2023

View reviewed changes

btw616 force-pushed the erofs-CR-and-rootfs-support branch 2 times, most recently from 4b4ffa6 to 11cdf38 Compare November 4, 2023 13:57

ayushr2 changed the title ~~Add the EROFS checkpoint/restore and rootfs support~~ Add the EROFS rootfs support Nov 4, 2023

btw616 force-pushed the erofs-CR-and-rootfs-support branch from 11cdf38 to ea6f72e Compare November 5, 2023 01:10

ayushr2 reviewed Nov 5, 2023

View reviewed changes

btw616 force-pushed the erofs-CR-and-rootfs-support branch from ea6f72e to cae689e Compare November 5, 2023 14:13

ayushr2 reviewed Nov 5, 2023

View reviewed changes

runsc/container/container.go Outdated Show resolved Hide resolved

runsc/container/container.go Outdated Show resolved Hide resolved

ayushr2 approved these changes Nov 6, 2023

View reviewed changes

runsc/container/container_test.go Outdated Show resolved Hide resolved

runsc/container/container_test.go Outdated Show resolved Hide resolved

runsc/container/container_test.go Show resolved Hide resolved

btw616 force-pushed the erofs-CR-and-rootfs-support branch from cae689e to 160b6e2 Compare November 6, 2023 10:59

ayushr2 reviewed Nov 6, 2023

View reviewed changes

runsc/container/container.go Outdated Show resolved Hide resolved

btw616 force-pushed the erofs-CR-and-rootfs-support branch 2 times, most recently from 0126e20 to e71f35b Compare November 6, 2023 16:18

ayushr2 added the ready to pull label Nov 6, 2023

btw616 force-pushed the erofs-CR-and-rootfs-support branch from e71f35b to bff1150 Compare November 7, 2023 00:18

copybara-service bot mentioned this pull request Nov 7, 2023

Add the EROFS rootfs support #9648

Closed

copybara-service bot merged commit 3ec902d into google:master Nov 9, 2023
5 checks passed

hsiangkao mentioned this pull request Nov 14, 2023

EROFS snapshotter & differ containerd/containerd#9361

Open

hsiangkao mentioned this pull request Jun 7, 2024

performance: what can image-spec do to improve handling of large images? opencontainers/image-spec#1190

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add the EROFS rootfs support #9486

Add the EROFS rootfs support #9486

btw616 commented Oct 7, 2023 •

edited

Loading

ayushr2 left a comment

btw616 commented Oct 20, 2023

btw616 commented Oct 23, 2023

ayushr2 left a comment

btw616 commented Oct 24, 2023

ayushr2 commented Oct 25, 2023

btw616 commented Oct 26, 2023

ayushr2 left a comment

ayushr2 left a comment

btw616 commented Nov 6, 2023

ayushr2 commented Nov 6, 2023

btw616 commented Nov 7, 2023

Add the EROFS rootfs support #9486

Add the EROFS rootfs support #9486

Conversation

btw616 commented Oct 7, 2023 • edited Loading

ayushr2 left a comment

Choose a reason for hiding this comment

btw616 commented Oct 20, 2023

btw616 commented Oct 23, 2023

ayushr2 left a comment

Choose a reason for hiding this comment

btw616 commented Oct 24, 2023

ayushr2 commented Oct 25, 2023

btw616 commented Oct 26, 2023

ayushr2 left a comment

Choose a reason for hiding this comment

ayushr2 left a comment

Choose a reason for hiding this comment

btw616 commented Nov 6, 2023

ayushr2 commented Nov 6, 2023

btw616 commented Nov 7, 2023

btw616 commented Oct 7, 2023 •

edited

Loading