Bug: stacker has problems running on a tmpfs

[smoser]

stacker version

v1.0.0-rc9-a576aa3

Describe the bug

whiteouts do not seem to work building on a tmpfs.
I saw this when trying to run the 'dir whiteouts work' test in test/dir-whiteout.bats on a tmpfs.

The following stacker.yaml

base:
    from:
        type: docker
        # url: oci:/tmp/sync-oci.d:busybox:latest
        url: docker://busybox:latest
    run: |
        mkdir /foo
        touch /foo/bar
mid:
    build_only: true
    from:
        type: built
        tag: base
    run: |
        rm -f /foo/bar
        rm -Rf /foo

I'm in a dir that is on a tmpfs.

$ df -h .
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           128G   13M  128G   1% /mnt/tmpfs

$ grep /mnt/tmpfs /proc/1/mountinfo 
1898 28 0:97 / /mnt/tmpfs rw,relatime shared:34 - tmpfs tmpfs rw,size=201326592k,inode64
856 1898 0:113 / /mnt/tmpfs rw,relatime shared:464 - tmpfs tmpfs rw,size=134217728k,inode64

$ uname -r
6.2.0-31-generic

Running 'stacker build' produces:

$ stacker build
preparing image base...
loading docker://busybox:latest
Copying blob 3f4d90098f5b skipped: already exists  
Copying config 5ed23df91f done  
Writing manifest to image destination
Storing signatures
found cached layer base
preparing image mid...
cache miss because layer definition was changed
+ rm -f /foo/bar
+ rm -Rf /foo
rm: can't remove '/foo': Input/output error
error: run commands failed: execute failed: exit status 1
error: exit status 1

It fails the same way privileged (via sudo).

It generates some dmesg output

$ dmesg | tail
[7867619.584151] overlayfs: failed to set xattr on upper
[7867619.584158] overlayfs: ...falling back to xino=off.
[7867619.667236] overlayfs: failed to set xattr on upper
[7867619.667243] overlayfs: ...falling back to xino=off.

To reproduce

No response

Expected behavior

No response

Screenshots

No response

Additional context

No response

[smoser]

With --debug

$ stacker --debug build
stacker version v1.0.0-rc9-a576aa3
usernsexec-ing [u 0 1001 1 1 231073 65535 g 0 1001 1 1 231073 65535 -- /usr/local/bin/stacker --internal-userns --debug build]
stacker version v1.0.0-rc9-a576aa3
initializing stacker recipe: stacker.yaml
substituting $STACKER_ROOTFS_DIR to /mnt/tmpfs/tmp/smt/roots
substituting $STACKER_STACKER_DIR to /mnt/tmpfs/tmp/smt/.stacker
substituting $STACKER_OCI_DIR to /mnt/tmpfs/tmp/smt/oci
substituting $STACKER_WORK_DIR to
stacker build order:
0 build /mnt/tmpfs/tmp/smt/stacker.yaml: requires: []
building: 0 /mnt/tmpfs/tmp/smt/stacker.yaml
substituting $STACKER_ROOTFS_DIR to /mnt/tmpfs/tmp/smt/roots
substituting $STACKER_STACKER_DIR to /mnt/tmpfs/tmp/smt/.stacker
substituting $STACKER_OCI_DIR to /mnt/tmpfs/tmp/smt/oci
substituting $STACKER_WORK_DIR to
Dependency Order [base mid]
preparing image base...
overlay-dirs, possibly modified after import: []
loading oci:/tmp/sync-oci.d:busybox:latest
Copying blob 3f4d90098f5b skipped: already exists
Copying config 5ed23df91f done
Writing manifest to image destination
Storing signatures
unpacking to /mnt/tmpfs/tmp/smt/roots/base
lxc rootfs overlay arg overlayfs:/mnt/tmpfs/tmp/smt/roots/sha256_3f4d90098f5b5a6f6a76e9d217da85aa39b2081e30fa1f7d287138d6e7bf0ad7/overlay:/mnt/tmpfs/tmp/smt/roots/base/overlay
stacker version v1.0.0-rc9-a576aa3
stacker subcommand: [/usr/local/bin/stacker --oci-dir /mnt/tmpfs/tmp/smt/oci --roots-dir /mnt/tmpfs/tmp/smt/roots --stacker-dir /mnt/tmpfs/tmp/smt/.stacker --storage-type overlay --debug internal-go check-aa-profile lxc-container-default-cgns]
bind mounting /mnt/tmpfs/tmp/smt/.stacker/imports/base into container at /stacker/imports
+ mkdir /foo
+ touch /foo/bar
Generating overlay_dirs layers for base
generated tar layer sha256:c1d2d91ae410ec74861bf986841312a769ecd878fe1e245201d45bdc04d7f7e7 from /mnt/tmpfs/tmp/smt/roots/base/overlay
renaming /mnt/tmpfs/tmp/smt/roots/base/overlay -> /mnt/tmpfs/tmp/smt/roots/sha256_c1d2d91ae410ec74861bf986841312a769ecd878fe1e245201d45bdc04d7f7e7/overlay
filesystem base built successfully
preparing image mid...
overlay-dirs, possibly modified after import: []
lxc rootfs overlay arg overlayfs:/mnt/tmpfs/tmp/smt/roots/base/overlay:/mnt/tmpfs/tmp/smt/roots/sha256_c1d2d91ae410ec74861bf986841312a769ecd878fe1e245201d45bdc04d7f7e7/overlay:/mnt/tmpfs/tmp/smt/roots/sha256_3f4d90098f5b5a6f6a76e9d217da85aa39b2081e30fa1f7d287138d6e7bf0ad7/overlay:/mnt/tmpfs/tmp/smt/roots/mid/overlay
stacker version v1.0.0-rc9-a576aa3
stacker subcommand: [/usr/local/bin/stacker --oci-dir /mnt/tmpfs/tmp/smt/oci --roots-dir /mnt/tmpfs/tmp/smt/roots --stacker-dir /mnt/tmpfs/tmp/smt/.stacker --storage-type overlay --debug internal-go check-aa-profile lxc-container-default-cgns]
bind mounting /mnt/tmpfs/tmp/smt/.stacker/imports/mid into container at /stacker/imports
+ rm -f /foo/bar
+ rm -Rf /foo
rm: can't remove '/foo': Input/output error
error: run commands failed: execute failed: exit status 1
stackerbuild.io/stacker/pkg/stacker.(*Builder).build
        /stacker-tree/pkg/stacker/build.go:502
stackerbuild.io/stacker/pkg/stacker.(*Builder).BuildMultiple
        /stacker-tree/pkg/stacker/build.go:622
main.doBuild
        /stacker-tree/cmd/stacker/build.go:118
github.com/urfave/cli/v2.(*Command).Run
        /stacker-tree/.build/gopath/pkg/mod/github.com/urfave/cli/v2@v2.25.0/command.go:273
github.com/urfave/cli/v2.(*Command).Run
        /stacker-tree/.build/gopath/pkg/mod/github.com/urfave/cli/v2@v2.25.0/command.go:266
github.com/urfave/cli/v2.(*App).RunContext
        /stacker-tree/.build/gopath/pkg/mod/github.com/urfave/cli/v2@v2.25.0/app.go:332
github.com/urfave/cli/v2.(*App).Run
        /stacker-tree/.build/gopath/pkg/mod/github.com/urfave/cli/v2@v2.25.0/app.go:309
main.main
        /stacker-tree/cmd/stacker/main.go:328
runtime.main
        /usr/lib/go/src/runtime/proc.go:267
runtime.goexit
        /usr/lib/go/src/runtime/asm_amd64.s:1650
error: exit status 1
stackerbuild.io/stacker/pkg/container.MaybeRunInNamespace
        /stacker-tree/pkg/container/userns.go:102
main.main.func3
        /stacker-tree/cmd/stacker/main.go:323
github.com/urfave/cli/v2.(*Command).Run
        /stacker-tree/.build/gopath/pkg/mod/github.com/urfave/cli/v2@v2.25.0/command.go:213
github.com/urfave/cli/v2.(*App).RunContext
        /stacker-tree/.build/gopath/pkg/mod/github.com/urfave/cli/v2@v2.25.0/app.go:332
github.com/urfave/cli/v2.(*App).Run
        /stacker-tree/.build/gopath/pkg/mod/github.com/urfave/cli/v2@v2.25.0/app.go:309
main.main
        /stacker-tree/cmd/stacker/main.go:328
runtime.main
        /usr/lib/go/src/runtime/proc.go:267
runtime.goexit
        /usr/lib/go/src/runtime/asm_amd64.s:1650