Pre-built architectural enclaves contain redundant NixOS ld-linux path

[berrange]

Attempting to reproduce the architectural enclaves on a non-NixOS platform fails, despite having the same toolchain versions. Debugging revealed that this is a result of the built enclaves containing a reference to the NixOS ld-linux path.

This can be easily seen by querying the INTERP section of the binaries

$ readelf -l libsgx_*.signed.so | grep --after 2 INTERP
  INTERP         0x00000000000002a8 0x00000000000002a8 0x00000000000002a8
                 0x0000000000000054 0x0000000000000054  R      0x1
      [Requesting program interpreter: /nix/store/scd5n7xsn0hh0lvhhnycr9gx0h8xfzsl-glibc-2.34-210/lib/ld-linux-x86-64.so.2]
--
  INTERP         0x00000000000002a8 0x00000000000002a8 0x00000000000002a8
                 0x0000000000000054 0x0000000000000054  R      0x1
      [Requesting program interpreter: /nix/store/scd5n7xsn0hh0lvhhnycr9gx0h8xfzsl-glibc-2.34-210/lib/ld-linux-x86-64.so.2]
--
  INTERP         0x00000000000002a8 0x00000000000002a8 0x00000000000002a8
                 0x0000000000000054 0x0000000000000054  R      0x1
      [Requesting program interpreter: /nix/store/scd5n7xsn0hh0lvhhnycr9gx0h8xfzsl-glibc-2.34-210/lib/ld-linux-x86-64.so.2]
--
  INTERP         0x00000000000002a8 0x00000000000002a8 0x00000000000002a8
                 0x0000000000000054 0x0000000000000054  R      0x1
      [Requesting program interpreter: /nix/store/scd5n7xsn0hh0lvhhnycr9gx0h8xfzsl-glibc-2.34-210/lib/ld-linux-x86-64.so.2]

IIUC, the ld-linux interpretor is irrelevant for SGX, since the SGX SDK contains native logic for loading code into the enclaves. If this is indeed the case, then I'd request that the reference to the NixOX ld-linux is removed from future pre-built enclave binaries.

Looking at the build system this could potentially be achieved by extending the 'strip' command usage to also purge the '.interp' section of the ELF binary.

$(STRIP) --strip-unneeded --remove-section=.comment --remove-section=.note --remove-section=.interp $(SONAME)

Alternatively the -Wl,-dynamic-linker arg could be used to set the .interp section contents to some dummy value to indicate it is redundant / unused e.g.

LDTFLAGS  = -L$(SGX_LIB_DIR) -Wl,--whole-archive $(TRTSLIB) -Wl,--no-whole-archive \
            -Wl,--start-group $(EXTERNAL_LIB) -Wl,--end-group -Wl,--build-id       \
            -Wl,--version-script=$(ROOT_DIR)/build-scripts/enclave.lds $(ENCLAVE_LDFLAGS) \
            -Wl,dynamic-linker,/bin/false

There are probably other options too - I don't mind as long as this (apparently) redundant NixOS path can be removed from future pre-built enclave binaries.

[lzha101]

@berrange The AE reproducibility build is based on the provided docker file with Nix env. So we assume the reproducibility verification is also based on the same environment from the provided docker.
May I know if the NixOS ld-path is the only issue that prevent you from reproducing the AEs in non Nix env?

[berrange]

@berrange The AE reproducibility build is based on the provided docker file with Nix env. So we assume the reproducibility verification is also based on the same environment from the provided docker. May I know if the NixOS ld-path is the only issue that prevent you from reproducing the AEs in non Nix env?

I'm aiming to distribute the pre-built AE's in Fedora, but as a pre-requisite for proposing that be allowed by Fedora, I need to be able to reproduce their build process within Fedora, not NixOS. This isn't easy, as obviously you've never expected/intended for this to be done. The three biggest issues that affect reproducibility which I'm having to patch

• This (apparently redundant) ld-linux path gets added to the AE .so binaries
• The asserts in the cpp library get generated with an absolute path which I have to rewrite with -fmacro-prefix-map GCC arg
• The NASM created .o files in ipp-crypto get an absolute path included which I have to rewrite by editting the ELF file to rewrite a symbol

I've not had time to investigate whether there's an easy way to avoid the absolute files paths in the last 2 cases, so I've not filed any issues requesting changes for those yet.

FYI, I've currently got reproducibility succeeding for SGX 2.22 release, and now working on 2.24 release for the qve.so AE that was re-issued. https://gitlab.com/berrange/fedora-sgx-ng-copr/-/tree/main/linux-sgx-enclaves-2_22?ref_type=heads Of course this all relies on me packaging the same versions of GCC, binutils, nasm that your NixOS env uses, which I've done in that same git repo.