SGX 2.24 reproducible build process is not fully using the NixOS package versions declared in Dockerfile

[berrange]

In 2.23 the SGX build env Dockerfile was referencing:

• GCC 8.5.0
• Binutils 2.38
• NASM 2.15.05
• GLibC 2.34
• Linux 5.17

In 2.24 the SGX build env Dockerfile shows changes for GCC, NASM and GLibC versions:

• GCC 9.5.0
• Binutils 2.38
• NASM 2.16.01
• GLibC 2.38
• Linux 5.17

With the changes between 2.23 and 2.24 build env shown by this diff

$ git diff -r sgx_2.23 -r sgx_2.24_reproducible  linux/reproducibility/Dockerfile
diff --git a/linux/reproducibility/Dockerfile b/linux/reproducibility/Dockerfile
index c7ae0f6d..4e28a01f 100644
--- a/linux/reproducibility/Dockerfile
+++ b/linux/reproducibility/Dockerfile
@@ -29,14 +29,14 @@ stdenvNoCC.mkDerivation { \n\
 \t\t/nix/store/ja1iw47v9jkfgnj97l5cjivzn84160w2-file-5.41 \n\
 \t\t/nix/store/h2yqbmb203mb75rnqpg0x72w8fyw0vmz-cmake-3.22.3 \n\
 \t\t/nix/store/h3811zhzwfjzl3csh82xhr31hd5awaiz-gnum4-1.4.19 \n\
-\t\t/nix/store/cgvn7nmqbkkr4pzr02ypr6421z17mmgh-openssl-1.1.1o \n\
-\t\t/nix/store/nlbwsch1j3vcs1yixrrc6zw6nf3rjs64-openssl-1.1.1o-dev \n\
+\t\t/nix/store/xk45sn1w4fcs5vrn0zjinjswscwf52mx-openssl-3.1.4 \n\
+\t\t/nix/store/8m90kan2g80r6s1brn4zzmlghbb74s6m-openssl-3.1.4-dev \n\
 \t\t/nix/store/0gz8dq0z6gx6lka4dbnmvbh47x1piri3-gnumake-4.2.1 \n\
 \t\t/nix/store/1j1i0x8p9xajqdlf0h0m38dpmb8ipxa6-linux-headers-5.17 \n\
-\t\t/nix/store/scd5n7xsn0hh0lvhhnycr9gx0h8xfzsl-glibc-2.34-210 \n\
+\t\t/nix/store/xmprbk52mlcdsljz66m8yf7cf0xf36n1-glibc-2.38-44 \n\
 \t\t/nix/store/wp7kc5y840ldi5sd8j81fj4dabccff2d-binutils-wrapper-2.38 \n\
-\t\t/nix/store/n435g2xggqp22jl66ybfmjxc61d729v1-gcc-wrapper-8.5.0 \n\
-\t\t/nix/store/msf8mgq5i2v5yha67aykmnqf2rd00r4f-nasm-2.15.05 \n\
+\t\t/nix/store/95nvd0v2xzf9vcv3kk4nmkjlcalc1jn7-gcc-wrapper-9.5.0 \n\
+\t\t/nix/store/bwc29jqdwk3gqya220f2lr3yvyw02rrm-nasm-2.16.01 \n\
 \t]; \n\
 \n\
 \tshellHook = '' \n\
@@ -57,15 +57,15 @@ RUN touch .bash_profile \
 && nix-env -i /nix/store/ja1iw47v9jkfgnj97l5cjivzn84160w2-file-5.41 \
 && nix-env -i /nix/store/h2yqbmb203mb75rnqpg0x72w8fyw0vmz-cmake-3.22.3 \
 && nix-env -i /nix/store/h3811zhzwfjzl3csh82xhr31hd5awaiz-gnum4-1.4.19 \
-&& nix-env -i /nix/store/cgvn7nmqbkkr4pzr02ypr6421z17mmgh-openssl-1.1.1o \
-&& nix-env -i /nix/store/nlbwsch1j3vcs1yixrrc6zw6nf3rjs64-openssl-1.1.1o-dev \
+&& nix-env -i /nix/store/xk45sn1w4fcs5vrn0zjinjswscwf52mx-openssl-3.1.4 \
+&& nix-env -i /nix/store/8m90kan2g80r6s1brn4zzmlghbb74s6m-openssl-3.1.4-dev \
 && nix-env -i /nix/store/0gz8dq0z6gx6lka4dbnmvbh47x1piri3-gnumake-4.2.1 \
 && nix-env -i /nix/store/1j1i0x8p9xajqdlf0h0m38dpmb8ipxa6-linux-headers-5.17 \
-&& nix-env -i /nix/store/scd5n7xsn0hh0lvhhnycr9gx0h8xfzsl-glibc-2.34-210 \
+&& nix-env -i /nix/store/xmprbk52mlcdsljz66m8yf7cf0xf36n1-glibc-2.38-44 \
 && nix-env -i /nix/store/wp7kc5y840ldi5sd8j81fj4dabccff2d-binutils-wrapper-2.38 \
 && nix-env --set-flag priority 10 binutils-wrapper-2.38 \
-&& nix-env -i /nix/store/n435g2xggqp22jl66ybfmjxc61d729v1-gcc-wrapper-8.5.0 \
-&& nix-env -i /nix/store/msf8mgq5i2v5yha67aykmnqf2rd00r4f-nasm-2.15.05
+&& nix-env -i /nix/store/95nvd0v2xzf9vcv3kk4nmkjlcalc1jn7-gcc-wrapper-9.5.0 \
+&& nix-env -i /nix/store/bwc29jqdwk3gqya220f2lr3yvyw02rrm-nasm-2.16.01
 
 
 #config nix-shell

I can successfully reproduce the SGX AE build for 2.22 and 2.23, but when I attempt to reproduce the AE build using the 2.24 release and its declared NixOX package versions, it fails.

Upon investigating this I've found that the build process is not in fact correctly using the glibc and binutils versions declared in the Dockerfile.

Code is being assembled with binutils 2.40, NOT 2.38.

This is because the update to GCC 9.5.0 has pulled in binutils 2.40, despite the NixOS config requesting 2.38. In fact both 2.38 and 2.40 appear to be installed in parallel, but 2.40 is the version being used.

We can see what is in $PATH, which matches the Dockerfile request for binutils 2.38:

$ which as
/nix/store/v72qcv0mz67akqxm2j8n799j3fvrxg9r-wp7kc5y840ldi5sd8j81fj4dabccff2d-binutils-wrapper-2.38/bin/as

$ as --version
GNU assembler (GNU Binutils) 2.38
Copyright (C) 2022 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or later.
This program has absolutely no warranty.
This assembler was configured for a target of `x86_64-unknown-linux-gnu'.

Actually running GCC though tells us the truth about what is used:

$ cat > hello.c
void foo(void) {}

$ gcc  -v -c hello.c 2>&1  | grep -E '(/as|Binutils)'
 /nix/store/95nvd0v2xzf9vcv3kk4nmkjlcalc1jn7-gcc-wrapper-9.5.0/bin/as -v --64 -o /tmp/cchCB1RW.o /tmp/ccXEP3BW.s
GNU assembler version 2.40 (x86_64-unknown-linux-gnu) using BFD version (GNU Binutils) 2.40

$ ls -al  /nix/store/95nvd0v2xzf9vcv3kk4nmkjlcalc1jn7-gcc-wrapper-9.5.0/bin/as
lrwxrwxrwx 1 user user 72 Jan  1  1970 /nix/store/95nvd0v2xzf9vcv3kk4nmkjlcalc1jn7-gcc-wrapper-9.5.0/bin/as -> /nix/store/505l13qqnpp0rvrja5s7rgwz5nzir579-binutils-wrapper-2.40/bin/as

IOW, GCC is using the binutils version symlinked from the gcc-wrapper-9.5.0 directory, not the version in $PATH requested in the SGX Dockerfile.

IPP Crypto is being built with older glibc 2.34, while the rest of SGX code is built with newer glibc 2.38

In this case the build output clearly shows an explicit include for glibc 2.34 -I/nix/store/dwsi3wsqpqm0hpzdm9fsxc7q732p9xwi-glibc-2.34-210-dev/include:

This glibc 2.34 is not what the Dockerfile requested, which was 2.38

Looking at the -save-temps output for other parts of the build shows the glibc 2.38 is correctly being used

After some investigation I found the cause is the file external/ippcp_internal/ipp-crypto/sources/ippcp/CMakeLists.txt which references CMAKE_SYSTEM_INCLUDE_PATH:

set (C_INCLUDE_DIRECTORIES
    ${IPP_CRYPTO_SOURCES_DIR}
    ${IPP_CRYPTO_SOURCES_DIR}/ecnist
    ${IPP_CRYPTO_SOURCES_DIR}/sm2
    ${IPP_CRYPTO_SOURCES_INCLUDE_DIR}
    ${IPP_CRYPTO_INCLUDE_DIR}
    ${INTERNAL_INCLUDE_DIR}
    # RSA_SB (ifma) uses crypto_mb headers
    ${IPP_CRYPTO_SOURCES_DIR}/crypto_mb/include
    $<$<C_COMPILER_ID:Intel>:$ENV{ROOT}/compiler/include $ENV{ROOT}/compiler/include/icc>
    $<$<NOT:$<C_COMPILER_ID:Intel>>:${CMAKE_SYSTEM_INCLUDE_PATH}>
    $<$<OR:$<C_COMPILER_ID:Intel>,$<BOOL:${MSVC_IDE}>>:$ENV{INCLUDE}>
    )

And the version of cmake requested in NixOS has harcoded this to glibc 2.34

$ cd /nix/store/1vjlia5qpa3ak7rsw9hhc7gxcra2yz9d-h2yqbmb203mb75rnqpg0x72w8fyw0vmz-cmake-3.22.3
$ grep --after 1 -r CMAKE_SYSTEM_INCLUDE_PATH  | grep glibc
share/cmake-3.22/Modules/Platform/UnixPaths.cmake-  /nix/store/dwsi3wsqpqm0hpzdm9fsxc7q732p9xwi-glibc-2.34-210-dev/include

I'm pretty surprised that IPP Crypto and SGXSSL were allowed to reference glibc system headers at all, given that they're intended to be used in enclaves with the custom SDK C library. None the less, having them use different glibc system headers in the same SGX reproducible build feels like a bad idea.

I'm unclear what the reasons were for updating glibc and gcc, but NOT updating binutils, in 2.24 release, but I'd at least expect the reprodicible build process to be using the versions declared in linux/reproducibility/Dockerfile rather than different versions pulled in as a side-effect of other packages.

It looks like the root cause here is the attempt to cherry-pick updates for just a subset of the build toolchain components. If all of cmake, gcc, binutils, glibc, nasm were updated to the same point in time for NixOS, then they would likely be a consistent set.

[lzha101]

I'm unclear what the reasons were for updating glibc and gcc, but NOT updating binutils, in 2.24 release, but I'd at least expect the reprodicible build process to be using the versions declared in linux/reproducibility/Dockerfile rather than different versions pulled in as a side-effect of other packages

The Dockerfile change in 2.24 is caused by the IPP-crypto upgrade ( OpenSSL 3.x is required and an upgraded glibc is required ). Will check the details and back here.