JIT unable to find symbols from libc_nonshared.a

[hahnjo]

Right now, the JIT seems to be unable to find symbols from the static libc_nonshared.a, that is linked together with libc.so. Until ~2020, this library contained a number of wrappers, including the *stat functions. On these systems, for example EL 8 but also not-latest Ubuntus, the following C code:

#include <sys/stat.h>

int main() {
  struct stat s;
  lstat("build.ninja", &s);
  return 0;
}

when compiled into IR and then fed to lli fails with:

JIT session error: Symbols not found: [ lstat ]
./bin/lli: Failed to materialize symbols: { (main, { main }) }

The good news is that recent glibc heavily reduced the number of symbols in that static library, up to the point that it only contains at_quick_exit, atexit, pthread_atfork, and __stack_chk_fail_local. It would be nice to improve the situation for older systems, but not sure if we can do much about it...

[llvmbot]

@llvm/issue-subscribers-orcjit

[lhames]

@hahnjo Have you tried adding a StaticLibraryDefinitionGenerator for libc_nonshared.a? If the functions that you mentioned aren't linked and exported in your binary then the right thing to do is add the archive to your JITDylib and let the JIT pull them in.

[hahnjo]

Yes perfect, adding --extra-archive=/usr/lib64/libc_nonshared.a to the command line works! Should this be the default in LLJIT?

[lhames]

I think so. I think we want some sort of "add standard runtimes" utility, probably as part of SetUpExecutorNativePlatform in https://reviews.llvm.org/D144276. This could pull in compiler-rt and the MSVC runtime too.

We'd need to get he behavior right, e.g. what library search order should it use on each platform, and what should it do if a library is not found?

We could always starting experimenting with the idea in the lli tool itself --the stakes are lower there.

[lhames]

I chatted to @vgvassilev a little more about this on discord.

Adding the static archives to the JITDylib is the fully general solution: it will handle archives that have static constructors and destructors, etc.

I suspect that 99% of the time this is overkill, and it's sufficient to point these symbols at copies linked into the main executable, and that's what we've been living on for years. For in-process JITs it's sufficient to define absoluteSymbols pointing at the desired functions, but this won't work for out-of-process JITs. For the out-of-process case we can solve this once we have a pre-linked ORC runtime library: we can add a get_libc_workaround_symbols function to that which returns a mapping that can be added to the JIT in the controller process.

I'll leave this open for now to track the idea discussed above: having the platform (or SetUpExecutorNativePlatform utility) set up the archive automatically.

[hahnjo]

Thanks Lang, we've now implemented a workaround using absoluteSymbols in ROOT/Cling: root-project/root#13286