Robust bindgen support #140

lifthrasiir · 2023-06-09T09:11:14Z

It turned out that #74, which added bindgen support, was pretty much incomplete. For example the following C code will fail to compile during bindgen.

#include <stdlib.h>

It should be noted that Zig contains multiple libc header files:

zig/lib/libc/glibc/include etc. are used to build libc on demand. They are not meant to be directly included, and in fact contain a small subset of libc headers which are used by other libc subsystems. Consequently files in this directory will most likely fail to compile; this includes stdlib.h, errno.h, pthread.h and signal.h.
zig/lib/include contains header files shared by all targets. They are mostly platform-specific intrinsics.
zig/lib/libc/include/$TARGET contains header files specific to given target. For given target there may be multiple such directories, and some directories are referenced from Zig but not present and ignored by clang driver.
Finally, zig/lib/libcxx/include and zig/lib/libcxxabi/include are included before any other headers because libc++ itself contains some C headers that are not necessarily in libc (e.g. uchar.h).

Right now cargo-zigbuild only concerns with 1, so while some headers won't compile at all, other headers will inadvertently refer to the system library. This PR makes cargo-zigbuild to include 2 and 3, and if required, 4 as well. There are now explicit tests to include (almost) all standard C and C++ header files.

As noted from a lengthy comment that I don't really repeat here, the current strategy of overriding BINDGEN_EXTRA_CLANG_ARGS is not really ideal. I have considered alternative approaches, and unfortunately none of them would work out of the box. In the long term we may get clang-sys fixed and then put target-specific zig cc wrappers to PATH so that clang-sys can infer a correct search path by itself, but it will take a long time for every one to use a fixed clang-sys. So I've resorted to this hack for now.

lifthrasiir · 2023-06-09T09:34:33Z

Whoops... I realized that this approach alone won't work in general because bindgen will still look at older clang anyway (which we can't currently change). So if you have a recent enough clang (I believe, 13 or later), this would indeed work, but this PR is still not complete yet.

messense · 2023-06-11T03:45:45Z

It'd be great if we can merge a solution even if it only works for newer clang versions.

TheButlah · 2023-06-13T22:07:08Z

log.txt
I tried this while compiling to --target aarch64-unknown-linux-gnu and couldn't get it to work. I can't disclose my -sys crate but it was pretty simple, and I've printed the full expansion of the bindgen builder and my environment variables.

lifthrasiir · 2023-06-14T02:13:15Z

log.txt I tried this while compiling to --target aarch64-unknown-linux-gnu and couldn't get it to work. I can't disclose my -sys crate but it was pretty simple, and I've printed the full expansion of the bindgen builder and my environment variables.

Huh, this looks pretty bad indeed. So to summarize what had happened here, we are trying to build aarch64-unknown-linux-gnu from an aarch64-apple-darwin host, but my version of zigbuild completely failed to detect include directories from zig and only the Xcode include directories are used.

@TheButlah, can you post what the following command prints? It should say something like #include <...> search starts here: and a bunch of include directories, which should have been picked up.

$ /Users/ryan.butler/Library/Caches/cargo-zigbuild/0.16.10/zigcc-aarch64-unknown-linux-gnu.sh -E -x c /dev/null -v

TheButlah · 2023-06-14T04:41:03Z

Sure, here you go. I messed with some stuff (specifically installing and uninstalling cross-compilation toolchains, llvm, adding llvm to path, etc) since my original message (I think? I don't remember) and wanted both logs to be in sync, so I just reran the cargo zigbuild for the below logs.

command you asked for
corresponding cargo zigbuild --target aarch64-unknown-linux-gnu

All wrapper.h does currently is #include <stdio.h>

TheButlah · 2023-06-14T04:52:26Z

❯ brew list llvm --versions
llvm 16.0.5
                                                                                   
❯ llvm-config --libdir --includedir --prefix
/opt/homebrew/Cellar/llvm/16.0.5/lib
/opt/homebrew/Cellar/llvm/16.0.5/include
/opt/homebrew/Cellar/llvm/16.0.5
                                                                                   
❯ echo $PATH
/opt/homebrew/opt/mongodb-community@4.4/bin:/opt/homebrew/opt/llvm/bin:/Users/ryan.butler/.local/bin:/Users/ryan.butler/Library/Caches/fnm_multishells/66198_1686694763407/bin:/Users/ryan.butler/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/Users/ryan.butler/.local/bin:/Users/ryan.butler/Library/Caches/fnm_multishells/86670_1685625262144/bin:/Users/ryan.butler/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/Library/Frameworks/Python.framework/Versions/3.10/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Users/ryan.butler/.cargo/bin
                                                                                   
❯ echo $CPPFLAGS

❯ echo $LDFLAGS

❯ clang --version
Homebrew clang version 16.0.5
Target: arm64-apple-darwin22.4.0
Thread model: posix
InstalledDir: /opt/homebrew/opt/llvm/bin
                                                                                   
❯ xcode-select -p
/Library/Developer/CommandLineTools
                                                                                   
❯ zig cc --version
Homebrew clang version 15.0.7
Target: arm64-apple-darwin22.4.0
Thread model: posix
InstalledDir: /usr/bin

❯ zig env
{
 "zig_exe": "/opt/homebrew/Cellar/zig/0.10.1/bin/zig",
 "lib_dir": "/opt/homebrew/Cellar/zig/0.10.1/lib/zig",
 "std_dir": "/opt/homebrew/Cellar/zig/0.10.1/lib/zig/std",
 "global_cache_dir": "/Users/ryan.butler/.cache/zig",
 "version": "0.10.1",
 "target": "aarch64-macos.13.3.1...13.3.1-none"
}

❯ /usr/bin/clang --version
Apple clang version 14.0.3 (clang-1403.0.22.14.1)
Target: arm64-apple-darwin22.4.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

lifthrasiir · 2023-06-14T08:01:00Z

command you asked for

Thank you. Unfortunately all lines have been truncated past 83 columns so I can't easily see the exact commands zig has composed, but it seems clear that zigcc-aarch64-unknown-linux-gnu doesn't work as we want, especially given that it has no glibc include directories at all. And it turned out that -x c is only properly recognized in the development version of Zig! It was only added this year and I didn't realize that. For now, you can reinstall Zig with the development version (brew install zig --HEAD I believe) and try again.

TheButlah · 2023-06-14T13:14:10Z

weird, sorry. Here is the un-truncated output on the not-HEAD version of zig. I'll post the HEAD version shortly.
command.txt

Edit: brew install zig --HEAD doesn't build for me, so I can't run that.

The previous approach to use `--sysroot` was pretty much incomplete, as zig uses multiple include directories to reduce the bundle size. This commit implements a future-proof strategy to determine correct compile options by using `zig cc` itself, which is also what bindgen (transitively via clang-sys) does internally. Also fixes a couple of minor issues: - Handles all three kinds of `BINDGEN_EXTRA_CLANG_ARGS` environment variable correctly (one of them was missing previously). - Existing `BINDGEN_EXTRA_CLANG_ARGS` no longer blocks bindgen support. Collected options are instead appended to them.

Zstd-rs didn't include enough headers and the incomplete bindgen support was simply overlooked. For example, `stdlib.h` didn't work.

Bindgen should have accepted a correct TARGET environment variable, so it makes sense to set only target-specific environment variables. This also fixes a small bug when only the pan-target variable is set, in which case the newly set target-specific variable should first inherit the pan-target value and then more options should have been appended, but the pan-target value was discarded previously.

Only the master version of Zig recognizes it, all existing releases don't and adds proper include directories only when a file with a corresponding extension is added. Note: this will write additional empty files to the cache directory, which should be writable (otherwise everything else will fail anyway). Since the cache directory may end up being cwd, this commit tried to use a specific enough file name to avoid collision.

This is specific to Zig because while Zig ships with the recent enough glibc it dynamically sets `__GLIBC_MINOR__` according to the target. At this point it seems more robust to collect *all* options from `zig cc` output, but this is not as viable because all we can see is cc1 options (`-###` or `-v`), not clang driver options from Zig. It seems that we have no direct means to collect the latter, so we have to be conservative about which options to collect.

It turned out that unused argument warnings (`-Wunused-command-line- argument`) come from cc1, not from driver, and as long as all options are "claimed"---not necessarily used---it doesn't issue warnings. On the other hands, Zig internally adds `-nostdinc` but we didn't because we wanted header-only C++ libraries in the host to be usable, but it turned out that the presence of `__has_include` made this horribly fragile (e.g. some versions of libc++ used `__has_include( <pthread.h>)` to determine pthread support), so we gave up. If this facility is really desired, consider using `-idirafter /usr/include`.

The latter is also a recent addition, and `-nostdinc` already implies `nostdinc++` so we no longer need guarantees from `-stdlib++-isystem`. (We can't still replace duplicate `-c-isystem` and `-cxx-isystem` into a single `-isystem` due to the relocation.)

.github/workflows/CI.yml

This in particular affects `__has_include` new in C++17, which is also how the test internally works. Tested files are chosen so that there is no chance that other OSes have matching header files by accident.

lifthrasiir · 2023-06-17T09:40:40Z

I'm now modestly confident about this PR, having passed the CI. Newer commits should work from all supported clang versions. I've also manually checked against 0.9.0 from my local machine (WSL Ubuntu 18.04 LTS, clang 10.0, glibc 2.27), additional verifications would be appreciated though.

@TheButlah I think this PR should work now without any changes---I've now worked around the -x issue.

messense

Thanks! Looks great!

TheButlah · 2023-06-21T19:45:18Z

Seems to work! Thanks :3

rust-cross/cargo-zigbuild#140 https://github.com/PyO3/maturin/blob/v1.2.0/Cargo.toml#L84

lifthrasiir marked this pull request as draft June 9, 2023 09:30

TheButlah mentioned this pull request Jun 13, 2023

Document use with bindgen #144

Closed

lifthrasiir added 9 commits June 17, 2023 16:47

Add explicit bindgen tests.

60ae2eb

Zstd-rs didn't include enough headers and the incomplete bindgen support was simply overlooked. For example, `stdlib.h` didn't work.

Update the digression about zig cc.

ef51fe2

Better diagnostics when bindgen-exhaustive test fails.

81f13a7

lifthrasiir force-pushed the robust-bindgen branch from dce74cc to 832e423 Compare June 17, 2023 07:49

Conditionally test C++ headers that are only in some Zig versions.

84436d6

lifthrasiir force-pushed the robust-bindgen branch from 832e423 to e5caa37 Compare June 17, 2023 07:56

messense reviewed Jun 17, 2023

View reviewed changes

.github/workflows/CI.yml Outdated Show resolved Hide resolved

messense added the CI-no-fail-fast label Jun 17, 2023

lifthrasiir force-pushed the robust-bindgen branch from e5caa37 to 87f5092 Compare June 17, 2023 08:41

Explicitly test stray include paths from the host.

998830e

This in particular affects `__has_include` new in C++17, which is also how the test internally works. Tested files are chosen so that there is no chance that other OSes have matching header files by accident.

lifthrasiir force-pushed the robust-bindgen branch from 87f5092 to 998830e Compare June 17, 2023 09:09

lifthrasiir marked this pull request as ready for review June 17, 2023 09:36

messense approved these changes Jun 17, 2023

View reviewed changes

messense merged commit c04fbf3 into rust-cross:main Jun 17, 2023
30 checks passed

messense linked an issue Jun 17, 2023 that may be closed by this pull request

Document use with bindgen #144

Closed

lifthrasiir deleted the robust-bindgen branch June 17, 2023 09:45

messense mentioned this pull request Jun 17, 2023

Update cargo-zigbuild to 0.16.11 PyO3/maturin#1666

Merged

Porkepix mentioned this pull request Jun 17, 2023

cargo-zigbuild 0.16.11 Homebrew/homebrew-core#134025

Merged

furiosamg added a commit to furiosamg/mlperf-postprocess that referenced this pull request Dec 13, 2023

Use maturin~=1.2.0 for resolving workaround

23fffa3

rust-cross/cargo-zigbuild#140 https://github.com/PyO3/maturin/blob/v1.2.0/Cargo.toml#L84

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Robust bindgen support #140

Robust bindgen support #140

lifthrasiir commented Jun 9, 2023

lifthrasiir commented Jun 9, 2023

messense commented Jun 11, 2023

TheButlah commented Jun 13, 2023 •

edited

Loading

lifthrasiir commented Jun 14, 2023

TheButlah commented Jun 14, 2023 •

edited

Loading

TheButlah commented Jun 14, 2023 •

edited

Loading

lifthrasiir commented Jun 14, 2023

TheButlah commented Jun 14, 2023 •

edited

Loading

lifthrasiir commented Jun 17, 2023

messense left a comment

TheButlah commented Jun 21, 2023

Robust bindgen support #140

Robust bindgen support #140

Conversation

lifthrasiir commented Jun 9, 2023

lifthrasiir commented Jun 9, 2023

messense commented Jun 11, 2023

TheButlah commented Jun 13, 2023 • edited Loading

lifthrasiir commented Jun 14, 2023

TheButlah commented Jun 14, 2023 • edited Loading

TheButlah commented Jun 14, 2023 • edited Loading

lifthrasiir commented Jun 14, 2023

TheButlah commented Jun 14, 2023 • edited Loading

lifthrasiir commented Jun 17, 2023

messense left a comment

Choose a reason for hiding this comment

TheButlah commented Jun 21, 2023

TheButlah commented Jun 13, 2023 •

edited

Loading

TheButlah commented Jun 14, 2023 •

edited

Loading

TheButlah commented Jun 14, 2023 •

edited

Loading

TheButlah commented Jun 14, 2023 •

edited

Loading