Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable LLVM zlib #72696

Merged
merged 1 commit into from
Jun 21, 2020
Merged

Enable LLVM zlib #72696

merged 1 commit into from
Jun 21, 2020

Conversation

jethrogb
Copy link
Contributor

@jethrogb jethrogb commented May 28, 2020

Compilers may generate ELF objects with compressed sections (although rustc currently doesn't do this). Currently, when linking these with rust-lld, you'll get this error:

rust-lld: error: ...: contains a compressed section, but zlib is not available

This enables zlib when building LLVM.

@rust-highfive
Copy link
Collaborator

r? @Mark-Simulacrum

(rust_highfive has picked a reviewer for you, use r? to override)

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label May 28, 2020
@@ -175,6 +174,12 @@ impl Step for Llvm {
}
}

if builder.config.lld_enabled {
cfg.define("LLVM_ENABLE_ZLIB", "ON");
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not clear to me if this setting has repercussions beyond LLD.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you think of a reason we wouldn't want to support zlib even if lld isn't enabled? I imagine it's more broadly useful and not measurably slower to compile.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked at the git history and couldn't find a reason for it being off. It may just have been a case of "we don't use it, so why enabled it?"

@@ -175,6 +174,12 @@ impl Step for Llvm {
}
}

if builder.config.lld_enabled {
cfg.define("LLVM_ENABLE_ZLIB", "ON");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you think of a reason we wouldn't want to support zlib even if lld isn't enabled? I imagine it's more broadly useful and not measurably slower to compile.

@Mark-Simulacrum
Copy link
Member

@bors r+

@bors
Copy link
Contributor

bors commented May 28, 2020

📌 Commit 8541213c95ef77bd91f73c75771652710940fc9d has been approved by Mark-Simulacrum

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 28, 2020
@Mark-Simulacrum
Copy link
Member

@bors r+

I think we raced there :)

@bors
Copy link
Contributor

bors commented May 28, 2020

📌 Commit 44b5f53 has been approved by Mark-Simulacrum

@jethrogb jethrogb changed the title Enable LLVM zlib when building LLD Enable LLVM zlib May 28, 2020
RalfJung added a commit to RalfJung/rust that referenced this pull request May 31, 2020
…acrum

Enable LLVM zlib

Compilers may generate ELF objects with compressed sections (although rustc currently doesn't do this). Currently, when linking these with `rust-lld`, you'll get this error:

`rust-lld: error: ...: contains a compressed section, but zlib is not available`

This enables zlib when building LLVM.
@RalfJung
Copy link
Member

I think this is responsible for the failure in #72809 (comment)
@bors r-

error: linking with `x86_64--netbsd-gcc-sysroot` failed: exit code: 1
  |
  = note: "x86_64--netbsd-gcc-sysroot" "-Wl,--as-needed" "-m64" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-netbsd/lib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-netbsd/release/deps/rustc_binary-b59a9e32d9cc1353.rustc_binary.5vy141hy-cgu.0.rcgu.o" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-netbsd/release/deps/rustc_binary-b59a9e32d9cc1353.rustc_binary.5vy141hy-cgu.1.rcgu.o" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-netbsd/release/deps/rustc_binary-b59a9e32d9cc1353" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro" "-Wl,-znow" "-Wl,-O1" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-netbsd/release/deps" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/release/deps" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-netbsd/release/build/psm-e5dfce7cf4d6a957/out" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-netbsd/release/build/rustc_llvm-f7516fab485c4cb0/out" "-L" "/checkout/obj/build/x86_64-unknown-netbsd/llvm/build/lib" "-L" "/x-tools/x86_64-unknown-netbsd/sysroot/usr/lib" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-netbsd/lib" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-netbsd/release/deps" "-lrustc_driver-7f25206d2ccc28bf" "-Wl,--start-group" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-netbsd/lib" "-lstd-ddcb9910f53a8ac3" "-Wl,--end-group" "-Wl,-Bstatic" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-netbsd/lib/libcompiler_builtins-b492342e7e22b0e0.rlib" "-Wl,-Bdynamic" "-lutil" "-lrt" "-lutil" "-lpthread" "-lrt" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-lrt" "-lutil" "-Wl,-rpath,$ORIGIN/../lib"
  = note: /checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-netbsd/release/deps/librustc_driver-7f25206d2ccc28bf.so: undefined reference to `compress2'
          /checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-netbsd/release/deps/librustc_driver-7f25206d2ccc28bf.so: undefined reference to `crc32'
          /checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-netbsd/release/deps/librustc_driver-7f25206d2ccc28bf.so: undefined reference to `compressBound'
          /checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-netbsd/release/deps/librustc_driver-7f25206d2ccc28bf.so: undefined reference to `uncompress'
          collect2: error: ld returned 1 exit status

@bors bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels May 31, 2020
@RalfJung
Copy link
Member

@bors rollup=never

@dvdplm
Copy link

dvdplm commented Jun 1, 2020

Compilers may generate ELF objects with compressed sections

Out of curiosity, can you elaborate a bit further on this? How can a program compiled with rustc end up with compressed sections? Or do you mean other compilers that produce artifacts that are linked with rust-lld?

EDIT: this is from the SGX toolchain I guess, but can you elaborate on which parts of it that end up generating compressed sections?

@jethrogb
Copy link
Contributor Author

jethrogb commented Jun 2, 2020

You're right the Rust compiler doesn't do this. But C compilers do, for example I believe GCC does this by default on Ubuntu 18.04. So if you're trying to link native code and you're using rust-lld, you'll run into this issue. I don't think this is related to SGX in particular, except that it uses rust-lld by default so the issue is more apparent.

@jethrogb
Copy link
Contributor Author

jethrogb commented Jun 2, 2020

Who is the NetBSD maintainer? @gandro ?

How would we install zlib in the NetBSD build container?

@gandro
Copy link
Contributor

gandro commented Jun 2, 2020

@jethrogb

Who is the NetBSD maintainer? @gandro ?

Unfortunately I have not been actively involved in Rust on NetBSD for the last few years. I do not know who is currently maintaining it.

@jethrogb
Copy link
Contributor Author

jethrogb commented Jun 2, 2020

cc @MartinHusemann @jakllsch @bgermann @Niacat since you had a NetBSD PR somewhat recently

@bgermann
Copy link
Contributor

bgermann commented Jun 3, 2020

I do not think the problem is that zlib is not installed. The check for the header says: "Looking for zlib.h - found". Maybe it is a missing -lz linker flag.
Searching through the other targets' logs on the same CI run I find some of the zlib.h header and compress2 linking checks failing. So these targets do not fail even if zlib is not installed. You could try setting the ZLIB config option to FORCE_ON to see them fail as well.

@alarixnia
Copy link
Contributor

alarixnia commented Jun 3, 2020

rust maintainer for NetBSD is @he32. i'm responsible for repackaging rust binary releases, however.

zlib is part of the NetBSD base system so you shouldn't need to install anything. instead the problem seems to be that the -lz linker flag is missing.

semi-relatedly from a NetBSD packaging point of view I'm wondering why rust binaries for NetBSD aren't linked statically, or at a minimum with --static-libstdc++

@jethrogb
Copy link
Contributor Author

jethrogb commented Jun 9, 2020

So which libraries need to be linked is determined by llvm-config in https://github.com/rust-lang/rust/blob/master/src/librustc_llvm/build.rs. If -lz is needed for LLVM on NetBSD but is not being output by llvm-config, I think that's an LLVM bug. I've disabled zlib for NetBSD for now.

@Niacat Looks like NetBSD is actually specifically excluded from statically linking libstdc++. Although there's some other netbsd-specific stuff at https://github.com/rust-lang/rust/blob/master/src/librustc_llvm/build.rs#L270 .

@rust-highfive
Copy link
Collaborator

The job mingw-check of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
##[section]Starting: Linux mingw-check
##[section]Starting: Initialize job
Agent name: 'Azure Pipelines 8'
Agent machine name: 'fv-az578'
Current agent version: '2.170.1'
##[group]Operating System
16.04.6
LTS
LTS
##[endgroup]
##[group]Virtual Environment
Environment: ubuntu-16.04
Version: 20200517.1
Included Software: https://github.com/actions/virtual-environments/blob/ubuntu16/20200517.1/images/linux/Ubuntu1604-README.md
##[endgroup]
Agent running as: 'vsts'
Prepare build directory.
Set build variables.
Download all required tasks.
Download all required tasks.
Downloading task: Bash (3.163.3)
Checking job knob settings.
   Knob: AgentToolsDirectory = /opt/hostedtoolcache Source: ${AGENT_TOOLSDIRECTORY} 
   Knob: AgentPerflog = /home/vsts/perflog Source: ${VSTS_AGENT_PERFLOG} 
Start tracking orphan processes.
##[section]Finishing: Initialize job
##[section]Starting: Configure Job Name
==============================================================================
---
========================== Starting Command Output ===========================
[command]/bin/bash --noprofile --norc /home/vsts/work/_temp/5033bc9b-667f-4bdd-a4f3-42ada1eb5595.sh

##[section]Finishing: Disable git automatic line ending conversion
##[section]Starting: Checkout rust-lang/rust@refs/pull/72696/merge to s
Task         : Get sources
Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version      : 1.0.0
Author       : Microsoft
---
##[command]git remote add origin https://github.com/rust-lang/rust
##[command]git config gc.auto 0
##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
##[command]git config --get-all http.proxy
##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/72696/merge:refs/remotes/pull/72696/merge
---
 ---> 3adb0605cc65
Step 6/7 : ENV RUN_CHECK_WITH_PARALLEL_QUERIES 1
 ---> Using cache
 ---> 28dbc326cb7f
Step 7/7 : ENV SCRIPT python3 ../x.py test src/tools/expand-yaml-anchors &&            python3 ../x.py check --target=i686-pc-windows-gnu --host=i686-pc-windows-gnu &&            python3 ../x.py build --stage 0 src/tools/build-manifest &&            python3 ../x.py test --stage 0 src/tools/compiletest &&            python3 ../x.py test src/tools/tidy &&            python3 ../x.py doc --stage 0 src/libstd &&            /scripts/validate-toolstate.sh
 ---> 537a01811900
Successfully built 537a01811900
Successfully tagged rust-ci:latest
Built container sha256:537a018119009dc218456238dec90b5530050db1e2a1e166550c218003f6159d
---
   Compiling bootstrap v0.0.0 (/checkout/src/bootstrap)
error[E0308]: mismatched types
   --> src/bootstrap/native.rs:171:13
    |
170 | /         if !target.contains("netbsd") {
171 | |             cfg.define("LLVM_ENABLE_ZLIB", "ON")
    | |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `()`, found `&mut cmake::Config`
172 | |         } else {
173 | |             cfg.define("LLVM_ENABLE_ZLIB", "OFF")
    | |_________- expected this to be `()`
    |
help: try adding a semicolon
    |
    |
171 |             cfg.define("LLVM_ENABLE_ZLIB", "ON");
help: consider using a semicolon here
    |
174 |         };
    |          ^
    |          ^

error[E0308]: mismatched types
   --> src/bootstrap/native.rs:173:13
    |
170 | /         if !target.contains("netbsd") {
171 | |             cfg.define("LLVM_ENABLE_ZLIB", "ON")
172 | |         } else {
173 | |             cfg.define("LLVM_ENABLE_ZLIB", "OFF")
    | |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `()`, found `&mut cmake::Config`
    | |_________- expected this to be `()`
    |
help: try adding a semicolon
    |
    |
173 |             cfg.define("LLVM_ENABLE_ZLIB", "OFF");
help: consider using a semicolon here
    |
174 |         };
    |          ^
---
  local time: Tue Jun  9 19:20:46 UTC 2020
  network time: Tue, 09 Jun 2020 19:20:46 GMT
== end clock drift check ==

##[error]Bash exited with code '1'.
##[section]Finishing: Run build
##[section]Starting: Checkout rust-lang/rust@refs/pull/72696/merge to s
Task         : Get sources
Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version      : 1.0.0
Author       : Microsoft
Author       : Microsoft
Help         : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199)
==============================================================================
Cleaning any cached credential from repository: rust-lang/rust (GitHub)
##[section]Finishing: Checkout rust-lang/rust@refs/pull/72696/merge to s
Cleaning up task key
Start cleaning up orphan processes.
Terminate orphan process: pid (4295) (python)
##[section]Finishing: Finalize Job

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @rust-lang/infra. (Feature Requests)

@Mark-Simulacrum
Copy link
Member

@bors r+

@bors
Copy link
Contributor

bors commented Jun 11, 2020

📌 Commit 8caa14f has been approved by Mark-Simulacrum

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 11, 2020
@MartinHusemann
Copy link
Contributor

Uhhm, this is all very cryptic and totally undebuggable (from my POV).

To make it possible to file a bug against llvm upstream, what invocation of llvm-config is supposed to output -lz (and why?)

@mati865
Copy link
Contributor

mati865 commented Jun 11, 2020

It's not LLVM bug.
--system-libs is not passed to llvm-config when cross-compiling:

if !is_crossed {
cmd.arg("--system-libs");
}

@Mark-Simulacrum
Copy link
Member

To be clear I'm happy to (in the future) accept a PR fixing this and changing things, but I don't want to block progress for fixes to tier-2 platforms that seem non trivial.

@jethrogb
Copy link
Contributor Author

jethrogb commented Jun 11, 2020

According to the git history, passing --system-libs is not appropriate when cross-compiling. 5266406#diff-b0767d8366aa7e0a83a847d53032dfd4R105-R113 . (Not saying that's correct, but that's why it's not being done today)

@Manishearth
Copy link
Member

@bors p=1

letting rollup=never PRs flush out

@bors
Copy link
Contributor

bors commented Jun 21, 2020

⌛ Testing commit 8caa14f with merge 349f6bf...

@bors
Copy link
Contributor

bors commented Jun 21, 2020

☀️ Test successful - checks-azure
Approved by: Mark-Simulacrum
Pushing 349f6bf to master...

@bors bors added the merged-by-bors This PR was explicitly merged by bors. label Jun 21, 2020
@bors bors merged commit 349f6bf into rust-lang:master Jun 21, 2020
witchof0x20 added a commit to witchof0x20/nixpkgs that referenced this pull request Jun 23, 2020
(Rust now has a dynamic library dependence on zlib. (see rust-lang/rust#72696))
Mic92 pushed a commit to NixOS/nixpkgs that referenced this pull request Jul 7, 2020
(Rust now has a dynamic library dependence on zlib. (see rust-lang/rust#72696))
nielx added a commit to nielx/rust that referenced this pull request Sep 1, 2020
PR rust-lang#72696 enabled the option LLVM_ENABLE_ZLIB for the LLVM builds. On Haiku, zlib is linked as a shared library. When cross-compiling LLVM, rustbuild should be instructed to explicitly linking to libz.
bors added a commit to rust-lang-ci/rust that referenced this pull request Sep 4, 2020
…ulacrum

Disable zlib in LLVM on Haiku

PR rust-lang#72696 enabled the option LLVM_ENABLE_ZLIB for the LLVM builds. Like NetBSD and aarch64-apple-darwin (see PR rust-lang#75500), the LLVM build system not explicitly linking to libz on these platforms cause issues. For Haiku, this meant the runtime loader complaining about undefined symbols..
nielx added a commit to nielx/rust that referenced this pull request Oct 16, 2020
PR rust-lang#72696 enabled the option LLVM_ENABLE_ZLIB for the LLVM builds. Like NetBSD and aarch64-apple-darwin (see PR rust-lang#75500), the LLVM build system not explicitly linking to libz on these platforms cause issues. For Haiku, this meant the runtime loader complaining about undefined symbols..
@inferiorhumanorgans
Copy link

So which libraries need to be linked is determined by llvm-config in https://github.com/rust-lang/rust/blob/master/src/librustc_llvm/build.rs. If -lz is needed for LLVM on NetBSD but is not being output by llvm-config, I think that's an LLVM bug. I've disabled zlib for NetBSD for now.

@jethrogb I've hit this while futzing around with DragonFly and Solaris builds as well. From what I can tell the issue is that in a cross compiling context (e.g. the NetBSD builds) the host llvm-config is called and thus the library flags are irrelevant to the cross target. This also caused heartburn where I was cross compiling with a host that used libc++ for a target that used libstdc++. See also: #114078.

I think it's potentially more appropriate to check for a cross compilation and link to libz in that case unless the target is blacklisted, however I don't know what architectures don't have libz enabled. For instance with macOS, -lz is included in the output of llvm-config, but can't be added in a cross context.

e.g.

$ ./build/x86_64-apple-darwin/llvm/bin/llvm-config --system-libs
-lm -lz
diff --git a/compiler/rustc_llvm/build.rs b/compiler/rustc_llvm/build.rs
index b0783d75..2bfc421b 100644
--- a/compiler/rustc_llvm/build.rs
+++ b/compiler/rustc_llvm/build.rs
@@ -251,7 +251,7 @@ fn main() {
     } else if target.contains("windows-gnu") {
         println!("cargo:rustc-link-lib=shell32");
         println!("cargo:rustc-link-lib=uuid");
-    } else if target.contains("netbsd") || target.contains("haiku") || target.contains("darwin") {
+    } else if is_crossed {
         println!("cargo:rustc-link-lib=z");
     }
     cmd.args(&components);

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
merged-by-bors This PR was explicitly merged by bors. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
Projects
None yet
Development

Successfully merging this pull request may close these issues.