Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Port PGO/LTO/BOLT optimized build pipeline to Rust #112235

Merged
merged 1 commit into from
Jul 9, 2023
Merged

Port PGO/LTO/BOLT optimized build pipeline to Rust #112235

merged 1 commit into from
Jul 9, 2023

Conversation

Kobzol
Copy link
Contributor

@Kobzol Kobzol commented Jun 3, 2023 •
edited
Loading

This PR ports the stage-build.py PGO/LTO/BOLT optimization script from Python to Rust, to make it easier to use dependencies, and make it a bit more robust. The PR switches both the Linux and Windows dist runners to the Rust script and removes the old Python script.

Funnily enough, the Rust port has less lines of code than the Python script :) I think that clearly shows that the Python script really lacked dependencies.

@rustbot
Copy link
Collaborator

rustbot commented Jun 3, 2023

r? @Mark-Simulacrum

(rustbot has picked a reviewer for you, use r? to override)

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Jun 3, 2023
@rust-log-analyzer

This comment has been minimized.

Sign in to view
@rustbot rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. labels Jun 3, 2023
@rust-log-analyzer

This comment has been minimized.

Sign in to view
@rust-log-analyzer

This comment has been minimized.

Sign in to view
@rust-log-analyzer

This comment has been minimized.

Sign in to view
@rust-log-analyzer

This comment has been minimized.

Sign in to view
@Kobzol
Copy link
Contributor Author

Kobzol commented Jun 3, 2023

@bors try

@bors
Copy link
Contributor

bors commented Jun 3, 2023

⌛ Trying commit 408578eea211c890c45c32baa3983dc1abd64776 with merge da5b8ab81a6270a5a4e49c3725bfde81a7faf6a7...

@rust-log-analyzer

This comment has been minimized.

Sign in to view
@rust-log-analyzer

This comment has been minimized.

Sign in to view
@bors
Copy link
Contributor

bors commented Jun 3, 2023

💔 Test failed - checks-actions

@bors bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 3, 2023
@Kobzol
Copy link
Contributor Author

Kobzol commented Jun 3, 2023

@bors try

@bors
Copy link
Contributor

bors commented Jun 3, 2023

⌛ Trying commit 21c5a4f1533a34c1611ab68d1a835b1c8f3f8284 with merge 3babef129a16a88b15d4852eb488c50661b97be2...

@rust-log-analyzer

This comment has been minimized.

Sign in to view
@rust-log-analyzer

This comment has been minimized.

Sign in to view
@bors
Copy link
Contributor

bors commented Jun 3, 2023

💔 Test failed - checks-actions

@Kobzol
Copy link
Contributor Author

Kobzol commented Jun 4, 2023

@bors try

@bors
Copy link
Contributor

bors commented Jun 4, 2023

⌛ Trying commit e81b1be849ddf3974e4b374fc2ced4b1fabf0562 with merge 53b17b880d3f17a78cd8ee30eb7833c2e27abe3e...

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jul 1, 2023
@Mark-Simulacrum
Copy link
Member

r=me with this rebased

@Kobzol
Copy link
Contributor Author

Kobzol commented Jul 9, 2023

@bors r=Mark-Simulacrum

@bors
Copy link
Contributor

bors commented Jul 9, 2023

📌 Commit 91d2fb2 has been approved by Mark-Simulacrum

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 9, 2023
@bors
Copy link
Contributor

bors commented Jul 9, 2023

⌛ Testing commit 91d2fb2 with merge 1861088...

@bors
Copy link
Contributor

bors commented Jul 9, 2023

☀️ Test successful - checks-actions
Approved by: Mark-Simulacrum
Pushing 1861088 to master...

@bors bors added the merged-by-bors This PR was explicitly merged by bors. label Jul 9, 2023
@bors bors merged commit 1861088 into rust-lang:master Jul 9, 2023
11 checks passed
@rustbot rustbot added this to the 1.73.0 milestone Jul 9, 2023
@Kobzol Kobzol deleted the opt-dist branch July 9, 2023 11:08
@bors bors mentioned this pull request Jul 9, 2023
Closed
@rust-timer
Copy link
Collaborator

Finished benchmarking commit (1861088): comparison URL.

Overall result: ✅ improvements - no action needed

@rustbot label: -perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)		 - - 0
Regressions ❌
(secondary)		 0.3% [0.3%, 0.3%] 1
Improvements ✅
(primary)		 -0.9% [-1.3%, -0.5%] 38
Improvements ✅
(secondary)		 -0.8% [-2.3%, -0.3%] 11
All ❌✅ (primary) -0.9% [-1.3%, -0.5%] 38

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)		 - - 0
Regressions ❌
(secondary)		 1.3% [1.3%, 1.3%] 1
Improvements ✅
(primary)		 - - 0
Improvements ✅
(secondary)		 - - 0
All ❌✅ (primary) - - 0

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)		 - - 0
Regressions ❌
(secondary)		 2.5% [2.1%, 2.9%] 2
Improvements ✅
(primary)		 -1.5% [-1.5%, -1.5%] 1
Improvements ✅
(secondary)		 - - 0
All ❌✅ (primary) -1.5% [-1.5%, -1.5%] 1

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 655.641s -> 658.609s (0.45%)

@lqd
Copy link
Member

lqd commented Jul 9, 2023

This looks to me like the #113485 (comment) noise returning to steady state.

@lqd lqd mentioned this pull request Jul 9, 2023
Merged
@wesleywiser
Copy link
Member

This is really awesome work @Kobzol! 💯

@Kobzol Kobzol mentioned this pull request Jul 14, 2023
Merged
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request Jul 14, 2023
@matthiaskrgr
Rollup merge of rust-lang#113685 - Kobzol:opt-dist-binary-sizes, r=Ma…
f021136 
…rk-Simulacrum

Print artifact sizes in `opt-dist`

The Python PGO script printed a nice table of artifact sizes (`librustc_driver.so`, `libLLVM.so`, ...) at the end of the CI run, which was useful to quickly see the sizes of important files. I forgot to port this functionality into the Rust (`opt-dist`) version in rust-lang#112235. This PR fixes that.

r? bootstrap
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request Jul 14, 2023
@matthiaskrgr
Rollup merge of rust-lang#113685 - Kobzol:opt-dist-binary-sizes, r=Ma…
dae9e40 
…rk-Simulacrum

Print artifact sizes in `opt-dist`

The Python PGO script printed a nice table of artifact sizes (`librustc_driver.so`, `libLLVM.so`, ...) at the end of the CI run, which was useful to quickly see the sizes of important files. I forgot to port this functionality into the Rust (`opt-dist`) version in rust-lang#112235. This PR fixes that.

r? bootstrap
github-actions bot pushed a commit to rust-lang/miri that referenced this pull request Jul 16, 2023
@matthiaskrgr
Rollup merge of #113685 - Kobzol:opt-dist-binary-sizes, r=Mark-Simula…
f75e01f 
…crum

Print artifact sizes in `opt-dist`

The Python PGO script printed a nice table of artifact sizes (`librustc_driver.so`, `libLLVM.so`, ...) at the end of the CI run, which was useful to quickly see the sizes of important files. I forgot to port this functionality into the Rust (`opt-dist`) version in rust-lang/rust#112235. This PR fixes that.

r? bootstrap
wip-sync pushed a commit to NetBSD/pkgsrc-wip that referenced this pull request Oct 6, 2023
@jperkin
rust: Upgrade to 1.73.0.
3a92f61 
Language
--------

- [Uplift `clippy::fn_null_check` lint as `useless_ptr_null_checks`.]
  (rust-lang/rust#111717)
- [Make `noop_method_call` warn by default.]
  (rust-lang/rust#111916)
- [Support interpolated block for `try` and `async` in macros.]
  (rust-lang/rust#112953)
- [Make `unconditional_recursion` lint detect recursive drops.]
  (rust-lang/rust#113902)
- [Future compatibility warning for some impls being incorrectly
  considered not overlapping.]
  (rust-lang/rust#114023)
- [The `invalid_reference_casting` lint is now **deny-by-default**
  (instead of allow-by-default)]
  (rust-lang/rust#112431)

Compiler
--------

- [Write version information in a `.comment` section like GCC/Clang.]
  (rust-lang/rust#97550)
- [Add documentation on v0 symbol mangling.]
  (rust-lang/rust#97571)
- [Stabilize `extern "thiscall"` and `"thiscall-unwind"` ABIs.]
  (rust-lang/rust#114562)
- [Only check outlives goals on impl compared to trait.]
  (rust-lang/rust#109356)
- [Infer type in irrefutable slice patterns with fixed length as array.]
  (rust-lang/rust#113199)
- [Discard default auto trait impls if explicit ones exist.]
  (rust-lang/rust#113312)
- Add several new tier 3 targets:
    - [`aarch64-unknown-teeos`]
      (rust-lang/rust#113480)
    - [`csky-unknown-linux-gnuabiv2`]
      (rust-lang/rust#113658)
    - [`riscv64-linux-android`]
      (rust-lang/rust#112858)
    - [`riscv64gc-unknown-hermit`]
      (rust-lang/rust#114004)
    - [`x86_64-unikraft-linux-musl`]
      (rust-lang/rust#113411)
    - [`x86_64-unknown-linux-ohos`]
      (rust-lang/rust#113061)
- [Add `wasm32-wasi-preview1-threads` as a tier 2 target.]
  (rust-lang/rust#112922)

Refer to Rust's [platform support page][platform-support-doc]
for more information on Rust's tiered platform support.

Libraries
---------

- [Add `Read`, `Write` and `Seek` impls for `Arc<File>`.]
  (rust-lang/rust#94748)
- [Merge functionality of `io::Sink` into `io::Empty`.]
  (rust-lang/rust#98154)
- [Implement `RefUnwindSafe` for `Backtrace`]
  (rust-lang/rust#100455)
- [Make `ExitStatus` implement `Default`]
  (rust-lang/rust#106425)
- [`impl SliceIndex<str> for (Bound<usize>, Bound<usize>)`]
  (rust-lang/rust#111081)
- [Change default panic handler message format.]
  (rust-lang/rust#112849)
- [Cleaner `assert_eq!` & `assert_ne!` panic messages.]
  (rust-lang/rust#111071)
- [Correct the (deprecated) Android `stat` struct definitions.]
  (rust-lang/rust#113130)

Stabilized APIs
---------------

- [Unsigned `{integer}::div_ceil`]
  (https://doc.rust-lang.org/stable/std/primitive.u32.html#method.div_ceil)
- [Unsigned `{integer}::next_multiple_of`]
  (https://doc.rust-lang.org/stable/std/primitive.u32.html#method.next_multiple_of)
- [Unsigned `{integer}::checked_next_multiple_of`]
  (https://doc.rust-lang.org/stable/std/primitive.u32.html#method.checked_next_multiple_of)
- [`std::ffi::FromBytesUntilNulError`]
  (https://doc.rust-lang.org/stable/std/ffi/struct.FromBytesUntilNulError.html)
- [`std::os::unix::fs::chown`]
  (https://doc.rust-lang.org/stable/std/os/unix/fs/fn.chown.html)
- [`std::os::unix::fs::fchown`]
  (https://doc.rust-lang.org/stable/std/os/unix/fs/fn.fchown.html)
- [`std::os::unix::fs::lfchown`]
  (https://doc.rust-lang.org/stable/std/os/unix/fs/fn.lchown.html)
- [`LocalKey::<Cell<T>>::get`]
  (https://doc.rust-lang.org/stable/std/thread/struct.LocalKey.html#method.get)
- [`LocalKey::<Cell<T>>::set`]
  (https://doc.rust-lang.org/stable/std/thread/struct.LocalKey.html#method.set)
- [`LocalKey::<Cell<T>>::take`]
  (https://doc.rust-lang.org/stable/std/thread/struct.LocalKey.html#method.take)
- [`LocalKey::<Cell<T>>::replace`]
  (https://doc.rust-lang.org/stable/std/thread/struct.LocalKey.html#method.replace)
- [`LocalKey::<RefCell<T>>::with_borrow`]
  (https://doc.rust-lang.org/stable/std/thread/struct.LocalKey.html#method.with_borrow)
- [`LocalKey::<RefCell<T>>::with_borrow_mut`]
  (https://doc.rust-lang.org/stable/std/thread/struct.LocalKey.html#method.with_borrow_mut)
- [`LocalKey::<RefCell<T>>::set`]
  (https://doc.rust-lang.org/stable/std/thread/struct.LocalKey.html#method.set-1)
- [`LocalKey::<RefCell<T>>::take`]
  (https://doc.rust-lang.org/stable/std/thread/struct.LocalKey.html#method.take-1)
- [`LocalKey::<RefCell<T>>::replace`]
  (https://doc.rust-lang.org/stable/std/thread/struct.LocalKey.html#method.replace-1)

These APIs are now stable in const contexts:

- [`rc::Weak::new`]
  (https://doc.rust-lang.org/stable/alloc/rc/struct.Weak.html#method.new)
- [`sync::Weak::new`]
  (https://doc.rust-lang.org/stable/alloc/sync/struct.Weak.html#method.new)
- [`NonNull::as_ref`]
  (https://doc.rust-lang.org/stable/core/ptr/struct.NonNull.html#method.as_ref)

Cargo
-----

- [Encode URL params correctly for `SourceId` in `Cargo.lock`.]
  (rust-lang/cargo#12280)
- [Bail out an error when using `cargo::` in custom build script.]
  (rust-lang/cargo#12332)

Compatibility Notes
-------------------

- [Update the minimum external LLVM to 15.]
  (rust-lang/rust#114148)
- [Check for non-defining uses of return position `impl Trait`.]
  (rust-lang/rust#112842)

Internal Changes
----------------

These changes do not affect any public interfaces of Rust, but they represent
significant improvements to the performance or internals of rustc and related
tools.

- [Remove LLVM pointee types, supporting only opaque pointers.]
  (rust-lang/rust#105545)
- [Port PGO/LTO/BOLT optimized build pipeline to Rust.]
  (rust-lang/rust#112235)
- [Replace in-tree `rustc_apfloat` with the new version of the crate.]
  (rust-lang/rust#113843)
- [Update to LLVM 17.]
  (rust-lang/rust#114048)
- [Add `internal_features` lint for internal unstable features.]
  (rust-lang/rust#108955)
- [Mention style for new syntax in tracking issue template.]
  (rust-lang/rust#113586)
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Nov 16, 2023
@he32
rust: Upgrade to version 1.73.0.
9081e36 
Pkgsrc changes:
 * Adjust patches and cargo checksums to new versions.
 * For an external LLVM, set dependency of llvm >= 15, in accordance
   with the upstream changes.
 * Add a patch with a backport from LLVM 17.0.3 fixing codegen for
   PPC, ref. rust-lang/rust#116845

Upstream changes:

Version 1.73.0 (2023-10-05)
==========================

Language
--------

- [Uplift `clippy::fn_null_check` lint as `useless_ptr_null_checks`.]
  (rust-lang/rust#111717)
- [Make `noop_method_call` warn by default.]
  (rust-lang/rust#111916)
- [Support interpolated block for `try` and `async` in macros.]
  (rust-lang/rust#112953)
- [Make `unconditional_recursion` lint detect recursive drops.]
  (rust-lang/rust#113902)
- [Future compatibility warning for some impls being incorrectly
  considered not overlapping.]
  (rust-lang/rust#114023)
- [The `invalid_reference_casting` lint is now **deny-by-default**
  (instead of allow-by-default)]
  (rust-lang/rust#112431

Compiler
--------

- [Write version information in a `.comment` section like GCC/Clang.]
  (rust-lang/rust#97550)
- [Add documentation on v0 symbol mangling.]
  (rust-lang/rust#97571)
- [Stabilize `extern "thiscall"` and `"thiscall-unwind"` ABIs.]
  (rust-lang/rust#114562)
- [Only check outlives goals on impl compared to trait.]
  (rust-lang/rust#109356)
- [Infer type in irrefutable slice patterns with fixed length as array.]
  (rust-lang/rust#113199)
- [Discard default auto trait impls if explicit ones exist.]
  (rust-lang/rust#113312)
- Add several new tier 3 targets:
    - [`aarch64-unknown-teeos`]
      (rust-lang/rust#113480)
    - [`csky-unknown-linux-gnuabiv2`]
      (rust-lang/rust#113658)
    - [`riscv64-linux-android`]
      (rust-lang/rust#112858)
    - [`riscv64gc-unknown-hermit`]
      (rust-lang/rust#114004)
    - [`x86_64-unikraft-linux-musl`]
      (rust-lang/rust#113411)
    - [`x86_64-unknown-linux-ohos`]
      (rust-lang/rust#113061)
- [Add `wasm32-wasi-preview1-threads` as a tier 2 target.]
  (rust-lang/rust#112922)

Refer to Rust's [platform support page][platform-support-doc]
for more information on Rust's tiered platform support.

Libraries
---------

- [Add `Read`, `Write` and `Seek` impls for `Arc<File>`.]
  (rust-lang/rust#94748)
- [Merge functionality of `io::Sink` into `io::Empty`.]
  (rust-lang/rust#98154)
- [Implement `RefUnwindSafe` for `Backtrace`]
  (rust-lang/rust#100455)
- [Make `ExitStatus` implement `Default`]
  (rust-lang/rust#106425)
- [`impl SliceIndex<str> for (Bound<usize>, Bound<usize>)`]
  (rust-lang/rust#111081)
- [Change default panic handler message format.]
  (rust-lang/rust#112849)
- [Cleaner `assert_eq!` & `assert_ne!` panic messages.]
  (rust-lang/rust#111071)
- [Correct the (deprecated) Android `stat` struct definitions.]
  (rust-lang/rust#113130)

Stabilized APIs
---------------

- [Unsigned `{integer}::div_ceil`]
  (https://doc.rust-lang.org/stable/std/primitiv e.u32.html#method.div_ceil)
- [Unsigned `{integer}::next_multiple_of`]
  (https://doc.rust-lang.org/stable/std/primitive.u32.html#method.next_multiple_of)
- [Unsigned `{integer}::checked_next_multiple_of`]
  (https://doc.rust-lang.org/stable/std/primitive.u32.html#method.checked_next_multiple_of)
- [`std::ffi::FromBytesUntilNulError`]
  (https://doc.rust-lang.org/stable/std/ffi/struct.FromBytesUntilNulError.html)
- [`std::os::unix::fs::chown`]
  (https://doc.rust-lang.org/stable/std/os/unix/fs/fn.chown.html)
- [`std::os::unix::fs::fchown`]
  (https://doc.rust-lang.org/stable/std/os/unix/fs/fn.fchown.html)
- [`std::os::unix::fs::lfchown`]
  (https://doc.rust-lang.org/stable/std/os/unix/fs/fn.lchown.html)
- [`LocalKey::<Cell<T>>::get`]
  (https://doc.rust-lang.org/stable/std/thread/struct.LocalKey.html#method.get)
- [`LocalKey::<Cell<T>>::set`]
  (https://doc.rust-lang.org/stable/std/thread/struct.LocalKey.html#method.set)
- [`LocalKey::<Cell<T>>::take`]
  (https://doc.rust-lang.org/stable/std/thread/struct.LocalKey.html#method.take)
- [`LocalKey::<Cell<T>>::replace`]
  (https://doc.rust-lang.org/stable/std/thread/struct.LocalKey.html#method.replace)
- [`LocalKey::<RefCell<T>>::with_borrow`]
  (https://doc.rust-lang.org/stable/std/thread/struct.LocalKey.html#method.with_borrow)
- [`LocalKey::<RefCell<T>>::with_borrow_mut`]
  (https://doc.rust-lang.org/stable/std/thread/struct.LocalKey.html#method.with_borrow_mut)
- [`LocalKey::<RefCell<T>>::set`]
  (https://doc.rust-lang.org/stable/std/thread/struct.LocalKey.html#method.set-1)
- [`LocalKey::<RefCell<T>>::take`]
  (https://doc.rust-lang.org/stable/std/thread/struct.LocalKey.html#method.take-1)
- [`LocalKey::<RefCell<T>>::replace`]
  (https://doc.rust-lang.org/stable/std/thread/struct.LocalKey.html#method.replace-1)

These APIs are now stable in const contexts:

- [`rc::Weak::new`]
  (https://doc.rust-lang.org/stable/alloc/rc/struct.Weak.html#method.new)
- [`sync::Weak::new`]
  (https://doc.rust-lang.org/stable/alloc/sync/struct.Weak.html#method.new)
- [`NonNull::as_ref`]
  (https://doc.rust-lang.org/stable/core/ptr/struct.NonNull.html#method.as_ref)

Cargo
-----

- [Encode URL params correctly for `SourceId` in `Cargo.lock`.]
  (rust-lang/cargo#12280)
- [Bail out an error when using `cargo::` in custom build script.]
  (rust-lang/cargo#12332)

Misc
----

Compatibility Notes
-------------------

- [Update the minimum external LLVM to 15.]
  (rust-lang/rust#114148)
- [Check for non-defining uses of return position `impl Trait`.]
  (rust-lang/rust#112842)

Internal Changes
----------------

These changes do not affect any public interfaces of Rust, but they
represent significant improvements to the performance or internals
of rustc and related tools.

- [Remove LLVM pointee types, supporting only opaque pointers.]
  (rust-lang/rust#105545)
- [Port PGO/LTO/BOLT optimized build pipeline to Rust.]
  (rust-lang/rust#112235)
- [Replace in-tree `rustc_apfloat` with the new version of the crate.]
  (rust-lang/rust#113843)
- [Update to LLVM 17.]
  (rust-lang/rust#114048)
- [Add `internal_features` lint for internal unstable features.]
  (rust-lang/rust#108955)
- [Mention style for new syntax in tracking issue template.]
  (rust-lang/rust#113586)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lqd lqd lqd left review comments

@the8472 the8472 the8472 left review comments

@klensy klensy klensy left review comments

@jyn514 jyn514 jyn514 left review comments

@Noratrieb Noratrieb Noratrieb left review comments

Assignees

@pnkfelix pnkfelix

@wesleywiser wesleywiser

@Mark-Simulacrum Mark-Simulacrum

Labels
A-testsuite Area: The testsuite used to check the correctness of rustc merged-by-bors This PR was explicitly merged by bors. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.
Projects
None yet
Milestone
1.73.0
Development

Successfully merging this pull request may close these issues.