Add build.print_step_rusage to config.toml
#82532
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Intended to piggy-back on output from existing build.print_step_timings setting.
On non-unix platforms, does not try to call `getrusage` (and does not attempt to implement its own shim; that could be follow-on work, though its probably best to not invest too much effort there, versus using separate dedicated tooling). On unix platforms, calls libc::rusage and attempts to emit the subset of fields that are supported on Linux and Mac OS X. Omits groups of related stats which appear to be unsupported on the platform (due to them all remaining zero). Adjusts output to compensate for Mac using bytes instead of kb (a well known discrepancy on Mac OS X). However, so far I observe a lot of strange values (orders of magnitude wrong) reported on Mac OS X in some cases, so I would not trust this in that context currently.
|
(rust-highfive has picked a reviewer for you, use r? to override) |
rust-highfive
added
the
S-waiting-on-review
pnkfelix
added
the
T-bootstrap
|
@bors r+ rollup |
|
📌 Commit f2d70c5 has been approved by |
bors
added
S-waiting-on-bors
bors
added a commit
to rust-lang-ci/rust
that referenced
this pull request
Mar 1, 2021
Rollup of 10 pull requests Successful merges: - rust-lang#82309 (Propagate RUSTDOCFLAGS in the environment when documenting) - rust-lang#82403 (rustbuild: print out env vars on verbose rustc invocations) - rust-lang#82507 (Rename the `tidy` binary to `rust-tidy`) - rust-lang#82531 (Add GUI tests) - rust-lang#82532 (Add `build.print_step_rusage` to config.toml) - rust-lang#82543 (fix env var name in CI) - rust-lang#82622 (Propagate `--test-args` for `x.py test src/tools/cargo`) - rust-lang#82628 (Try to clarify GlobalAlloc::realloc documentation comment.) - rust-lang#82630 (Fix a typo in the `find_anon_type` doc) - rust-lang#82643 (Add more proc-macro attribute tests) Failed merges: r? `@ghost` `@rustbot` modify labels: rollup
Dylan-DPC-zz
pushed a commit
to Dylan-DPC-zz/rust
that referenced
this pull request
Mar 18, 2021
Attempt to gather similar stats as rusage on Windows A follow up to rust-lang#82532. This is a bit hacked in because I think we need to discuss this before merging, but this is an attempt to gather similar metrics as `libc::rusage` on Windows. Some comments on differences: * Currently, we're passing `RUSAGE_CHILDREN` to `rusage` which collects statistics on all children that have been waited on and terminated. I believe this is currently just the invocation of the real `rustc` that the shim is wrapping. Does `rustc` itself spawn children processes? The windows version gets the child processes handle when spawning it, and uses that to collect the statistics. For maxrss, `rusage` will return "the resident set size of the largest child, not the maximum resident set size of the process tree.", the Windows version will only collect statistics on the wrapped `rustc` child process directly even if some theoretical sub process has a larger memory footprint. * There might be subtle differences between `rusage`'s "resident set" and Window's "working set". The "working set" and "resident set" should both be the number of pages that are in memory and which would not cause a page fault when accessed. * I'm not yet sure how best to get the same information that `ru_minflt`, `ru_inblock`, `ru_oublock`, `ru_nivcsw ` and `ru_nvcsw` provide. r? `@pnkfelix`
Dylan-DPC-zz
pushed a commit
to Dylan-DPC-zz/rust
that referenced
this pull request
Mar 18, 2021
Attempt to gather similar stats as rusage on Windows A follow up to rust-lang#82532. This is a bit hacked in because I think we need to discuss this before merging, but this is an attempt to gather similar metrics as `libc::rusage` on Windows. Some comments on differences: * Currently, we're passing `RUSAGE_CHILDREN` to `rusage` which collects statistics on all children that have been waited on and terminated. I believe this is currently just the invocation of the real `rustc` that the shim is wrapping. Does `rustc` itself spawn children processes? The windows version gets the child processes handle when spawning it, and uses that to collect the statistics. For maxrss, `rusage` will return "the resident set size of the largest child, not the maximum resident set size of the process tree.", the Windows version will only collect statistics on the wrapped `rustc` child process directly even if some theoretical sub process has a larger memory footprint. * There might be subtle differences between `rusage`'s "resident set" and Window's "working set". The "working set" and "resident set" should both be the number of pages that are in memory and which would not cause a page fault when accessed. * I'm not yet sure how best to get the same information that `ru_minflt`, `ru_inblock`, `ru_oublock`, `ru_nivcsw ` and `ru_nvcsw` provide. r? ``@pnkfelix``
Dylan-DPC-zz
pushed a commit
to Dylan-DPC-zz/rust
that referenced
this pull request
Mar 19, 2021
Attempt to gather similar stats as rusage on Windows A follow up to rust-lang#82532. This is a bit hacked in because I think we need to discuss this before merging, but this is an attempt to gather similar metrics as `libc::rusage` on Windows. Some comments on differences: * Currently, we're passing `RUSAGE_CHILDREN` to `rusage` which collects statistics on all children that have been waited on and terminated. I believe this is currently just the invocation of the real `rustc` that the shim is wrapping. Does `rustc` itself spawn children processes? The windows version gets the child processes handle when spawning it, and uses that to collect the statistics. For maxrss, `rusage` will return "the resident set size of the largest child, not the maximum resident set size of the process tree.", the Windows version will only collect statistics on the wrapped `rustc` child process directly even if some theoretical sub process has a larger memory footprint. * There might be subtle differences between `rusage`'s "resident set" and Window's "working set". The "working set" and "resident set" should both be the number of pages that are in memory and which would not cause a page fault when accessed. * I'm not yet sure how best to get the same information that `ru_minflt`, `ru_inblock`, `ru_oublock`, `ru_nivcsw ` and `ru_nvcsw` provide. r? ```@pnkfelix```
Dylan-DPC-zz
pushed a commit
to Dylan-DPC-zz/rust
that referenced
this pull request
Mar 19, 2021
Attempt to gather similar stats as rusage on Windows A follow up to rust-lang#82532. This is a bit hacked in because I think we need to discuss this before merging, but this is an attempt to gather similar metrics as `libc::rusage` on Windows. Some comments on differences: * Currently, we're passing `RUSAGE_CHILDREN` to `rusage` which collects statistics on all children that have been waited on and terminated. I believe this is currently just the invocation of the real `rustc` that the shim is wrapping. Does `rustc` itself spawn children processes? The windows version gets the child processes handle when spawning it, and uses that to collect the statistics. For maxrss, `rusage` will return "the resident set size of the largest child, not the maximum resident set size of the process tree.", the Windows version will only collect statistics on the wrapped `rustc` child process directly even if some theoretical sub process has a larger memory footprint. * There might be subtle differences between `rusage`'s "resident set" and Window's "working set". The "working set" and "resident set" should both be the number of pages that are in memory and which would not cause a page fault when accessed. * I'm not yet sure how best to get the same information that `ru_minflt`, `ru_inblock`, `ru_oublock`, `ru_nivcsw ` and `ru_nvcsw` provide. r? ````@pnkfelix````
bors
added a commit
to rust-lang-ci/rust
that referenced
this pull request
Mar 19, 2021
Attempt to gather similar stats as rusage on Windows A follow up to rust-lang#82532. This is a bit hacked in because I think we need to discuss this before merging, but this is an attempt to gather similar metrics as `libc::rusage` on Windows. Some comments on differences: * Currently, we're passing `RUSAGE_CHILDREN` to `rusage` which collects statistics on all children that have been waited on and terminated. I believe this is currently just the invocation of the real `rustc` that the shim is wrapping. Does `rustc` itself spawn children processes? The windows version gets the child processes handle when spawning it, and uses that to collect the statistics. For maxrss, `rusage` will return "the resident set size of the largest child, not the maximum resident set size of the process tree.", the Windows version will only collect statistics on the wrapped `rustc` child process directly even if some theoretical sub process has a larger memory footprint. * There might be subtle differences between `rusage`'s "resident set" and Window's "working set". The "working set" and "resident set" should both be the number of pages that are in memory and which would not cause a page fault when accessed. * I'm not yet sure how best to get the same information that `ru_minflt`, `ru_inblock`, `ru_oublock`, `ru_nivcsw ` and `ru_nvcsw` provide. r? `@pnkfelix`
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds
build.print_step_rusageto config.toml, which is meant to be an easy way to let compiler developers get feedback on the terminal during bootstrap about resource usage during each step.The output is piggy-backed on
[PRINT-STEP-TIMINGS], mostly because the functionality seemed to naturally fit there in the overall control-flow and output structure (even if very little is shared between the implementations themselves).Some sample output (from my Linux box, where I believe the
max rssoutput to be somewhat trust-worthy...):