Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: Bootstrap no longer documents compiler crates #117430

Closed
fmease opened this issue Oct 31, 2023 · 24 comments · Fixed by #117471 or #117535
Closed

CI: Bootstrap no longer documents compiler crates #117430

fmease opened this issue Oct 31, 2023 · 24 comments · Fixed by #117471 or #117535
Labels
A-contributor-roadblock Area: Makes things more difficult for new contributors to rust itself A-docs Area: documentation for any part of the project, including the compiler, standard library, and tools C-bug Category: This is a bug. regression-from-stable-to-nightly Performance or correctness regression from stable to nightly. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)

Comments

@fmease
Copy link
Member

fmease commented Oct 31, 2023

Bootstrap no longer documents any rustc_* / compiler/ crates or rustdoc in CI.
The index now only contains a meager 20 or so crates.
This has to have regressed fairly recently.

@fmease fmease added T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) A-docs Area: documentation for any part of the project, including the compiler, standard library, and tools C-bug Category: This is a bug. A-contributor-roadblock Area: Makes things more difficult for new contributors to rust itself labels Oct 31, 2023
@rustbot rustbot added the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Oct 31, 2023
@fmease fmease removed the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Oct 31, 2023
@Nemo157
Copy link
Member

Nemo157 commented Oct 31, 2023

It regressed in the latest nightly, we have CI tests on docs.rs checking for some of these and the last daily test succeeded.

@ChrisDenton
Copy link
Member

@Mark-Simulacrum
Copy link
Member

This appears to have regressed in #117328... but I'm pretty confused by that.

$ for commit in $(git log --first-parent --format=%H | head -n30); do echo $commit; curl -sI "https://ci-artifacts.rust-lang.org/rustc-builds/$commit/rustc-docs-nightly-x86_64-unknown-linux-gnu.tar.xz" | grep content-length; done
ffb7ed9fa420e9bcd98d84b431b2009445b7b967
content-length: 5454792
650991d62c3a2c80ba27009d06839adbb038bf5e
content-length: 5456880
a395214a3a88be976eec9256987d5eb6b3584bdc
content-length: 5454420
31bc7e2c47e82798a392c770611975a6883132c8
content-length: 5456600
236ac911de2f4b6210ba99e25c6c4843c9381a1a
content-length: 5454984
e6e931dda5fffbae0fd87c5b1af753cc95556880
content-length: 5468104
6d1fc53cf4f61b0dce5fc84cebae3b9fb8a45436
content-length: 5468832
e324cf0f73f09388d494986550f46dc9e12badd3
content-length: 5468796
91bbdd927a5e53a2fe126304fe8adbedf339616c
content-length: 5468436
1dfb6b162be402d8ca37e8aad4f58898b44e3a15
content-length: 5468796
bcb5798dd890a691644af9d371f3bd7fcc465584
content-length: 22395756
608e9682f0a6482903de8d3332770104a0ad943c
content-length: 22390916
ec2b311914a4880235862f3cd65fe24e92e6d3a5
content-length: 22391560
88ae8c93854a1a67c6c0a45971c7198477456c33
content-length: 22396416
83c9732e0c7e1cae5f039677da4c51ee1d9b19b0
content-length: 22393792
160fd3ceb5143100d4e06f324b02af1f9e786372
content-length: 22375124
bbcc1691a4646a6d2ebb2189bdff5b878407fbe6

@lqd
Copy link
Member

lqd commented Oct 31, 2023

This appears to have regressed in #117328... but I'm pretty confused by that.

Since #117328 mostly allowed #81746 to have an effect, it could be that the issue is in #81746.

@Noratrieb
Copy link
Member

cc @bjorn3 and also @onur-ozkan maybe you happen to have an idea too

@Noratrieb Noratrieb added the regression-from-stable-to-nightly Performance or correctness regression from stable to nightly. label Oct 31, 2023
@rustbot rustbot added the I-prioritize Issue: Indicates that prioritization has been requested for this issue. label Oct 31, 2023
@Noratrieb Noratrieb pinned this issue Oct 31, 2023
@bjorn3
Copy link
Member

bjorn3 commented Oct 31, 2023

I don't have a clue how either PR could have affected building docs.

@onur-ozkan
Copy link
Member

onur-ozkan commented Oct 31, 2023

I would do quick check between the recently merged PRs as there are not many. It seems not to be so hard to find and fix the problem. But I can only do that later tonight.. I don't have access to my computers right now.

@fmease fmease changed the title Bootstrap no longer documents compiler crates CI/CD: Bootstrap no longer documents compiler crates Oct 31, 2023
@onur-ozkan
Copy link
Member

Ok, so the problem is when we use cranelift as a backend with llvm, documentation outputs generated by these backends seem to conflict with each other during the invocations. It appears that one of them(most likely cranelift) is not generating the compiler documentation. As we purge the previously generated documentation output at

self.clear_if_dirty(&my_out, &rustdoc);

we end up with output that doesn't have the compiler documentation.

I am currently unsure of the exact reason why cranelift is causing this issue. I will open a temporary fix PR and investigate further once I get more time.

onur-ozkan added a commit to onur-ozkan/rust that referenced this issue Oct 31, 2023
Temporary fix for rust-lang#117430

Signed-off-by: onur-ozkan <work@onurozkan.dev>
onur-ozkan added a commit to onur-ozkan/rust that referenced this issue Oct 31, 2023
Temporary fix for rust-lang#117430

Signed-off-by: onur-ozkan <work@onurozkan.dev>
@fmease fmease changed the title CI/CD: Bootstrap no longer documents compiler crates CI: Bootstrap no longer documents compiler crates Nov 1, 2023
@apiraino apiraino removed the I-prioritize Issue: Indicates that prioritization has been requested for this issue. label Nov 1, 2023
@RalfJung
Copy link
Member

RalfJung commented Nov 2, 2023

Indeed the search at https://doc.rust-lang.org/nightly/nightly-rustc/ is completely useless right now. If a fix would take more than a day, reverting the offending PR should also be considered, given that those docs are quite crucial for rustc contributors.

@onur-ozkan
Copy link
Member

Indeed the search at https://doc.rust-lang.org/nightly/nightly-rustc/ is completely useless right now. If a fix would take more than a day, reverting the offending PR should also be considered, given that those docs are quite crucial for rustc contributors.

Fix PR is already open at #117471

@RalfJung
Copy link
Member

RalfJung commented Nov 2, 2023

Yeah I've seen that, but I can't tell if that's the lowest-risk approach. 🤷 Reverting (or un-setting CODEGEN_BACKENDS on whatever builder builds the docs) is something we know will obviously work.

@onur-ozkan
Copy link
Member

I am quite sure that PR contains no risk, but I am fine with reverting codegen environment variable updates in the runner containers

@onur-ozkan
Copy link
Member

#117471 did fix it locally, but seems didn't fix it on actual environment..

@onur-ozkan onur-ozkan reopened this Nov 2, 2023
@onur-ozkan
Copy link
Member

onur-ozkan commented Nov 2, 2023

To fix this, we can either disable cranelift on the gh action step which we build nightly documentation or revert #117328 until we find out and fix the problem of compiler documentations with cranelift.

@RalfJung
Copy link
Member

RalfJung commented Nov 3, 2023

disable cranelift on the gh action step which we build nightly documentation

Which one is that?

@onur-ozkan
Copy link
Member

disable cranelift on the gh action step which we build nightly documentation

Which one is that?

I guess (never worked on the infra side before) these ones

- name: upload artifacts to S3
run: src/ci/scripts/upload-artifacts.sh
env:
AWS_ACCESS_KEY_ID: "${{ env.ARTIFACTS_AWS_ACCESS_KEY_ID }}"
AWS_SECRET_ACCESS_KEY: "${{ secrets[format('AWS_SECRET_ACCESS_KEY_{0}', env.ARTIFACTS_AWS_ACCESS_KEY_ID)] }}"
if: "success() && !env.SKIP_JOB && (github.event_name == 'push' || env.DEPLOY == '1' || env.DEPLOY_ALT == '1')"

- name: upload artifacts to S3
run: src/ci/scripts/upload-artifacts.sh
env:
AWS_ACCESS_KEY_ID: "${{ env.ARTIFACTS_AWS_ACCESS_KEY_ID }}"
AWS_SECRET_ACCESS_KEY: "${{ secrets[format('AWS_SECRET_ACCESS_KEY_{0}', env.ARTIFACTS_AWS_ACCESS_KEY_ID)] }}"
if: "success() && !env.SKIP_JOB && (github.event_name == 'push' || env.DEPLOY == '1' || env.DEPLOY_ALT == '1')"

- name: upload artifacts to S3
run: src/ci/scripts/upload-artifacts.sh
env:
AWS_ACCESS_KEY_ID: "${{ env.ARTIFACTS_AWS_ACCESS_KEY_ID }}"
AWS_SECRET_ACCESS_KEY: "${{ secrets[format('AWS_SECRET_ACCESS_KEY_{0}', env.ARTIFACTS_AWS_ACCESS_KEY_ID)] }}"
if: "success() && !env.SKIP_JOB && (github.event_name == 'push' || env.DEPLOY == '1' || env.DEPLOY_ALT == '1')"

@RalfJung
Copy link
Member

RalfJung commented Nov 3, 2023

Those steps just do the uploading, the docs are most likely created before. We need to figure out which runner is doing this and disable cranelift there.

@onur-ozkan
Copy link
Member

It's done in multiple place(creating and uploading). I think the question is which one is getting deployed to https://doc.rust-lang.org/nightly/nightly-rustc/index.html.

@RalfJung
Copy link
Member

RalfJung commented Nov 3, 2023

It's been 3 days without docs, clearly this is not entirely trivial to fix. Let's revert.
#117535

bors added a commit to rust-lang-ci/rust that referenced this issue Nov 3, 2023
Revert "Auto merge of rust-lang#117328 - lqd:cranelift-rocket, r=Mark-Simulacrum"

This reverts commit 1dfb6b1, reversing changes made to bcb5798.

That commit broke generating nightly rustc docs. Revert it until we can figure out how to have both, cranelift and docs.

Fixes rust-lang#117430
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Nov 3, 2023
…zkan

Revert "bootstrap: do not purge docs on CI environment"

This reverts commit 6198e88.

ref rust-lang#117430 (comment), rust-lang#117471 (comment)
@bors bors closed this as completed in adda05f Nov 3, 2023
rust-timer added a commit to rust-lang-ci/rust that referenced this issue Nov 3, 2023
Rollup merge of rust-lang#117533 - onur-ozkan:revert-117471, r=onur-ozkan

Revert "bootstrap: do not purge docs on CI environment"

This reverts commit 6198e88.

ref rust-lang#117430 (comment), rust-lang#117471 (comment)
@lqd
Copy link
Member

lqd commented Nov 4, 2023

The docs are back.

@lqd lqd unpinned this issue Nov 4, 2023
@ChrisDenton
Copy link
Member

While the immediate issue is addressed, do we need an issue to track the underlying problem?

@lqd
Copy link
Member

lqd commented Nov 4, 2023

We could. At the same time, @bjorn3 and @onur-ozkan are also still working on this, e.g. in this zulip thread as it's quite important to fix. There's little chance we'll forget about this, and it should hopefully be fixed very soon.

@onur-ozkan
Copy link
Member

The underlying problem is that symlinks are being dropped in a wrong way between invocations. I can easily fix this by copying the document outputs instead of linking them, but I would prefer to invest a couple more hours in fixing the linking issue rather than replacing it.

bors added a commit to rust-lang-ci/rust that referenced this issue Nov 4, 2023
…g, r=<try>

re-enable cranelift on CI

First commit addresses the linking issue with compiler crates. Second one ensures that compiler crates are linked correctly, allowing us to detect these hard-to-catch bugs on CI. The remaining three commits cherry-picked from rust-lang#117328 to re-enable the Cranelift backend on CI.

More info: rust-lang#117430

cc `@bjorn3` `@RalfJung`
bors added a commit to rust-lang-ci/rust that referenced this issue Nov 4, 2023
…g, r=<try>

re-enable cranelift on CI

First commit addresses the linking issue with compiler crates. Second one ensures that compiler crates are linked correctly, allowing us to detect these hard-to-catch bugs on CI. The remaining three commits cherry-picked from rust-lang#117328 to re-enable the Cranelift backend on CI.

More info: rust-lang#117430

cc `@bjorn3` `@RalfJung`
bors added a commit to rust-lang-ci/rust that referenced this issue Nov 4, 2023
…g, r=<try>

re-enable cranelift on CI

First commit addresses the linking issue with compiler crates. Second one ensures that compiler crates are linked correctly, allowing us to detect these hard-to-catch bugs on CI. The remaining three commits cherry-picked from rust-lang#117328 to re-enable the Cranelift backend on CI.

More info: rust-lang#117430

cc `@bjorn3` `@RalfJung`
bors added a commit to rust-lang-ci/rust that referenced this issue Nov 5, 2023
…g, r=<try>

re-enable cranelift on CI

First commit addresses the linking issue with compiler crates. Second one ensures that compiler crates are linked correctly, allowing us to detect these hard-to-catch bugs on CI. Following three commits cherry-picked from rust-lang#117328 to re-enable the Cranelift backend on CI.

More info: rust-lang#117430

cc `@bjorn3` `@RalfJung`
bors added a commit to rust-lang-ci/rust that referenced this issue Nov 5, 2023
…g, r=<try>

re-enable cranelift on CI

First commit addresses the linking issue with compiler crates. Second one ensures that compiler crates are linked correctly, allowing us to detect these hard-to-catch bugs on CI. Following three commits cherry-picked from rust-lang#117328 to re-enable the Cranelift backend on CI.

More info: rust-lang#117430

cc `@bjorn3` `@RalfJung`
bors added a commit to rust-lang-ci/rust that referenced this issue Nov 5, 2023
…g, r=<try>

re-enable cranelift on CI

First commit addresses the linking issue with compiler crates. Second one ensures that compiler crates are linked correctly, allowing us to detect these hard-to-catch bugs on CI. Following three commits cherry-picked from rust-lang#117328 to re-enable the Cranelift backend on CI.

More info: rust-lang#117430

cc `@bjorn3` `@RalfJung`
bors added a commit to rust-lang-ci/rust that referenced this issue Nov 5, 2023
…g, r=Mark-Simulacrum

improve compiler&tool documenting and re-enable cranelift on CI

First commit addresses the linking issue with compiler crates. Second one ensures that compiler crates are linked correctly (with later commits we added this check for tools as well), allowing us to detect these hard-to-catch bugs on CI. Following three commits cherry-picked from rust-lang#117328 to re-enable the Cranelift backend on CI.

More info: rust-lang#117430

cc `@bjorn3` `@RalfJung`
github-actions bot pushed a commit to rust-lang/miri that referenced this issue Nov 15, 2023
…-Simulacrum

improve compiler&tool documenting and re-enable cranelift on CI

First commit addresses the linking issue with compiler crates. Second one ensures that compiler crates are linked correctly (with later commits we added this check for tools as well), allowing us to detect these hard-to-catch bugs on CI. Following three commits cherry-picked from #117328 to re-enable the Cranelift backend on CI.

More info: rust-lang/rust#117430

cc `@bjorn3` `@RalfJung`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-contributor-roadblock Area: Makes things more difficult for new contributors to rust itself A-docs Area: documentation for any part of the project, including the compiler, standard library, and tools C-bug Category: This is a bug. regression-from-stable-to-nightly Performance or correctness regression from stable to nightly. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)
Projects
None yet