Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spurious failure in issue-26092 #43402

Closed
alexcrichton opened this issue Jul 22, 2017 · 10 comments · Fixed by #45170
Closed

Spurious failure in issue-26092 #43402

alexcrichton opened this issue Jul 22, 2017 · 10 comments · Fixed by #45170
Labels
A-spurious Area: Spurious failures in builds (spuriously == for no apparent reason) C-tracking-issue Category: A tracking issue for an RFC or an unstable feature.

Comments

@alexcrichton
Copy link
Member

Looks like:

failures:
---- [run-make] run-make\issue-26092 stdout ----
	
error: make failed
status: exit code: 2
command: "make"
stdout:
------------------------------------------
PATH="/c/projects/rust/build/x86_64-pc-windows-gnu/test/run-make/issue-26092.stage2-x86_64-pc-windows-gnu:C:\projects\rust\build\x86_64-pc-windows-gnu\stage2\bin:/c/projects/rust/build/x86_64-pc-windows-gnu/stage2-tools/x86_64-pc-windows-gnu/release/deps:/c/projects/rust/build/x86_64-pc-windows-gnu/stage2/lib/rustlib/x86_64-pc-windows-gnu/lib:/c/Program Files (x86)/Inno Setup 5:/c/Python27:/c/projects/rust/mingw64/bin:/usr/bin:/c/Perl/site/bin:/c/Perl/bin:/c/Windows/system32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0:/c/Program Files/7-Zip:/c/Program Files/Microsoft/Web Platform Installer:/c/Tools/GitVersion:/c/Tools/PsTools:/c/Program Files/Git LFS:/c/Program Files (x86)/Subversion/bin:/c/Program Files/Microsoft SQL Server/120/Tools/Binn:/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/110/Tools/Binn:/c/Program Files (x86)/Microsoft SQL Server/120/Tools/Binn:/c/Program Files/Microsoft SQL Server/120/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/120/Tools/Binn/ManagementStudio:/c/Tools/WebDriver:/c/Program Files (x86)/Microsoft SDKs/TypeScript/1.4:/c/Program Files (x86)/Microsoft Visual Studio 12.0/Common7/IDE/PrivateAssemblies:/c/Program Files (x86)/Microsoft SDKs/Azure/CLI/wbin:/c/Ruby193/bin:/c/Tools/NUnit/bin:/c/Tools/xUnit:/c/Tools/MSpec:/c/Tools/Coverity/bin:/c/Program Files (x86)/CMake/bin:/c/go/bin:/c/Program Files/Java/jdk1.8.0/bin:/c/Python27:/c/Program Files/nodejs:/c/Program Files (x86)/iojs:/c/Program Files/iojs:/c/Users/appveyor/AppData/Roaming/npm:/c/Program Files/Microsoft SQL Server/130/Tools/Binn:/c/Program Files (x86)/MSBuild/14.0/Bin:/c/Tools/NuGet:/c/Program Files (x86)/Microsoft Visual Studio 14.0/Common7/IDE/CommonExtensions/Microsoft/TestWindow:/c/Program Files/Microsoft DNX/Dnvm:/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/130/Tools/Binn:/c/Program Files (x86)/Microsoft SQL Server/130/Tools/Binn:/c/Program Files (x86)/Microsoft SQL Server/130/DTS/Binn:/c/Program Files/Microsoft SQL Server/130/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/110/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/120/DTS/Binn:/c/Program Files (x86)/Apache/Maven/bin:/c/Python27/Scripts:/c/Tools/NUnit3:/c/Program Files/Mercurial:/c/Program Files/LLVM/bin:/c/Program Files/dotnet:/c/Program Files/erl8.3/bin:/c/Tools/curl/bin:/c/Program Files/Amazon/AWSCLI:/c/Program Files/Git/cmd:/c/Program Files/Git/usr/bin:/c/Program Files (x86)/Microsoft SQL Server/140/DTS/Binn:/c/Program Files (x86)/Microsoft Visual Studio 14.0/Common7/IDE/Extensions/Microsoft/SQLDB/DAC/140:/c/ProgramData/chocolatey/bin:/c/Program Files (x86)/nodejs:/c/Program Files (x86)/Yarn/bin:/c/Program Files/Microsoft Service Fabric/bin/Fabric/Fabric.Code:/c/Program Files/Microsoft SDKs/Service Fabric/Tools/ServiceFabricLocalClusterManager:/c/Users/appveyor/AppData/Roaming/npm:/c/Users/appveyor/AppData/Local/Yarn/bin:/c/Program Files/AppVeyor/BuildAgent:/c/projects/rust:/c/projects/rust/handle" 'C:\projects\rust\build\x86_64-pc-windows-gnu\stage2\bin\rustc.exe' --out-dir /c/projects/rust/build/x86_64-pc-windows-gnu/test/run-make/issue-26092.stage2-x86_64-pc-windows-gnu -L /c/projects/rust/build/x86_64-pc-windows-gnu/test/run-make/issue-26092.stage2-x86_64-pc-windows-gnu  -o "" blank.rs 2>&1 | \
		grep -i 'No such file or directory'
------------------------------------------
stderr:
------------------------------------------
make: *** [Makefile:4: all] Error 1
------------------------------------------
thread '[run-make] run-make\issue-26092' panicked at 'explicit panic', src\tools\compiletest\src\runtest.rs:2473:8
note: Run with `RUST_BACKTRACE=1` for a backtrace.
failures:
    [run-make] run-make\issue-26092
test result: FAILED. 151 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out
@alexcrichton alexcrichton added the A-spurious Area: Spurious failures in builds (spuriously == for no apparent reason) label Jul 22, 2017
@Mark-Simulacrum Mark-Simulacrum added the C-tracking-issue Category: A tracking issue for an RFC or an unstable feature. label Jul 28, 2017
@Gankra
Copy link
Contributor

Gankra commented Aug 15, 2017

I just ran into this locally on mac, if that's relevant to anyone.

@philipc
Copy link
Contributor

philipc commented Oct 3, 2017

Not sure if this is related, but nightly is now writing thread panics to stderr:

$ echo 'fn main() {}' | rustup run nightly rustc -o "" -
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: FatalError', /checkout/src/libcore/result.rs:906:4
note: Run with `RUST_BACKTRACE=1` for a backtrace.
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: FatalError', /checkout/src/libcore/result.rs:906:4
error: could not write output to : No such file or directory

error: aborting due to previous error

thread '<unnamed>' panicked at 'aborting due to worker thread panic', /checkout/src/librustc_trans/back/write.rs:1643:20

whereas stable only gives:

$ echo 'fn main() {}' | rustup run stable rustc -o "" -
error: could not write output to : No such file or directory

error: aborting due to previous error

The order of the panic messages changes, so is it possible that one is being written in the middle of the 'No such file or directory'?

aidanhs added a commit to aidanhs/rust that referenced this issue Oct 4, 2017
Fixes rust-lang#43402 now there's no multithreaded panic printouts

Also update a comment
bors added a commit that referenced this issue Oct 5, 2017
…hton

Don't unwrap work item results as the panic trace is useless

Fixes #43402 now there's no multithreaded panic printouts

Also update a comment

--------

Likely regressed in #43506, where the code was changed to panic in worker threads on error.

Unwrapping gives zero extra information since the stack trace is so short, so we may as well just surface that there was an error and exit the thread properly. Because there are then no multithreaded printouts, I think it should mean the output of the test for #26199 is deterministic and not interleaved (thanks to @philipc #43402 (comment) for a hint).

Sadly the output is now:
```
thread '<unnamed>' panicked at 'aborting due to worker thread panic', src/librustc_trans/back/write.rs:1643:20
note: Run with `RUST_BACKTRACE=1` for a backtrace.
error: could not write output to : No such file or directory

error: aborting due to previous error
```
but it's an improvement over the multi-panic situation before.

r? @alexcrichton
@aidanhs
Copy link
Member

aidanhs commented Oct 5, 2017

I'm hoping this is fixed after the merge above. If anyone sees it again, please ping me.

@kennytm
Copy link
Member

kennytm commented Oct 6, 2017

@aidanhs The test failed again on 32-bit Linux in #44818.

@arielb1 arielb1 reopened this Oct 6, 2017
@aidanhs
Copy link
Member

aidanhs commented Oct 6, 2017

Ok, the error emitter itself is on a different thread. I'll see if I can figure out a fix.

bors added a commit that referenced this issue Oct 9, 2017
…chton

Don't panic in the coordinator thread, bubble up the failure

Fixes #43402 (take 2)

Followup to #45019, this makes the coordinator thread not panic on worker failures since they can be reported reasonably back in the main thread.

The output also now has no evidence of backtraces at all, unlike the previous PR:
```
$ ./build/x86_64-unknown-linux-gnu/stage1/bin/rustc -o "" x.rs
error: could not write output to : No such file or directory

error: aborting due to previous error
```

r? @alexcrichton
@aidanhs
Copy link
Member

aidanhs commented Oct 9, 2017

Slightly more hopeful that this is fixed this time, but again - ping me if not :)

@kennytm
Copy link
Member

kennytm commented Oct 10, 2017

@aidanhs 😢 #45160 (comment)

@aidanhs
Copy link
Member

aidanhs commented Oct 10, 2017

Hmm, maybe need to do some testing on windows.

@aidanhs aidanhs reopened this Oct 10, 2017
@aidanhs
Copy link
Member

aidanhs commented Oct 10, 2017

Well this seems to be cause now (happens very intermittently):

$ rustc -o "" blank.rs  2>&1
error: could not write output to 

error: aborting due to previous error
$ rustc -o "" blank.rs  2>&1
error: could not write output to : No such file or directory

error: aborting due to previous error

aidanhs added a commit that referenced this issue Oct 10, 2017
This is a big hammer, but should be effective at completely removing a
few issues, including inconsistent error messages and segfaults when
LLVM workers race to report results

LLVM_THREAD_LOCAL has been present in LLVM since 8 months before 3.7
(the earliest supported LLVM version that Rust can use)

Maybe fixes #43402 (third time lucky?)
bors added a commit that referenced this issue Oct 10, 2017
…al, r=alexcrichton

Band-aid fix to stop race conditions in llvm errors

This is a big hammer, but should be effective at completely removing a
few issues, including inconsistent error messages and segfaults when
LLVM workers race to report results

`LLVM_THREAD_LOCAL` has been present in LLVM since 8 months before 3.7
(the earliest supported LLVM version that Rust can use)

Maybe fixes #43402 (third time lucky?)

r? @alexcrichton

------

You can see that in https://github.com/rust-lang/rust/blob/5f578dfad0dd5d43b28eff71a7e857d10c3f55fe/src/librustc_trans/back/write.rs#L75-L100 there's a small window where the static global error message (made thread local in this PR) could be altered by another thread.

Note that we can't use `thread_local` because gcc 4.7 (permitted according to the readme) does not support it.

Maybe ideally all the functions should be modified to not use a global, but this PR makes things deterministic at least. My only hesitation is whether errors are checked in different threads to where they occur, but I figure that's probably unlikely (and is less bad than racing code).

As an aside, segfault evidence before this patch when I was doing some debugging:
```
$ while grep 'No such file or directory' log2; do RUST_LOG=debug ./build/x86_64-unknown-linux-gnu/stage1/bin/rustc -o "" y.rs >log2 2>&1; done
error: could not write output to : No such file or directory
error: could not write output to : No such file or directory
error: could not write output to : No such file or directory
error: could not write output to : No such file or directory
error: could not write output to : No such file or directory
error: could not write output to : No such file or directory
error: could not write output to : No such file or directory
Segmentation fault (core dumped)
error: could not write output to : No such file or directory
error: could not write output to : No such file or directory
```
@aidanhs
Copy link
Member

aidanhs commented Oct 10, 2017

Another fix attempt merged (third time lucky?), same drill as usual!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-spurious Area: Spurious failures in builds (spuriously == for no apparent reason) C-tracking-issue Category: A tracking issue for an RFC or an unstable feature.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants