CI: Migrate to windows-2022

[ehuss]

GitHub is starting a migration of windows-latest to windows-2022:
https://github.blog/changelog/2022-01-11-github-actions-jobs-running-on-windows-latest-are-now-running-on-windows-server-2022/
actions/runner-images#4856

Migration starts now, and should finish by March 6. It is not clear if that timetable includes the -xl runners. Based on the fact that the ubuntu-latest-xl runners have not yet migrated to 20.04, which was "completed" in 2021-03-18 (and it is now 2022-01-15), the timing for the -xl images might be much slower.

UPDATE from GitHub: Our first goal is to GA Windows 2022 this quarter and then we will start the process of getting the XL pools ready.

It may be possible to lock to an older version with windows-2019-xl (as it appears those version-specific qualifiers work with ubuntu), but I have not tested that.

I have started testing some builds in 2022 and they are not going well, I'll try to document some issues here.

[ehuss]

Issue: dos2unix does not work

The src/ci/scripts/verify-line-endings.sh script fails with the error:

      1 [main] dos2unix (4780) C:\Program Files\Git\usr\bin\dos2unix.exe: *** fatal error - cygheap base mismatch detected - 0x180348408/0x180347408.
This problem is probably due to using incompatible versions of the cygwin DLL.
Search for cygwin1.dll using the Windows Start->Find/Search facility
and delete all but the most recent version.  The most recent version *should*
reside in x:\cygwin\bin, where 'x' is the drive on which you have
installed the cygwin distribution.  Rebooting is also suggested if you
are unable to find another cygwin DLL.

I haven't investigated much, but I suspect it might be a conflict with one of our manually downloaded msys/mingw things. actions/runner-images#594 also contains some context.

As a side note: The install-mingw step installs and updates a bunch of packages, which the 2019 images do not (all the packages are already installed and up-to-date).

[ehuss]

Issue: errors in run-make-fulldeps

Many of the run-make-fulldeps tests are failing. Many are probably related to the same underlying problem of handling gnu vs windows paths.

x86_64-pc-windows-gnu and x86_64-pc-windows-msvc fail with about 125 errors, roughly of this form:

D:arustrustbuildx86_64-pc-windows-gnutestrun-make-fulldepsa-b-a-linker-guarda-b-a-linker-guard/b: No such file or directory

could not find native static library `foo`, perhaps an -L flag is missing?

D:\a\rust\rust\build\x86_64-pc-windows-gnu\test\run-make-fulldeps\arguments-non-c-like-enum\arguments-non-c-like-enum/libnonclike.a: No such file or directory

error: linking with `x86_64-w64-mingw32-gcc` failed: exit code: 1
  |
  = note: "x86_64-w64-mingw32-gcc" "-Wl,C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\rustcgZbYmn\\list.def" "-fno-use-linker-plugin" "-Wl,--dynamicbase" "-Wl,--disable-auto-image-base" "-m64" "-Wl,--high-entropy-va" "C:\\MORE_SPACE\\x86_64-pc-windows-gnu\\stage2\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\rsbegin.o" "D:arustrustbuildx86_64-pc-windows-gnutestrun-make-fulldepsc-dynamic-dylibc-dynamic-dylib\\foo.foo.172d8985-cgu.0.rcgu.o" "D:arustrustbuildx86_64-pc-windows-gnutestrun-make-fulldepsc-dynamic-dylibc-dynamic-dylib\\foo.21ett6z3hgvj0xmp.rcgu.rmeta" "-L" "D:arustrustbuildx86_64-pc-windows-gnutestrun-make-fulldepsc-dynamic-dylibc-dynamic-dylib" "-L" "C:\\MORE_SPACE\\x86_64-pc-windows-gnu\\stage2\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib" "-lcfoo" "-Wl,--start-group" "-L" "C:\\MORE_SPACE\\x86_64-pc-windows-gnu\\stage2\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib" "-lstd-6a8d029579976682" "-Wl,--end-group" "-Wl,-Bstatic" "C:\\MORE_SPACE\\x86_64-pc-windows-gnu\\stage2\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\libcompiler_builtins-94cb478f7048ff38.rlib" "-Wl,-Bdynamic" "-lkernel32" "-lws2_32" "-lbcrypt" "-ladvapi32" "-luserenv" "-lkernel32" "-lgcc_s" "-lmsvcrt" "-lmingwex" "-lmingw32" "-lgcc" "-lmsvcrt" "-luser32" "-lkernel32" "-Wl,--nxcompat" "-L" "C:\\MORE_SPACE\\x86_64-pc-windows-gnu\\stage2\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib" "-o" "D:arustrustbuildx86_64-pc-windows-gnutestrun-make-fulldepsc-dynamic-dylibc-dynamic-dylib\\foo.dll" "-shared" "-Wl,--out-implib=D:arustrustbuildx86_64-pc-windows-gnutestrun-make-fulldepsc-dynamic-dylibc-dynamic-dylib\\libfoo.dll.a" "-nodefaultlibs" "C:\\MORE_SPACE\\x86_64-pc-windows-gnu\\stage2\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\rsend.o"
  = note: D:/a/rust/rust/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find -lcfoo
          collect2.exe: error: ld returned 1 exit status

cp: cannot stat 'D:\a\rust\rust\build\x86_64-pc-windows-gnu\test\run-make-fulldeps\extern-multiple-copies\extern-multiple-copies/libfoo1.rlib': No such file or directory

error: error writing dependencies to `D:arustrustbuildx86_64-pc-windows-gnutestrun-make-fulldepsinclude_bytes_depsinclude_bytes_deps\main.d`: The system cannot find the path specified. (os error 3)

/usr/bin/sh: line 1: D:arustrustbuildx86_64-pc-windows-gnutestrun-make-fulldepscompile-stdincompile-stdin/rust_out: No such file or directory

[Kay-kay2019]

It probably needs an update but I'll look at it when I get home. Any recommendations?

[ehuss]

Unfortunately I don't know why it is failing. We install a custom mingw and msys2 (https://github.com/rust-lang/rust/blob/master/src/ci/scripts/install-mingw.sh and https://github.com/rust-lang/rust/blob/master/src/ci/scripts/install-msys2.sh), which might be interfering with the system-provided binaries. Or it could be something else, I haven't investigated.

[dpaoliello]

I'd like to have a go at getting this working

@rustbot claim

[dpaoliello]

@ehuss is there something that I need to do to get access to Server 2022 build agents?
I tried modifying the PR build, but it seems to be stuck waiting for agents: https://github.com/rust-lang/rust/actions/runs/8758234487/job/24038585875?pr=124174

[dpaoliello]

First blocking issue:

CI currently installs Clang14 to build LLVM:

rust/src/ci/scripts/install-clang.sh

Line 13 in ce3263e

LLVM_VERSION="14.0.5"

But the STL in VS 2022 requires Clang 16+:
https://github.com/rust-lang/rust/actions/runs/8758452222/job/24039301488#step:27:652

I've raised this on Zulip with wg-llvm: https://rust-lang.zulipchat.com/#narrow/stream/187780-t-compiler.2Fwg-llvm/topic/Updating.20the.20version.20of.20LLVM.20used.20to.20build.20LLVM.3F/near/434435620

[dpaoliello]

Pinged t-infra for XL Server 2022 images: https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra/topic/Availability.20of.20XL.20Windows.20Server.202022.20runners/near/435038176

[dpaoliello]

Current status

Draft PR: #124174

Pending Fixes

I'm waiting on the intra team to help out with two things (would appreciate some help to expidite these):

• Setting up XL Server 2022 runners
• Getting an updated version of Clang on the mirror

There also one test that was failing which I've submitted a PR to fix: #124378

Build status

Working

• i686-mingw
• x86_64-mingw
• dist-i686-mingw
• dist-x86_64-mingw

Works with updated Clang (16+)

• x86_64-msvc
• i686-msvc
• x86_64-msvc-ext
• dist-i686-msvc
• dist-x86_64-msvc-alt

Broken

• dist-x86_64-msvc - Keeps timing out on the standard build machines, but I don't foresee any other issues.
• dist-aarch64-msvc - Requires an LLVM fix which is in 19, but I'm trying to backport to 18 backporting is not possible as it would be an ABI break.

[ChrisDenton]

This was only partially fixed.