Migrate "ui" test suite from legacy compiletest-style directives // to ui_test-style directives //@

[jieyouxu]

Preface

There's an on-going effort to rewrite parts of or the entirety of compiletest
(rust-lang/compiler-team#536). A step towards this involve migrating
ui tests to use the ui_test framework, which involves
changing compiletest directives in // <directive-name> style to ui_test //@ <directive-name>
style (rust-lang/compiler-team#512).

This PR aims to implement the directive-style change from // to //@ for ui tests only and
make compiletest only accept //@ directives in the "ui" test suite (only).

Key Changes

1. All ui test // directives are replaced by //@ directives.
2. Only accept //@ directives for "ui" test suite.
3. Errors if a comment could be interpreted as a legacy-style // directive.

Diff Generation

The diff is generated by:

• Collecting directives from ui tests via hacking on compiletest.
• Using a migration tool to replace // directives in ui tests with //@.

Reproduction Steps

0. Delete the temporary directory $RUSTC_REPO_PATH/build/x86_64-apple-darwin/test/ui/__directive_lines and the temporary file $RUSTC_REPO_PATH/build/x86_64-apple-darwin/test/ui/__directive_lines.txt (if you ran the collection script before).
1. Use the https://github.com/jieyouxu/rust/tree/collect-test-directives collect-test-directives
   script, which outputs temporary files recording headers occuring in each ui test.
   • You need to checkout this branch: git checkout collect-test-directives.
   • You might need to rebase on lastest master and ensure there are no conflicts.
   • You likely need to run ./x test tests/ui --stage 1 --force-rerun to generate the temporary
     files consistently.
2. Checkout the migrate-ui-test-directives branch.
3. Run the migration tool https://github.com/jieyouxu/compiletest-ui_test-header-migration.
   • You will need to first generate a migration_config.toml via cargo run -- generate-config under $CWD.
   • Then, update manual_directives = ["// should-fail"] in migration_config.toml. This is required because the collection script doesn't deal with some special meta ui tests and there are no other // should-fail occurrences.
4. Check that the migration at least does not cause UI test failures if you change compiletest to
   accept //@ directives for ui tests only.
   • RUSTC_TEST_FAIL_FAST=1 ./x test tests/ui --stage 1 --bless
5. Confirm that there is no difference after running the migration tool when you are on the
   migrate-ui-test-directives branch.

Next Steps

• Need to implement some kind of warning or tidy script to help contributors catch old-style
  // <directive-name> directives, while only accepting ui_test-style //@ <directive-name>
  directives. An error is emitted if a comment that could be interpreted as legacy-style test directive is encountered.
• Need to properly document this change in e.g. rustc-dev-guide (Update docs about ui tests now using //@ headers rustc-dev-guide#1885).
  • Add a README.md to tests/ui describing the directive style change.

[fmease]

Lol, did rustbot just give up?
r? compiler

[oli-obk]

r? @oli-obk

[oli-obk]

cc @asquared31415 @pietroalbini

[asquared31415]

I no longer have the energy to work on my efforts for that, but I have pushed the rest of my changes to my fork and this was part of an effort to allow compiletest to properly parse ui_test style headers, not just change the syntax. Feel free to use any of this code for any purpose.

I think it's a better move to just move to ui_test style comments without changing the actual commands. However I would like to see a tidy warning and fix suggestion (that can auto apply on bless), to make sure that developers don't accidentally use the old style comments and the test doesn't actually test everything it's meant to.

(I have some code in my branch that has tidy fixing things, for reference. However I ended up completely changing how the parsing works, so it may or may not be usable.)

[jieyouxu]

I think it's a better move to just move to ui_test style comments without changing the actual commands. However I would like to see a tidy warning and fix suggestion (that can auto apply on bless), to make sure that developers don't accidentally use the old style comments and the test doesn't actually test everything it's meant to.

I agree that a tidy warning would be a good idea. I'm not entirely sure on the auto-apply fix suggestion though, because there might be unfortunate cases like

// This comment serves to mention that
// ignore-test is a very useful directive
// which ignores the test.

and then the test can become unintentionally ignored. (Even though is this the current behavior.)

[oli-obk]

It may be simpler to parse twice, once with @ and once without, and error if any successfully parse without.

[jieyouxu]

It may be simpler to parse twice, once with @ and once without, and error if any successfully parse without.

I added a very dumb check, trying to parse a // comment as a legacy directive. Currently it simply reports an error message and panics on the first such encountered directive.

[bors]

☔ The latest upstream changes (presumably #120980) made this pull request unmergeable. Please resolve the merge conflicts.

[rustbot]

Changes to the code generated for builtin derived traits.

cc @nnethercote

[jieyouxu]

Sorry there's something weird with my diff, might have accidentally done something...

[jieyouxu]

I can squash the commits after the relevant migration parts are reviewed.
@rustbot review

[oli-obk]

lgtm! Please squash.

will the collect-test-directives branch still work on top of this or does it include the relevant parts to skip tidy?

[rust-timer]

Finished benchmarking commit (bccb9bb): comparison URL.

Overall result: ✅ improvements - no action needed

@rustbot label: -perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-0.3%	[-0.3%, -0.3%]	1
All ❌✅ (primary)	-	-	0

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-1.2%	[-1.2%, -1.2%]	1
All ❌✅ (primary)	-	-	0

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 638.862s -> 639.086s (0.04%)
Artifact size: 306.32 MiB -> 308.39 MiB (0.67%)