integ-runner: complaining Please use the IntegTest construct to configure the test

[pahud]

Describe the bug

integ-runner is complaining

Error during integration test: Error: integ.main is a new test. Please use the IntegTest construct to configure the test

while my test has been using IntegTest in the test.

Expected Behavior

integ-runner should not complain the error

Current Behavior

It's complaining the error(see reproduction steps below)

Reproduction Steps

in test/integ/integ.main.ts

/// !cdk-integ TestStack
import { IntegTest } from '@aws-cdk/integ-tests-alpha';
import {
  App, RemovalPolicy, Stack, StackProps,
  aws_s3 as s3,
} from 'aws-cdk-lib';
import { Construct } from 'constructs';

export class DemoStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    new s3.Bucket(this, 'DummyBucket', {
      removalPolicy: RemovalPolicy.DESTROY,
    });
  }
}

const app = new App();
const stack = new Stack(app, 'TestStack');
new IntegTest(app, 'cdk-integ', {
  testCases: [stack],
});

When I run npx integ-runner --directory test/integ, I got

Verifying integration test snapshots...

  NEW        integ.main 2.034s

Snapshot Results: 

Tests:    1 failed, 1 total
Error: Some tests failed!
To re-run failed tests run: integ-runner --update-on-failed

And run it again with --update-on-failed

$ npx integ-runner --directory test/integ --update-on-failed

I got

Verifying integration test snapshots...

  NEW        integ.main 1.722s

Snapshot Results: 

Tests:    1 failed, 1 total

Running integration tests for failed tests...

Running in parallel across regions: us-east-1, us-east-2, us-west-2
Running test /Users/xxx/Downloads/repro2/test/integ/integ.main.ts in us-east-1
  ERROR      /Users/xxx/Downloads/repro2/test/integ/integ.main.ts (undefined/us-east-1) 1.688s
      Error during integration test: Error: integ.main is a new test. Please use the IntegTest construct to configure the test
https://github.com/aws/aws-cdk/tree/main/packages/%40aws-cdk/integ-tests

Test Results: 

Tests:    1 failed, 1 total
Error: Some integration tests failed!
    at main (/Users/xxx/Downloads/repro2/node_modules/@aws-cdk/integ-runner/lib/index.js:10367:15)

But I am actually using IntegTest.

Possible Solution

No response

Additional Information/Context

integ-runner version "2.72.1"
integ-tests-alpha version "2.124.0-alpha.0"

CDK CLI Version

2.124.0 (build 4b6724c)

Framework Version

No response

Node.js Version

v18.16.0

OS

Mac OS X

Language

TypeScript

Language Version

No response

Other information

No response

[GavinZZ]

For visibility, there’s a recent change that was released in 2.124.0 to only allow integ-runner to run javascript test file in experimental modules and aws-cdk-lib, so all the new ts test file needs to be built through yarn build into js test file. This could be the reason that caused the failure.

[pahud]

@GavinZZ Thank you but I guess it might not be related.

This works for me with the following steps with cdk 2.125.0

// initiate a new project
$ cdk init -l typescript
$ mkdir test/integ
// edit test/integ/integ.main.ts and past all content mentioned above in the ts file and save.
$ yarn add @aws-cdk/integ-tests-alpha  
$ yarn add @aws-cdk/integ-runner
$ npx integ-runner --directory test/integ 

You see the error message

Verifying integration test snapshots...

  NEW        integ.main 1.999s

Snapshot Results: 

Tests:    1 failed, 1 total
Failed: /Users/xxx/repos/repro5/test/integ/integ.main.ts
Error: Some tests failed!
To re-run failed tests run: integ-runner --update-on-failed
    at main (/Users/xxx/repos/repro5/node_modules/@aws-cdk/integ-runner/lib/index.js:10424:13)

Now, run again with --update-on-failed. It works with snapshots generated.

% npx integ-runner --directory test/integ --update-on-failed

Verifying integration test snapshots...

  NEW        integ.main 4.581s

Snapshot Results: 

Tests:    1 failed, 1 total
Failed: /Users/xxx/repos/repro5/test/integ/integ.main.ts

Running integration tests for failed tests...

Running in parallel across regions: us-east-1, us-east-2, us-west-2
Running test /Users/xxx/repos/repro5/test/integ/integ.main.ts in us-east-1
  SUCCESS    integ.main-cdk-integ/DefaultTest 14.391s
       NO ASSERTIONS

Test Results: 

Tests:    1 passed, 1 total

However, if you initiate your project with projen

$ npx projen new awscdk-app-ts

edit .projenrc.js

const { awscdk } = require('projen');
const project = new awscdk.AwsCdkTypeScriptApp({
  cdkVersion: '2.1.0',
  defaultReleaseBranch: 'main',
  name: 'repro5',
  devDeps: [
    '@aws-cdk/integ-tests-alpha',
    '@aws-cdk/integ-runner',
  ],
});
project.synth();

$ npx projen
$ mkdir test/integ
// edit test/integ/integ.main.ts and save
$ npx integ-runner --directory test/integ  // got the error
$ npx integ-runner --directory test/integ --update-on-failed

Then I get the error

% npx integ-runner --directory test/integ --update-on-failed

Verifying integration test snapshots...

  NEW        integ.main 1.754s

Snapshot Results: 

Tests:    1 failed, 1 total

Running integration tests for failed tests...

Running in parallel across regions: us-east-1, us-east-2, us-west-2
Running test /Users/xxx/repos/repro5/test/integ/integ.main.ts in us-east-1
  ERROR      /Users/xxx/repos/repro5/test/integ/integ.main.ts (undefined/us-east-1) 1.645s
      Error during integration test: Error: integ.main is a new test. Please use the IntegTest construct to configure the test
https://github.com/aws/aws-cdk/tree/main/packages/%40aws-cdk/integ-tests

Test Results: 

Tests:    1 failed, 1 total
Error: Some integration tests failed!
    at main (/Users/xxx/repos/repro5/node_modules/@aws-cdk/integ-runner/lib/index.js:10367:15)

[moltar]

This is a very confusing error. I have upgraded from 2.100 to 2.122 and got completely destroyed with the integ-runner.

I get the alpha status and all, but it'd be great to get some more context on these errors. These errors are not readable and do not point to any docs.

Error 1:

  If your app has multiple stacks, specify which stack to select by adding this to your test source:

      /// !cdk-integ STACK ...

That's just super weird and feels like a hack.

Error 2:

  ERROR      .../foo.itest.ts (undefined/us-east-1) 0.411s
      Error during integration test: Error: foo.itest is a new test. Please use the IntegTest construct to configure the test
https://github.com/aws/aws-cdk/tree/main/packages/%40aws-cdk/integ-tests-alpha

u wot m8?

[hoegertn]

After a long search I found it:

You need to upgrade the "@aws-cdk/integ-runner" to a newer version. By default it picks something around 2.72 which does not work...

[pahud]

@hoegertn Thank you for your time. I will validate that again.