update gherkin etc, validate against cck

[davidjgoss]

More or less copied the approach from cucumber-jvm.

To run:

$ yarn
$ yarn cck-test

(Or, run compatibility/cck_spec.ts from your IDE.)

Done:

• Expose World and IWorldOptions in public API
• Add a log function to the World which attaches text with the cucumber log media type
• Emit attachments in new structure
• Add some methods to DataTable
• Emit Meta message at start of run - fixes Emit a Meta message at the beginning of the run. #1310
• Keep a stopwatch of the test run, use it for a timestamp on various messages
• Emit messages for hooks and step definitions
• Emit messages for parameter type definitions
• stepMatchArgumentsLists are now being emitted within testCase.testSteps
• pickleAccepted and pickleRejected messages are now not emitted
• testCaseFinished.testResult is now not emitted (see Move or remove will_be_retried from TestStepResult common#902)
• testRunFinished.success is now not emitted

Questions:

• testCase messages per CCK fixtures should be emitted together at start of run, rather than before each test case is started (non-trivial refactor - do we need this right away?)

[aslakhellesoy]

That all sounds good to me David.

[davidjgoss]

And we are green.

[davidjgoss]

@charlierudolph good to merge?

[charlierudolph]

Awesome having cck and formatter tests passing! Let me take a look.

[charlierudolph]

Are we expecting all of these to be excluded long term? I would hope with predictable ids that big block could be fixed. Do you think its worth having an issue to try to drive this list down.

[davidjgoss]

I agree - when I was working against a slightly earlier CDK version I think it was only the testCase ordering that made the ids not line up.

But cck has changed from using predictable ids to uuids so not sure what the intent is there or where that leaves us.

[aslakhellesoy]

The predictable ids and timestamps are a dead end. The cck no longer uses them.

It's a dead end because various Cucumber implementations emit messages in slightly different order than what fake-cucumber (the engine behind the cck) emits. It would be too much work to make them all emit in the same order.

So what we opted for instead is a validation mechanism where certain values (ids, timestamps) are not compared. We pair this with smoke tests that send the generated messages through 2 of the most advanced message consumers we have - cucumber-html and json-formatter. If these don't crash and a cursory eye-balling looks OK, we're confident enough that the emitted messages are ok.

[charlierudolph]

Time reporting looks a bit off:

⋊> ~/p/cucumber-js on cck ◦ time node ./bin/cucumber-js
...
239 scenarios (239 passed)
1201 steps (1201 passed)
0m00.449s
       38.04 real        23.48 user         3.40 sys

Update: think this is due to the result of a testCase being the worst case step result and using that duration instead of the sum of the steps. I'll work on a fix for this.

Update 2 pushed fix:

⋊> ~/p/cucumber-js on cck ◦ time node ./bin/cucumber-js
...
239 scenarios (239 passed)
1201 steps (1201 passed)
0m25.237s
       38.10 real        23.52 user         3.50 sys

Timing still seems a bit off though as the timestamps in messages goes up to 34 (I'm guessing difference between pure user code and some cucumber overhead but not sure)

[davidjgoss]

Will hit merge on this in the morning if no other issues

[charlierudolph]

Woohoo! Thanks for all your work on this @davidjgoss