Selecting Features/Scenarios at run time based on external data.

[svarunbe]

Hi @ptrthomas,

Karate is really a robust framework for web service testing. We are currently using this framework at our organisation for the feasibility study. However, I am facing few challenges when utilizing this framework effectively. Please help if you can solve these challanges.

We have basically multi-country multi-service structure and we have written each feature file for each service. After done writting, we want to execute few services in a particular country. We cannot take help from tags since the country-service mapping is dynamic and we have an excel matrix for this use case. More over, we cannot edit the source code or feature file once it is pushed to server.

My requirement is that either I should pick up the all the feature files based on the excel matrix and run them or Once a feature is picked up, I should check whether it exist in excel matrix and if not, I should be able to skip the feature. (The feature should not be marked as failed in the report as I did not run it)

The same requirement is present for Scenario level. I should be able to control from excel data ( simple yes or no flag).

On exploring this option in google, ## I find that JUNIT AssumeTrue(false) will mark the scenario or feature as skipped in normal cucumber-jvm runner. But the same is not happening in Karate. It is marking the scenario or feature as failed. Hence I understood that this can be an issue or missing featur in karate.

Kindly look into this use case and advice if any solution is available.

Thanks,
Arun Kumar V

[ptrthomas]

I may not have understood your need correctly, in that case please follow these instructions: https://github.com/intuit/karate/wiki/How-to-Submit-an-Issue

here's what I suggest. don't use excel - read this answer on stack overflow: https://stackoverflow.com/questions/47954754/how-to-read-input-data-from-an-excel-spreadsheet-and-pass-it-json-payload-in-kar/47954946#47954946

all you have to do is dynamically create a JSON and then iterate over it and call features. so now you have control over the features you need to execute.

[svarunbe]

Hi @ptrthomas,

Thanks for the reply and sorry for the confusing requirements.

My need is simple.
I would like to skip my feature or a scenario based on a flag at run time.
In Junit, there is AssumeTrue(false) method. I expect this feature to be available in karate.

Kindly let me know if it is possible to have this feature.

Thanks,
Arun Kumar V

[ptrthomas]

my answer is simple: no but seriously, i suspect you have not understood karate enough. i have suggested a much more elegant solution above. honestly i have never heard of this assumeTrue you speak of. anyway, if you are insisting that we make some Java method available (not clear why) the answer is no. look for alternate options please.

…

On 12-Jun-2018, at 4:03 PM, svarunbe ***@***.***> wrote: Hi @ptrthomas, Thanks for the reply and sorry for the confusing requirements. My need is simple. I would like to skip my feature or a scenario based on a flag at run time. In Junit, there is AssumeTrue(false) method. I expect this feature to be available in karate. Kindly let me know if it is possible to have this feature. Thanks, Arun Kumar V — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[svarunbe]

Hi @ptrthomas,

I understood that your provided solution is for handling data inside a feature.

My concern is that I have 200 feature files and I want to run few of them at a given period. Could you please clarify how to do this other than hardcoding the list of feature files in the Runner.java.

Sorry to ask many times, I'm trying to learn karate better.

Thanks.

[ptrthomas]

i don’t think you still read the stack overflow link i posted. please. - karate can call other features from a feature (refer doc) - karate can loop over a JSON array and use each row to build a call any questions ?

…

On 12-Jun-2018, at 5:13 PM, svarunbe ***@***.***> wrote: Hi @ptrthomas, I understood that your provided solution is for handling data inside a feature. My concern is that I have 200 feature files and I want to run few of them at a given period. Could you please clarify how to do this other than hardcoding the list of feature files in the Runner.java. Sorry to ask many times, I'm trying to learn karate better. Thanks. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[svarunbe]

Hi @ptrthomas,

I have read the documentation and I am spending my last 15 days exploring Karate framework.

• I know that karate can call other features from a feature, but the in the reports, it will be displayed as Steps not Features( Am I correct?)
• I am not sure on the second point.

Sorry to insist my use case again. This could be common scenario in most of the organisation.

1. Select a set of features at run time to execute
2. Skip few scenarios within a single feature based on a data
3. Report should display the Pass/Fail status for the selected features, and all other features should be marked as skipped.

I beleive that this could be acheived if there is a provision to skip a feature or scenario at run time.

kindly have this checked.
https://stackoverflow.com/questions/16037999/is-it-possible-to-skip-a-scenario-with-cucumber-jvm-at-run-time/16176975

https://groups.google.com/forum/#!topic/cukes/sl_LVlFJ8EQ

Thanks.

[ptrthomas]

sorry both the links above don't work.

in the reports, it will be displayed as Steps not Features( Am I correct?)

yes. but you are achieving your goal of testing a bunch of features. right ?

anyway, I understand that you might be particular that you want a report broken down by different features. personally I think you are over-engineering your tests but I agree that this is a good feature to have so have opened #417

but I have one question for you:

Skip few scenarios within a single feature based on a data

Skipping features is fine, but how do you intend to skip a Scenario unless you use tags ? Sorry please don't say that it should be via JUnit. waiting for your reply.

[svarunbe]

Hi @ptrthomas,

Thank you for accepting this feature.

The link is working fine when copy and paste in browser tabs. I am not sure why it is not working when directly clicking at them.

Skip few scenarios within a single feature based on a data

The rationale behind this is that, I have 100 scenarios within a feature, and I don't want to run all of the 99 scenario and background steps if my first Scenario fails knowing that rest 99 scenarios will fail.

Because running a single features takes 30 - 40 minutes and we have 200 feature files. We dont want to wait that longer to know that the api endpoint url is wrong.

As we cannot change our tag values dynamically, there should be a mechanism to mark the scenario as skipped like "karate.skip" or like throwing a skip exception.

This way we can control our scenario execution even from outside of runner file.

Thanks,

[ptrthomas]

@svarunbe thanks for being clear :)

I think you have what you need - this is an undocumented new feature of Karate which I feel is too dangerous - but maybe you can prove that this is useful.

you can call * eval karate.abort() to skip a scenario at any time and it will not mark the test as failed. here is an example: https://github.com/intuit/karate/blob/develop/karate-netty/src/test/java/com/intuit/karate/netty/client.feature#L73

so in your case, if you have set a flag, you can do:

* def condition = true
* eval if (condition) karate.abort()

I'm going to release a 0.8.0.RC7 in a few minutes and would like you to test it. the API to choose features is ready

[svarunbe]

@ptrthomas

Great. Will check it up and update you.
May I know why it is considered dangerous?

Thanks

[ptrthomas]

@svarunbe here you go: https://martinfowler.com/articles/nonDeterminism.html

[svarunbe]

Hi @ptrthomas,

While using Karate.abort(), I have found out that the step result is marked as 'Undefined'.
Also noticed that, all the results for the steps below karate.abort() statement is marked as 'passed'.

Because of this, Cucumber-report considered the skipped scenario as 'Failed'.

Could you please change the step result for the karate.abort() as well as the steps below to it as 'skipped'?

Thanks,

[ptrthomas]

ok. raise a ticket.

[ptrthomas]

@svarunbe and can you please do some research if needed and confirm if "Skipped" will also cause the same problem. should we go with "Passed" ?

[svarunbe]

@ptrthomas
Have raised a ticket. #418

I have checked on few reports implementation that if all the steps in a scenario is marked as skipped, they will mark the scenario as 'Skipped'.
If any steps as 'Undefined' or 'Pending' then it will be shown as 'Failed'

I think we should not go with 'Passed' because it will show the wrong number of passed scenario count.

[ptrthomas]

@svarunbe the "Undefined" thing is fixed. can't commit on when we make the next release so you can try and build from source if needed. can't help observing that you also haven't had the courtesy to confirm if everything else is working as expected.

we'll wait in case you come up with some more demands.

[svarunbe]

@ptrthomas

Thanks for the fix.
We are collecting the information to verify the run time feature selection api. I will give update on that.

As for as feature request, I have few in mind.

1. Soft Assertion like TestNG
2. Steps starting with '*', should not be displayed in the report.

I will add if anything else needed.

Thanks,

[ptrthomas]

@svarunbe please take your time :)

soft assertions.

no we will not support this. refer cucumber/common#79

Steps starting with '*', should not be displayed in the report.

interesting suggestion. please refer this duplicate ticket and add your comments if necessary: #397

[svarunbe]

@ptrthomas

Sorry for taking this long time.
We have verified the run time feature selection api in many possible combination and it is working as expected. Now I can dynamically able to select the list of features at run time :)

Regarding the karate.abort()
Now the result for the particular step is marked as 'SKIPPED', but the results for the steps below it still shown as 'PASSED'. Because of this, the scenario is shown as 'Failed'

Is it possible to mark the rest of the step result as 'SKIPPED'?

Thanks,

[ptrthomas]

@svarunbe which version. if you have tried 0.8.0.RC8 and still have an issue I need to be sure I understand what you are saying and what the issue is. so if you want any changes now, you HAVE to follow this process: https://github.com/intuit/karate/wiki/How-to-Submit-an-Issue

[svarunbe]

Hi @ptrthomas,

Yes I have tried with 0.8.0.RC8 and the issue was present in it.
Now it has been fixed in 0.8.0.1. (by the ticket #464).

Thank you for the fix.