Feature tests and generation of test code should use CLI

[kanigsson]

On #713, I am modifying the interface for the code generation slightly, as was discussed during review. The parser now also produces an Integration object that needs to be passed to the generator.

Instead of adapting the feature_test.py and generate_spark_test_code.py to the new interface, I think these scripts should run the rflx CI instead.

I would go even slightly further and transform the feature tests into tests that execute a script similar to this:

• code generation (also check if identical to baseline)
• compilation
• execution
• proof

The script would not proceed to the next step if one step fails. This would replace the current test_compilability, test_executability etc.

Arguments in favor:

• avoid code duplication
• transforms feature tests into end-to-end tests that also test the CLI

Arguments against:

• Need to check if pytest still computes coverage correctly, but it's supposed to work: https://pytest-cov.readthedocs.io/en/latest/subprocess-support.html
• anything I missed?

What's your opinion, @treiher?

[kanigsson]

By the way, I used the label "bug" for lack of a better label. Maybe we should add a label "QA", or "quality", or "testing"?

[treiher]

By the way, I used the label "bug" for lack of a better label. Maybe we should add a label "QA", or "quality", or "testing"?

Yes, I think that could be helpful, I will add one.

Instead of adapting the feature_test.py and generate_spark_test_code.py to the new interface, I think these scripts should run the rflx CI instead.

The input for generate_spark_test_code.py contains Python models, which cannot be used as input for rflx. We could consider converting the Python models to specifications. I'm not sure yet, if it would be possible to completely replace the models by specifications or if it would make sense to have both and check that they are in sync.

I would go even slightly further and transform the feature tests into tests that execute a script similar to this:

• code generation (also check if identical to baseline)
• compilation
• execution
• proof

The script would not proceed to the next step if one step fails. This would replace the current test_compilability, test_executability etc.

The reason for having separate test functions (like test_equality and test_executability) is the possibility to perform steps separately. I think that is a useful feature during development. For example, I'm not always interested in having "equality", when testing the effects of a changed code generation on the functional correctness or on the proof.

Arguments against:

• Need to check if pytest still computes coverage correctly, but it's supposed to work: https://pytest-cov.readthedocs.io/en/latest/subprocess-support.html

I also couldn't find a definitive answer. The linked documentation doesn't explicitly mention sub-processes started using the subprocess module.

Arguments in favor:

• avoid code duplication
• transforms feature tests into end-to-end tests that also test the CLI

That are worthwhile goals, but I have the feeling that the necessary work outweigh the benefits at the moment. It would be an optimization, but it doesn't solve a concrete problem. I suppose the changes needed for #713 should be quite simple, even if they have to be done at multiple places (correct me if I'm wrong). I think we should keep this issue open and reconsider its implementation at a later time.

[senier]

We don't currently see a benefit in implementing this ticket.