Improving the test harness

[amotl]

Dear mqttwarn community,

this will be all about my proposal for finally giving mqttwarn a test harness in order to support the nice people caring about it, to make it more robust and to encourage future development, cleanups and refactoring of its internals.

Introduction

Along the lines of #127, @jpmens already asked for:

4. We (I) should probably also think about creating a simple test harness. Maybe just a short shell script which publishes topics/payloads that match a configuration created by make-config so that people can see how it all fits together.

He got full acknowledgement:

Absolutely agree with that. Let's have exactly this as a basic quick turnaround test harness for the upcoming 0.15.0 release and let's build a full-fledged and thorough test environment for the 1.0.0 release later this year.

As "later this year" might be just now and as i also was feeling your recent pain with #327 (thanks again, @robdejonge, @rgitzel and @jpmens), i again recognized the mental toll everyone pays each time when diving into the internals of mqttwarn in order to tweak something. Note to self: I can barely remember i came here for improving that - we finally should get the "develop" branch ready ;].

Tests FTW

So i just gave this a shot in the tests-ftw branch. It is the foundation for a full-fledged test harness based on the lovely and powerful pytest. The test harness is now able to boot mqttwarn in-process and run the message reception and transformation process through its core machinery - everything without any MQTT connection at all - making these kinds of tests real unit tests.

Test internals

You will find some basic unit tests in test_util.py and some more advanced ones in test_core.py. The latter ones work by simulating a message delivery to the test/log-1 topic registered by selftest.ini while capturing the log output of the whole process. After that, the outcome is validated by checking for appropriate content in the log output to proof everything worked fine. This is the unit test variant of full end-to-end testing.

Please note that mqttwarn is a multithreaded program, so the test suite has to account for asynchronous behavior of the "unit under test". Currently, it just issues a time.sleep(0.05) after the simulated message delivery in order to let the worker thread fulfill its job processing the message. It's a very low value which we might have to increase in the future depending on different systems, environments and quantities. By now, i chose this low value on purpose in order to make things snappy, which is very important to me (14 tests passed in 0.16 seconds).

Usage

Running the test suite should be as easy as issuing

$ make test

Outlook

In order to improve the end-to-end testing even further, we will add full integration tests based on the even more lovely lovely.testlayers along the lines. This will spin up Mosquitto and mqttwarn as daemons and run the message delivery through regular MQTT without any mocking.

Have fun!

Cheers,
Andreas.

-- https://hooktube.com/watch?v=0TYnoYl1JfY

[rgitzel]

@amotl This is a great start!

One thing though. I think it would be helpful to distinguish between different types of tests.

• "unit tests" don't require anything to be running, indeed all the test does is call one isolated function
• "Integration tests" will use several functions or classes together, but still do not require a fully running application (though they might interact with a database or such)
• "system tests" involve actually running the app, with or without associated infrastructure

All of these are valuable.

From a quick read, it seems the tests you've added are more of the "system" variety. I'll try in the next day or so to write some "unit" tests with pytest, to compare.

[jpmens]

Let's please not scare away new module contributers (and old ones like myself) by making the test framework too complicated or complex to add to.

One of our strengths in the past has been that we've had lots of service contributions, and I'd like that to remain thusly. Whatever test framework(s) we add should be as simple as possible to add to, please.

[amotl]

@jpmens: I hear you and second your wish not to overengineer the software tests in order not to lose anyone on that. It's quite the opposite i am aiming at: To get everyone started to embrace the new tooling the mqttwarn code base will offer.

My short-term goal is to gain a reasonable code coverage of core.py. By now, i won't go into the fuzz about testing the service plugins yet as we would have to go for greater lengths there and provide the test framework with different kinds of mocking facilities.

Whether we will make up a contribution policy like "code + docs + tests" or relaxed variants thereof: Time will tell, we should decide later. Personally, i don't have strong opinions in this area, service plugins probably will continue to stay completely untested for a while, so there will be no barrier enforced to them in any way.

The main intention is to test the core of mqttwarn thoroughly as safety tooling primarily used by us and all the core developers who dare to go into the details there in order to tame its intertwingulation. "How testing works" will be thoroughly documented as well, we can use the introduction above as a blueprint for that.

I would like to hear some comments on the tests implemented by now. Modulo inline documentation, which i will definitively add, i find them reasonably easy to follow:

• test_util.py
• test_core.py

What do you think?

[amotl]

The test harness just succeeded on CircleCI for the first time [1]. Based on pytest [2], tox [3] and docker-tox [4], the respective CircleCI configuration file [5] is really short and concise.

Thanks a bunch @hpk42, @nicoddemus, @RonnyPfannschmidt, @blueyed, @benjaminp, @asottile, @gaborbernat, @obestwalter, @themattrix and everyone who contributed.

[1] https://app.circleci.com/jobs/github/jpmens/mqttwarn/4
[2] https://pytest.org/
[3] https://tox.readthedocs.io/
[4] https://github.com/themattrix/docker-tox
[5] https://github.com/jpmens/mqttwarn/blob/master/.circleci/config.yml