A pytest
plugin to help run only a sample subset of all tests in each
session.
Sometimes, it may not be feasible to run all tests of a project in a single test session if the tests are time-intensive. Nevertheless, as long as it is possible to run a few tests with each test session while ensuring that the different sessions execute different tests, it should still be possible to get good test coverage.
This package is a pytest
plugin and as the name suggests, it enables
the execution of samples of the entire set of tests that pytest
may
collect. Furthermore, the runtime of the entire test session will be
limited by some timeout duration. Once it expires, all remaining tests
will be marked as skipped, so that only a certain set of tests will be
run.
See also this Stack Overflow question.
Currently, there are two modes of operation implemented. In either mode,
the command line flag --samples-soft-timeout
can be used to set the
timeout after which no further test should be run. The default value for
the timeout can be read from pytest -h
.
Note
Running tests will not be terminated immediately if the timeout expires during their execution. Instead, whether the timeout has occurred is only checked after each test. This also means that at least one test item will always run before the timeout expires.
Tip
To parse the provided string into a timedelta
, the
pytimeparse
package is used. See its documentation for possible formats.
As the name suggests, this mode does not record any information
regarding which test have passed, failed etc. In order to nevertheless
allow a different set of tests to be executed with each session, the
list of all tests will be randomly shuffled. This way, the first few
tests run before the timeout expires will most likely be different with
each session. The mode is activated by enabling the plugin with
--samples=nostate
passed to the pytest
command line.
Caution
The command line flag --samples-seed
can be used to set the seed of
the internal random number generator used for shuffling the test items.
However, this is problematic if the seed is hard-coded since the order
of tests can then be identical in each session leading to strongly
reduced coverage. To prevent this from happening, a warning is
emitted if this flag is provided. This can be deactivated with the
--samples-nostate-seeded
flag.
Other flags, such as the flags used for the stateful mode described
below, will be ignored by this mode, except for the --samples-db-path
path flag. If this flag is provided in stateful mode, the program
terminates with an error.
Tip
The pytest
plugins
pytest-randomly
and
pytest-random-order
also implement random reordering of tests with more control over how
the tests are actually reordered. If you are only trying to eliminate
effects introduced by the ordering of the tests, these plugins might
be better suited. The shuffling implemented here does not make
considerations regarding the hierarchy of test modules, test
classes and test functions and shuffles all tests at function
level.
In stateful mode, activated via --samples=stateful
, information
regarding passed, xfailed and xpassed tests is written into
an SQLite database file. This information is used to provide a more
optimized ordering of the tests for each test run. Failed test will not
be recorded since it is assumed that they will be fixed and therefore
changed as soon as possible. The written information is used to
re-schedule tests that have had successful runs after all tests that
were not yet run or have failed. Furthermore, the tests with successful
run will be re-scheduled in order from last to most recent successful
run.
Note
The database will store the paths of the files containing the tests
relative to the pytest
root directory. Changing this directory
between sessions may therefore invalidate all test items.
Note
If a test passes, but a fixture it has requested causes an error on teardown, that test will not be written to or will be removed from the database since it must be assumed that something else might have gone wrong.
Note
The database stores all time information with respect to the UTC time zone.
The path to which the database is written must be provided with the
--samples-db-path
flag. If it is not provided, the program terminates
with an error.
Caution
Currently, there is no validation of the actual schema of the database in the provided file against the expected schema implemented. If the provided path points to a valid SQLite database, additional tables may be created or existing ones may get corrupted.
Caution
To overwrite the file if it cannot be opened, the
--samples-overwrite-broken-db
can be provided. However, this may
overwrite the provided file path and can lead to data loss.
By default, the results of the session are bulk-written after all tests
have finished or the timeout expires. The recorded "last run" time will
then approximately correspond to the time when the session ended.
Alternatively, the --samples-write-immediately
flag can be set. Then,
the database will be accessed immediately after the execution of the
individual test item succeeds and the "last run" time will be much
closer to the actual time the test has finished.
There are several flags to modify the ordering of the tests:
--samples-randomize
: If this flag is provided, the order of tests that are not found in the database and therefore did not have a successful execution before will be randomized, similar to what happens in nostate mode. However, the tests with a successful run will still be executed after all tests without a successful run in the order from last to most recent time of their last successful execution.--samples-seed
can be used to set the seed of the internal random number generator used for shuffling the test items, as is the case in nostate mode. The--samples-nostate-seeded
is ignored here since setting the seed does not necessarily mean that the tests executed in each session will be identical due to the information store in the database.- The
--samples-hash-testfiles
flag can be used to enable storing of MD5 hashes of the test files in the database alongside. This way, changes of the test files can be inferred. If such a change is detected, all tests in these files will be dropped from the database at the start of the test session since it must be assumed that at least one of them has changed. If there was no hash information stored in the database from a previous session, but is requested in a later session, this will be interpreted as the file having changed, too. - Once the database has stored all test items, only their last run time
will be used as a means of determining the test order. To avoid this,
--samples-rest-on-saturation
can be set to drop all entries from the database once all tests are found in it. Whether all tests have had a successful execution is checked both before and after the test run, meaning that the database might end up empty after the session.
Caution
The algorithm used for hashing the files when
--samples-hash-testfiles
is provided is not safe for cryptographical
application
Important
The hashing mechanism provided by --samples-hash-testfiles
is not
sufficient to detect all changes that may occur to a test function.
For example, if a test requests a fixture defined in a file that is
not otherwise tracked, changes to this fixture function will not cause
all tests depending on it to be dropped from the database.
Furthermore, although unlikely, hash collisions may obfuscate changes
made to a test file.
Note
The hash produced for the file may depend on the line endings (CRLF, LF) used. This must be taken into account when switching between systems with differing line endings.
Regardless of whether the --samples-hash-testfiles
flag is set, all
tests and all files will be removed from the database if they are no
longer collected by pytest
. This prevents the database from being
cluttered with out-of-date entries. Test entries are removed before each
test run, while files are removed after all test items were executed or
the time has run out. To prevent this cleanup from happening, the flag
--samples-no-pruning
can be set which will preserve stale entries that
no longer exist. Note that if --samples-hash-testfiles
is set and a
file changes, the associated test item entries will still be removed.
If you need to debug the plugin, you can set the flag
--samples-enable-db-logging
to enable sqlalchemy
's logging.
You can install the latest version of this package directly from GitHub:
pip install git+https://github.com/devds96/pytest-samples
Alternatively, you can clone the git repo and run in its root directory:
pip install .
Some further ideas that may be implemented in the future:
- Instead of computing the MD5 hash of test files, it may be faster to
use hash functions provided by packages such as
metrohash
. - Instead of using a soft timeout method, where whether it has expired
is only checked after each test, it is possible to use a hard
timeout which immediately (with some limitations) terminates the
tests using the mechanisms implemented in
pytest-timeout
. - Currently, it seems to be relatively difficult to perform validations
of the database schema using
sqlalchemy
. However, this validation would be a necessary step to protect erroneously passed database files which are SQLite databases but are not the intended target. As mentioned above, the default behavior ofsqlalchemy
is to simply create the required tables if they are not present.