Run arroyo test suite in ducktape #5499
Conversation
VladLazar
force-pushed
the
arroyo-tests
branch
2 times, most recently
from
d43c53c
to
2c96216
Compare
| Run the arroyo test suite against a redpanda cluster in | ||
| a ducktape environment. | ||
|
|
||
| The test suite lives here under tests/ in https://github.com/getsentry/arroyo. |
There was a problem hiding this comment.
we should respond on twitter once this lands.
| @@ -163,6 +163,10 @@ RUN apt update && \ | |||
| file && \ | |||
| rm -rf /var/lib/apt/lists/* | |||
|
|
|||
| RUN mkdir /root/external_test_suites && \ | |||
| git -C /root/external_test_suites clone --depth=1 https://github.com/getsentry/arroyo.git && \ | |||
There was a problem hiding this comment.
based on the rest of the Dockerfile it looks like we try to pin everything to a specific sha1/version
There was a problem hiding this comment.
Yeah. It's definitely a trade-off. It's more stable if we pin, but then someone needs to remember to update it manually every now and then. That being said, that test suite doesn't seem to change much over time. I'll pin it to the current tip of the main branch.
There was a problem hiding this comment.
yeh, agree about updating. i think we need some out-of-band process by which we periodically upgrade the dependencies. at least this way we keep flakiness minimized
This commit extends the ducktape test container to clone and install arroyo (https://github.com/getsentry/arroyo) in order to run its test suite against a Redpanda cluster.
Pytest is required in order to run the arroyo tests.
This commit adds a ducktape test that runs the arroyo test suite against a redpanda cluster. All test output is logged and test failures are parsed from the pytest output. The test fails if there were pytest failures or if pytest returns an error code indicating that the tests couldn't run.
|
Failure found #342 |
Cover letter
This PR adds a ducktape test that runs part of the arroyo test suite (not all of the tests run against a live cluster and we're only interested in those).
In order to avoid forking the tests (which leads to license complications and having to maintain the fork) I've made it such that the environment clones the external repository and runs the tests from there.
Unfortunately, the arroyo tests use pytest, which makes it difficult to deal with logging and error reporting. The approach I've taken is to log all the output from the pytest invocation and to parse that for failures.
Perhaps these test should only run in the nightly. Any opinions on that?
Fixes #4575
Notes
force_push:
--depth 1when cloning the external repoforce_push: