Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert the test suite to 100% pytest-based #641

Closed
2 tasks done
jaimergp opened this issue Jan 31, 2023 · 1 comment · Fixed by #675
Closed
2 tasks done

Convert the test suite to 100% pytest-based #641

jaimergp opened this issue Jan 31, 2023 · 1 comment · Fixed by #675
Labels
locked [bot] locked due to inactivity type::feature request for a new feature or capability type::tech-debt identifies or resolves some technical debt type::testing issues about tests or the test infrastructure

Comments

@jaimergp
Copy link
Contributor

Checklist

  • I added a descriptive title
  • I searched open requests and couldn't find a duplicate

What is the idea?

Our current test infrastructure uses some pytest logic, but the main logic happens in the run_examples.py script, which depends on the content of examples/. This is a homemade, ad-hoc script that has grown a bit too much, with many features and cases. Every time we add a new variable to check it's shoehorned in and it is not getting any better.

I suggest we use a pytest-only test infra, powered by fixtures and other pytest goodies. The good folks at MNE (@larsoner more specifically) did suggest something along these lines a while ago (see comment here for more details and ideas).

This would allow us to test better, clearer and more efficiently. Debugging should also get easier thanks to the -k flag.

Why is this needed?

  • Better development experience
  • Long-term maintainability
  • Readability of the test infra

What should happen?

  • All the examples should be covered by a pytest case, with a docstring explaining what we are aiming for in that test.
  • Post-installation checks wouldn't need the post_install.sh hacks, and could be done with pytest idioms. Testing with Bash makes me feel bad :D

Additional Context

No response

@jaimergp jaimergp added type::feature request for a new feature or capability type::tech-debt identifies or resolves some technical debt type::testing issues about tests or the test infrastructure labels Jan 31, 2023
@goanpeca
Copy link
Contributor

I can also help with this! :)

@github-actions github-actions bot added the locked [bot] locked due to inactivity label Jun 19, 2024
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jun 19, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
locked [bot] locked due to inactivity type::feature request for a new feature or capability type::tech-debt identifies or resolves some technical debt type::testing issues about tests or the test infrastructure
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

2 participants