Roadmap

Summer 2022

April

• Updated & Freeze Grafana Example
• Stattest library for Num and Cat features
• TestSuites Architecture Drafted
• TestSuites Draft with: JSON and HTML formats, default configuration (run test suites with no specification), alerts
• Implemented TestSuite for DataDrift
• Designed API of the library (including potential plans for refactoring of Analyzers, Dashboards, Profile, Options and creation of new entities like TestSuited, Metric or Report)

May

• TestSuites Alfa with manual configuration: users can specify alert conditions manually or read them from json config
• TestSuites customisation: users can create custom suites from metrics implemented in evidently or custom implemented ones
• Fixed API
• "Alert On" mode for Dashboards. Dashboard(tabs=[DataDriftReport(alerts=True)])
• Profiles with updated structure (json output)
• Examples:
  • profiles log in DB;
  • profiles concatenation: Evidently Dashboard with trend analysis;
  • profiles concatenation: External (Tableau-like) Dashboard with trend analysis

note: 20% resources – tool quality

June

• TestSuites Beta with Reference-based configuration: users can specify alert conditions based on reference data
• New API implemented (this basically means refactoring) evidently 0.2
• Technical Documentation
• Updated Gitbook Documentation
• Example: Airflow integration with Evidently TestSuite
• Example: Grafana integration with Evidently TestSuite

evidently 0.2 public release

July

• Test API, collect feedback

Checkpoint for decision making

Possible Directions

• TestSuites redesign
• Suites Gallery
• Evidently service for multiple models and profiles visualisation in time (like $ run evidently)

---

Winter 2021

Reports

• Dataset Profiling
• [backlog] Historical Drift
• advanced configurations

Algorithms & Stats

• Anderson-Darling test for Data and Target Drift
• Barnard's exact test for Data and Target Drift
• Target Drift Analysis: add Kendall and Spearman correlation
• [backlog] Regression Performance: segments of underperformance
• [backlog] "Small Dataset" option for reports
• [backlog] "Big Dataset" option for reports

Examples & Tutorials

• Update all the examples & tutorials with readme
• Update the structure of examples folder

Monitoring

• metrics calculation logic for Production Data Stream
• Grafana: Data Drift Dashboard
• Grafana: Numerical Target Drift Dashboard
• Grafana: Regression Performance Dashboard
• Grafana: Categorical Target Drift Dashboard
• Grafana: Classification Performance Dashboard
• Grafana: Probabilistic Classification Performance Dashboard

Evidently Community

• evidently_community repo for custom reports
• evidently_community contributor Guide
• tests for evidently_community project

Environments

• interactive reports in pylab
• interactive reports in google colaboratory notebooks
• interactive reports in kaggle notebooks

User Interface

• reduce HTML reports size (at least twice, better more)
• [backlog] UI configuration (color-schema change, etc)
• [backlog] color blind mode for reports
• fix UI building issues

[backlog] Run App

• [backlog] run evidently dashboards from terminal in a "service" mode
• [backlog] host it somewhere

Testing, CI/CD

• python unit tests for analyzers
• python unit tests for widgets
• python unit tests for dashboard
• automation of running selected examples as a test
• package deployment

Documentation & Repo

• Developer's guide
• Contributor's guide
• Release notes
• Good first issues, issue templates