Skip to content

Roadmap

Jump to bottom
emeli-dral edited this page Apr 20, 2022 · 7 revisions

Summer 2022

April

  • Updated & Freeze Grafana Example
  • Stattes library for Num and Cat features
  • TestSuites Architecture Drafted
  • TestSuites Draft with: JSON and HTML formats, default configuration (run test suit with no specification), alerts
  • Implemented TestSuite for DataDrift
  • Designed API of the library (including potential plans for refactoring of Analyzers, Dashboards, Profile, Options and creation of new entities like TestSuited, Metric or Report)

May

  • TestSuites Alfa with manual configuration: users can specify alert conditions manually or read them from json config)
  • TestSuites customisation: users can create custom suites from evidently or custom metrics
  • Fixed API
  • Dashboards with alerts like DataDriftReport(alerts=True)
  • Profiles with updated structure
  • Examples: profiles log in DB; profiles concatenation: Evidently Dashboard with trend analysis; profiles concatenation: External (Tableau-like) Dashboard with trend analysis 20% resources – tool quality

June

  • TestSuites Beta with Reference-based configuration:users can specify alert conditions based on reference data
  • New API implemented (this basically means refactoring) evidently 0.2
  • Technical Documentation
  • Updated Gitbook Documentation
  • Example: Airflow integration with Evidently TestSuite
  • Example: Grafana integration with Evidently TestSuite evidently 0.2 public release

July

  • Test API, collect feedback

Checkpoint for decision making:

  • TestSuites V2 (redesign)
  • Content for TestSuites
  • Suites Gallery
  • Evidently interface for multiple models (like $ run evidently)
  • Dashboards update: auto sampling and stuff etc
Winter 2021

Reports

  • Dataset Profiling
  • [backlog] Historical Drift
  • advanced configurations

Algorithms & Stats

  • Anderson-Darling test for Data and Target Drift
  • Barnard's exact test for Data and Target Drift
  • Target Drift Analysis: add Kendall and Spearman correlation
  • [backlog] Regression Performance: segments of underperformance
  • [backlog] "Small Dataset" option for reports
  • [backlog] "Big Dataset" option for reports

Examples & Tutorials

  • Update all the examples & tutorials with readme
  • Update the structure of examples folder

Monitoring

  • metrics calculation logic for Production Data Stream
  • Grafana: Data Drift Dashboard
  • Grafana: Numerical Target Drift Dashboard
  • Grafana: Regression Performance Dashboard
  • Grafana: Categorical Target Drift Dashboard
  • Grafana: Classification Performance Dashboard
  • Grafana: Probabilistic Classification Performance Dashboard

Evidently Community

  • evidently_community repo for custom reports
  • evidently_community contributor Guide
  • tests for evidently_community project

Environments

  • interactive reports in pylab
  • interactive reports in google colaboratory notebooks
  • interactive reports in kaggle notebooks

User Interface

  • reduce HTML reports size (at least twice, better more)
  • [backlog] UI configuration (color-schema change, etc)
  • [backlog] color blind mode for reports
  • fix UI building issues

[backlog] Run App

  • [backlog] run evidently dashboards from terminal in a "service" mode
  • [backlog] host it somewhere

Testing, CI/CD

  • python unit tests for analyzers
  • python unit tests for widgets
  • python unit tests for dashboard
  • automation of running selected examples as a test
  • package deployment

Documentation & Repo

  • Developer's guide
  • Contributor's guide
  • Release notes
  • Good first issues, issue templates
Clone this wiki locally