Future: Configuration Simulation & Testing

[lcjohnso]

Problem: Users configure combinations of extractors, reducers, and rules+effects, but testing a completed config is cumbersome. Testing data sets may be linked to workflow and results viewed through hidden workflow/subjects/<ID> route, but problems include single user filtering (reductions often set to discard multiple classifications, but for single test user edit is necessary) and subject selection randomization & queue size (requires careful data input to mock various test conditions). Furthermore, testing on live in-production workflows (when configuration performed after launch) is fraught, but duplication of configuration to a separate test workflow is labor intensive.

Solution: A testing / simulation tool to aid in extractor / reducer / rule setup that shows how classifications are processed at each step -- from annotations to extractions to reductions.
Helpful for extractor/reducer selection and debugging. Allow teams to test and correct configuration bugs early.

More broadly: exposing classification/annotation payload in project builder may also pay dividends in terms of exposing project teams to data format and content early on, during workflow construction.