TODO.md

To do

Here's what I'm currently working on:

First release

• write tutorial

• write motivation

• move todo and roadmap to proper tickets

• website on mobile?

Second release

• add auth test
  • --gen-env UserName=... --gen-env Password=...?

Documentation

Website

• language reference?
• faq
  • different from prior work
    • https://github.com/microsoft/restler-fuzzer https://www.code-intelligence.com/blog/stateful-rest-api-fuzzing
    • https://github.com/Fuzzapi/API-fuzzer
    • https://github.com/KissPeter/APIFuzzer
    • https://github.com/TNO-S3/WuppieFuzz
    • https://endava.github.io/cats/docs/intro
    • https://smithy.io
    • TypeSpec
    • https://www.oxlip-lang.org/doc/overview.html
    • Testing RESTful APIs: A Survey
    • A Survey of Protocol Fuzzing
    • https://github.com/schemathesis/schemathesis
    • https://github.com/mockoon/mockoon
    • https://github.com/usebruno/bruno
    • https://fizzbee.io/
• news/blog

Other

• ARCHITECTURE.md?
• contributing.md
  • https://x.com/mitchellh/status/1852039089547309552
• A systematic approach to technical documentation authoring

Features

Spex

• literate spex

  • literate markdown -> markdown with code blocks annotated with outputs e.g. example.spex.md:

    foo : GET /bar -> String
    

    Should translate into:

    paths:
      bar:
        method: GET
        op: foo

    But one should also be able to write something like:

    foo : GET /bar -> String
    >>> ```yaml
    paths:
      bar:
        method: GET
        op: foo
    

    where >>> lets us include the output in the source, in that case spex should ensure that the output is correct.

  • hide (setup) code from the output

  • comments in the source code that doesn't show up in the output

• more types

  • path: string, int, float, path (string + /), uuid

• allow for control of # of test cases vs test case length

• change size over time

  • verify
  • mock

• add max size?

• use duration rather than numTests?

• print progress while testing

• test database with migration when generators change

• coverage-driven generation

• Nice errors

  • Introduce FancyError datatype and display it in LibMain?
  • https://elm-lang.org/news/compiler-errors-for-humans
  • https://hackage.haskell.org/package/diagnose-2.5.1/docs/Error-Diagnose.html
  • https://medium.com/designing-atlassian/10-design-principles-for-delightful-clis-522f363bac87
  • https://github.com/charmbracelet/bubbletea
  • https://gleam.run/

• ensure shrinking shrinks to the same status code, i.e. doesn't uncover another error?

• check for and forbid overlapping ops (same method and path)

• diff two specs

• Linking Unit Tests and Properties

Spexup

• cache releases.json?

• check what versions are locally installed already

• basic ui:

  • spexup [update] [spexup|spex] -- install latest version
  • spexup list -- lists available releases
  • spexup install (spexup|spex)

Bugs

• abstract types don't need to pass scope checking, but they fail during generation:

  component PetStore where
  
  addPet : POST /pet @Pet
  getPet : GET /pet/{petId : Int} -> Pet
  

  spex: genUserDefined: impossible, scope checker CallStack (from HasCallStack): error, called at src/Spex/Verifier/Generator.hs:198:20 in spex-0.0.0-inplace:Spex.Verifier.Generator

  They should be generated by reuse only...

• abstract types get generated, if there are not previous values to draw from. Fix by allowing generation to throw?

• unique types don't get checked

• normal types should sometimes generate/sometimes reuse

Refactor

CI

• cache packages and remove second cabal update? use >> GITHUB_ENV if needed
  • caching should affect the need for the second update, are we mounting the right thing?

Build system

• fix broken app image and ci smoke test

• make linux work without docker locally?

  • cache files between docker and not should probably not be shared?

• -O2 release builds?

• enable concurrent RTS on executables?

• changelog generator (semantic commit messages)

• commit hook for conventional commits?

• move petstore to separate repo?

Tooling

• https://pandoc.org/chunkedhtml-demo/13-syntax-highlighting.html
• Highlighting: https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks#syntax-highlighting