cli_test_dir and assert_cli collaboration

[epage]

These two libs serve similar purposes but take different approaches

• cli_test_dir
  • Relies on adding to Command
  • Integrates file system checks
  • Requires you run the program in a certain directory
• https://github.com/killercup/assert_cli
  • Doesn't support file system stuff

In addition, @killercup has been exploring additional ways of testing programs:

My general goal is to make a library that makes writing tests like these easy and fun. It'll probably involve generating some test boilerplate code, and I'm not entirely sure what I want that API to look like. I'm not even sure if that's in scope for assert_cli. What do you think?

source

Our thought has been to compose smaller tools with limited policy into a fancier more BDD style tool that is a bit more opinionated.

I know from cli_test_dir, I'd like some of the filesystem stuff (and contribute my own) in lower level integration tests

[emk]

Hello!

cli_test_dir has been around for a while. It's based on the test patterns in @BurntSushi's xsv and rg projects, which really impressed me.

Current users include:

• My https://github.com/emk/substudy and https://github.com/emk/subtitles-rs projects.
• Many of Faraday's open source Rust projects, like https://github.com/faradayio/scrubcsv and https://github.com/faradayio/geochunk. One big user is http://cage.faraday.io/.
• Some pretty complex in-house CLI tools at Faraday, including daemon processes that run independently while I test them over HTTP, for example.

In practice, the Command extensions make it easy to assemble complex commands and assert that they work (or fail). Features include:

• Accessing child process objects when necessary, so that I can test servers.
• Manipulating the environment.
• Building complex argument lists using multiple invocations of arg and args.
• Reading a command's output into a string and parsing it (often using serde_json), then using that output to verify that (say) the listed database record IDs correspond to newly created records.
• Preparing input and output files in the test directory, so that I can test commands that read and write files.

A big goal here is maintaining a significant amount of testing flexibility (because I test a wide variety of tools), and to be familiar to anybody who uses Command.

I've looked at assert_cli's documentation, and I don't see how to do many of the tasks that I mention above. I really like the assert_cli API, but it looks like the underlying Command objects are well-hidden. If true, this would make it necessary for me to eventually clone most of Command's API to do the kind of things that I mention above.

At this point, the API of cli_test_dir is unlikely to break backwards compatibility in any major ways. I'm happy to add new features to it that people need. And if anybody would like to re-export parts of it, I'm happy to help with that, too.

[epage]

Thanks for the feedback.

So it seems that the target for each is

• assert_cli: fluent testing of one-shot programs that relies on other creates for additional features (tempdir, dir comparison, file system asserts)
• cli_test_dir: Complex Command interactions and validation, including file system assertions.
  • It is nice having all that test harness ready to go.
  • It is nice that everything is exposed to the user so they can test things how they want.
  • Personally, I'm not as much of a fan of it tying everything together, making it harder to compose or handle specific types of needs

Compatibility does put a damper on finding ways to merge paths.

My thoughts moving forward

• Create some standalone file system assertions like cli_test_dir to be able to use with assert_cli or other places.