Remove Behave and give up on BDD

[anttivs]

The BDD stuff is really mostly useless in the context of Mittn. For example, the use of Behave and Gherkin in the Burp scanning case is just syntactic sugar, which just makes you fat. BDD-Security (http://www.continuumsecurity.net/bdd-intro.html) is a more ideologically pure BDD-driven testing solution.

Many testers would just want to use whichever test runner they use now (e.g., nose) and see Mittn as a wrapper for test tools, with an interface that looks like the current Behave step library. Mittn should then be packaged to be installable with pip.

I see this is a fairly major pivot opportunity but it will also decrease dependencies and improve testability of the tool itself.

If anyone has opinions on how the tool should be:

• packaged (remember there are non-free binary dependencies)
• configurable
• callable from unit/system tests,

please comment below.

[iriusrisk]

Would it be possible to support both BDD and other testing libraries? E.g. with the existing Behave step library you could reference those steps from a different testing framework and then effectively support that framework too.

That's the approach I took for BDD-Security, e.g. the auth test written for TestNG instead of JBehave: https://github.com/continuumsecurity/bdd-security/blob/master/src/test/java/net/continuumsecurity/testng/web/AuthenticationTest.java

(although I haven't been updating the testNG tests as regularly as the BDD versions)

A neater solution might to write an intermediate class that sits above the Behave steps and hides any nastiness that using a pure Behave step class would expose.

[anttivs]

Actually I just today thought about this and probably I'm going to create some sort of class that wraps the tools and their configuration, and modify the currently existing Behave steps to use those classes instead. So the existing Behave steps would continue to work. I probably do it this way around because the current Behave steps just assert False on failures, and I'd like to emit fancier exceptions for other test frameworks.

The thing I am not sure about how this plays along with making the whole thing pip-installable.

[anttivs]

Here's a bit more background for getting rid of Behave. The tool is supposed to be used in a system where, when new APIs are added, injection and fuzz tests are just automagically extended over those APIs. Now, one needs to give valid case examples of API calls in a Gherkin file, but I want to have an object that you can just programmatically feed valid case examples from somewhere (e.g., the same place where automatically created API docs are being created).

[iriusrisk]

Don't know anything about pip, but if it's like Jars you could create a number of different dependent packages, so one "core" distribution which doesn't include any testing libraries or frameworks, and then separate pip's for every supported testing framework?

[iriusrisk]

RE: programmatically feeding API examples, could this be a separate utility script that converts the API into a Gherkin Data Table file, to be used in an existing story. So you keep the story the same, all that changes is the Data Table.

[anttivs]

Re packaging, I am mainly just worried about maintaining the directory hierarchy because I know some projects clone the repo and assume stuff is where it is now; and on the other hand, new users who would just install the package, whether they get the behave stuff at all or not.

The utility script to create Gherkin is how I thought it would be done, but this just really adds a layer that is not needed. I was told in no uncertain terms by QA automation folks they don't need extra layers like that.

[mathias-nyman]

This is finally happening! As this a complete backwards-compatibility break, it will be worked on on the develop branch.

[juhakivekas]

We're continuing this work on a new branch of develop in another repository and breaking backwards-compatibility even more.

We're starting with the httpfuzzer, and the new interface is going to be something simple like this:

import mittn
m = mittn('mittn.conf')
m.addtarget('http://target.io/api', '{"valid":"json"}')
m.fuzz()

Most of the configuration done in the feature files will be done in a "defaults are fine" way to lessen the effort that the user needs to put into setting up the tool.