Best practice / example for integration tests with pytest

[hille721]

Till now my integration tests were only Ansible tasks and running directly with ansible-test integration.
With this setup they seems to be triggered via pytest. I see clearly the advantage (e.g. easily trigger them out of VSCode), but I don't know how to best migrate from "Ansible approach" to the pytest approach. I guess I need a pytest which will then trigger the Ansible tasks?
Is there any documentation or good examples of how to achieve that. Could there be something generic, which could be used in every collection?

[cidrblock]

I hacked something together for the ansible.scm collection, but I don't like it.

I keep wondering about how to do this better myself, in a repeatable fashion. localhost isn't really the issue, it's the ability to stand up the necessary infrastructure to run the tests, or selectively run individual tests leaving the infrastructure up if needed.

I'm thinking molecule might be a better collection test runner long term but I need to do thinking about how to glue it all together without just a single giant scenario, or one scenario for every role in the tests/integration directory.

My gut tells me we need to teach molecule to be collection integration test aware, I'm just not sure how today.

I'll probably be throwing some discussions/issues up and will let you know

[hille721]

I saw you implantation in ansible.scm collection but if you don't like it I won't use it as my best practice example.

I also see Molecule as the tool for integration testing, thus probably no need to have both, molecule and ansible-test integration.

Imo it would be just good if in the end tox-ansible somehow magically put such integration tests up, without that I have to carry about to write pytest Python code to execute my integration plabooks / molecule scenarios.

[greg-hellings]

Running Molecule tests was the original function of tox-ansible. If you need that functionality you'll need to use the tox 3.x series and tox-ansible 1.x series together. The 2.0 rewrite of tox-ansible has not yet added back in support for Molecule based environment and scenarios.

[hille721]

No, I need it for ansible-test, that's why I also not used it before.
My question was about the integration tests (ansible-test integration), so far I triggered them directly with that command, but here they are now triggered via pytest.
And I also generally see them redundant with molecule (which I not yet used but where I'm now starting with the great changes from v6).
Thus the question is also more open, what's the way forwared :)

[cidrblock]

I did some work here, and think I like where it is going........

So the integration support in tox is really doing nothing more than running pytest test/integration, based on the assumption pytest can run the integration tests.

The issue here is the integration tests are actually ansible roles.

Molecule is a good candidate here since it can stand up and tear down infrastructure, so we just need to teach molecule how to build scenarios, one for each integration test role and then paramertize them for pytest.

Although I think molecule should be able to do this by default, here's a PR with a default molecule scenario that will build molecule scenarios on the fly, one for each integration test, running molecule create

https://github.com/ansible-collections/ansible.scm/pull/227/files

and the bridge between pytest and molecule is here, a new fixture embedded in the pytest ansible plugin:

ansible/pytest-ansible#147

this will require the user add a small pytest test to the integration directory:

https://github.com/ansible-collections/ansible.scm/blob/733a67ac78ba32a87fe2c3ec0664578ccb16c145/tests/integration/test_integration.py

(it's all still WIP)

The reason I think I like it is because the roles and responsiblities of each tool are pretty sane, and are core functionality

tox: manages the matrix of python/ansible versions
pytest: runs tests, well integration w/ vscode as a UI
molecule: wraps ansible roles, builds and tears down infrastucture necessary for ansible testing
integration tests: ansible roles (although if I were starting from scratch with a collection, I'm not sure roles would be necessary, but they are a convenient way to box the tests

Right now, the tox-ansible plugin is "installing" the collection in the tox venv for both integration and unit, and molecule is doing the same into it's ephemeral directory structure......... If we assume molecule is in play here, it shouldn't need to do it... will put some thought to this.

(edit: I should note I like the idea of molecule here because it can be used to test not just the integration tests within a collection but user facing content within the collection too, e.g. roles and playbooks, couple with tox, molecule will now get run across multiple python intepreters and ansible versions but both integration and user facing content)

your thoughts appreciated here trying to get this right

[cidrblock]

The ansible.scm collection has been updated to use molecule via pytest via tox-ansible as the integration test runner.

https://github.com/ansible-collections/ansible.scm

Still a little more than the user should need to do but much better than what was there.

[hille721]

Just had time to look into this with more detail again. I really like the direction in which this is going as this streamlines Ansible collection testing and makes it also easy to provide further tests.

Although I think molecule should be able to do this by default, here's a PR with a default molecule scenario that will build molecule scenarios on the fly, one for each integration test, running molecule create

Yes, imo molecule should be able to detect integration targets in the same manner as ansible-test integration was doing it. This would be then really convenient. But the workaround you are sharing is also a good starting point already as new integration tests can be added and will be picked up automatically from molecule.
Only one thing I don't understand is, why you then committed the molecule scenarios which will be created on the fly: https://github.com/ansible-collections/ansible.scm/tree/main/extensions/molecule?

And one further question: Why have you chosen to store the molecule scenarios in extensions/molecule and not only molecule? Are there any technical reasons or just kind of best practice?
Could the molecule scenarios also added to tests/molecule? Then all testing part would be in the same root directory.

[cidrblock]

I commited them becasue I wasn't ready to modify the the tox-ansible plugin or GHA to run "molecule create" yet. If we do that, the scenarios for each target could be left out of the repo.

The reason I didn't want to count on default being the first scenario to run is that if we are using pytest-xdist for multiproc, there isn't a really a guarentee that default would run first. (I think it almost always would but I didn't want to count on it)

I've added a bunch of RFEs for molecule related to this, e.g. ansible/molecule#4003

Use of the extension directory is an "emerging" best practise. When event driven ansible content was added to the collections, it was put into the extensions directory. Although not really documented formally anywhere it's going to be the place moving forward. The molecule docs were updated for v6 to recommend it.

I don't think putting them in the "tests" directory would be an immediate issue, but I could see a time in the future where something like ansible lint or the vscode extension assume the extension/molecule directory as a default to treat that content differently.

Still looking into a couple other things to improve the whole process, will keep you posted :)