Replacing tornado with asyncio

[sphuber]

We have just merged PR #4317 which replaces the tornado library with asyncio. These libraries are used extensively by the engine to make it possible to handle multiple processes asynchronously and so make it responsive and scale. The reason for removing is tornado is that we were stuck to an older version since newer versions introduced a lot of backwards incompatible changes. However, limiting the version of tornado caused a conflict with other tools that required a more recent version, such as jupyter. We could have upgraded the code to be compatible with more recent versions of tornado but instead decided to replace it entirely by asyncio, which provides the same functionality, but is a Python built-in module.

The work was performed in collaboration with @unkcpz during the Google Summer of Code of 2020. He did not only update our own lower lying libraries plumpy and kiwipy to use asyncio, but also contributed to the external library circus (which is used by our daemon) to make it compatible with recent tornado versions. Finally, he migrated the aiida-core code to use asyncio and this work is now almost ready to be merged and will be released with v1.6 towards the end of this year. I want to take this opportunity to thank Jason once more for a great job done!

The reason that I am writing this post is that despite the extensive testing that we have done, the changes are quite extensive and touch almost every part of the engine, which is itself a critical component of aiida-core. We therefore want to inform users, that may actually run aiida-core off of the develop branch, that soon there will be a significant change. We actually encourage you to try out the develop branch against your own workflows, but maybe not in a production database. It would be great to collect any information on suspected unintended changes in behavior or bugs in here. We can then split real bugs off into issues so we can fix them.

[chrisjsewell]

Thanks @sphuber, I like this use case for discussions 👍

[ltalirz]

Just mentioning this to @aiidateam/plugin-developers

[ramirezfranciscof]

I don't wan't to derail this thread too much, but just felt the need to congratulate both @sphuber and @unkcpz for finally merging this PR. I know it was a collosal task which, as @sphuber mentioned, involved a lot of work not only in this package but also in some of our dependencies as well. Moreover the change is of great importance because indeed our dependency on tornado was starting to bring some important problems and limitations, thus becoming sort of "shackles" impeeding the engine.

Kudos to both!

[zhubonan]

Finally v6 jupyter notebooks 😀

I have been using newer notebooks by putting the server into a different conda environment that just have juptyer installed.
May be the official documentation should mention this approach? @ltalirz
Juypter itself has many dependencies so it is better to keep it separate from the others for the future.

[chrisjsewell]

It’s funny that you should say that because, that’s exactly what I have been trying to set up with aiidalab in quantum mobile. I was having some issues with getting the terminal to also point towards the kernel (I.e the aiida environment)?
It is an option, but it is still a pain in the backside.

[zhubonan]

It’s funny that you should say that because, that’s exactly what I have been trying to set up with aiidalab in quantum mobile. I was having some issues with getting the terminal to also point towards the kernel (I.e the aiida environment)?
It is an option, but it is still a pain in the backside.

Hmm I see. I use conda environments and !verdi status works fine in the notebook (it shows the info in the target environment with aiida installed). Maybe it works differently for virtualenv.

[greschd]

Another datapoint: I use a conda env for jupyter but virtualenv for AiiDA - the !verdi always works AFAICT, but opening a Terminal in Jupyter uses the "right" virtualenv only because it's activated in my .bashrc.
I also have a symlink at ~/.local/bin/verdi to <virtualenv>/bin/verdi, but don't remember why; in any case, that works without activating the virtualenv.

[chrisjsewell]

Thanks for the feedback guys. I’m going to take another stab at this soonish and will let you know how it goes

[espenfl]

@sphuber No complaints from tests or regular runs with aiida-vasp on recent develop. All updates now merged into develop?

[sphuber]

Yes, develop contains the entire migration. There are just one or two open issues that have been reported that need to be looked into before we can release it as v1.6