Incompatible cluster sshd configuration MaxSessions 1

[mp-cms]

I am unable to create a working AiiDA computer for our cluster. The following error message is shown

verdi computer test vsc4
Report: Testing computer<vsc4> for user<###@###.ac.at>...
* Opening connection... [OK]
* Checking for spurious output... 03/06/2023 08:08:18 AM <50038> paramiko.transport: [ERROR] Secsh channel 1 open FAILED: open failed: Connect failed
[Failed]: ChannelException: ChannelException(2, 'Connect failed')
  Use the `--print-traceback` option to see the full traceback.
# Repeated 3 more times
Warning: 4 out of 5 tests failed

I have been trying to get this to work for quite some time. First by varying every combination of conceivable AiiDA Flags for verdi computer configure core.ssh vsc4 and by looking into the source code. Then, through a blog post online I found out that this type of error occurs when a particular sshd configuration flag on the server (cluster) is set.

Now, I want to mention that I am not familiar with ssh server administration so excuse me if I get some things wrong. Specifically, when MaxSessions 1 in /etc/sshd_config, only one ssh channel can be opened simultaneously. AiiDA labels the first channel with number 0, that's why * Opening connection... [OK] works, and every other test fails. Curiously I never ran into issues like this when I open multiple terminals with ssh connections to the cluster.

I contacted the cluster admins concerning this and they confirmed that they have set MaxSessions 1 for security reasons, i.e. they are not going to change this flag.

The last thing I tried was verdi config set transport.task_maximum_attempts both 1 and 10 but to no avail.

I would greatly appreciate any help or advice you could offer. On that note, I would like to thank Jusong Yu and Marnik Bercx for previous discussions.

[zhubonan]

The test opens multiple channels without closing I think so it fails. During the actual operation of AiiDA there should be only one connection is used per daemon worker so you might be fine. However, it won't work if you also login from you computer via SSH.

The normal ssh works fine probably because you have enabled the control master and hence extra connections will use multiplexing I think. Unfortunately the underlying python ssh library does not currently support control masterl like your system ssh.

Things to try:

• there is a fork on the paramiko library that does support multiplexing Mux transport for client (rebased, the second) paramiko/paramiko#1931

• just a thought - you can login from your computer and use port forwarding to forward a port (PORT) and then run your own ssh server (without admin, there are guides on the internet) listening this port. The primary ssh session must be kept open all the time. Then you instruct AiiDA to connect to your local computer's PORT. This connects to your own ssh server that does not have and restrictions. I have not done this but in theory it should work.

• nudge the admins again....

[mp-cms]

Apologies for not responding sooner.

Thank you for you suggestions. Answering them directly:

• I installed the fork of paramiko that you suggested but I got the same error message
• Please correct me if I am wrong, but from your suggestion I understood that I could install a separate ssh server on the cluster and force AiiDA to connect to it, all without having root access on the cluster. I searched for tutorials but every one that I found needed root access on the remote machine at some point.
• I think the chance that they will change this flag is very low, but I can try. What number for MaxSessions should I suggest?

Here's one more workaround that I thought of. For the time being, I would be satisfied to just use AiiDAs provenance graph/database functionality. Is there a way that I do my VASP calculations on the cluster, download the results and add them to the database manually? Sorry if this is not the right place to ask this.

Thanks again for helping me out!

[zhubonan]

Sorry for the late reply:

I installed the fork of paramiko that you suggested but I got the same error message.

I think it may need some configuration. I have not tried that fork myself. It maybe easier to first test without AiiDA but directly using their Python API and see if you can connect to the remote using the ssh control master.

Please correct me if I am wrong, but from your suggestion I understood that I could install a separate ssh server on the cluster and force AiiDA to connect to it, all without having root access on the cluster. I searched for tutorials but every one that I found needed root access on the remote machine at some point.

Yes, that is what I meant. There are older versions of OpenSSH that seems to do not have the issue with the root access. See here: https://serverfault.com/questions/344295/is-it-possible-to-run-sshd-as-a-normal-user. Again I have not verified this myself.

I think the chance that they will change this flag is very low, but I can try. What number for MaxSessions should I suggest?

I think something in the range of ~20 would be safe for the operation of AiiDA.

Here's one more workaround that I thought of. For the time being, I would be satisfied to just use AiiDAs provenance graph/database functionality. Is there a way that I do my VASP calculations on the cluster, download the results and add them to the database manually? Sorry if this is not the right place to ask this.

I don't it would work because out-of-the box, beacuse AiiDA has to be interfaced with the HPC for running jobs, or at least it is designed to run this way for the provenance. However, there is a one workaround I think you can still try - setup a Slurm scheduler on your local machine or some other workstation (Slurm is easy to install, others should work as well), then let AiiDA submit jobs to there. The jobs should not be run automatically but instead be sets to be on-hold by default. Then you can write a script that monitor any such new job, copy the input files to your remote cluster and submit for it to run. After your remote cluster finishes the jobs, have the file automatically transfer to the machine where the Slurm scheduler is installed and set the job state to be finished. Then AiiDA will pull the outputs and parse them from this machine, as if it is run there.

[ltalirz]

Just one more thought: it's of course possible in principle to install AiiDA in user space on the cluster as well (e.g. you can get postgresql and rabbitmq via mamba/conda install -c conda-forge aiida-core.services ) - not sure whether they like that of course.

From the cluster admin perspective I think in the medium term they should decide whether they allow access for workflow management tools like AiiDA from outside the cluster or allow running them inside; but one of the two options should be supported.