Guacamole not connecting to Linux vm

[bwiseman]

Have a proof of concept TRE up and running. Windows vm look fine,. My Linux vm deploy with no errors but when I try to connect I get a flash of 'connected to guacamole' and then a red screen saying 'the remote desktop server is currently unreachable'.

The vm is in the same workgroup and same project as the windows vm. I'm accessing it as the same user with tre user, workspace researcher and owner permissions. Looked at some logs on azure and api/docs but couldn't see anything. It feels like I've missed something simple in the deployment for permissions?

make user_resource_bundle BUNDLE=guacamole-azure-linuxvm WORKSPACE_SERVICE=guacamole

and then created the user resourcet.

Are there guacamole logs hiding somewhere in the TRE or Azure?
I had a look at Virtual Machine User Login role but that didn't change anything. Assignments look the same between linux and windows. Tried updating, restarting, logging off/on. I haven't dropped it from a height yet... that's next level

[marrobi]

Have you got the nexus package mirror installed? The Linux VMs use it to access apt packages.

[bwiseman]

Yes nexus installed. I tried redeploying and creating the service. Deleted and created a new Linix vm. Still can't connect to it with guacamole. I have logged into the nexus admin vm. I can run sudo apt get ... and it seems to work ok. Guacamole still gives the red 'desktop server is currently unavailable' error if I try to connect to a project vm though. I can't get the secret to try to login via the azure portal to check logs on the vm as the kv is locked down by network access. Reluctant to mess about with that if the guacamole problem is something obvious to someone.

I noticed a couple of things that look a bit odd on the nexus service and nexus vm although maybe not related

• The deployed nexus tre-shared-service-certs stays in 'pending' and status says '[Nexus: renew] so that's maybe normal if it's checking for ssl updates? The operations tab says it's deployed successfully but it flags Resource locked while it updates. I can't get to actions to update, disable or delete it.

• It has an eth0 interface with an ip in the azuretre network which is expected. It has a docker0 interface with an ip address outwith the azuretre network. It is using an ip in another vnet which is assigned for another purpose. It shouldn't have anything to do with the TRE and isn't in the main config.yaml I didn't assign a docker0 ip. I don't see it in the conf files. Where would it pick that up from? What is the docker0 interface used for?

[marrobi]

Can you check the cloud init logs on nexus and the linux VM?

https://microsoft.github.io/AzureTRE/unreleased/troubleshooting-faq/cloud-init/

It could be is an issue with your nexus deployment, can you bring up nexus in a browser from your Windows VM?

[bwiseman]

Yes I can bring up nexus in a browser from a Windows vm connection.

I guess I need to edit the kv access policy and network access to allow me to copy the secret for a problem linux vm? I don't have any other access. Is that wehere I need to look at the cloud-init logs and scripts?

[marrobi]

I typically reset the VM credentials/create a new user via the portal menu to troubleshoot, then use serial console to login and view the logs.

It's odd that Nexus is up, but the Linux VM is failing somewhere. Imagine something transient is going on.

[bwiseman]

To reset the vm credentials to something known.. whats the best way to do that?. Looked at the reset password script but wasn't sure how to specify username on the vm without having access

[marrobi]

As below:

[bwiseman]

Hmm ok, so running the cloud init scripts manually worked. Was asked to replace a key.... yes. Let it run through and then tested the guacamole connection... ya dancer, it worked!

Checking the log beforehand I saw "failed to connect to nexus-xxxx.cloudapp.azure.com port 443 connection refused, no valid OpenPGP found"

Every one of my Linux vm's has this problem, windows work fine. What's failing in the deploy that creates the pgp stuff for linix?

[marrobi]

So the same happens with a new Linux VM?

When you browse to nexus is the SSL certificate in the browser valid?

In the install scripts you create a certificate using the certs service, and pass the details to Nexus. Can you check the logs for cloud-init on Nexus? Maybe the cert install failed.

Actually we did have a recent issue that only occasionally appears, fixed in main, #3492, but isn't in a release yet, but will be in the coming weeks, maybe you are seeing this. I'm thinking it must be.

[bwiseman]

so

• yes it happens with every linux vm I create. Tried different workspaces, different images
• yes the SSL cert if I browse to nexus is valid

Running the scripts allows the guacamole connection to work.

The Linux DSVM image complains about docker when I run the init script and then doesn't have the dsvm apps for the dsvm image. Don't know if this is related.

  Process: 14699 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=1/FAILURE)
 Main PID: 14699 (code=exited, status=1/FAILURE)

Jul 14 20:24:02 linuxvm2ef3 systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Jul 14 20:24:02 linuxvm2ef3 systemd[1]: docker.service: Failed with result 'exit-code'.
Jul 14 20:24:02 linuxvm2ef3 systemd[1]: Failed to start Docker Application Container Engine.
dpkg: error processing package docker-ce (--configure):

Jul 14 20:43:19 linuxvm2ef3 systemd[1]: Started Jupyter Notebook.
Jul 14 20:43:19 linuxvm2ef3 systemd[19775]: jupyter.service: Failed to determine user credentials: No such process
Jul 14 20:43:19 linuxvm2ef3 systemd[19775]: jupyter.service: Failed at step USER spawning /anaconda/bin/jupyter-notebook: No such process
Jul 14 20:43:19 linuxvm2ef3 systemd[1]: jupyter.service: Main process exited, code=exited, status=217/USER
Jul 14 20:43:19 linuxvm2ef3 systemd[1]: jupyter.service: Failed with result 'exit-code'.

[marrobi]

Can you attach the cloud init logs for nexus and a new Linux VM please. If cant upload them, let me know and we'll find another way.

Can you also try deploying a plain Ubuntu, rather than DSVM image?

[bwiseman]

dsvm-cloud-init-output.log

The plain ubuntu had the same guacamole issue but now, as if by magic... deploys and connects.
The dsvm deploys and like the plain, guacamole connects without jumping through any hoops. Doesn't load data science tools though.

Attached is the cloud-init log from the linux dsvm that doesn't load the data science tools.

[marrobi]

Very strange. I wonder if a certificate was out of date on one of the package sources.

In production we highly recommend using your own custom VM images as reduces the external dependencies hence scope for problems and improves deployment time.

[marrobi]

Looks like a few issues in the cloud init log:

Get:1 https://nexus-azuretre.uksouth.cloudapp.azure.com/repository/docker bionic/stable amd64 containerd.io amd64 1.6.21-1 [28.3 MB]
Get:2 https://nexus-azuretre.uksouth.cloudapp.azure.com/repository/docker bionic/stable amd64 docker-buildx-plugin amd64 0.10.5-1ubuntu.18.04bionic [26.1 MB]
Get:3 https://nexus-azuretre.uksouth.cloudapp.azure.com/repository/docker bionic/stable amd64 docker-ce-cli amd64 5:24.0.2-1ubuntu.18.04bionic [13.3 MB]
Get:4 https://nexus-azuretre.uksouth.cloudapp.azure.com/repository/docker bionic/stable amd64 docker-ce amd64 5:24.0.2-1ubuntu.18.04bionic [22.9 MB]
Get:5 https://nexus-azuretre.uksouth.cloudapp.azure.com/repository/docker bionic/stable amd64 docker-ce-rootless-extras amd64 5:24.0.2-1ubuntu.18.04bionic [9014 kB]
Get:6 https://nexus-azuretre.uksouth.cloudapp.azure.com/repository/docker bionic/stable amd64 docker-compose-plugin amd64 2.18.1-1ubuntu.18.04bionic [10.9 MB]
dpkg-preconfigure: unable to re-open stdin: No such file or directory

Jul 16 00:40:36 linuxvm2dd8 dockerd[11290]: unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives don't match any configuration option: graph

[bwiseman]

Thanks, yes, something not right with the docker daemon config on the vm so I'll have a closer look at that.

Looking at custom images is next on my list but I haven't found any starter documentation for creating them. I can see the template structure but it looks a bit daunting to try to create that from scratch!

[SvenAelterman]

PS: same experience here + with at least one customer. Will try some of the suggestions outlined.

[bwiseman]

What have you seen sven?

[SvenAelterman]

Linux VMs that we can't connect to. I haven't been able to troubleshoot on my deployment or the customer's due to time constraints (and knowledge constraints; so the above will help).

[marrobi]

@bwiseman apologies for the delay, this could be related to this : #3642

Seems new nexus deploys are failing.

For future, if you hit issues, it's best to create an Issue rather than discussion, as they are triaged etc. Thanks.

[bwiseman]

Thanks for the link... I just assumed I was doing something wrong (usually the most likely assumptions are the best). I'll check out the nexus issue and fixes. barbara

…

On 2023-08-03 09:20, Marcus Robinson wrote: @bwiseman [1] apologies for the delay, this could be related to this : #3642 [2] Seems new nexus deploys are failing. For future, if you hit issues, it's best to create an Issue rather than discussion, as they are triaged etc. Thanks. -- Reply to this email directly, view it on GitHub [3], or unsubscribe [4]. You are receiving this because you were mentioned.Message ID: ***@***.***> Links: ------ [1] https://github.com/bwiseman [2] #3642 [3] #3612 (comment) [4] https://github.com/notifications/unsubscribe-auth/AHQNUARWSEJSPXE5DRFY34DXTNNMPANCNFSM6AAAAAA2IQWFKU