Bug: 'Unable to connect to terminal'

[jordan-bravo]

Describe the bug

When attempting to launch the built-in Lightning node terminal within Polar, an error message is displayed:

Unable to connect to terminal
Docker container 'polar-n2-alice' not found

To Reproduce

Steps to reproduce the behavior:

1. Open Polar
2. Start network (in my case, two LND nodes)
3. Click on a Lightning node (in my case, an LND node called 'alice')
4. Click on Actions
5. Under the section Terminal, click Launch.

Expected behavior

The Polar Terminal should launch and provide the ability to run commands on the node.

Actual behavior

The Polar Terminal launches and shows the error listed above.

Screenshots

Desktop (please complete the following information):

• OS: Fedora 37
• Polar Version: 1.4.0
• Docker Version: 20.10.21
• Docker Compose Version: 2.12.2

[jamaljsr]

Are you able to deposit funds and open a channel from alice?

Does docker ps show the alice container running?

[jordan-bravo]

• docker ps shows the alice container is running.
• I restarted Polar and I'm also experiencing a permission issue with admin.macaroon (see screenshot below).

[jamaljsr]

Setting up docker on Linux can be pretty tricky when it comes to permissions. Did you install Docker Desktop for Linux or install using a package? For the packages or scripts, there are sometimes post-install steps that need to be done. Is Docker running as root or a user account?

Unfortunately, I dont have a Fedora machine to test or troubleshoot with.

[jordan-bravo]

To test it out, I did the following:

• A fresh install of Ubuntu 22.10.
• Installed Docker Desktop v4.15.0 (which installs Docker version 20.10.21 and Docker Compose version v2.13.0)
• Downloaded the Polar AppImage, made it executable, then opened, which shows the following, no matter how many times I click "Check Again":

Any advice? Is Polar compatible with Docker Desktop?

Here is the output of the terminal command docker version:

[jordan-bravo]

Update

I created a symbolic link using the command sudo ln -s /home/${USER}/.docker/desktop/docker.sock /var/run/docker.sock, and now Polar recognizes Docker, but still shows the following error for Docker Compose (see screenshot).

{
  "errno": -2,
  "code": "ENOENT",
  "syscall": "spawn docker-compose",
  "path":"docker-compose",
  "spawnargs": ["version","--short"]
}

It looks like Polar is calling docker-compose, which is the syntax for the deprecated api for Docker Compose V1. Docker recommends using the Docker Compose V2 syntax: docker compose (a space between the words, instead of a hyphen).

I tried creating an alias in my ~/.bashrc file:

alias docker-compose="docker compose"

However Polar is still showing the same error. I'm not sure how to make Polar use the alias.

[jordan-bravo]

Update 2

There is an option in Docker Desktop to create a link from docker-compose to docker compose, this solved the startup issue and now Polar can find Docker and Docker Compose.

However, there is still a permission issue with admin.macaroon. Any suggestions?

[jamaljsr]

Thanks a bunch for all of the details and screenshots. I have an Ubuntu VM with Polar working but I haven't installed from scratch in a long while. Unfortunately, Docker Desktop doesn't work inside a VM, so I won't be able to reproduce your exact scenario myself.

For the permissions issue, can you share what the current permissions and owner are for the bob dir? Polar should create this dir before starting the network in order to make your host user the owner of the dir. This should grant you read/write permission to the subdirs as well when the LND node writes to the Docker volume. It appears something is going wrong there.

[jordan-bravo]

Here's what the permissions look like. My (host) user name is j but the owner is 100999.

[jamaljsr]

Ahh, that is interesting. This isn't behavior I've ever seen before. My hunch is that Docker Desktop is handling the file permissions differently now that it is using a VM internally. Docker Server for Linux doesn't use a VM.

Can you try stopping the network, changing the owner/permissions of the volumes dir to your j user, then restart the network? I'm curious to see if Docker will modify the perms again after it boots up.

[jordan-bravo]

I performed the following steps:

• In Polar, I stopped the network.
• In a terminal, cd to the directory ~/.polar/networks/1/volumes/lnd, then run the commands:

sudo chown -R j:j alice
sudo chown -R j:j bob

• In Polar, start the network.

Result: the network starts, and I can launch the Polar terminal with no permission issues.

Summary

There are three outstanding issues here when using Docker Desktop:

1. Polar is expecting docker at /var/run/docker.sock. A temporary workaround is to created a symbolic link using the command sudo ln -s /home/${USER}/.docker/desktop/docker.sock /var/run/docker.sock, however this does not persist across system reboots.
2. Polar is using the deprecated V1 Docker Compose API which expects the command docker-compose instead of the V2 version docker compose. The workaround for this is a setting in Docker Desktop, check the box for Enable Docker Compose V1/V2 compatibility mode.
3. Inside the Docker Desktop VM, the owner for the items in the lnd directory are not the host user. The workaround for this is to recursively own the directories with sudo chown -R ${USER}:${USER} ~/.polar/networks.

[jamaljsr]

Thanks for testing and confirming that behavior. I agree that Polar needs to be updated to resolve those few issues. I will try to get a native Linux machine setup to work on better compatibility with Docker Desktop. I really appreciate the troubleshooting here.

[jordan-bravo]

I'm a TypeScript and React dev, I'd be happy to submit a bug fix if I'm able to figure it out. Could you point me in the right direction with regards to where in the codebase these Docker related issues might be located?

[jamaljsr]

I appreciate the offer to help.

Polar uses the dockerode and docker-compose NPM packages to communicate with Docker on the machine. All of this interaction is done in the dockerService.ts file. That's where you'd want to start.

Regarding each issue, here are my thoughts on how I would initially attempt to solve them:

• Polar is expecting docker at /var/run/docker.sock. A temporary workaround is to created a symbolic link using the command sudo ln -s /home/${USER}/.docker/desktop/docker.sock /var/run/docker.sock, however this does not persist across system reboots.

By default, dockerode uses multiple methods to attempt to connect to Docker on the machine. You can see this logic in the docker-modem repo. Since it appears that dockerode isn't automatically detecting the socket file at /home/${USER}/.docker/desktop/docker.sock, Polar could check for the existence of this file and if it exists, pass it as the socketPath to the Dockerode constructor. I created #609 which allows the user to specify the paths for the Docker socket and docker-compose executable, but I would really prefer the detection to be automatic. This is why I still haven't merged that in yet.

• Polar is using the deprecated V1 Docker Compose API which expects the command docker-compose instead of the V2 version docker compose. The workaround for this is a setting in Docker Desktop, check the box for Enable Docker Compose V1/V2 compatibility mode.

I haven't found a clear solution to this, but after just a bit of searching I found that the dockerode dev also created the dockerode-compose package which has some functionality to call docker compose commands via the Docker API. Unfortunately, it looks like it doesn't have full API coverage. It only supports up, down, and pull. Polar also needs the ability to start, stop, and remove individual services in the docker network. A thought I had was possibly use the dockerode-compose package for some of the functionality and fill the gaps in the Polar codebase using the dockerode package, which should be possible. There may also be other packages out there but I haven't spent a lot of time researching.

• Inside the Docker Desktop VM, the owner for the items in the lnd directory are not the host user. The workaround for this is to recursively own the directories with sudo chown -R ${USER}:${USER} ~/.polar/networks.

I haven't spent any time troubleshooting this issue since I don't have a machine running Docker Desktop on Linux. I'd need to learn more about why the volume file permissions aren't being set in the same way as Docker Engine. It could just be setting additional flags in the compose file, I'm not sure. One idea is to have Polar reset the file perms after starting the nodes but this feels a bit hacky. There may be a cleaner solution but would require a better understanding of why Docker behaves differently.

[jordan-bravo]

Thanks for the info. I found something that might shed light on the ownership situation.

The Docker Desktop Docs describe this issue but it's not immediately obvious to me what the solution would be.

In particular, the last paragraph of the section is relevant:

In this scenario if a shared file is chowned inside a Docker Desktop container owned by a user with a UID of 1000, it shows up on the host as owned by a user with a UID of 100999. This has the unfortunate side effect of preventing easy access to such a file on the host. The problem is resolved by creating a group with the new GID and adding our user to it, or by setting a recursive ACL (see setfacl(1)) for folders shared with the Docker Desktop VM.

[jamaljsr]

Wow, this got ugly. I assumed Docker Desktop would make things easier, not harder. Especially with a something as common as sharing files with the host.

After reading that docs, my initial thought is to first try updating those two files to confirm that does indeed solve the problem. If that works, I'd see if there's a way to detect if the user is running Docker Engine or Docker Desktop. If it's Desktop, I'd see if Polar/Electron could read/write to the subuid and subgid files. If so, it could check to see if the username is in that file. If it's not, then display a modal in Polar on startup requesting the user's permission to update those files. The modal should have a brief explanation of the problem, solution, and a link out to that Docker docs page.

Obviously, there are a lot of IFs in this approach. Any one of them failing would mean you'd need to reevaluate and pivot, or possibly come up with another idea. But you'd likely learn a lot more in the process of working on this and may come up with a cleaner solution.

[bxplorer]

Thanks a bunch for all of the details and screenshots. I have an Ubuntu VM with Polar working but I haven't installed from scratch in a long while. Unfortunately, Docker Desktop doesn't work inside a VM, so I won't be able to reproduce your exact scenario myself.

For the permissions issue, can you share what the current permissions and owner are for the bob dir? Polar should create this dir before starting the network in order to make your host user the owner of the dir. This should grant you read/write permission to the subdirs as well when the LND node writes to the Docker volume. It appears something is going wrong there.

I've got Linux Mint installed and running Docker Desktop on a VirtualBox 6.1 by setting up on Settings -> System -> Processor -> Extended Features -> Enable Nested VT-x/AMD-V.
https://i.stack.imgur.com/0bM6m.png

[bxplorer]

Update 2

There is an option in Docker Desktop to create a link from docker-compose to docker compose, this solved the startup issue and now Polar can find Docker and Docker Compose.

However, there is still a permission issue with admin.macaroon. Any suggestions?

I installed docker-composer following the instructions in https://docs.docker.com/compose/install/other/ to have Polar find it.

[bxplorer]

Here's what the permissions look like. My (host) user name is j but the owner is 100999.

I did a work arround to this issue by giving all permissions to everybody:
$ sudo chmod -R a+rwx ~/.polar/networks/1/volumes/lnd/

But now I'm getting the error "Failed do fetch the bitcoin block height connect;ECONNREFUSED 127.0.0.1:18443' even tough polar-n1-backend1 container is up and running and listening to this port.

[iangregsondev]

Yep, I can confirm there are some issues with running this under Ubuntu (actually, my distro is Zorin which is based on ubuntu)

Tried many many things and I just can't get it to work.

The docker process creates the permissions as root under "data" etc..

The only way I could get it to work is to start the network, and it would fail on LND for example.

Stop the network..

do a CHOWN on all files below .polar and now it works.

[iangregsondev]

Oh, damn :-( So that does work, but once you exit Polar and come back in, permissions are lost on all network files again

user and group seems to be set back to 100999

[iangregsondev]

Ok, sorry for the noise, last update :-) Confirmed.

I am able to do a CHMOD to 777 for everything under network and this seems to be maintained after restarting of polar and stopping and starting the network.

sudo chmod 777 -R .

[jamaljsr]

Appreciate the additional details. It's very helpful. I hope to be able to spend some time on these Linux issues week.

@bxplorer What host OS are you using which supports you running Docker Desktop inside of a VM? I'll try to use VirtualBox myself instead of VMWare Fusion.

@iangregsondev Are you using Zorin + Docker Desktop on bare hardware or inside of a VM? If inside a VM, what host OS are you using as well?

[iangregsondev]

hi @jamaljsr ,

Yes, are bare hardware. No VM.

Its basically Zorin as Desktop, with Docker Desktop (for linux) installed.

[iangregsondev]

A couple of things to look at that may work.

1. Create the "bind" volume directories manually, don't let docker create them. For example, "data", if you create this manually under the current user and then bind that volume via docker the usual way - then everything created under that directory by docker itself should inherit that user.

2. pass --user or in a docker compose, you can specify the user. I think the current user you can get use $UID - not sure

Cheers.

[iangregsondev]

Ok - a quick update, I have fixed the issues on my side. I will try and explain the reason behind it.

1. I am on Linux, so I don't need to run Docker Desktop, although it would be nice.
2. On the docker website it states that docker desktop for linux runs in a VM, I think this is to enable side by side installation of docker engine and docker desktop.
3. When ONLY docker desktop is installed then the docker.sock is NOT found, I tried creating a symlink as suggested by others and this did work but it reset for some reason, maybe on reboot ? Anyway, the docker.sock location is actually displayed inside of the "docker context ls", - Docker desktop is installed in a different docker context - I assume to allow running both DE and docker desktop at the same time.

There is more info here https://docs.docker.com/desktop/faqs/linuxfaqs/#what-is-the-difference-between-docker-desktop-for-linux-and-docker-engine

and specifically, it states the following which is 100% to do with our issue. I didn't follow these instructions for creating a GID - I just went with docker engine.

In this scenario if a shared file is chowned inside a Docker Desktop container owned by a user with a UID of 1000, 
it shows up on the host as owned by a user with a UID of 100999. This has the unfortunate side effect of preventing 
easy access to such a file on the host. The problem is resolved by creating a group with the new GID and adding our 
user to it, or by setting a recursive ACL (see setfacl(1)) for folders shared with the Docker Desktop VM.

So I removed Docker Desktop completely, and I am using Docker engine. Everything works great.

No messing with chown or chmod.

[iangregsondev]

@jamaljsr - Although everything is working now with regards to permissions etc etc..

I can confirm that the issue with the viewing of the "Logs" (I created another github issue for this) - is still here.

The above fix does NOT fix that.

Best regards

Ian

[wksantiago]

d Docker Desktop completely, and I am using Docker engine

How did you get Polar to use Docker engined vs Docker desktop after removing docker desktop?

Here is the answer:

The issue is that during the installation a wrong entry in ~/.docker/config.json was created.

Namely credsStore instead of credStore.

Just change the entry in ~/.docker/config.json like this, and you’re good to go:

{
"stackOrchestrator" : "swarm",
"experimental" : "disabled",
"credStore" : "desktop"
}

[bxplorer]

Appreciate the additional details. It's very helpful. I hope to be able to spend some time on these Linux issues week.

@bxplorer What host OS are you using which supports you running Docker Desktop inside of a VM? I'll try to use VirtualBox myself instead of VMWare Fusion.

@iangregsondev Are you using Zorin + Docker Desktop on bare hardware or inside of a VM? If inside a VM, what host OS are you using as well?

Sorry for taking so long to answer.

I'm using VirtualBox 6.1 on a Windows 10 Pro as a host OS, and Linux Mint 20.3 as guest OS. I had 2 issues on doing that:
1-I can't have docker and VirtualBox on Windows at the same time, so I decided not to run docker on Windows, running it inside a VM instead.
2-I had to Enable Nested VT-x/AMD-V as described before.

Anyway I got Polar running by uninstalling everything, and reinstalling all over again replacing docker-desktop for docker-ce.
It seems docker-desktop was the issue.

[jamaljsr]

Appreciate the follow-up. That is good information. I ended up getting an Ubuntu machine setup running on native hardware in order to test Docker Desktop. The new file sharing mechanism is pretty complex and I haven't gotten Polar working with it just yet. So for the time being, Docker Desktop isn't supported. I will continue trying to find a solution.