502 Bad Gateway response from edgeshark container

[mortenvf]

Running edgeshark out of the box, I get a 502 Bad Gateway response when accessing http://localhost:5001/. (Running on Ubuntu 22.04 in VmWare virtual machine)

Command

wget -q --no-cache -O -   https://github.com/siemens/edgeshark/raw/main/deployments/wget/docker-compose-localhost.yaml   | docker compose -f - up

Log output from container shows

edgeshark-1  | DEBU[2024-05-14T13:34:54Z] handling "/"...                              
edgeshark-1  | DEBU[2024-05-14T13:34:54Z] forwarding/reverse proxying to "http://gostwire.ghost-in-da-edge:5000/" 
edgeshark-1  | 2024/05/14 13:34:54 http: proxy error: EOF

Pinging and connecting with netcat to gostwire.ghost-in-da-edge:5000 from the edgeshark container works just fine.

Update: This also occurs with ./edgeshark.sh up.

What could be causing this issue?

Best regards,
Morten

[thediveo]

Hi mortenvf, I don't think we've seen this so far. What does the log from the gostwire service say? It should mention GET requests and also log that the discovery service came up correctly. The webui gets served by the g(h)ostwire service, and packetflix forwards proxy request not addressing the capture API endpoint to it.

In your vm, do you use Canonical's snap Docker or Docker's Docker CE installation?

[mortenvf]

Ghostwire is not logging anything that indicates that it has received the forwarded request.

I have enabled --debug mode in both edgeshark and ghostwire and additionally on edgeshark --log-headers --log-requests. I get this log output (minus the initial INFO and DEBUG messages which look inconspicuous).

...
gostwire-1   | DEBU[2024-05-15T11:18:30Z] gostwire discovery finished                  
edgeshark-1  | INFO[2024-05-15T11:19:01Z] 172.23.0.1 - - [15/May/2024:11:19:01 +0000] "GET / HTTP/1.1" -1 -1 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36" 
edgeshark-1  | DEBU[2024-05-15T11:19:01Z] handling "/"...                              
edgeshark-1  | DEBU[2024-05-15T11:19:01Z] forwarding/reverse proxying to "http://gostwire.ghost-in-da-edge:5000/" 
edgeshark-1  | 2024/05/15 11:19:01 http: proxy error: EOF

I am using the docker apt package.

One thing that I noted was that docker-compose complained unsupported option services.gostwire.cgroup, so I installed the docker compose plugin instead.

Is packetflix performing the forwarding, or is some 3rd party reverse proxy installed in the image handling forwarding?

[thediveo]

thank you very much for filling down further!

as you wrote, as there is a discovery finished, the ghostwire services appears healthy.

the Unsupported cgroup option should be no issue, as there is a workaround in the code for older Docker engines.

packetflix itself does the forwarding, using Go's http proxy type, so this is part of the Go binary.

the packaged then should be the docker.io deb package; I haven't tested against this, but only against the docker-ce package. in the past we've seen seccomp profile issues, but I'm unsure of this could be related to this problem here. for 22.04lts the debian package has Docker 20.10 with some security fix backports. my only guess at this time would be giving the upstream docker-ce package straight from Docker itself a test.

[mortenvf]

If I expose ghostwire port 5000, then I can access the ghostwire web UI directly and it seems to be working as intended, showing the results of discovery.

[thediveo]

I'm unfortunately atm lost for any idea.

[thediveo]

just to be sure, did you do a "docker compose ... down" in order to clean up?

[mortenvf]

Yes, I did docker compose down.

Another observation: something weird seemed to be going with networking. Some IPs became unreachable while edgeshark was running and this resolved when I shut edgeshark down.

I will see if I can find time to test with docker-ce.

[thediveo]

I wonder if watching the engine events could reveal more?

[thediveo]

I tested on an Ubuntu 22.04.4LTS on kvm. I kept the snap-based Docker engine, which is 24.0.5 atm. I deployed using the host-exposing compose file and was able to successfully access the web UI at the VM's host address and port 5001.

Did you put your VM on Broadcom to sleep/hibernation? I'm still having issues on a VMware Workstation Pro on Kubuntu VMs with VMware tools totally trashing the bridge IP configuration, even with the exceptions verified to be in VMware's configuration file.