piped-backend container labelled as "unhealthy"

[Jerroder]

Describe your question

Hi,

I've had Piped installed for months but since yesterday, when I access it (through a reverse proxy), I get the front-end but it's loading forever, the loading wheel spins indefinitely and nothing happens. When accessing it locally through its IP, I get the "Welcome to nginx!" message as if nothing was configured.

I tried deleting all the containers and re-creating them (using the nginx AIO script, as I did the first time) but it hasn't solved the problem. When running docker ps, I can see the piped-backend container as "unhealthy". I guess that the reason but I have no idea how to solve it. I don't know where to find logs either.

Any idea?

[kovdan01]

I'm also facing the issue. I tried to find something useful in docker-compose logs output, but the only piped-backend lines are the following which appear on startup:

piped-backend    | Oct 12, 2023 7:04:19 PM liquibase.database
piped-backend    | INFO: Set default schema name to public
piped-backend    | Oct 12, 2023 7:04:22 PM liquibase.changelog
piped-backend    | INFO: Reading from public.databasechangelog
piped-backend    | Database is up to date, no changesets to execute
piped-backend    | Oct 12, 2023 7:04:23 PM liquibase.changelog
piped-backend    | INFO: Reading from public.databasechangelog
piped-backend    | Oct 12, 2023 7:04:23 PM liquibase.util
piped-backend    | INFO: UPDATE SUMMARY
piped-backend    | Oct 12, 2023 7:04:23 PM liquibase.util
piped-backend    | INFO: Run:                          0
piped-backend    | Oct 12, 2023 7:04:23 PM liquibase.util
piped-backend    | INFO: Previously run:               3
piped-backend    | Oct 12, 2023 7:04:23 PM liquibase.util
piped-backend    | INFO: Filtered out:                 0
piped-backend    | Oct 12, 2023 7:04:23 PM liquibase.util
piped-backend    | INFO: -------------------------------
piped-backend    | Oct 12, 2023 7:04:23 PM liquibase.util
piped-backend    | INFO: Total change sets:            3
piped-backend    |
piped-backend    |
piped-backend    | UPDATE SUMMARY
piped-backend    | Run:                          0
piped-backend    | Previously run:               3
piped-backend    | Filtered out:                 0
piped-backend    | -------------------------------
piped-backend    | Total change sets:            3
piped-backend    |
piped-backend    | Oct 12, 2023 7:04:23 PM liquibase.util
piped-backend    | INFO: Update summary generated
piped-backend    | Oct 12, 2023 7:04:23 PM liquibase.lockservice
piped-backend    | INFO: Successfully released change log lock
piped-backend    | Oct 12, 2023 7:04:23 PM liquibase.command
piped-backend    | INFO: Command execution complete
piped-backend    | ThrottlingCache: 0 entries
piped-backend    | SLF4J: No SLF4J providers were found.
piped-backend    | SLF4J: Defaulting to no-operation (NOP) logger implementation
piped-backend    | SLF4J: See https://www.slf4j.org/codes.html#noProviders for further details.
piped-backend    | Oct 12, 2023 7:04:23 PM org.hibernate.Version logVersion
piped-backend    | INFO: HHH000412: Hibernate ORM core version [WORKING]
piped-backend    | Oct 12, 2023 7:04:25 PM org.hibernate.cache.internal.RegionFactoryInitiator initiateService
piped-backend    | INFO: HHH000026: Second-level cache disabled
piped-backend    | Oct 12, 2023 7:04:26 PM org.hibernate.engine.jdbc.connections.internal.ConnectionProviderInitiator initiateService
piped-backend    | INFO: HHH000130: Instantiating explicit connection provider: org.hibernate.hikaricp.internal.HikariCPConnectionProvider
piped-backend    | Logged in as user: null
piped-backend    | Room ID: !CfXSiQMnWTEPfnBNuK:matrix.org
piped-backend    | Filter ID: null
piped-backend    | Oct 12, 2023 7:04:26 PM org.hibernate.engine.jdbc.dialect.internal.DialectFactoryImpl constructDialect
piped-backend    | WARN: HHH90000025: PostgreSQLDialect does not need to be specified explicitly using 'hibernate.dialect' (remove the property setting and it will be selected by default)
piped-backend    | Oct 12, 2023 7:04:31 PM org.hibernate.engine.transaction.jta.platform.internal.JtaPlatformInitiator initiateService
piped-backend    | INFO: HHH000489: No JTA platform available (set 'hibernate.transaction.jta.platform' to enable JTA platform integration)
piped-backend    | Database connection is ready!
piped-backend    | PubSub: queue size - 9 channels
piped-backend    | Cleanup: Removed 0 old videos

[Jerroder]

I get the same logs, nothing useful and no errors at all.

[kovdan01]

It just magically started working for me again. @rderensy Please try to update and restart docker containers - maybe your instance will become working as well.

[Jerroder]

Lucky you! I just pruned all the docker and reinstalled them, the backend still appears as "unhealthy" and nothing is loading.

[brodokk]

So... I have the same problem, @kovdan01 solution doesn't seems to work for me but when i check the page where there is the piped build https://hub.docker.com/r/1337kavin/piped/tags

Using openj9 is fine for me... My guess @rderensy would be to try a different tag?

[Jerroder]

That worked, thank you @brodokk!

[kovdan01]

@rderensy FYI: piped:latest also stopped working for me, unhealthy as previosly ¯_(ツ)_/¯
@brodokk Thanks for solution! piped:openj9 works for me

[graphik55]

@brodokk thank you, this works for me too. I tried to debug multiple days to find a solution and did not recognize this issue :S

For further usage:
The container itselfs starts with the 'latest' tag without any issue. The 'healthcheck' endpoint works until you open the frontend. As soon as this is opened the 'healthcheck' fails and the container becomes 'unhealthy'.
The java process inside of the 'piped-backend' container is still running and the TCP listener is still acitve.
Validated with:
ps faux
for the process and the following snipped for the listener:

awk 'function hextodec(str,ret,n,i,k,c){
    ret = 0
    n = length(str)
    for (i = 1; i <= n; i++) {
        c = tolower(substr(str, i, 1))
        k = index("123456789abcdef", c)
        ret = ret * 16 + k
    }
    return ret
}
function getIP(str,ret){
    ret=hextodec(substr(str,index(str,":")-2,2)); 
    for (i=5; i>0; i-=2) {
        ret = ret"."hextodec(substr(str,i,2))
    }
    ret = ret":"hextodec(substr(str,index(str,":")+1,4))
    return ret
} 
NR > 1 {{if(NR==2)print "Local - Remote";local=getIP($2);remote=getIP($3)}{print local" - "remote}}' /proc/net/tcp 

(source: https://staaldraad.github.io/2017/12/20/netstat-without-netstat/)

I wasn't able to find any error message, I guess until it is possible to start the 'piped.jar' in debug mode this becomes difficult to analyze further in a normal environment.

[graphik55]

Due to a chat discussion in the matrix room I tried to find anything using a java profiler.
I am abele to attach to the docker process but I am too unexperienced in debugging java like that.
So if there is anything I can provide just let me know. With some brief instructions I might provide everything what is need to find the root cause.

[danielmotaleite]

piped:openj9 is not listed anymore in docker hub and all other tags give the same result, the healthcheck locks up as soon we start to get traffic
Any other workaround found?

[FireMasterK]

Is there any kind of hardware/server configuration that I could deploy Piped on to reproduce this issue? I'm unfortunately unable to reproduce this locally, despite trying many different things

[Jerroder]

FWIW I'm running Debian 12 virtualized on Proxmox.

@FireMasterK Do you want to troubleshoot together or something?

[danielmotaleite]

Ok, found that starting everything, not accessing the site, actually blocking the incoming traffic and wait for some time (30min on my first attempt, have to do more tests) and then try again, everything was working and didn't failed until now. I'm also using a proxmox VM, maybe this is related with the CPU, less resources, slower startup and receiving traffic when not everything was ready, blocking the backend. Waiting enough time for the backend/java to start everything correctly may be the key... a VM with little CPU takes more time to startup and have a longer time range to trigger the problem, a full server with many cpu may startup quickly and harder to trigger the problem

[Jerroder]

FWIW I reinstalled the OS completely and this time, I used docker compose up -d instead of docker-compose up -d (V2 instead of V1) and it works. Now the reinstall might have solved it, but perhaps using the new compose makes a difference too.