Unable to start and issues with generation on v1.74.0

[abc]

Hi there,

A couple of issues:

1. It would appear that the TLS certificate and DHparams are not being generated as part of the generate process.
2. Even when I create those files manually, the container still will not start.

Before creating certificates manually:

-=> start turn
-=> start matrix
Cannot create pid file: /var/run/turnserver.pid: Permission denied
0: : 
RFC 3489/5389/5766/5780/6062/6156 STUN/TURN Server
Version Coturn-4.5.2 'dan Eider'
0: : log file opened: /var/tmp/turn_8_2023-01-02.log
0: : 
Max number of open files/sockets allowed for this process: 1048576
0: : 
Due to the open files/sockets limitation,
max supported number of TURN Sessions possible is: 524000 (approximately)
0: : 
==== Show him the instruments, Practical Frost: ====
0: : TLS supported
0: : DTLS supported
0: : DTLS 1.2 supported
0: : TURN/STUN ALPN supported
0: : Third-party authorization (oAuth) supported
0: : GCM (AEAD) supported
0: : OpenSSL compile-time version: OpenSSL 3.0.5 5 Jul 2022 (0x30000050)
0: : 
0: : SQLite supported, default database location is /var/lib/turn/turndb
0: : Redis supported
0: : PostgreSQL supported
0: : MySQL supported
0: : MongoDB is not supported
0: : 
0: : Default Net Engine version: 3 (UDP thread per CPU core)
=====================================================
0: : Domain name: 
0: : Default realm: turn.matrix
0: : 
CONFIGURATION ALERT: You specified --lt-cred-mech and --use-auth-secret in the same time.
Be aware that you could not mix the username/password and the shared secret based auth methods. 
Shared secret overrides username/password based auth method. Check your configuration!
0: : ERROR: 
CONFIG ERROR: Empty cli-password, and so telnet cli interface is disabled! Please set a non empty cli-password!
0: : WARNING: cannot find certificate file: /data/matrix.tls.crt (1)
0: : WARNING: cannot start TLS and DTLS listeners because certificate file is not set properly
0: : WARNING: cannot find private key file: /data/matrix.tls.key (1)
0: : WARNING: cannot start TLS and DTLS listeners because private key file is not set properly
0: : WARNING: cannot find DH key file: /data/matrix.tls.dh (1)
0: : NO EXPLICIT LISTENER ADDRESS(ES) ARE CONFIGURED
0: : ===========Discovering listener addresses: =========
0: : Listener address to use: 127.0.0.1
0: : Listener address to use: 172.17.0.6
0: : Listener address to use: ::1
0: : =====================================================
0: : Total: 1 'real' addresses discovered
0: : =====================================================
0: : NO EXPLICIT RELAY ADDRESS(ES) ARE CONFIGURED
0: : ===========Discovering relay addresses: =============
0: : Relay address to use: 172.17.0.6
0: : Relay address to use: ::1
0: : =====================================================
0: : Total: 2 relay addresses discovered
0: : =====================================================
0: : Cannot create pid file: /var/run/turnserver.pid
0: : pid file created: /var/tmp/turnserver.pid
0: : IO method (main listener thread): epoll (with changelist)
0: : Wait for relay ports initialization...
0: :   relay 172.17.0.6 initialization...
0: :   relay 172.17.0.6 initialization done
0: :   relay ::1 initialization...
0: :   relay ::1 initialization done
0: : Relay ports initialization done
Cannot create relay thread
: Operation not permitted

After creating certificates manually:

-=> start turn
-=> start matrix
Cannot create pid file: /var/run/turnserver.pid: Permission denied
0: : 
RFC 3489/5389/5766/5780/6062/6156 STUN/TURN Server
Version Coturn-4.5.2 'dan Eider'
0: : log file opened: /var/tmp/turn_8_2023-01-02.log
0: : 
Max number of open files/sockets allowed for this process: 1048576
0: : 
Due to the open files/sockets limitation,
max supported number of TURN Sessions possible is: 524000 (approximately)
Cannot create relay thread
: Operation not permitted
0: : 
==== Show him the instruments, Practical Frost: ====
0: : TLS supported
0: : DTLS supported
0: : DTLS 1.2 supported
0: : TURN/STUN ALPN supported
0: : Third-party authorization (oAuth) supported
0: : GCM (AEAD) supported
0: : OpenSSL compile-time version: OpenSSL 3.0.5 5 Jul 2022 (0x30000050)
0: : 
0: : SQLite supported, default database location is /var/lib/turn/turndb
0: : Redis supported
0: : PostgreSQL supported
0: : MySQL supported
0: : MongoDB is not supported
0: : 
0: : Default Net Engine version: 3 (UDP thread per CPU core)
=====================================================
0: : Domain name: 
0: : Default realm: turn.matrix
0: : ERROR: 
CONFIG ERROR: Empty cli-password, and so telnet cli interface is disabled! Please set a non empty cli-password!
0: : SSL23: Certificate file found: /data/matrix.tls.crt
0: : SSL23: Private key file found: /data/matrix.tls.key
0: : TLS1.0: Certificate file found: /data/matrix.tls.crt
0: : TLS1.0: Private key file found: /data/matrix.tls.key
0: : TLS1.1: Certificate file found: /data/matrix.tls.crt
0: : TLS1.1: Private key file found: /data/matrix.tls.key
0: : TLS1.2: Certificate file found: /data/matrix.tls.crt
0: : TLS1.2: Private key file found: /data/matrix.tls.key
0: : TLS cipher suite: HIGH
0: : DTLS: Certificate file found: /data/matrix.tls.crt
0: : DTLS: Private key file found: /data/matrix.tls.key
0: : DTLS1.2: Certificate file found: /data/matrix.tls.crt
0: : DTLS1.2: Private key file found: /data/matrix.tls.key
0: : DTLS cipher suite: HIGH
0: : NO EXPLICIT LISTENER ADDRESS(ES) ARE CONFIGURED
0: : ===========Discovering listener addresses: =========
0: : Listener address to use: 127.0.0.1
0: : Listener address to use: 172.17.0.6
0: : Listener address to use: ::1
0: : =====================================================
0: : Total: 1 'real' addresses discovered
0: : =====================================================
0: : NO EXPLICIT RELAY ADDRESS(ES) ARE CONFIGURED
0: : ===========Discovering relay addresses: =============
0: : Relay address to use: 172.17.0.6
0: : Relay address to use: ::1
0: : =====================================================
0: : Total: 2 relay addresses discovered
0: : =====================================================
0: : Cannot create pid file: /var/run/turnserver.pid
0: : pid file created: /var/tmp/turnserver.pid
0: : IO method (main listener thread): epoll (with changelist)
0: : Wait for relay ports initialization...
0: :   relay 172.17.0.6 initialization...
0: :   relay 172.17.0.6 initialization done
0: :   relay ::1 initialization...
0: :   relay ::1 initialization done
0: : Relay ports initialization done

[andreaspeters]

Hi @abc thanks for your issue. If you do not need coturn, please try to disable it COTURN_ENABLE="false".

[abc]

Hi @andreaspeters, thanks for the suggestion - I tried that and it still wasn't working, it was failing due to a lack of permissions, and no matter what I tried I couldn't get it to work - including running it as the root user.

I've gone ahead with just getting etke.cc to set up matrix for me since I don't really know what I'm doing, but I'd be happy to work with you to resolve this (potential) bug or to figure out what the issue is so it can be added to the readme.

[andreaspeters]

The permission denied at "/var/run/turnserver.pid" does not bother coturn. :-) Can you share the matrix config files and how you start the container? It would be easier for me to figure out whats wrong. :-) If you don't want publish it here, you can send it to me via email or my matrix support chat (both at: https://www.aventer.biz).

[abc]

To generate config files:

sudo docker run -v /srv/matrix/data:/data --rm --user 1004:1004 -e SERVER_NAME=matrix -e REPORT_STATS=no -e COTURN_ENABLE="false" avhost/docker-matrix:v1.74.0 generate

I haven't edited the config files at all for this example, just left them at the defaults.

To start the container:

sudo docker run -d --user 1004:1004 -e COTURN_ENABLE="false" -p 8448:8448 -p 8008:8008 -p 3478:3478 -v /srv/matrix/data:/data avhost/docker-matrix:v1.74.0 start

I then get these logs:

-=> start matrix
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.10/dist-packages/synapse/app/homeserver.py", line 386, in <module>
    main()
  File "/usr/local/lib/python3.10/dist-packages/synapse/app/homeserver.py", line 376, in main
    hs = setup(sys.argv[1:])
  File "/usr/local/lib/python3.10/dist-packages/synapse/app/homeserver.py", line 284, in setup
    config = HomeServerConfig.load_or_generate_config(
  File "/usr/local/lib/python3.10/dist-packages/synapse/config/_base.py", line 773, in load_or_generate_config
    obj.parse_config_dict(
  File "/usr/local/lib/python3.10/dist-packages/synapse/config/_base.py", line 794, in parse_config_dict
    self.invoke_all(
  File "/usr/local/lib/python3.10/dist-packages/synapse/config/_base.py", line 393, in invoke_all
    res[config_class.section] = getattr(config, func_name)(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/synapse/config/repository.py", line 141, in read_config
    self.media_store_path = self.ensure_directory(
  File "/usr/local/lib/python3.10/dist-packages/synapse/config/_base.py", line 242, in ensure_directory
    os.makedirs(dir_path, exist_ok=True)
  File "/usr/lib/python3.10/os.py", line 225, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/media_store'

The user with uid/gid 1004 has full permissions to the /srv/matrix and /srv/matrix/data directories but they maybe need some more permissions I don't know of.

The distro is CentOS 7

[mvgorcum]

What should happen is that on /srv/matrix/data you see a folder media_store that uid/gid 1004 has full rights to, it's possible that synapse is configured to try to place it's media store in /media_store inside the container, which would indeed cause this crash. It's also possible that 1004 doesn't have access to the media_store folder in /srv/matrix/data/media_store.

[redgoat650]

I think the problem here is that a few months back, synapse upstream changed their config yaml generation such that media_store_path (for one) no longer has its value in quotes.

The awk invocation in configure_homeserver_yaml is explicitly looking for the value "/media_store", in quotes.

After running generate, I see media_store_path: /media_store in the config file, no quotes. It was supposed to have been replaced by awk to read "/data/media_store" if the path had been in quotes. Instead we launch with /media_store, and we hit permission issues trying to mkdir in the root directory.

I tried poking around for a workaround, via environment variables or something, but no dice; seems we'll have to update the paths in the yaml file manually, and the fix will have to be made here. The awk can be updated to expect optional " for the values it's substituting, should do the trick.

Anyone know if the data directory can be parameterized when calling python3 -m synapse.app.homeserver? If so, that might be a more robust solution than an awk substitution after the generation completes.