Landscape is down after psql patroni.service restart

[anna-savchenko]

Steps to reproduce

1. Deploy a landscape bundle with Postgresql
2. Restart snap.charmed-postgresql.patroni.service on the leader unit

Expected behavior

Landscape responds.

Actual behavior

Landscape does not respond.
We suspect that the issue started after the restart of patroni.service with sudo systemctl restart snap.charmed-postgresql.patroni.service.
We did not see when exactly the landscape stopped responding, but there weren't any other changes to the system apart from this restart.

Versions

Operating system: Ubuntu 22.04.4 LTS

Juju CLI: 3.5.3-genericlinux-amd64

Juju agent: 3.5.2

Charm revision: 14/stable rev.429

LXD: N/A

Log output

Aug 1 12:17:12 appserver-2 ERR https://redacted_VIP/index.html request_id=b0239d98-061e-438e-baba-7d17257ab1b4#012Traceback (most recent call last):#12 File "/usr/lib/python3/dist-packages/storm/exceptions.py", line 165, in wrap_exceptions#012 yield#012 File "/usr/lib/python3/dist-packages/storm/database.py", line 187, in wrapper#012 return attr(*args, **kwargs)#12 ^^^^^^^^^^^^^^^^^^^^^#012psycopg2.errors.ReadOnlySqlTransaction: cannot execute SELECT FOR UPDATE in a read-only transaction#012CONTEXT: SQL statement "SELECT generation, access_time FROM session_data#012 WHERE session_data.client_id = provided_client_id FOR UPDATE"#012PL/pgSQL function get_generation(bytea,interval,interval) line 18 at SQL statement#012#012#012The above exception was the direct cause of the following exception:#12#012Traceback (most recent call last):#12 File "/usr/lib/python3/dist-packages/zope/publisher/publish.py", line 145, in publish#012 result = publication.callObject(request, obj)#12 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#12 File "/usr/lib/python3/dist-packages/zope/app/publication/zopepublication.py", line 207, in callObject#012 return mapply(ob, request.getPositionalArguments(), request)#12 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#12 File "/usr/lib/python3/dist-packages/zope/publisher/publish.py", line 118, in mapply#012 return debug_call(obj, args)#12 ^^^^^^^^^^^^^^^^^^^^^#12 File "/usr/lib/python3/dist-packages/zope/publisher/publish.py", line 125, in debug_call#012 return obj(*args)#12 ^^^^^^^^^^#12 File "/opt/canonical/landscape/canonical/landscape/ui/root/page.py", line 250, in call#12 errors = session["landscape.authentication.error"]#12 ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#12 File "/usr/lib/python3/dist-packages/zope/session/session.py", line 527, in getitem#12 sd = sdc[self.client_id]#12 ~~~^^^^^^^^^^^^^^^^#12 File "/opt/canonical/landscape/canonical/landscape/ui/session/session.py", line 74, in getitem#12 result = store.execute(#12 ^^^^^^^^^^^^^^#12 File "/usr/lib/python3/dist-packages/storm/store.py", line 118, in execute#012 return self._connection.execute(statement, params, noresult)#12 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#12 File "/usr/lib/python3/dist-packages/storm/databases/postgres.py", line 306, in execute#012 return Connection.execute(self, statement, params, noresult)#12 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#12 File "/usr/lib/python3/dist-packages/storm/database.py", line 323, in execute#012 raw_cursor = self.raw_execute(statement, params)#12 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#12 File "/usr/lib/python3/dist-packages/storm/database.py", line 457, in raw_execute#012 self._run_execution(raw_cursor, args, params, statement)#12 File "/usr/lib/python3/dist-packages/storm/database.py", line 471, in _run_execution#012 self._check_disconnect(raw_cursor.execute, *args)#12 File "/usr/lib/python3/dist-packages/storm/database.py", line 537, in _check_disconnect#012 return function(*args, **kwargs)#12 ^^^^^^^^^^^^^^^^^^^^^^^^^#12 File "/usr/lib/python3/dist-packages/storm/database.py", line 186, in wrapper#012 with wrap_exceptions(self._database):#12 File "/usr/lib/python3.12/contextlib.py", line 158, in exit#12 self.gen.throw(value)#12 File "/usr/lib/python3/dist-packages/storm/exceptions.py", line 185, in wrap_exceptions#012 six.raise_from(wrapped.with_traceback(tb), e)#12 File "", line 3, in raise_from#012 File "/usr/lib/python3/dist-packages/storm/exceptions.py", line 165, in wrap_exceptions#012 yield#012 File "/usr/lib/python3/dist-packages/storm/database.py", line 187, in wrapper#012 return attr(*args, **kwargs)#12 ^^^^^^^^^^^^^^^^^^^^^#012storm.database.ReadOnlySqlTransaction: cannot execute SELECT FOR UPDATE in a read-only transaction#012CONTEXT: SQL statement "SELECT generation, access_time FROM session_data#012 WHERE session_data.client_id = provided_client_id FOR UPDATE"#012PL/pgSQL function get_generation(bytea,interval,interval) line 18 at SQL statement
Aug 1 12:17:12 appserver-2 ERR Dumping ['OOPS-0843e065730d53b777f2a1b5352b9f06'].

Additional context

The snap.charmed-postgresql.patroni.service is Active on all 3 units

[github-actions]

https://warthogs.atlassian.net/browse/DPE-5020

[marceloneppel]

Discussion taking place on Matrix: https://matrix.to/#/!BukWfnyOTgQSKAxdtT:ubuntu.com/$BqPCgChzCkr_IcxqwCSikswiFHUsyq2iFXLOtfNXIjc?via=ubuntu.com&via=matrix.org&via=laquadrature.net

[taurus-forever]

@anna-savchenko what should we do with this ticket?
The Matrix troubleshooting went into lack of LDAP support, but I prefer a dedicated Jira Epic for @7annaba3l to sign first.
LDAP story is not a bugfix to handle here. Tnx!

[anna-savchenko]

Hey @taurus-forever
I plan to deploy a Landscape bundle on a testing environment without LDAP integration and try patroni.service restart to see if I can reproduce this.

[dragomirp]

Hi, @anna-savchenko, were you able to reproduce?

[taurus-forever]

@anna-savchenko we have no reply here for a while and meanwhile (it looks like) Landscape has addressed the reported here issue in https://bugs.launchpad.net/landscape-charm/+bug/2076143

I am resolving this issue, feel free to reopen if still topical. Tnx!