Add connection pooling for dagster_postgres lib

[Lukashab]

Recently, I ran into an issue with dagster_postgres library #19222 . When I use postgres as a storage for my dagster architecture, I sometimes get "too many clients" error, especially when multiple jobs are running in parallel and there are concurrency slots set up for some ops.
I was wondering why there is no pooling mechanism when connecting to postgres instance.
For example here:

dagster/python_modules/libraries/dagster-postgres/dagster_postgres/event_log/event_log.py

Line 91 in d3720b4

self._engine = create_engine(

Is there any reason behind this logic? Because for dagster instance with multiple shcedules and jobs running, this can cause failures very quickly and there is simply no other way how to avoid it than setting external pooling mechanism or extending the connection limit of the postgres instance.
Thank you

[the4thamigo-uk]

I suppose by 'external pooling mechanism' you mean https://www.pgbouncer.org/ or similar?

[Lukashab]

Yes exactly. Pgbouncer is what I meant. But, as I said, i would rather have possibility to pool the connections on the dagster's side as well.

[the4thamigo-uk]

In also would love this feature as we are struggling with similar issues.