PG::ConnectionBadqc:work PQconsumeInput() connection not open

[udfalkso]

How can I fix this issue? It's happening in production but not development. There is an issue from 2013 that seems relevant, but slightly different. My setup with rails is totally straightforward, no shared connections, etc. Any ideas would be appreciated. Thanks!

bundler/gems/queue_classic-4cdc9b8e804b/lib/queue_classic/conn_adapter.rb:62:in `wait_for_notify': PQconsumeInput() connection not open
 (PG::ConnectionBad)
    from bundler/gems/queue_classic-4cdc9b8e804b/lib/queue_classic/conn_adapter.rb:62:in `block in wait_for_notify'
    from bundler/gems/queue_classic-4cdc9b8e804b/lib/queue_classic/conn_adapter.rb:61:in `tap'
    from bundler/gems/queue_classic-4cdc9b8e804b/lib/queue_classic/conn_adapter.rb:61:in `wait_for_notify'
    from bundler/gems/queue_classic-4cdc9b8e804b/lib/queue_classic/conn_adapter.rb:34:in `block in wait'
    from bundler/gems/queue_classic-4cdc9b8e804b/lib/queue_classic/conn_adapter.rb:31:in `synchronize'
    from bundler/gems/queue_classic-4cdc9b8e804b/lib/queue_classic/conn_adapter.rb:31:in `wait'
    from bundler/gems/queue_classic-4cdc9b8e804b/lib/queue_classic/worker.rb:96:in `lock_job'
    from bundler/gems/queue_classic-4cdc9b8e804b/lib/queue_classic/worker.rb:72:in `work'
    from bundler/gems/queue_classic-4cdc9b8e804b/lib/queue_classic/worker.rb:46:in `start'
    from bundler/gems/queue_classic-4cdc9b8e804b/lib/queue_classic/tasks.rb:27:in `block (2 levels) in <main>'
    from gems/rake-12.3.1/lib/rake/task.rb:271:in `block in execute'
    from gems/rake-12.3.1/lib/rake/task.rb:271:in `each'
    from gems/rake-12.3.1/lib/rake/task.rb:271:in `execute'
    from gems/bugsnag-6.11.1/lib/bugsnag/integrations/rake.rb:18:in `execute_with_bugsnag'
    from gems/rake-12.3.1/lib/rake/task.rb:213:in `block in invoke_with_call_chain'
    from /usr/local/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
    from gems/rake-12.3.1/lib/rake/task.rb:193:in `invoke_with_call_chain'
    from gems/rake-12.3.1/lib/rake/task.rb:182:in `invoke'
    from gems/rake-12.3.1/lib/rake/application.rb:160:in `invoke_task'
    from gems/rake-12.3.1/lib/rake/application.rb:116:in `block (2 levels) in top_level'
    from gems/rake-12.3.1/lib/rake/application.rb:116:in `each'
    from gems/rake-12.3.1/lib/rake/application.rb:116:in `block in top_level'
    from gems/rake-12.3.1/lib/rake/application.rb:125:in `run_with_threads'
    from gems/rake-12.3.1/lib/rake/application.rb:110:in `top_level'
    from gems/rake-12.3.1/lib/rake/application.rb:83:in `block in run'
    from gems/rake-12.3.1/lib/rake/application.rb:186:in `standard_exception_handling'
    from gems/rake-12.3.1/lib/rake/application.rb:80:in `run'
    from gems/rake-12.3.1/exe/rake:27:in `<top (required)>'
    from bin/rake:23:in `load'
    from bin/rake:23:in `<top (required)>'
    from /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli/exec.rb:74:in `load'
    from /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli/exec.rb:74:in `kernel_load'
    from /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli/exec.rb:28:in `run'
    from /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli.rb:463:in `exec'
    from /usr/local/lib/ruby/site_ruby/2.5.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
    from /usr/local/lib/ruby/site_ruby/2.5.0/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
    from /usr/local/lib/ruby/site_ruby/2.5.0/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
    from /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli.rb:27:in `dispatch'
    from /usr/local/lib/ruby/site_ruby/2.5.0/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
    from /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli.rb:18:in `start'
    from /usr/local/bin/bundle:30:in `block in <main>'
    from /usr/local/lib/ruby/site_ruby/2.5.0/bundler/friendly_errors.rb:124:in `with_friendly_errors'
    from /usr/local/bin/bundle:22:in `<main>'

[siegy22]

I'm running into an issue where PG::ConnectionBad: connection is closed happens. I recently upgraded to rails 5.2 which seems to be the issue here.

Do you use rails or just plain queue_classic? If so what version? Do you use the automatic connection sharing, or did you disable it by setting QC_RAILS_DATABASE=false?

I think to help resolve this, the maintainers will need a lot more information that just stacktrace 😄

[vongruenigen]

I'm running into an issue where PG::ConnectionBad: connection is closed happens. I recently upgraded to rails 5.2 which seems to be the issue here.
...

I run into the same issue currently and have no idea how to fix it besides fiddling with queue_classic internas. Happened after I've upgraded our application from rails 4.x to 5.2. It seems to only happen from time to time, non-deterministic, and also just in production but not in development.

[siegy22]

@vongruenigen Use QC_RAILS_DATABASE=true, fixed it for me..

I was able to reproduce it, when a high load appeared on my web site. After a while of lower load, the PG::ConnectionBad appeared. (Probably some garbage collector cleaning up AR connections without queue classic noticing it)

[vongruenigen]

@siegy22 Thanks, I'll immediately try QC_RAILS_DATABASE=true. I also have the problem that the problem occurs with really low load, namely only me being the only user manually testing stuff on our app on a test server.

We'll probably switch to specifying the database via QC_DATABASE_URL in the long run, but I'll have to wait until one of our DevOps show up to delegate this task 😄

[siegy22]

@vongruenigen You can also try out my fork, I've a applied a small patch. Which should make the connection sharing with active record work again. 😄

https://github.com/siegy22/queue_classic

[vongruenigen]

@siegy22, thanks, I'll probably try it later somewhen 👍 Is the fix hacky? If not, maybe you could consider opening a PR.

[vongruenigen]

@siegy22, have you tried setting the ENV['QC_DATABASE_URL'] directly in application.rb? Like so:

if ENV['QC_RAILS_DATABASE'].blank? && ENV['QC_DATABASE_URL'].blank?
  db_conf = config.database_configuration[Rails.env].symbolize_keys
  db_url = "#{db_conf[:adapter]}://#{db_conf[:username]}"

  # more string fiddling ...

  ENV['QC_DATABASE_URL'] = db_url.freeze
  ENV['QC_RAILS_DATABASE'] = false.to_s
end

This seems to work for me too, but I'm not happy with building the database URL by concatenating strings tbh.

[siegy22]

Yeah I've tried several ways to do what you're attempting to do, but always found a new way which made it fail and ended up running from my fork for now.

I'm trying to get this fixed upstream, and maybe even get a release 😄

[ukd1]

@siegy22 I'm checking out your code ^ -- looks like it might be backwards incompatible?

[siegy22]

Check my comment in your PR: #312
@ukd1 😄

[kolen]

@siegy22 this might be due to connections taken with ActiveRecord::Base.connection are released to the pool by Action Pack's request handling cycle. To lease connection properly (according to AR docs), ActiveRecord::Base.connection_pool.checkout should be used.

I opened #317, as it looks distinct from the problem described initially here: PQconsumeInput() might be caused just by TCP connection closed or other causes, as opposed to PG::ConnectionBad: connection is closed which is caused only by .close being called on connection object: this closed state is distinct from disconnect and can't be recovered by reset. (Pool might close connections it in e.g. flush).

[pkotnis]

Any chances rails/rails#36473 (released 2 days ago as part of Rails 2.5.4) fixed the problem?

[kolen]

@pkotnis no, QueueClassic does not call connections method (it doesn't need a list of all connections).