SSL_connect SYSCALL returned=5 errno=0 state=SSLv3/TLS write finished

[shootingfly]

Background:
I had four amazon cloud machines (docker) s1, s2, s3, s4, I meet a exception in one of amazon cloud machines
It was a holiday, the exception happened suddenly in s1. When I restart my rails server (docker) in 2 days ago, everything is normal.

Bunny Version: 2.19.0
Rabbitmq Version: 3.8.34

Exception Message:

OpenSSL::SSL::SSLError
SSL_connect SYSCALL returned=5 errno=0 state=SSLv3/TLS write finished

Exception Stack: 
OpenSSL::SSL::SSLError: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3/TLS write finished
  from bunny (2.19.0) lib/bunny/transport.rb:87:in `connect'
  from bunny (2.19.0) lib/bunny/transport.rb:87:in `connect'
  from bunny (2.19.0) lib/bunny/session.rb:326:in `start'
  from sneakers (2.11.0) lib/sneakers/publisher.rb:31:in `connect!'
  from sneakers (2.11.0) lib/sneakers/publisher.rb:24:in `block in ensure_connection!'
  from sneakers (2.11.0) lib/sneakers/publisher.rb:23:in `synchronize'
  from sneakers (2.11.0) lib/sneakers/publisher.rb:23:in `ensure_connection!'
  from sneakers (2.11.0) lib/sneakers/publisher.rb:15:in `publish'
  from sneakers_packer (0.1.5) lib/sneakers_packer.rb:26:in `block in publish'
  from sneakers_packer (0.1.5) lib/sneakers_packer.rb:25:in `synchronize'
  from sneakers_packer (0.1.5) lib/sneakers_packer.rb:25:in `publish'
  from app/workers/runner_worker.rb:62:in `perform'
  from sidekiq (6.5.1) lib/sidekiq/processor.rb:196:in `execute_job'
  from sidekiq (6.5.1) lib/sidekiq/processor.rb:164:in `block (2 levels) in process'
  from sidekiq (6.5.1) lib/sidekiq/middleware/chain.rb:144:in `block in invoke'
  from request_store-sidekiq (0.1.0) lib/request_store/sidekiq/server_middleware.rb:5:in `call'
  from sidekiq (6.5.1) lib/sidekiq/middleware/chain.rb:146:in `block in invoke'
  from sentry-raven (3.1.2) lib/raven/integrations/sidekiq/cleanup_middleware.rb:7:in `call'
  from sidekiq (6.5.1) lib/sidekiq/middleware/chain.rb:146:in `block in invoke'
  from sidekiq (6.5.1) lib/sidekiq/middleware/chain.rb:149:in `invoke'
  from sidekiq (6.5.1) lib/sidekiq/processor.rb:163:in `block in process'
  from sidekiq (6.5.1) lib/sidekiq/processor.rb:136:in `block (6 levels) in dispatch'
  from sidekiq (6.5.1) lib/sidekiq/job_retry.rb:115:in `local'
  from sidekiq (6.5.1) lib/sidekiq/processor.rb:135:in `block (5 levels) in dispatch'
  from sidekiq (6.5.1) lib/sidekiq/rails.rb:14:in `block in call'
  from activesupport (7.0.3.1) lib/active_support/execution_wrapper.rb:92:in `wrap'
  from activesupport (7.0.3.1) lib/active_support/reloader.rb:72:in `block in wrap'
  from activesupport (7.0.3.1) lib/active_support/execution_wrapper.rb:92:in `wrap'
  from activesupport (7.0.3.1) lib/active_support/reloader.rb:71:in `wrap'
  from sidekiq (6.5.1) lib/sidekiq/rails.rb:13:in `call'
  from sidekiq (6.5.1) lib/sidekiq/processor.rb:131:in `block (4 levels) in dispatch'
  from sidekiq (6.5.1) lib/sidekiq/processor.rb:257:in `stats'
  from sidekiq (6.5.1) lib/sidekiq/processor.rb:126:in `block (3 levels) in dispatch'
  from sidekiq (6.5.1) lib/sidekiq/job_logger.rb:13:in `call'
  from sidekiq (6.5.1) lib/sidekiq/processor.rb:125:in `block (2 levels) in dispatch'
  from sidekiq (6.5.1) lib/sidekiq/job_retry.rb:82:in `global'
  from sidekiq (6.5.1) lib/sidekiq/processor.rb:124:in `block in dispatch'
  from sidekiq (6.5.1) lib/sidekiq/job_logger.rb:39:in `prepare'
  from sidekiq (6.5.1) lib/sidekiq/processor.rb:123:in `dispatch'
  from sidekiq (6.5.1) lib/sidekiq/processor.rb:162:in `process'
  from sidekiq (6.5.1) lib/sidekiq/processor.rb:78:in `process_one'
  from sidekiq (6.5.1) lib/sidekiq/processor.rb:68:in `run'
  from sidekiq (6.5.1) lib/sidekiq/component.rb:8:in `watchdog'
  from sidekiq (6.5.1) lib/sidekiq/component.rb:17:in `block in safe_thread'

[michaelklishin]

See RabbitMQ logs (on all nodes) for clues, as well as Troubleshooting TLS.

I cannot suggest more without seeing your code, certificate and server logs. Guessing is a very expensive way of troubleshooting infrastructure.

[michaelklishin]

Some googling around suggests this is a symptom of the server process (a RabbitMQ node or a proxy of some kind) running out of file handles.

See Open File Handle Limit.

Finally, RabbitMQ 3.8 has reached EOL one year ago.

[shootingfly]

@michaelklishin Thanks, I will study your suggestions. I will check Open File Handle Limit and TLS problem.
The log was lost after restart server, and the rabbitmq server is belongs to my customer, not me.