Pub/sub message queue refactor

I am highly available now! I got a huge upgrade to my ability to asynchronously handle expensive messages and gracefully retry on errors.

before

I used the Slack Real Time Messaging API to maintain a stateful websocket connection with Slack and handled messages as they came.

This made me incredibly stateful. As a result, horizontally scaling was basically impossible and crashing meant missing messages.

after

Now I use the Slack Events API and a publish/subscribe event queue worker model to handle messages.

At the cost of the deployment diagram becoming a bit more complex:

• My updates can rolled out incrementally with high availability!
• Plenty of chances to gracefully retry failed messages.