Publish answers to question topic

[cortadocodes]

Summary

This pull request removes the use of answer topics by publishing answer messages to the service revision (formerly known as question) topic and filtering subscriptions to only receive a) questions or b) response messages to a specific question. This speeds up the question asking process, reduces cloud infrastructure requirements and the permissions surface, and allows us to avoid topic number limits.

Also added is validation of all messages and their attributes against a new publicly available schema. This ensures services are communicating as they should and opens up the possibility of writing services in other languages and creating emulators.

As this, by itself, constitutes an inter-service communication breaking change, we've taken the opportunity to reduce the complexity of the codebase by removing backwards compatibility patches for service communication (i.e. we've grouped multiple breaking changes together into one).

Contents (#603)

IMPORTANT: There are 7 breaking changes.

New features

• Validate messages and their attributes against new service communication schema (see #614 for changelog - it was merged into this branch)
• Allow diagnostics (formerly known as crash diagnostics) to always be switched on for a service

Enhancements

• 💥 BREAKING CHANGE: Publish responses to questions to the service revision (question) topic instead of creating a separate answer topic
• 💥 BREAKING CHANGE: Store message number in message attributes instead of in message data
• 💥 BREAKING CHANGE: Remove question UUID from log record message body
• 💥 BREAKING CHANGE: Remove inter-service communication backwards compatibility code
• 💥 BREAKING CHANGE: Make input and output values and manifest optional
• 💥 BREAKING CHANGE: Replace boolean allow_save_diagnostics_data_on_crash argument with string/enum save_diagnostics argument in Service.ask and related methods
• Add ability to filter subscriptions
• Add question UUID attribute to all messages
• Send more possible errors to parent in Service.answer
• Add kind field to question messages
• Add sender_type attribute to all messages
• Add ability to instantiate Runner from service/app configurations

Fixes

• Stop double-JSON-encoding output manifests

Dependencies

• Update octue version in template apps' dependencies

Refactoring

• 💥 BREAKING CHANGE: Rename crash diagnostics to diagnostics
• Group message attributes in Service._send_message and MockMessage under explicit attributes argument
• Make OrderedMessageHandler._waiting_messages attribute public
• Rename various message attributes

Testing

• Store mock pub/sub messages against subscriptions instead of topics
• Add missing type field to emulated Pub/Sub questions

Operations

• Fix add-issues-to-octue-board workflow
• Stop automatically building docker images for registry in release workflow
• Add ReadTheDocs config file to fix documentation building

---

Upgrade instructions

💥 Update all Octue services in your network to use this version of octue so they're still able to communicate. Postpone upgrading until you can upgrade all services simultaneously.

💥 Replace allow_save_diagnostics_data_on_crash with save_diagnostics set to one of these values: "SAVE_DIAGNOSTICS_OFF", "SAVE_DIAGNOSTICS_ON_CRASH", or "SAVE_DIAGNOSTICS_ON"

💥 Crash diagnostics rename:

• Use the octue get-diagnostics CLI command instead of the octue get-crash-diagnostics command
• Rename crash_diagnostics_cloud_path in your service configurations to diagnostics_cloud_path

[codecov-commenter]

Codecov Report

Attention: 16 lines in your changes are missing coverage. Please review.

Comparison is base (19cfd4c) 95.03% compared to head (1902194) 94.90%.
Report is 1 commits behind head on main.

Files	Patch %	Lines
octue/cloud/validation.py	73.91%	6 Missing ⚠️
octue/cloud/pub_sub/events.py	87.50%	2 Missing ⚠️
octue/cloud/pub_sub/service.py	95.83%	2 Missing ⚠️
octue/runner.py	88.88%	2 Missing ⚠️
...cloud/deployment/google/answer_pub_sub_question.py	80.00%	1 Missing ⚠️
octue/cloud/emulators/_pub_sub.py	97.14%	1 Missing ⚠️
octue/cloud/pub_sub/message_handler.py	96.15%	1 Missing ⚠️
octue/diagnostics.py	75.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #603      +/-   ##
==========================================
- Coverage   95.03%   94.90%   -0.14%     
==========================================
  Files          75       77       +2     
  Lines        3322     3394      +72     
==========================================
+ Hits         3157     3221      +64     
- Misses        165      173       +8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.