Opensearch dual write

[conorbros]

No description provided.

[benbromhead]

As discussed yesterday, what does the failure path look like? Could we have a failure subchain where failed writes can get either logged, or written to a Kafka queue or something?

[conorbros]

My plans for failure retries was to either:

• build a Retry transform, where failed messaged would be sent down the chain again for a specified number of times
• build the retry functionality into the OpenSearchSink

When the retries fail, I think logging should be sufficient. Logging means that the failed writes to specific documents or indexes could be included in a reindex command after the Shotover dual writing operation is finished even. https://opensearch.org/docs/latest/im-plugin/reindex-data/