Add decision about the HA model to be used with the s3gw with Longhorn

[giubacc]

Describe your changes

Add decision about the HA model to be used with the s3gw with Longhorn

Signed-off-by: Giuseppe Baccini [email protected]

Issue ticket number and link

Related to: https://github.com/aquarist-labs/s3gw/issues/361

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• CHANGELOG.md has been updated should there be relevant changes in this PR.

[tserong]

I know the full details are in the research docs, but I wonder if we should add a very brief sentence to provide a little more detail about each mode? Maybe something like this (assuming I've managed to capture the essence):

• Active/Active (multiple s3gw instances concurrently serving the same data)
• Active/Warm Standby (multiple s3gw instances, one serving data, others able to take over if active instance fails)
• Active/Standby (single s3gw instance, with Kubernetes restarting/redeploying as necessary on failure)

Otherwise, LGTM. I assume we'll update the PR links later to point to the research docs once that PR is merged.

[giubacc]

I know the full details are in the research docs, but I wonder if we should add a very brief sentence to provide a little more detail about each mode? Maybe something like this (assuming I've managed to capture the essence):

• Active/Active (multiple s3gw instances concurrently serving the same data)
• Active/Warm Standby (multiple s3gw instances, one serving data, others able to take over if active instance fails)
• Active/Standby (single s3gw instance, with Kubernetes restarting/redeploying as necessary on failure)

Otherwise, LGTM. I assume we'll update the PR links later to point to the research docs once that PR is merged.

that makes perfectly sense, will change; moreover, pointing the PR is something not ideal probably, it will make more sense point the HA document directly once the corresponding PR will be merged.

[jecluis]

would be nice to have a small bullet point list of what are the perceived "best" HA characteristics.

[jecluis]

either "Pull Request" or "pull request". I'm partial towards the latter.

[giubacc]

For the time being I will go with all downcase; anyway, I'm not sure we should point the PR; perhaps better link the HA document once merged?

[l-mb]

A key part that we also need to address in our stack is the ingress layer, since we can hide many problems behind a proxy that stalls the clients, and the time of making it connect to a new backend affects our timings as well.

[giubacc]

A key part that we also need to address in our stack is the ingress layer, since we can hide many problems behind a proxy that stalls the clients, and the time of making it connect to a new backend affects our timings as well.

Currently, with Traefik, we are getting a "NotFound: Not Found\n\tstatus code: 404, request id: , host id: " when the s3gw backend service is down.
Could be better to address Ingress specific investigations in a separate thread?
Maybe we can write in this ADR that the Ingress part needs a specific investigation effort and we reserve to do this with a separate dedicated issue/epic?

[irq0]

Currently, with Traefik, we are getting a "NotFound: Not Found\n\tstatus code: 404, request id: , host id: " when the s3gw backend service is down.

That is not useful - clients won't retry a 404. 503 Service Unavailable would be a good option as clients like boto3 retry them https://github.com/boto/boto3/blob/develop/docs/source/guide/retries.rst

[l-mb]

Basically, if we push the ingress out - just like node failure recovery, or pre-loading the images - the actual ADR boils down to "s3gw needs to be able to be crash-consistent and start up fast after such an outage, because our active/cold standby model relies on the framework around us to achieve HA."

Then, this ADR would set a worst-case recovery goal (crashed under heavy load, say) of <1-5s for the s3gw restart part of the full stack.

That's fair (and mostly true), but then we need to immediately start on the next LEP to discuss how to achieve HA goals (RTO < 30s or some such goal) end-to-end. (And how to measure/validate those numbers as part of our E2E system testing, perhaps using k6?)

[giubacc]

Basically, if we push the ingress out - just like node failure recovery, or pre-loading the images - the actual ADR boils down to "s3gw needs to be able to be crash-consistent and start up fast after such an outage, because our active/cold standby model relies on the framework around us to achieve HA."

Then, this ADR would set a worst-case recovery goal (crashed under heavy load, say) of <1-5s for the s3gw restart part of the full stack.

That's fair (and mostly true), but then we need to immediately start on the next LEP to discuss how to achieve HA goals (RTO < 30s or some such goal) end-to-end. (And how to measure/validate those numbers as part of our E2E system testing, perhaps using k6?)

I have the impression/feeling that the activity on the ingress thing (testing, researching modes that fit us, etc) is something that needs a proper allocation in terms of time at management level. That is something that must be agreed, especially now we are more committed in integrating with LH.

[jecluis]

@l-mb is there anything else you'd like to see done here?

[giubacc]

@jecluis @l-mb ping

[jecluis]

no idea how this is passing linting. Regardless, mind fixing this line to a proper column size? Maybe move the link to a ref at the end of the document to mitigate the noise?

[giubacc]

I've instead linked the now available HA document directly with its natural project path.
That is much more short and it looks definitely better.