connection resume/recover #167
Conversation
sacOO7
force-pushed
the
refactor/resume-recover
branch
2 times, most recently
from
1ecc934 to
d63fa70
Compare
sacOO7
force-pushed
the
refactor/resume-recover
branch
from
d63fa70 to
ef36b77
Compare
| ** @(RTN15c)@ The system's response to a resume request (with a non-empty connectionKey) will be one of the following: | ||
| **** @(RTN15c6)@ A @CONNECTED@ @ProtocolMessage@ with the same @connectionId@ as the current client (and no @error@ property). This indicates that the resume attempt was valid. | ||
| **** @(RTN15c7)@ @CONNECTED@ @ProtocolMessage@ with a new @connectionId@ and an @ErrorInfo@ in the @error@ field. In this case, the resume was invalid, and the error indicates the cause. The @error@ should be set as the @reason@ in the @CONNECTED@ event, and as the @Connection#errorReason@. The internal @msgSerial@ counter should be reset so that the first message published to Ably will contain a @msgSerial@ of @0@. | ||
| **** @(RTN15c8)@ Irrespective of success/failure for a resume request, the client library should move all channels that were in the @ATTACHING@, @ATTACHED@, or @SUSPENDED@ states to the @ATTACHING@ state, and initiate an @RTL4c@ attach sequence for each. The connection should also process any messages queued per @RTL6c2@ (there is no need to wait for the attaches to finish before processing queued messages). |
There was a problem hiding this comment.
But this item is a child of a spec item saying "The system's response to a resume request (with a non-empty connectionKey) will be one of the following:". Every other child is one of the four things that can happen:
(RTN15c) The system’s response to a resume request will be one of the following:
↳ CONNECTED, same connId
↳ CONNECTED, new connId
↳ token ERROR
↳ other ERROR
Adding an item in the middle that's actually is a shared spec item for the first two of the items doesn't make sense
There was a problem hiding this comment.
Okay, though the first 2 do exhibit a common behavior for attaching channels and processing pending messages.
There was a problem hiding this comment.
I can remove this if not needed. Though I am not sure if resume failure applies here too as per #167 (comment)
| *** @(RTN19a2)@ In the case of an @RTN15c6@ successful resume, the @msgSerial@ of the reattempted @ProtocolMessage@s should remain the same as for the original attempt. In the case of an @RTN15c7@ failed resume, the message must be assigned a new @msgSerial@ from the SDK's internal counter. | ||
| ** @(RTN19a)@ Any @ProtocolMessage@ that is awaiting an @ACK@/@NACK@ on the old transport will not receive the @ACK@/@NACK@ on the new transport (failed resume as per @RTN15c7@). The client library must therefore resend any @ProtocolMessage@ that is awaiting a @ACK@/@NACK@ to Ably in order to receive the expected @ACK@/@NACK@ for that message (subject to @RTN7c@/@RTN7d@). The Ably service is responsible for keeping track of messages, ignoring duplicates and responding with suitable @ACK@/@NACK@ messages | ||
| *** @(RTN19a1)@ One possible implementation of this requirement would be to add those @ACK@/@NACK@ messages to the @RTL6c2@ connection-wide queue of messages that will be sent once the connection becomes @CONNECTED@. | ||
| *** @(RTN19a2)@ In the case of an @RTN15c6@ successful resume, the @msgSerial@ of the reattempted @ProtocolMessage@s remains unchanged. In the case of an @RTN15c7@ failed resume, each message must be assigned a new incremental @msgSerial@ from the Connection#MessageSerial. |
There was a problem hiding this comment.
remains unchanged
why the change to the present tense? that sentence is saying "don't change it in this case (only in the other case)"
There was a problem hiding this comment.
Or we should change it to
@(RTN19a2)@ Only in case of new connection Id, each message must be assigned a new incremental @msgSerial@ from the Connection#MessageSerial. This should automatically get covered, when pending connection queue messages (added at RTN19a1) are processed.
There was a problem hiding this comment.
Okay, Correct me if I am wrong, server connectionState doesn't maintain pending ack/nack on the server. It just maintains pending messages.
There was a problem hiding this comment.
Only in case of new connection Id
I'd really prefer it refer back to the spec item that defines the case. The relevant principle is DRY (Don't Repeat Yourself): if you define the same requirement in multiple places, with slightly different wording each time, the same logic gets implemented multiple times, and then if that needs changing it's way too easy to miss a place it needs to be changed in.
Okay, Correct me if I am wrong, server connectionState doesn't maintain pending ack/nack on the server. It just maintains pending messages.
yes
textile/features.textile
Outdated
| *** @(RTN19a1)@ One possible implementation of this requirement would be to add any in-flight messages to the @RTL6c2@ connection-wide queue of messages that will be sent once the connection next becomes @CONNECTED@ | ||
| *** @(RTN19a2)@ In the case of an @RTN15c6@ successful resume, the @msgSerial@ of the reattempted @ProtocolMessage@s should remain the same as for the original attempt. In the case of an @RTN15c7@ failed resume, the message must be assigned a new @msgSerial@ from the SDK's internal counter. | ||
| ** @(RTN19a)@ Any @ProtocolMessage@ that is awaiting an @ACK@/@NACK@ on the old transport will not receive the @ACK@/@NACK@ on the new transport (failed resume as per @RTN15c7@). The client library must therefore resend any @ProtocolMessage@ that is awaiting a @ACK@/@NACK@ to Ably in order to receive the expected @ACK@/@NACK@ for that message (subject to @RTN7c@/@RTN7d@). The Ably service is responsible for keeping track of messages, ignoring duplicates and responding with suitable @ACK@/@NACK@ messages | ||
| *** @(RTN19a1)@ One possible implementation of this requirement would be to add those @ACK@/@NACK@ messages to the @RTL6c2@ connection-wide queue of messages that will be sent once the connection becomes @CONNECTED@. |
There was a problem hiding this comment.
No, we're not adding ack/nacks to the queue, the sdk never generates or sends an ack or nack
There was a problem hiding this comment.
I meant Ack.Nack contained messages, not ack itself
There was a problem hiding this comment.
I was not able to understand in-flight messages at all : (
It should explicitly mention type of mesages we are adding to the queue.
There was a problem hiding this comment.
I meant Ack.Nack contained messages
I don't understand what you mean by this
The types of messages that would be requeued are ones that the library is waiting to hear an ACK or NACK from the server for, which is MESSAGE and PRESENCE protocolmessages per RTN7a
There was a problem hiding this comment.
Then we should mention sending MESSAGE and PRESENCE type messages associated with acks/nacks.
textile/features.textile
Outdated
| @@ -553,9 +554,9 @@ h3(#realtime-connection). Connection | |||
| ** @(RTN17e)@ If the realtime client is connected to a fallback host endpoint, then for the duration that the transport is connected to that host, all HTTP requests, such as history or token requests, should be first attempted to the same datacenter the realtime connection is established with i.e. the same fallback host must be used as the default HTTP request host. If however the HTTP request against that fallback host fails, then the normal fallback host behavior should be followed attempting the request against another fallback host as described in "RSC15":#RSC15 | |||
| * @(RTN19)@ Transport state side effects - when a transport is disconnected for any reason: | |||
| ** @(RTN19a)@ Any @ProtocolMessage@ that is awaiting an @ACK@/@NACK@ on the old transport will not receive the @ACK@/@NACK@ on the new transport. The client library must therefore resend any @ProtocolMessage@ that is awaiting a @ACK@/@NACK@ to Ably in order to receive the expected @ACK@/@NACK@ for that message (subject to @RTN7c@/@RTN7d@). The Ably service is responsible for keeping track of messages, ignoring duplicates and responding with suitable @ACK@/@NACK@ messages | |||
There was a problem hiding this comment.
Imagine, connection was in suspendedState, after several retries, it's connected to the server ( with new transport )
As per RTN8c and RTN9c we are clearing connectionKey and connectionId. This means a new connection attempt is not a resume attempt. So, there's neither resume failure nor resume success as a part of a new connection.
In this case, what should happen to ack/nack messages? Are we supposed to treat it as a resume failure and send ack/nack messages with incremental message-serial ? Same we do it for channel attach in case of RTN15g3
There was a problem hiding this comment.
Also, it feels we are attaching channels (in attaching, attached or suspended state ) for almost every type of reconnection 🤔
There was a problem hiding this comment.
Update - I checked the code, seems we clear ack/nack messages on suspended state, so we don't really have to worry about sending them. But can there be a case where a re-connection attempt is not a resume attempt in disconnected state?
There was a problem hiding this comment.
In this case, what should happen to ack/nack messages?
they'll have been cleared already per RTN7c
can there be a case where a re-connection attempt is not a resume attempt in disconnected state?
yes: before you're connected for the first time
| ** @(RTN15b)@ In order for a connection to be resumed and connection state to be recovered, the client must have received a @CONNECTED@ ProtocolMessage which will include a private connection key. To resume that connection, the library reconnects to the "websocket":https://ably.com/topic/websockets endpoint with an additional querystring param: | ||
| *** @(RTN15b1)@ @resume@ is the @ProtocolMessage#connectionKey@ from the most recent @CONNECTED@ @ProtocolMessage@ received | ||
| *** @(RTN15b2)@ This clause has been deleted. It was valid up to and including specification version @1.2@. | ||
| ** @(RTN15c)@ The system's response to a resume request will be one of the following: | ||
| **** @(RTN15c6)@ A @CONNECTED@ @ProtocolMessage@ with the same @connectionId@ as the current client (and no @error@ property). This indicates that the resume attempt was valid. The client library should move all channels that were in the @ATTACHING@, @ATTACHED@, or @SUSPENDED@ states to the @ATTACHING@ state, and initiate an @RTL4c@ attach sequence for each. The connection should also process any messages queued per @RTL6c2@ (there is no need to wait for the attaches to finish before processing queued messages). | ||
| **** @(RTN15c7)@ @CONNECTED@ @ProtocolMessage@ with a new @connectionId@ and an @ErrorInfo@ in the @error@ field. In this case, the resume was invalid, and the error indicates the cause. The @error@ should be set as the @reason@ in the @CONNECTED@ event, and as the @Connection#errorReason@. The internal @msgSerial@ counter should be reset so that the first message published to Ably will contain a @msgSerial@ of @0@. The rest of the process is the same as for @RTN16c6@: The client library should move all channels that were in the @ATTACHING@, @ATTACHED@, or @SUSPENDED@ states to the @ATTACHING@ state, and initiate an @RTL4c@ attach sequence for each. The connection should also process any messages queued per @RTL6c2@. |
There was a problem hiding this comment.
what if there was no resume attempt as a part of suspended retry, are we supposed to treat is as resume failed? That means we need to reset msgSerial in this case too 🤔
There was a problem hiding this comment.
Also, we will get a new connectionId but there won't be an error in the errorField. I think this third case is coming for almost every other spec that mentions just a dual behavior for resume success/failure.
There was a problem hiding this comment.
I think resume failure should be new connectionId != prev connectionId. We shouldn't have extra checks for errors as such. So, if there is no resume attempt (null key ), it will be treated as a resume failure.
There was a problem hiding this comment.
what if there was no resume attempt as a part of suspended retry, are we supposed to treat is as resume failed? That means we need to reset msgSerial in this case too 🤔
RTN15g: "If a client has been disconnected for longer than the @connectionStateTtl@, it should not attempt to resume. Instead, it should clear the local connection state, and any connection attempts should be made as for a fresh connection" -- the msgSerial is part of the connection state, so it will have been reset in this case. (Perhaps the spec should explicitly list the properties that should be cleared, for clarity?)
(see also RTN15g1 which explains why RTN15g is strictly stronger than just doing this on going into SUSPENDED)
Also, we will get a new connectionId but there won't be an error in the errorField. I think this third case is coming for almost every other spec that mentions just a dual behavior for resume success/failure.
There's two cases. Either connection state (including connectionKey and msgSerial) is still there, in which case the library will try and do a resume, and the possible cases are listed in RTN15c. Or the connection state has been cleared, including the msgSerial, in which case the library is just doing a clean connection. The sdk never stops trying to resume but keeps its msgSerial.
There was a problem hiding this comment.
In the ably-js, I can find messageSerial being cleared as a part of clearing connection state.
But, I couldn't find any references to clearing messageSerial as a part of https://sdk.ably.com/builds/ably/specification/main/features/#RTN8c and https://sdk.ably.com/builds/ably/specification/main/features/#RTN9c
| ** @(RTN19b)@ If there are any pending channels i.e. in the @ATTACHING@ or @DETACHING@ state, the respective @ATTACH@ or @DETACH@ message should be resent to Ably | ||
| *** @(RTN19a1)@ One possible implementation of this requirement would be to add those @ACK@/@NACK@ messages to the @RTL6c2@ connection-wide queue of messages that will be sent once the connection becomes @CONNECTED@. | ||
| *** @(RTN19a2)@ In the case of an @RTN15c6@ successful resume, the @msgSerial@ of the reattempted @ProtocolMessage@s remains unchanged. In the case of an @RTN15c7@ failed resume, each message must be assigned a new incremental @msgSerial@ from the Connection#MessageSerial. | ||
| ** @(RTN19b)@ If there are any pending channels i.e. in the @ATTACHING@ or @DETACHING@ state, the respective @ATTACH@ or @DETACH@ message should be resent to Ably once reconnected. |
There was a problem hiding this comment.
Can we make this spec more clear?
Does this mean, we tried to attach on old transport, but before receiving attached/detached, connection was disconnected. So, on new transport (resumed/non-resumed) we should resend these messages?
There was a problem hiding this comment.
Update - getting state to detached will work on new transport only if detach message is explicitly sent to old transport and connection is resumed.
For attached, it doesn't matter if connection is resumed/not-resumed, for any of the new transport, attach will work.
There was a problem hiding this comment.
this spec item is basically obselete: the 'send an attach for any attaching' part of this item is no longer needed, redundant to rtn15c6/7. and of course there's no need to send a DETACH for detaching channels any more. So it can just say "If there were any channels in the DETACHING state they should be moved to the DETACHED state"
There was a problem hiding this comment.
They should be explicitly moved to detached state or those channels will automatically receive detached message from the server on new transport?
There was a problem hiding this comment.
Btw, this is still not clear, as per my observsation Only if detach is sent on old transport, channel on new transport will be detached by sending new detach message. Channel on new transport won't automatically move to detached state : (
textile/features.textile
Outdated
| @@ -553,9 +554,9 @@ h3(#realtime-connection). Connection | |||
| ** @(RTN17e)@ If the realtime client is connected to a fallback host endpoint, then for the duration that the transport is connected to that host, all HTTP requests, such as history or token requests, should be first attempted to the same datacenter the realtime connection is established with i.e. the same fallback host must be used as the default HTTP request host. If however the HTTP request against that fallback host fails, then the normal fallback host behavior should be followed attempting the request against another fallback host as described in "RSC15":#RSC15 | |||
| * @(RTN19)@ Transport state side effects - when a transport is disconnected for any reason: | |||
| ** @(RTN19a)@ Any @ProtocolMessage@ that is awaiting an @ACK@/@NACK@ on the old transport will not receive the @ACK@/@NACK@ on the new transport. The client library must therefore resend any @ProtocolMessage@ that is awaiting a @ACK@/@NACK@ to Ably in order to receive the expected @ACK@/@NACK@ for that message (subject to @RTN7c@/@RTN7d@). The Ably service is responsible for keeping track of messages, ignoring duplicates and responding with suitable @ACK@/@NACK@ messages | |||
There was a problem hiding this comment.
Statement says The Ably service is responsible for keeping track of messages, ignoring duplicate. Since, messages published via realtimechannel doesn't have ID ( assigned by server ) when ack/nacks are republished, we receive duplicate messages for the same. This means, as of now, any realtimechannel that doesn't receive ack/nack on same transport will publish duplicate messages on new transport : (
There was a problem hiding this comment.
Update -
-
When resume is successful, resent ack/nack doesn't have duplicates -> https://github.com/ably/ably-dotnet/blob/201feb14f3934b3a1c6778dcddbf14deb9c17027/src/IO.Ably.Tests.Shared/Realtime/ConnectionSandboxTransportSideEffectsSpecs.cs#L199
-
When resume is not successful, resent ack/nack have duplicates ->
https://github.com/ably/ably-dotnet/blob/201feb14f3934b3a1c6778dcddbf14deb9c17027/src/IO.Ably.Tests.Shared/Realtime/ConnectionSandboxTransportSideEffectsSpecs.cs#L190C19-L190C19
Not sure how server actually checks for duplicates : (
My only guess is by checking message id's which are not present in both cases. Though for resume success, we have same connection id.
There was a problem hiding this comment.
Update - for the second case, since we are adding ack/nack to pending messages, is it a desired behavior?
There was a problem hiding this comment.
this means, as of now, any realtimechannel that doesn't receive ack/nack on same transport will publish duplicate messages on new transport
This is what RTN19a2 is for, which requires the re-send attempt (for a successful resume) to use the same msgSerial.
The message id for a realtime connection is implicitly connectionId:msgSerial. If your resume is successful, and you resend with the same msgId, you will be correctly deduplicated.
In the case of a resume failure, you're out of luck, so yes you can get duplicates.
| @@ -526,16 +527,16 @@ h3(#realtime-connection). Connection | |||
| ** @(RTN20c)@ When @CONNECTING@, if the operating system indicates that the underlying internet connection is now available, the client should restart the pending connection attempt | |||
| * @(RTN16)@ @Connection@ recovery: | |||
| ** @(RTN16i)@ Connection recovery is similar to connection resumption (see "RTN15c":#RTN15c), except that instead of the library resuming from a time at which that library instance was previously connected, it is doing so from external state provided in the client options, "(TO3i)":#TO3i. Since the library has no state at the time of connection, the channels must be explicitly attached by the user; continuity preservation is achieved by the @channelSerial@s for each channel being stored in the recovery key. | |||
| ** @(RTN16f)@ When a library is instantiated with the @recover@ client option, it should initialize its internal @msgSerial@ counter to the @msgSerial@ component of the @recoveryKey@. (If the recover fails, the counter should be reset to 0 per "RTN15c7":#RTN15c7 ) | |||
| ** @(RTN16f)@ When a library is instantiated with the @recover@ client option, it should initialize its internal @connectionId@ and @msgSerial@ counter to the @connectionId@ and @msgSerial@ component of the @recoveryKey@. (If the recover fails, the counter should be reset to 0 per "RTN15c7":#RTN15c7 ) | |||
There was a problem hiding this comment.
If we don't set connectionId, connectionId will not match for new connected instance, means resume is not successful as per RTN15c6.
I think this will not hamper implementation much since spec is almost the same for resume success/failure.
But this doesn't feel the right way when someone tries to debug the code.
There was a problem hiding this comment.
You could store the connectionId if you like, yes. But it's not strictly necessary, because you can just use whether there was an error property in the CONNECTED to decide whether the recover succeeded.
It's true that the spec is unclear on this. Maybe a sentence like "In the case of recover, in the absence of a connectionId to compare, the library can use whether there is an @error@ in the CONNECTED message to determine whether the recover succeeded"?
There was a problem hiding this comment.
Or better set the connectionId, then it will be in sync with resume. We don't really have to add extra conditions for this. Having extra error is already confusing enough : (
| @@ -526,16 +527,16 @@ h3(#realtime-connection). Connection | |||
| ** @(RTN20c)@ When @CONNECTING@, if the operating system indicates that the underlying internet connection is now available, the client should restart the pending connection attempt | |||
| * @(RTN16)@ @Connection@ recovery: | |||
There was a problem hiding this comment.
We also need to add a case for recovery failure ->
Currently, we cover following cases
- setting messageserial to zero as a part of
RTN15c7. - We never set
connectionKeydirectly, it will be updated automatically for new connection.
We don't cover
reverting a change made as a part of `RTN16j` -> initialized channels with their serials
There was a problem hiding this comment.
it doesn't need to be reverted. the channelSerials are fine to keep. (similarly we don't clear any channel state in the case of an RTN15c7 resume failure)
| @@ -679,7 +680,7 @@ h3(#realtime-channel). RealtimeChannel | |||
| ** @(RTL10b)@ Additionally supports the param @untilAttach@, which if true, will only retrieve messages prior to the moment that the channel was attached or emitted an @UPDATE@ indicating loss of continuity. This bound is specified by passing the querystring param @fromSerial@ with the @RealtimeChannel#properties.attachSerial@ assigned to the channel in the @ATTACHED@ @ProtocolMessage@ (see "RTL15a":#RTL15a). If the @untilAttach@ param is specified when the channel is not attached, it results in an error | |||
| ** @(RTL10c)@ Returns a @PaginatedResult@ page containing the first page of messages in the @PaginatedResult#items@ attribute returned from the history request | |||
| ** @(RTL10d)@ A test should exist that publishes messages from one client, and upon confirmation of message delivery, a history request should be made on another client to ensure all messages are available | |||
| * @(RTL12)@ An attached channel may receive an additional @ATTACHED@ @ProtocolMessage@ from Ably at any point. (This is typically triggered following a transport being resumed to indicate a partial loss of message continuity on that channel, in which case the @ProtocolMessage@ will have a @resumed@ flag set to false). If and only if the @resumed@ flag is false, this should result in the channel emitting an @UPDATE@ event with a @ChannelStateChange@ object. The @ChannelStateChange@ object should have both @previous@ and @current@ attributes set to @attached@, the @reason@ attribute set to to the @error@ member of the @ATTACHED@ @ProtocolMessage@ (if any), and the @resumed@ attribute set per the @RESUMED@ bitflag of the @ATTACHED@ @ProtocolMessage@. (Note that @UPDATE@ should be the only event emitted: in particular, the library must not emit an @ATTACHED@ event if the channel was already attached, see @RTL2g@). | |||
| * @(RTL12)@ An attached channel may receive an additional @ATTACHED@ @ProtocolMessage@ from Ably at any point. (This is typically triggered following a transport being resumed to indicate a partial loss of message continuity on that channel, in which case the @ProtocolMessage@ will have a @resumed@ flag set to false). If the @resumed@ flag is false, this should result in the channel emitting an @UPDATE@ event with a @ChannelStateChange@ object, otherwise don't set the state or emit attached event. The @ChannelStateChange@ object should have both @previous@ and @current@ attributes set to @attached@, the @reason@ attribute set to to the @error@ member of the @ATTACHED@ @ProtocolMessage@ (if any), and the @resumed@ attribute set per the @RESUMED@ bitflag of the @ATTACHED@ @ProtocolMessage@. (Note that @UPDATE@ should be the only event emitted: in particular, the library must not emit an @ATTACHED@ event if the channel was already attached, see @RTL2g@). | |||
There was a problem hiding this comment.
This change has been raised as per https://github.com/ably/ably-go/pull/621/files#r1412495177
There was a problem hiding this comment.
I also had a question, where client sends explicit attach on already attached channel. I could see duplicate ATTACHED message received for every ATTACH sent from client side.
How does attach_resume flag play into this? Is resumed flag set for ATTACHED message is side effect of attach_resume sent from client side?
| **** @(RTN15c7)@ @CONNECTED@ @ProtocolMessage@ with a new @connectionId@ and an @ErrorInfo@ in the @error@ field. In this case, the resume was invalid, and the error indicates the cause. The @error@ should be set as the @reason@ in the @CONNECTED@ event, and as the @Connection#errorReason@. The internal @msgSerial@ counter should be reset so that the first message published to Ably will contain a @msgSerial@ of @0@. The rest of the process is the same as for @RTN16c6@: The client library should move all channels that were in the @ATTACHING@, @ATTACHED@, or @SUSPENDED@ states to the @ATTACHING@ state, and initiate an @RTL4c@ attach sequence for each. The connection should also process any messages queued per @RTL6c2@. | ||
| ** @(RTN15c)@ The system's response to a resume request (with a non-empty connectionKey) will be one of the following: | ||
| **** @(RTN15c6)@ A @CONNECTED@ @ProtocolMessage@ with the same @connectionId@ as the current client (and no @error@ property). This indicates that the resume attempt was valid. | ||
| **** @(RTN15c7)@ @CONNECTED@ @ProtocolMessage@ with a new @connectionId@ and an @ErrorInfo@ in the @error@ field. In this case, the resume was invalid, and the error indicates the cause. The @error@ should be set as the @reason@ in the @CONNECTED@ event, and as the @Connection#errorReason@. The internal @msgSerial@ counter should be reset so that the first message published to Ably will contain a @msgSerial@ of @0@. |
There was a problem hiding this comment.
Spec doesn't mention about value of channel Flag.resumed after channel attach @SimonWoolf ?
No description provided.