Networking connectivity integration test using local proxy

[jaley]

@cruickshankpg here's an empty test and a place to put the proxy implementation. Let's discuss the interface we're working to initial so we can both get started.

[jaley]

@cruickshankpg - this will now run a simple integration test, which publishes a bunch of location updates for a single Trackable, using an injected AblyRealtime instance. This should be enough for you to put a proxy in the middle and confirm that you can break the test by interfering with traffic? Can also disable TLS as required.

[jaley]

This branch now has failing tests, which I believe reproduce some of our open issues:

• faultDuringTracking[TcpConnectionRefused] currently fails because presence.leave() times out after two minutes if you call publisher.remove() while connectivity is interrupted. I believe this is similar to what's happening in Sending presence leave event times out whilst still connected #862, though it's important we cover this with more types of fault when those are implemented. cc: @lawrence-forooghian
• faultBeforeAddingTrackable[TcpConnectionRefused] fails after 10 seconds because we throw ConnectionException if initial requests to track occur while offline, this is Exceptions thrown for new publishing requests when offline #871 , cc: @davyskiba
• faultDuringTracking[TcpConnectionUnresponsive] fails because of com.ably.tracking.ConnectionException: Timeout was thrown when waiting for channel to attach, which looks very much like Publisher apps reporting "Timeout was thrown when waiting for channel to attach" #859, though I think even with this fixed there may be more failures to come. (no assignee yet)

I've not yet implemented a Layer 7 proxy, which is needed for the rest of the high priority fault types, so I'll try to get that done next. Tomorrow is really busy for me, but I'll see what I can do.

It might be a good idea to try to get this reviewed and ready to merge before then, if anyone wants to use the tests we have so far as a reproduction of the issues above to validate fixes? (Assuming that turns them green...)

Word of warning -- these are slow tests when they're failing. Generally, the pass in 2-3 seconds each, but waiting for ably-java timeouts when things are going wrong takes a few minutes. We may be able to tweak some of the timeouts in ClientOptions to speed up testing, but sadly I think some important ones are not client-configurable, e.g. the time taken for a channel to move to suspended state.

[QuintinWillison]

At 1e96b3a, locally from Android Studio, I am seeing:

With the following failures:

Test	Failure
faultDuringTracking[TcpConnectionRefused]	com.ably.tracking.ConnectionException: failed to connect to localhost/127.0.0.1 (port 13579): connect failed: ECONNREFUSED (Connection refused)
faultBeforeAddingTrackable[TcpConnectionRefused]	java.lang.AssertionError: Expectation 'attempt to add active Trackable while fault active' did not result in success.
faultDuringTracking[TcpConnectionUnresponsive]	com.ably.tracking.ConnectionException: Timeout was thrown when waiting for channel to attach
faultBeforeAddingTrackable[TcpConnectionUnresponsive]	java.lang.AssertionError: Expectation 'attempt to add active Trackable while fault active' did not result in success.

@jaley Is this what you would expect at this stage with this pull request? If so, then I would like to annotate those tests as ignored for now to make this pull request green and mergeable to main branch.

[jaley]

Nice work, @QuintinWillison!

Yes, I can confirm that I get the exact same failures, even on the #886 branch, as I don't think any of the relevant fixes have landed in main yet. In case you want to add comments/etc as to why these tests are ignored:

• faultDuringTracking[TcpConnectionRefused] - this is Publisher apps reporting "Timeout was thrown when waiting for channel to attach" #859 , which @ikbalkaya has linked to Re-attach fails due to previous detach request ably-java#885
• faultBeforeAddingTrackable[TcpConnectionRefused] - @davyskiba has PR Feature/remove add exceptions #883 open, which should resolve this error
• faultDuringTracking[TcpConnectionUnresponsive] - Publisher apps reporting "Timeout was thrown when waiting for channel to attach" #859 again
• faultBeforeAddingTrackable[TcpConnectionUnresponsive] - also fixed by Feature/remove add exceptions #883

(Of course, there's no guarantee that there won't be more issues masked by those...)

[davyskiba]

I have a few kotlinish suggestions, but we can address them later

[KacperKluka]

I skimmed over proxy implementation and test helper classes details, however, since the tests run and work I think it's safe to say that it looks good 👍 Also left a few suggestions/questions but they shouldn't block this PR 😉