Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Long poll completion buffer to prevent timeouts #4425

Conversation

vytautas-karpavicius
Copy link
Contributor

@vytautas-karpavicius vytautas-karpavicius commented Aug 26, 2021

What changed?
Leave some long poll completion room on the server side.
When little time is left until expiration we reserve some time longPollCompletionBuffer when creating a timer to return earlier, so that this request will be able to finish without timeout.
This PR comes with companion client PR cadence-workflow/cadence-go-client#1123. Which will prevent doing additional request all together when too little time is left.
If an older client is used, we still need a safeguard timeout error. Otherwise client would be stuck in long poll loop for the duration of completion buffer, which can result in additional request being made that return empty responses.

Related previous PR: #3830

Why?
The idea here is to prevent context timeout errors, which affects SLAs and is not truly an error.

How did you test it?
Locally. With go client changes and without go client changes.

Potential risks

Release notes

Documentation Changes

@coveralls
Copy link

coveralls commented Aug 26, 2021

Pull Request Test Coverage Report for Build 04d2b3e7-35d2-4c7e-b214-bbee20bad89d

  • 13 of 17 (76.47%) changed or added relevant lines in 1 file are covered.
  • 115 unchanged lines in 13 files lost coverage.
  • Overall coverage decreased (-0.06%) to 56.397%

Changes Missing Coverage Covered Lines Changed/Added Lines %
service/history/historyEngine.go 13 17 76.47%
Files with Coverage Reduction New Missed Lines %
client/history/client.go 2 41.25%
client/history/metricClient.go 2 47.05%
common/types/mapper/thrift/shared.go 2 64.98%
service/history/handler.go 2 48.15%
common/persistence/executionManager.go 4 75.5%
common/persistence/statsComputer.go 4 95.71%
service/frontend/workflowHandler.go 4 58.8%
service/history/queue/timer_queue_processor_base.go 4 78.78%
service/history/historyEngine.go 9 70.09%
common/persistence/sql/sqlExecutionStore.go 16 60.0%
Totals Coverage Status
Change from base Build 35aee8df-b422-435f-99b8-dad5f592c3ae: -0.06%
Covered Lines: 79064
Relevant Lines: 140191

💛 - Coveralls

@vytautas-karpavicius vytautas-karpavicius marked this pull request as ready for review August 26, 2021 13:01
@vytautas-karpavicius vytautas-karpavicius requested review from yycptt and a team August 26, 2021 13:01
@vytautas-karpavicius vytautas-karpavicius merged commit efb9f90 into cadence-workflow:master Aug 27, 2021
@vytautas-karpavicius vytautas-karpavicius deleted the long-poll-completion-buffer branch August 27, 2021 14:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants