[BUG] Failed to launch workflows due to too many requests

[pingsutw]

Describe the bug

I get the below error when I run a workflow with 100 Launch plans, and each launch plan has 100 tasks.

• max Parallelism: 100

Workflow[flytesnacks:development:workflow.agent.load_test.sleep_wf] failed. RuntimeExecutionError: max number of system 
retry attempts [52/50] exhausted. Last known status message: Workflow[] failed. ErrorRecordingError: failed to publish event, 
caused by: EventSinkError: Error sending event, caused by [rpc error: code = Unavailable desc = unexpected HTTP status 
code received from server: 429 (Too Many Requests); transport: received unexpected content-type "text/html"]

Expected behavior

Workflow should not fail

Additional context to reproduce

No response

Screenshots

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[pingsutw]

other error:

Workflow[flytesnacks:development:workflow.agent.load_test.load_test_wf] failed. RuntimeExecutionError: max number of system retry attempts [51/50] exhausted. Last known status message: Workflow[flytesnacks:development:workflow.agent.load_test.load_test_wf] failed. CausedByError: Failed to propagate Abort for workflow. Error:
0: [SystemError] system error, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Internal desc = rpc error: code = Internal desc = unexpected error type for: ERROR: duplicate key value violates unique constraint "execution_operations_pkey" (SQLSTATE 23505)]
1: [SystemError] system error, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Internal desc = rpc error: code = Internal desc = unexpected error type for: ERROR: duplicate key value violates unique constraint "execution_operations_pkey" (SQLSTATE 23505)]
2: [SystemError] system error, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Internal desc = rpc error: code = Internal desc = unexpected error type for: ERROR: duplicate key value violates unique constraint "execution_operations_pkey" (SQLSTATE 23505)]
3: [SystemError] system error, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Internal desc = rpc error: code = Internal desc = unexpected error type for: ERROR: duplicate key value violates unique constraint "execution_operations_pkey" (SQLSTATE 23505)]
4: [SystemError] system error, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Internal desc = rpc error: code = Internal desc = unexpected error type for: ERROR: duplicate key value violates unique constraint "execution_operations_pkey" (SQLSTATE 23505)]
5: [SystemError] system error, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Internal desc = rpc error: code = Internal desc = unexpected error type for: ERROR: duplicate key value violates unique constraint "execution_operations_pkey" (SQLSTATE 23505)]
6: [SystemError] system error, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Internal desc = rpc error: code = Internal desc = unexpected error type for: ERROR: duplicate key value violates unique constraint "execution_operations_pkey" (SQLSTATE 23505)]
7: [SystemError] system error, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Internal desc = rpc error: code = Internal desc = unexpected error type for: ERROR: duplicate key value violates unique constraint "execution_operations_pkey" (SQLSTATE 23505)]
8: [SystemError] system error, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Internal desc = rpc error: code = Internal desc = unexpected error type for: ERROR: duplicate key value violates unique constraint "execution_operations_pkey" (SQLSTATE 23505)]
9: [SystemError] system error, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Internal desc = rpc error: code = Internal desc = unexpected error type for: ERROR: duplicate key value violates unique constraint "execution_operations_pkey" (SQLSTATE 23505)]
10: [SystemError] system error, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Internal desc = rpc error: code = Internal desc = unexpected error type for: ERROR: duplicate key value violates unique constraint "execution_operations_pkey" (SQLSTATE 23505)]
11: [SystemError] system error, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Internal desc = rpc error: code = Internal desc = unexpected error type for: ERROR: duplicate key value violates unique constraint "execution_operations_pkey" (SQLSTATE 23505)]
12: [SystemError] system error, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Internal desc = rpc error: code = Internal desc = unexpected error type for: ERROR: duplicate key value violates unique constraint "execution_operations_pkey" (SQLSTATE 23505)]
13: [SystemError] system error, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Internal desc = rpc error: code = Internal desc = unexpected error type for: ERROR: duplicate key value violates unique constraint "execution_operations_pkey" (SQLSTATE 23505)]
14: [SystemError] system error, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Internal desc = rpc error: code = Internal desc = unexpected error type for: ERROR: duplicate key value violates unique constraint "execution_operations_pkey" (SQLSTATE 23505)]
15: [SystemError] system error, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Internal desc = rpc error: code = Internal desc = unexpected error type for: ERROR: duplicate key value violates unique constraint "execution_operations_pkey" (SQLSTATE 23505)]
16: [SystemError] system error, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Internal desc = rpc error: code = Internal desc = unexpected error type for: ERROR: duplicate key value violates unique constraint "execution_operations_pkey" (SQLSTATE 23505)]
17: [SystemError] system error, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Internal desc = rpc error: code = Internal desc = unexpected error type for: ERROR: duplicate key value violates unique constraint "execution_operations_pkey" (SQLSTATE 23505)]
18: [SystemError] system error, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Internal desc = rpc error: code = Internal desc = unexpected error type for: ERROR: duplicate key value violates unique constraint "execution_operations_pkey" (SQLSTATE 23505)]
19: [SystemError] system error, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Internal desc = rpc error: code = Internal desc = unexpected error type for: ERROR: duplicate key value violates unique constraint "execution_operations_pkey" (SQLSTATE 23505)]

[wild-endeavor]

For the 429 backoff, #5166 will help with this some. Can we try again locally after that is merged and see if we can repro?
https://github.com/flyteorg/flyte/blob/master/flytepropeller/events/config.go#L26-L27

The other issue is a bit different. We should probably make it a different ticket.

[hamersaw]

closing this as the aforementioned PR is the best mitigation.