fix: apply custom cursor pagination where workflows and archived workflows are merged

[sunyeongchoi]

Fixes #11715

Motivation

Workflows and archive workflows use different pagination logics.
However, in the current workflow page, if both workflows and archive workflows coexist, we want them to be retrieved together.
The issue arises from passing the pagination options used for workflows pagination directly to archive workflows, causing a mismatch between the types used in archive workflows and workflows, resulting in archive workflows not being retrieved.

Modifications

I have implemented the following logic to resolve this issue:

1. fetch the entire list of workflows and archived workflows separately.
2. merge the workflows and archived workflows.
3. perform cursor pagination on the merged result based on the resourceVersion.

After contemplating on how to address this issue, I concluded that the conventional approach of paginating workflows and archived workflows separately would not easily resolve this problem. Therefore, I implemented it in this manner.

Verification

• Confirming the pagination is functioning correctly.
• Checking if Archive workflows are retrieved successfully.
• Verifying that Workflows are retrieved correctly.
• Validating the proper functioning of tests with the 'make test' command.

[terrytangyuan]

This is great! Could you also look into adding some tests around cursorPaginationByResourceVersion?

[sunyeongchoi]

This is great! Could you also look into adding some tests around cursorPaginationByResourceVersion?

Sure! I'll write a test code and upload a commit tomorrow :)

[terrytangyuan]

Would this approach require fetching the entire list every time user clicks another page?

[sunyeongchoi]

Yes, the entire list is fetched every time the page changed.

[sunyeongchoi]

I have written and uploaded test code for cursorPaginationByResourceVersion. 859c0b2

I am concerned about the potential performance impact of my implementation because it retrieves the entire dataset with each page change. However, I can't think of a better way to solve this problem on the server-side other than this method.

In my opinion, unless there's a better approach, the server should send the entire list to the front-end once, and then pagination can be handled on the front-end. However, one drawback of this approach is that the data won't be updated on the front-end without refreshing the page. To achieve real-time data updates, we might need to consider using websockets or polling.

Do you have any suggestions or better ideas on how to handle this?

[terrytangyuan]

I don't have a better idea. I think correctness is important right now. Performance-wise, we should do something from the front-end like what you said to only refetch the whole list when refreshing. This can be something that's configurable.

[sunyeongchoi]

Then, after this PR, I will think about the front page pagination and give it a try.

Also, since I am curious about the performance of the currently implemented pagination, I will think about ways to test its performance.

I will share the test results later when they come out.

[terrytangyuan]

Great!

[terrytangyuan]

This looks good once the last comment and the conflicts are resolved. Great work! This should be included in the next RC #11381

[terrytangyuan]

Need to sort the imports

[terrytangyuan]

@sunyeongchoi Looks like the test is flaky. See https://github.com/argoproj/argo-workflows/actions/runs/6166062084/job/16734921912

[sunyeongchoi]

@sunyeongchoi Looks like the test is flaky. See https://github.com/argoproj/argo-workflows/actions/runs/6166062084/job/16734921912

Oh no, I'll quickly post a PR fixing that test. sorry.

[sunyeongchoi]

@sunyeongchoi Looks like the test is flaky. See https://github.com/argoproj/argo-workflows/actions/runs/6166062084/job/16734921912

I uploaded bugfix PR about this :) #11816
Thaks for your report.