Produce Results from PipelineTasks fanned out in a Matrix

[jerop]

In TEP-0090: Matrix, we introduced Matrix to support fanning out a PipelineTask with different combinations of Parameters.

This issue tracks the work to produce Results from PipelineTasks fanned out in a Matrix, which was a non-goal in TEP-0090. This work was scoped out as we waited for TEP-0075: Object Parameters and Results and TEP-0076: Array Results to land. See TEP-0090: Matrix - Results for further details.

The design work is ongoing in tektoncd/community#753.

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[jerop]

/lifecycle frozen

[EmmaMunley]

@Tomcli Do you have a use case for using results from Matrixed Tasks that is results from a looping functionality? Also, if you can share the use cases if you have any. Thanks!

cc: @pritidesai @jerop

[Tomcli]

The use case we have is hyperparameter turning for machine learning where we need to aggregate all the results from all the tasks and find/compute the desire result. We probably will not use the Tekton native functionality because in Kubeflow we have hundreds of loop/matrixed tasks. So Kubeflow Pipelines use the metadata service to store each task result into Kubeflow's database in this case instead of passing hundreds of results into one task in the pod spec.

[jerop]

@EmmaMunley all the use cases for Matrix in TEP-0090 apply to the need for Results from fanned out PipelineTasks

Just looking at the first use case from TEP-0090, the Kaniko Task produces a Result named "image-digest" -- how do get those Results when 3 TaskRuns were created from the PipelineTask?

[jerop]

There were prior discussions about this issue in tektoncd/community#753 but closed the PR because I didn't have the capacity to see it through, maybe that'd be a good starting point @EmmaMunley

[EmmaMunley]

/assign

[pritidesai]

We have discovered multiple concerns and some suggestions during designing an access to a result from an individual instance of fanned out tasks. We decided to descope the existing proposal so they no longer are applicable. I am highlighting them here so that we do not loose them while revisiting next time.

Concerns:

tektoncd/community#1047 (comment)
tektoncd/community#1047 (comment)

Suggestions:

tektoncd/community#1047 (comment)
tektoncd/community#1047 (comment)

Overall, we have an approval from the community to go ahead and support such use case:

With Matrix:

[EmmaMunley]

#7167 closes this issue
/close

[tekton-robot]

@EmmaMunley: You can't close an active issue/PR unless you authored it or you are a collaborator.

In response to this:

#7167 closes this issue
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jerop]

Thank you @EmmaMunley 🎉

/close

[tekton-robot]

@jerop: Closing this issue.

In response to this:

Thank you @EmmaMunley 🎉

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.