Is dagster suitable for highly concurrent workflows?

[alexeysofin]

If we have a situation when a job (dag) has many small CPU tasks, like image processing, let's say 500k images in a job, and then join tasks' results to form a final aggregated result, is dagster a suitable option for such cases? Can it be scaled horizontally based on the dag config?

[sryza]

I am not personally aware who has run jobs with 500k steps. It would not be uncommon though to chunk together sets of images into tens or hundreds of steps.