Use GCP Batch or Cloud Run Jobs for Runs?

[seandavi]

Google Cloud has a couple of batch processing environments:

• GCP Batch
• Cloud Run Jobs

Is anyone working on a Dagster integration for running jobs using either of these services, akin to the DockerRunLauncher?

[tim-carbonfarm]

I have set up a "GCP-native" Dagster deployment which:

• uses Cloud Run (services) for gRPC code location servers
• uses Cloud Run Jobs as run workers

So far it's working very well. If there is some interest I could look into open-sourcing it.

[baumann-t]

Perfect! PR #21509 is closed. I received an invitation from @timchap to contribute to his PR. I've copied my existing Example and pushed it to his repo. I should have more time later this week to adapt my original example to the new launcher.

@timchap, I think we could add both of our approaches for the gRPC code location servers in the Cloud Run Launcher Example:

• One example using a simple VM (like mine)
• One example using Cloud Run Services (like what you mentioned)

This would showcase two different options for running Dagster on GCP. Let me know what you think!

[leonardovff]

the pr was completely ignored :\

[garethbrickman]

Hey folks, can assure you the PR has not been ignored. Please check the latest reply here: #21864 (comment)

[garethbrickman]

Work on the GCP Cloud Runner has been migrated to our community-integrations repo: dagster-io/community-integrations#26

[garethbrickman]

The GCP Cloud Runner is now available in its community-integrations repo: dagster-contrib-gcp

And can be installed by running:

pip install dagster_contrib_gcp

[stickyhipp]

cc: @neilfulwiler