How to K8SRunLauncher locally?

[amarrella]

Hi, I'm trying to run my pipeline using local dagit and K8SRunLauncher on an existing cluster but the job coordinator seems to fail.

I have set up my (local) instance like this:

run_launcher:
    module: dagster_k8s.launcher
    class: K8sRunLauncher
    config:
        service_account_name: dagster
        dagster_home: "/opt/dagster/dagster_home"
        job_image:
            env: JOB_IMAGE
        instance_config_map: dagster-instance
        postgres_password_secret: dagster-postgresql-secret
        job_namespace: dagster
        load_incluster_config: False
        env_config_maps:
        - dagster-my-pipeline-user-env # this exists in the cluster

When I run the job, it successfully creates a pod, but the pod immediately fails with an error:

Traceback (most recent call last):
  File "/usr/local/bin/dagster", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/dagster/cli/__init__.py", line 41, in main
    cli(auto_envvar_prefix=ENV_PREFIX)  # pylint:disable=E1123
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/dagster/cli/api.py", line 93, in execute_run_with_structured_logs_command
    _execute_run_command_body(recon_pipeline, args.pipeline_run_id, instance, send_to_buffer)
  File "/usr/local/lib/python3.7/site-packages/dagster/cli/api.py", line 111, in _execute_run_command_body
    EngineEventData.in_process(pid, marker_end="cli_api_subprocess_init"),
  File "/usr/local/lib/python3.7/site-packages/dagster/core/instance/__init__.py", line 1019, in report_engine_event
    check.inst_param(pipeline_run, "pipeline_run", PipelineRun)
  File "/usr/local/lib/python3.7/site-packages/dagster/check/__init__.py", line 185, in inst_param
    obj, ttype, param_name, additional_message=additional_message
  File "/usr/local/lib/python3.7/site-packages/future/utils/__init__.py", line 446, in raise_with_traceback
    raise exc.with_traceback(traceback)
dagster.check.ParameterCheckError: Param "pipeline_run" is not a PipelineRun. Got None which is type <class 'NoneType'>.

When I describe the pods, it seems it's passing my local environment as a python environment (see the args below)

api
execute_run_with_structured_logs
{"__class__": "ExecuteRunArgs", "instance_ref": null, "pipeline_origin": {"__class__": "PipelinePythonOrigin", "pipeline_name": 
"my_pipeline", "repository_origin": {"__class__": "RepositoryPythonOrigin", "code_pointer": {"__class__": "ModuleCodePointer",
 "fn_name": "repository", "module": "bin.repository"}, "executable_path": "/Users/amarrella/Projects/my_project/.venv/bin/python"}}, 
"pipeline_run_id": "d69e4fbc-28a3-4ee5-89b1-5b8ff6c6fcd3"}

I'm not sure this is related but it's suspicious that it uses my local executable path.

I checked:

• The dagster guide for workspaces: https://docs.dagster.io/overview/repositories-workspaces/workspaces#main and tried to change the executable_path there, but the catch is that local dagit then won't work.
• The K8SRunLauncher API Doc: https://docs.dagster.io/_apidocs/libraries/dagster_k8s#dagster_k8s.K8sRunLauncher

To me all seems configured correctly, am I missing something?

I'm able to workaround the issue by using the dagit instance in the kubernetes cluster, but that requires updating the pods with the grpc server as well and i'd like to avoid this during fast iterations.

[catherinewu]

Hi @amarrella 👋 My suspicion is that the K8s run pod is not using the same run storage as your local instance; when launching a run, dagit's instance (in your case, the local instance) writes the pipeline run to the run storage, but the K8s pod might have a different run storage configured due to a different run storage on its instance config (resulting in None).

To check, could you inspect the instance of the k8s job (instance_config_map: dagster-instance) and see what run storage is used? And then can also manually check that the expected run_id exists there (ie "d69e4fbc-28a3-4ee5-89b1-5b8ff6c6fcd3" from above)

[amarrella]

Hi @catherinewu @johannkm 👋

thanks for bearing with me!

by "don't see the logs", does that refer to the events in the pipeline run view? do the events show up when refreshing the page after run succeeds? does this consistently happen or sporadically

I don't see the events/logs i emit with context.log and that doesn't happen even after refresh.

Hi @amarrella - With the dagster-k8s run launcher we create a single pod for the entire pipeline run, so config can't be specified per solid. Instead you can place the tags on the pipeline definition with something like

Ah that's a pity. I'm stuck with celery so, as I don't want to over-provision resources (especially if they are GPU nodes).

Thanks again for all your help!

[catherinewu]

Hmm, could you try with context.log.info? I'm surprised that you don't see an error, hmm

[amarrella]

Hmm, could you try with context.log.info? I'm surprised that you don't see an error, hmm

Hey @catherinewu , sorry i meant context.log.info not context.log. Unfortunately i can't see them.

Anyway, given that i can't really use this launcher for what i'm trying to do (i need different pods per solid and my understanding now is that the K8sRunLauncher triggers a pod per pipeline) I think i'll have use the CeleryK8sRunLauncher instead.

[amarrella]

I'm an idiot, I realized now why the run_coordinator was stuck... My docker image entrypoint in the image i was trying to run from localhost was starting dagit and ignoring everything else (the "live" image didn't have this issue). Removing the entrypoint fixed the issue

[catherinewu]

ohhhh i see, so glad that you found the issue!