Standardizing logging configuration in Dagster

[rexledesma]

Overview

Logging is central to Dagster's core infrastructure, providing observability to computations defined using the framework as well as to Dagster components that aid in executing the computations.

Here are some examples where logging is used in the system:

1. Capturing Python logs to send to the Dagster event log, allowing them to displayed in the default view of Dagit.
2. Displaying raw stdout and stderr logs using Dagster's compute log storage.
3. Providing observability in Dagster infrastructure components, such as:
   1. The Dagster server (the uvicorn server powering the Dagster UI and GraphQL interface),
   2. The Dagster daemon (a long-running process that support features like scheduling, sensors, and run queueing).
   3. The Dagster code locations (powered by gRPC) that host the user's Dagster definitions and interface with the Dagster UI and the daemon.

Problems

Existing User Issues

Here is a sample of user-reported issues regarding our logging configuration, mostly dealing with (3) providing observability in Dagster infrastructure components. Currently, configuring logging in these components is not well supported.

• Allow configuration of dagster_daemon logging #4170
• Change dagster-daemon log level #4299
• Dagster event logging not configurable through dagster api execute_run #4992
• Support more flexible configuration of Dagster core process loggers. #6770
• reduce spamminess of sensor daemon logs #8911

Proposed Solution

Standardization of Dagster's logging interface must occur. Here are a few high-level changes that need to happen:

• The names and default configuration of existing loggers should be documented.
• Expose a standardized interface to configure how logs are displayed, as well as what level of logs are displayed while maintaining the required server handlers to correctly route logs to the Dagster event logs.
  • This should follow best practices for Python frameworks that expose logging configuration. Ideally, something like uvicorn's --log-config setting, which takes in a user-defined object.
  • This interface should be exposed for each Dagster infrastructure component (e.g. the Dagster server, daemon, and code locations)

[markfickett]

Thanks for starting this discussion!

Another area I'd love to see included is the ability to better send logs from Dagster to additional handlers, and filter them along the way. I think exposing a more standard interface to logging configuration meshes with this. Existing issues:

• Preserve original LogRecord when logging to custom handler
• Preserve module/lineno/filename when sending log messages to @loggers

And, some improved scalability with context.log usage in bit fan-outs with DynamicOutput, both for the events API and viewing logs in Dagit: Improve performance in Dagit with large log volumes .

[rexledesma]

Hi @markfickett, thanks for your input. Indeed, custom handlers are definitely under the scope of this standardization: the ideal state is that they should:

1. Be configurable as specified by the Python logging documentation, and
2. Not have any interference with the internal Dagster logging machinery.

To be clear, are you talking about logs emitted from ops/assets in Dagster, using the context or user-specified loggers? Again, the framework will still inject its own handler to ensure that the specified logs are emitted to the Dagster event logs, but custom handlers should be injected in such a way that it processing logs that are free of any Dagster pre-processing.

[markfickett]

I'm interested in both.

One use case is customizing the Dagster console output. I wanted to adjust the formatting, as well as filter out some spcific noisy logs. So I want that handler to include both context.log calls I'm making, other Python logger calls in my application code, and logs from internal Dagster loggers.

Another use case is sending logs to a structured JSON file for fluentbit to pick up. Again, I want to include all of Dagster-framework ogs, context.log messages, and other Python loggers from my application.

So, if I'm able to use the standard Python logging framework to specify a custom Handler, as well as its filters, that sounds great!

I had been using managed_python_loggers to make sure all logs go to my handler, and python_log_level to exclude some Dagster framework debug logs.

[tha23rd]

Just throwing a comment in here to advocate for an enhancement on customizing logging in Dagster internals. Our use case is that we would like to export logs created by Daemons to Datadog for monitoring + alerting purposes. At a bare minimum it would be a huge win if we could simply export the logs in JSON format.

Thanks!

[sigwinch28]

There have been several releases since this discussion and since the related issues have last been updated. Could we get a progress/planning update on this, please?

[sean-stach-ff]

Any update on this? So far the only solution we have found is to manually set a custom logger for all our jobs.

[dorothychen]

Would also love to understand if there are planning updates here.

To describe our use case - we also use DataDog for monitoring, and the inability to globally override how Dagster system logs (in particular, error log stack traces) are formatted has been a big usability pain point for developers.

(I've spoken to some folks on the Dagster side about this on slack as well, but I want to tie that together with this more centralized thread)

[dorothychen]

From my (admittedly limited) perspective given our specific use case, some potential solutions that would cover the gap for us:

• if there was a way to indicate that a custom python handler should replace (instead of add to) the default dagster logger
• if there was a way to set a custom logger for all jobs in a code location by default, without needing to be specified in each job's runconfig
• if there was a way to customize/inject a default system logger directly, i.e. replace colored_console_logger here

[sean-stach-ff]

@dorothychen I was able to apply custom logger to the entire dagster instance using https://docs.dagster.io/concepts/logging/python-logging#configuring-python-log-handlers-

[dorothychen]

Have you been able to get the custom logger to replace dagster's system loggers, i.e. for error messages and stack traces? This is the mechanism I'm referring to in my first bullet point (had the wrong link originally; updated!), but unfortunately afaict using the custom logger is only additive (code pointer)

[sean-stach-ff]

My problem was that I needed all logs to be formatted in a way to be ingested. I used the above linked concept to add a new formatter which formats the log and outputs them to STDOUT.

I suppose this doesn't help replace the logs that are already in place but it does allow for a global custom logger without having to set it each time. However, since the logs are appearing in STDOUT and Dagster default logs appear in STDERR - it does have a clear separation.

[zyd14]

Any updates here? Logging with anything except the default loggers is still a huge pain, especially if you want global configuration. Also I'm not seeing an obvious way to pass loggers for assets and asset jobs - there's logger_defs for op-jobs, but I don't see anything similar for assets / asset jobs. Has anyone figured out how to provide custom loggers for assets?

[VerdantForge]

So far I've been using this hack to be able to log to dagster console and use our own custom logger (which forwards logs to an aggregator)

class MultiLogger(Logger):
    """Log to Dagster and to the default logger"""

    python_logger: Logger
    dagster_logger: Logger

    def __init__(
        self,
        name,
        python_logger: Logger,
        dagster_logger: Logger,
        level=settings.LOG.LEVEL,
    ):
        super().__init__(name, level=level)
        self.python_logger = python_logger
        self.dagster_logger = dagster_logger

    def _log(self, level, msg, args, **kwargs):
        self.python_logger.log(level, msg, *args, **kwargs)
        self.dagster_logger.log(level, msg, *args, **kwargs)


def get_dagster_and_python_logger(name: str) -> Logger:
    return MultiLogger(
        name,
        python_logger=logging.getLogger(name),
        dagster_logger=get_dagster_logger(name),
    )

[zyd14]

So then you just call get_dagster_and_python_logger to get a logger instead of using context.log? Unfortunately not really feasible for me as I'd have to convert a couple hundred ops / assets to use this instead of context.log.

[VerdantForge]

Usually a call to logger = get_dagster_and_python_logger(__name__) at the top of the module to get a module level logger, the methods then use logger.info, logger.error, etc...
A quick regex should be able to take care of your refactoring, no?

[v1gnesh]

Any thoughts on using eliot for logging?