use case: deep hierarchical structure of algorithm, configured dynamically at compile time

[hfrcomm]

I have a modeling/data processing algorithm that has deeply nested “hierarchical” structure, i.e. it is organized as a “Russian doll” : there is top level data processing , where there are a few computational blocks that are connected to each other through output-to-input data connections, while each of these blocks contains several more (up to twenty) blocks, and each of these blocks contains again several blocks etc. This hierarchy has up to 4 levels in my use case, and the structure of blocks used at each level is meant to be “dynamic” , i.e. not implemented once and for all and held constant for as long as this algorithm exists, but rather depend on “configurational” parameters that can be edited by the user prior to launching the algorithm (i.e. at compile time).

I was spending a lot of my time trying to implement this algorithm as Dagster prototype. Dagster seems to have a great promise for my purpose, and one important part of that promise is the ability to use Dagit to browse the hierarchy by opening each block and browsing its contents.

I am struggling to figure out though how this type of algorithm can be represented in Dagster, specifically whether I can configure the exact composition of hierarchy at compile time from some type of initialization parameters, held in config parameters or somewhere else. It seems that the only type of adequate representation of deeply nested hierarchy of algorithm is to use composite solids for all upper levels of hierarchy , and only use solids at the lowest level. This way, I can browse the contents of each of the composite_solid block in Dagit. However it is not clear how I can dynamically change the composition of a given “composite_solid” from some type of initialization parameters , and where these parameters would come from. I could easily accomplish something similar to this purpose for a Solid, whereby I have a “context” argument which could contain the required dynamic initialization parameters (config parameters), and I could use various if else switches inside the body of this solid function to call different functions depending on values of said initialization parameters. This would be close to what I need but not quite it: the if else blocks will be only evaluated at run time, not at compile time. The situation is much worse though for “Composite Solid”: I cannot use if else switches to call different constituent lower-level composite_solid functions depending on parameters, this seems to be not allowed in composite_solids.

Additionally, there is no “context” argument in composite_solid’s, so there is not place for initialization parameters to be held.

[sryza]

Hi @hfrcomm - the way Dagster currently works is that the topology of a pipeline is fixed before pipeline run configuration is applied. I.e. we don't have a workflow that enables someone to supply run configuration to a pipeline definition, determine the topology of that pipeline definition based on the run configuration, and then view that topology in Dagit.

If this is something that we end up seeing a lot of user appetite for, we would consider adding it.

Dagster does support conditional solid branching at run time. Not sure if that would apply to what you're trying to do, but might be helpful? https://docs.dagster.io/concepts/solids-pipelines/pipelines#conditional-branching.