limitations on using dict arguments of composite_solid are very damaging to dagster?

[hfrcomm]

I need to pass a dict or another complex object as argument to a composite_solid in a Dagster project. This dict or another complex object, is an output of another solid or composite_solid, and it contains a lot of information , and is organized as Russian doll, container within container within container. I just discovered that there seems to be a major limitation in Dagster that seems to prohibit doing this for composite_solid. It is a perfectly good practice for a solid, but not for a composite_solid, it seems. The problem was discussed here:

https://stackoverflow.com/questions/65022207/how-to-use-dictionary-yielded-from-other-solid-in-a-composite-solid

under the question "How to use dictionary yielded from other solid in a composite Solid?", and the answer was given by Gregory Van Seghbroeck. The resume of the answer in my interpretation is "this cannot be done, composite_solid is different from regular solid in how it treats its arguments, so dict arguments to composite_solid's are not allowed".

My question to Dagster developers on this issue is two-fold:

1. Is Gregory Van Seghbroeck's answer indeed correct?
2. If it is, how come Dagster decided to impose such a limitation when it should be pretty obvious that it would be a death toll for any chances of serious industrial scale use of Dagster?

Let me explain what I mean by second question. In my mind , a big benefit from a system like Dagster would come from large Python data handling algorithmic projects which ingest a lot of different data and do a lot of data transformations and computations with them. Which is incidentally what is happening in my area of applicability (statistical modeling/data science).

For these type of applications, one needs to ingest heterogeneous data sets and do lots of computations on them. It is critically important to organize the DAG hierarchically in a Russian doll manner, which would allow for visual representation in Dagit, browsing up and down the hierarchy, and general logical clarity. This is due to the fact that we cannot put say 100 solids in one pipeline and browse it all using Dagit, such diagram will make no sense and have no value . In order for visual representation to bring real value , it has to fit on one screen, so it should be no more than 10 (more likely 6) blocks (solids or composite solids) on one diagram. So what I would expect to see in a Dagster representation of any complex system is one high level pipeline which consists of 5 to 10 complex computational sub-blocks (composite_solid's in Dagster terminology) that exchange complex data sets between them by connecting output datasets from one sub-block (composite_solid) as input to another sub-block.

Each of these sub-blocks would itself contain a set of computational sub-blocks (between 5 and 10 of them) which would also be organized the same way. So it is a Russian doll type structure of algorithmic data handling, and Dagster seems on the surface of it to be perfectly positioned to deliver the implementation of this exact architecture, with Dagit tool and its ability of hierarchical structures browsing (composite_solid's handling). The main element of Dagster which one would rely upon to create such hierarchical structure of data handling is composite_solid element. In order for it to work the composite_solid element needs to be able to the same manipulations with its inputs that the solid can. It is particularly important for composite_solid to be able to take a complex Russian doll type input and unpack it into a sequence of smaller object.
Roughly it should work like this: at the highest level of hierarch, there are 6 major composite_solid blocks. Each of them receives say one input which is an output of another of these 6 blocks. Each of this inputs is a composite object that has say 6 components , which need to be extracted out of these inputs during computation of each of these composite_solid. One simple way of doing it is by passing dict's as inputs here.
What I am trying to say here is that composite_solid must have exactly the same ability of handling its inputs as the solid as.
This would be critically important for organizing complex multi-level hierarchical data structures and algorithmic data processing blocks.
I would actually expect that the class composite_solid should inherit from the class solid. This does not seem to be the case now and composite_solid seems to be radically different from solid.

So how come this composite_solid's have these limitations in Dagster?
What are we supposed to do then when handling large complex computations, are there any workarounds that are at least somewhat practical?
Is there a plan to resolve these shortcomings?

[sryza]

Hi @hfrcomm - in your ideal world, where do you envision the code that does the dictionary unpacking would execute?

In production Dagster setups, typically each solid executes in its own process (or K8s pod). Are you planning to use such a setup? If so, would you like the dictionary unpacking code to execute once in its own process, or once in each of the
processes of the solids that use the unpacked values?

[hfrcomm]

I am not quite sue I get the thrust of your questions. Ideally, I'd like to be able to pass complex containers (dicts and even custom classes if possible) as inputs to a high-level composite_solid, and write code inside such composite_solid that unpacks these container inputs , selects smaller sub-units and pass them again to solids and other composite_solids that are called inside the high-level composite_solid. Each composite_solid should be able to unpack its own inputs such as dicts in other words, same as solids can do.
Where exactly the code for such unpacking would execute is of minor concern for my particular purposes . I would be happy with the production setup that you describe. I am not sure I understand the mechanics of this duplication of unpacking that you are alluding to. That seems to be an issue of how to organize computation behind the scenes more efficiently. I am looking at it from the user's perspective though.
Right now though it seems that unpacking dict inputs in composite_solid is not possible at all , unless I am mistaken, and that's the most serious issue. If somehow it would become possible, we would worry about execution. Ideally it should be done once where it is called in composite_solid, but even if it were duplicated multiple times it is maybe not such a major flaw.

[sryza]

I think the core issue causing the difficulty here is that composite solids are only meant for organizing solids, not directly defining computations. We're actually discussing a rename that will make this more clear, clarifying that composite solids are not solids, but containers for solids: #2902.

Because all computations in a pipeline must happen inside a solid, the way that I would think about implementing this would be to include a solid that does the unpacking. E.g.

@solid(output_defs=[OutputDefinition(name="val1"), OutputDefinition(name="val2")])
def unpack_dict(context, input_dict):
    yield Output(input_dict["val1"], output_name="val1")
    yield Output(input_dict["val2"], output_name="val2")

@composite_solid
def my_composite_solid(input_dict):
    val1, val2 = unpack_dict(input_dict)
    do_something_with_val1(val1)
    do_something_with_val2(val2)

Does that make sense? Are there reasons I'm missing that this doesn't work well for you?