Experimental partition API arguments

[abkfenris]

I'm trying to use the new experimental partition APIs, but it appears that some of the arguments that the old APIs include aren't carried forwards, which makes it harder to adapt the partitions to different timing.

Specifically, I'm trying to use @monthly_partitioned_config, but I cannot access the current month. With the previous API you could set partition_months_offset=0 to get the current month.

Setup:

@graph
def example_graph():
    pass

@monthly_partitioned_config(start_date="2021-08-01")
def monthly_config(start, _end):
    return {}

job = example_graph.to_job(config=monthly_config)

mode = job.get_default_mode()
partition_def = mode.get_partition_set_def(job.name)

If I try to access the current month, I get an exception.

>>> partition = partition_def.get_partition(name="2021-09-01")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/lib/python3.9/site-packages/dagster/core/definitions/partition.py", line 411, in get_partition
    check.failed("Partition name {} not found!".format(name))
  File "/opt/conda/lib/python3.9/site-packages/dagster/check/__init__.py", line 119, in failed
    raise CheckError("Failure condition: {desc}".format(desc=desc))
dagster.check.CheckError: Failure condition: Partition name 2021-09-01 not found!

If I try to access the previous month, then it works fine:

>>> partition = partition_def.get_partition(name="2021-08-01")
>>> partition
<dagster.core.definitions.partition.Partition object at 0x7f5f9106f280>

In this case I'm using the partition with a sensor instead of as a sensor as I don't control the upstream data, instead I'm triggering as data becomes available and aggregating all of the currently available for the current month.

Is there a way to adapt the experimental partition APIs to allow access to the current month? (or future! I may have some forecasting data I'm working with soon that can 'backfill' dates ahead...)

This is with 0.12.8.

>>> dagster.__version__
'0.12.8'

[sryza]

@abkfenris - do your partitions represent time windows? If so, does the time window for the partition that you want to be named "2021-08-01" end on 8/1 or begin on 8/1?

[abkfenris]

Yes, I do want two partitions, so something like end_offset would make sense. I hadn't thought about the partition end times.

@monthly_partitioned_config(start_date="2021-08-01", end_offset=1)
def my_config(start, end):
     ...

To give [(start=2021-08-01, end=2021-09-01), (start=2021-09-01, end=2021-10-01)]

If I was working with forecasts of climatology for the next 10 years:

@monthly_partitioned_config(start_date="2021-08-01", end_offset=((12 * 10) + 1))
def my_config(start, end):
     ...

To give [(start=2021-08-01, end=2021-09-01), (start=2021-09-01, end=2021-10-01), ... (start=2031-08-01, end=2031-09-01), (start=2031-09-01, end=2031-10-01)]

---

Might it make it more sense to have it inclusive vs exclusive, and have the default then be -1?

@monthly_partitioned_config(start_date="2021-08-01", end_offset=0)
def my_config(start, end):
     ...

To give [(start=2021-08-01, end=2021-09-01), (start=2021-09-01, end=2021-10-01)]

10 year forecast

@monthly_partitioned_config(start_date="2021-08-01", end_offset=12 * 10)
def my_config(start, end):
     ...

To give [(start=2021-08-01, end=2021-09-01), (start=2021-09-01, end=2021-10-01), ... (start=2031-08-01, end=2031-09-01), (start=2031-09-01, end=2031-10-01)]

[sryza]

Got it - makes total sense. I'll look into adding an end_offset argument.

Might it make it more sense to have it inclusive vs exclusive, and have the default then be -1?

I don't think it would be crazy, but I think that, in general, zero is the expected default for offset params, so would prefer to keep it at that. For the 10 year forecast, you could do 12 * 10 + 1 if I'm not mistaken.

[abkfenris]

Awesome, thank you.

[sryza]

Here's a diff: https://github.com/dagster-io/dagster/pull/4741/files

[abkfenris]

That looks great. The updated doc strings help make it much more clear that a partition is a window, not just a single time.