Skip to content

Commit

Permalink
Add "tiers" to data_index (#1147)
Browse files Browse the repository at this point in the history
This is the first step in moving our `compute` infrastructure from a
recursive model to a looped one, which is potentially faster to compile
and requires less memory.

For now all this does is automatically assign a "tier" to each
computeable quantity that indicates how "deep" in the dependency tree
that quantity lies.

tier of 0 means no dependencies on other data, but may depend on
transforms, profiles etc.
tier of 1 means it depends on only tier 0 stuff,
tier of 2 means it depends on tier 0 and tier 1, etc etc.

The goal is that if you were to compute things in a loop with the order
determined by the tiers, then it would always compute dependencies in
the correct order.
  • Loading branch information
f0uriest authored Aug 11, 2024
2 parents 9f907ca + 2953d8b commit 5c309d8
Showing 1 changed file with 27 additions and 0 deletions.
27 changes: 27 additions & 0 deletions desc/compute/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -94,3 +94,30 @@ def _build_data_index():


_build_data_index()


def set_tier(name, p):
"""Determine how deep in the dependency tree a given name is.
tier of 0 means no dependencies on other data,
tier of 1 means it depends on only tier 0 stuff,
tier of 2 means it depends on tier 0 and tier 1, etc etc.
Designed such that if you compute things in the order determined by tiers,
all dependencies will always be computed in the correct order.
"""
if "tier" in data_index[p][name]:
return
if len(data_index[p][name]["full_with_axis_dependencies"]["data"]) == 0:
data_index[p][name]["tier"] = 0
else:
thistier = 0
for name1 in data_index[p][name]["full_with_axis_dependencies"]["data"]:
set_tier(name1, p)
thistier = max(thistier, data_index[p][name1]["tier"])
data_index[p][name]["tier"] = thistier + 1


for par in data_index.keys():
for name in data_index[par]:
set_tier(name, par)

0 comments on commit 5c309d8

Please sign in to comment.