Rework math handling (#639)

* Isolate _def_path, scenario overrides, and CalliopeMath object. * Removed _model_def_dict. * Move math to `build` step. * Add `ignore_mode_math` to not load any of the internal math. * Update example clustering so that representative days _must_ represent themselves. * `pd.notnull` -> `pd.notna`. --------- Co-authored-by: Bryn Pickering <[email protected]>
calliope-project · Sep 30, 2024 · 0fc4f5e · 0fc4f5e
1 parent 99e5211
commit 0fc4f5e
Show file tree

Hide file tree

Showing 48 changed files with 1,600 additions and 1,474 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -2,6 +2,8 @@
 
 ### User-facing changes
 
+|new| Math has been removed from `model.math`, and can now be accessed via `model.math.data` (#639).
+
 |new| (non-NaN) Default values and data types for parameters appear in math documentation (if they appear in the model definition schema) (#677).
 
 |changed| `data_sources` -> `data_tables` and `data_sources.source` -> `data_tables.data`.
@@ -63,6 +65,12 @@ Parameter titles from the model definition schema will also propagate to the mod
 
 ### Internal changes
 
+|changed| `model._model_def_dict` has been removed.
+
+|new| `CalliopeMath` is a new helper class to handle math additions, including separate methods for pre-defined math, user-defined math and validation checks.
+
+|changed| `MathDocumentation` has been extracted from `Model`/`LatexBackend`, and now is a postprocessing module which can take models as input.
+
 |new| `gurobipy` is a development dependency that will be added as an optional dependency to the conda-forge calliope feedstock recipe.
 
 |changed| Added any new math dicts defined with `calliope.Model.backend.add_[...](...)` to the backend math dict registry stored in `calliope.Model.backend.inputs.attrs["math"]`.

diff --git a/docs/examples/calliope_model_object.py b/docs/examples/calliope_model_object.py
@@ -36,33 +36,10 @@
 # Get information on the model
 print(m.info())
 
-# %% [markdown]
-# ## Model definition dictionary
-#
-# `m._model_def_dict` is a python dictionary that holds all the data from the model definition YAML files, restructured into one dictionary.
-#
-# The underscore before the method indicates that it defaults to being hidden (i.e. you wouldn't see it by trying a tab auto-complete and it isn't documented)
-
-# %%
-m._model_def_dict.keys()
-
-# %% [markdown]
-# `techs` hold only the information about a technology that is specific to that node
-
-# %%
-m._model_def_dict["techs"]["pv"]
-
-# %% [markdown]
-# `nodes` hold only the information about a technology that is specific to that node
-
-# %%
-m._model_def_dict["nodes"]["X2"]["techs"]["pv"]
-
 # %% [markdown]
 # ## Model data
 #
-# `m._model_data` is an xarray Dataset.
-# Like `_model_def_dict` it is a hidden prperty of the Model as you are expected to access the data via the public property `inputs`
+# `m._model_data` is an xarray Dataset, a hidden property of the Model as you are expected to access the data via the public property `inputs`
 
 # %%
 m.inputs

diff --git a/docs/examples/piecewise_constraints.py b/docs/examples/piecewise_constraints.py
@@ -68,22 +68,20 @@
 #
 
 # %%
-new_params = {
-    "parameters": {
-        "capacity_steps": {
-            "data": capacity_steps,
-            "index": [0, 1, 2, 3, 4],
-            "dims": "breakpoints",
-        },
-        "cost_steps": {
-            "data": cost_steps,
-            "index": [0, 1, 2, 3, 4],
-            "dims": "breakpoints",
-        },
-    }
-}
+new_params = f"""
+  parameters:
+    capacity_steps:
+      data: {capacity_steps}
+      index: [0, 1, 2, 3, 4]
+      dims: "breakpoints"
+    cost_steps:
+      data: {cost_steps}
+      index: [0, 1, 2, 3, 4]
+      dims: "breakpoints"
+"""
 print(new_params)
-m = calliope.examples.national_scale(override_dict=new_params)
+new_params_as_dict = calliope.AttrDict.from_yaml_string(new_params)
+m = calliope.examples.national_scale(override_dict=new_params_as_dict)
 
 # %%
 m.inputs.capacity_steps
@@ -95,55 +93,48 @@
 # ## Creating our piecewise constraint
 #
 # We create the piecewise constraint by linking decision variables to the piecewise curve we have created.
-# In this example, we require a new decision variable for investment costs that can take on the value defined by the curve at a given value of `flow_cap`.
+# In this example, we need:
+# 1. a new decision variable for investment costs that can take on the value defined by the curve at a given value of `flow_cap`;
+# 1. to link that decision variable to our total cost calculation; and
+# 1. to define the piecewise constraint.
 
 # %%
-m.math["variables"]["piecewise_cost_investment"] = {
-    "description": "Investment cost that increases monotonically",
-    "foreach": ["nodes", "techs", "carriers", "costs"],
-    "where": "[csp] in techs",
-    "bounds": {"min": 0, "max": np.inf},
-    "default": 0,
-}
-
-# %% [markdown]
-# We also need to link that decision variable to our total cost calculation.
-
-# %%
-# Before
-m.math["global_expressions"]["cost_investment_flow_cap"]["equations"]
-
-# %%
-# Updated - we split the equation into two expressions.
-m.math["global_expressions"]["cost_investment_flow_cap"]["equations"] = [
-    {"expression": "$cost_sum * flow_cap", "where": "NOT [csp] in techs"},
-    {"expression": "piecewise_cost_investment", "where": "[csp] in techs"},
-]
-
-# %% [markdown]
-# We then need to define the piecewise constraint:
-
-# %%
-m.math["piecewise_constraints"]["csp_piecewise_costs"] = {
-    "description": "Set investment costs values along a piecewise curve using special ordered sets of type 2 (SOS2).",
-    "foreach": ["nodes", "techs", "carriers", "costs"],
-    "where": "piecewise_cost_investment",
-    "x_expression": "flow_cap",
-    "x_values": "capacity_steps",
-    "y_expression": "piecewise_cost_investment",
-    "y_values": "cost_steps",
-}
-
-# %% [markdown]
-# Then we can build our optimisation problem:
+new_math = """
+  variables:
+    piecewise_cost_investment:
+      description: "Investment cost that increases monotonically"
+      foreach: ["nodes", "techs", "carriers", "costs"]
+      where: "[csp] in techs"
+      bounds:
+        min: 0
+        max: .inf
+      default: 0
+  global_expressions:
+    cost_investment_flow_cap:
+      equations:
+        - expression: "$cost_sum * flow_cap"
+          where: "NOT [csp] in techs"
+        - expression: "piecewise_cost_investment"
+          where: "[csp] in techs"
+  piecewise_constraints:
+    csp_piecewise_costs:
+      description: "Set investment costs values along a piecewise curve using special ordered sets of type 2 (SOS2)."
+      foreach: ["nodes", "techs", "carriers", "costs"]
+      where: "piecewise_cost_investment"
+      x_expression: "flow_cap"
+      x_values: "capacity_steps"
+      y_expression: "piecewise_cost_investment"
+      y_values: "cost_steps"
+"""
 
 # %% [markdown]
 # # Building and checking the optimisation problem
 #
-# With our piecewise constraint defined, we can build our optimisation problem
+# With our piecewise constraint defined, we can build our optimisation problem and inject this new math.
 
 # %%
-m.build()
+new_math_as_dict = calliope.AttrDict.from_yaml_string(new_math)
+m.build(add_math_dict=new_math_as_dict)
 
 # %% [markdown]
 # And we can see that our piecewise constraint exists in the built optimisation problem "backend"
@@ -190,65 +181,6 @@
 )
 fig.show()
 
-# %% [markdown]
-# ## YAML model definition
-# We have updated the model parameters and math interactively in Python in this tutorial, the definition in YAML would look like:
-
-# %% [markdown]
-# ### Math
-#
-# Saved as e.g., `csp_piecewise_math.yaml`.
-#
-# ```yaml
-# variables:
-#   piecewise_cost_investment:
-#     description: Investment cost that increases monotonically
-#     foreach: [nodes, techs, carriers, costs]
-#     where: "[csp] in techs"
-#     bounds:
-#       min: 0
-#       max: .inf
-#     default: 0
-#
-# piecewise_constraints:
-#   csp_piecewise_costs:
-#     description: >
-#       Set investment costs values along a piecewise curve using special ordered sets of type 2 (SOS2).
-#     foreach: [nodes, techs, carriers, costs]
-#     where: "[csp] in techs"
-#     x_expression: flow_cap
-#     x_values: capacity_steps
-#     y_expression: piecewise_cost_investment
-#     y_values: cost_steps
-#
-# global_expressions:
-#   cost_investment_flow_cap.equations:
-#     - expression: "$cost_sum * flow_cap"
-#       where: "NOT [csp] in techs"
-#     - expression: "piecewise_cost_investment"
-#       where: "[csp] in techs"
-# ```
-
-# %% [markdown]
-# ### Scenario definition
-#
-# Loaded into the national-scale example model with: `calliope.examples.national_scale(scenario="piecewise_csp_cost")`
-#
-# ```yaml
-# overrides:
-#   piecewise_csp_cost:
-#     config.init.add_math: [csp_piecewise_math.yaml]
-#     parameters:
-#       capacity_steps:
-#         data: [0, 2500, 5000, 7500, 10000]
-#         index: [0, 1, 2, 3, 4]
-#         dims: "breakpoints"
-#       cost_steps:
-#         data: [0, 3.75e6, 6e6, 7.5e6, 8e6]
-#         index: [0, 1, 2, 3, 4]
-#         dims: "breakpoints"
-# ```
-
 # %% [markdown]
 # ## Troubleshooting
 #

diff --git a/docs/hooks/dummy_model/model.yaml b/docs/hooks/dummy_model/model.yaml
@@ -2,8 +2,9 @@ overrides:
   storage_inter_cluster:
     config.init:
       name: inter-cluster storage
-      add_math: ["storage_inter_cluster"]
       time_cluster: cluster_days.csv
+    config.build:
+      add_math: ["storage_inter_cluster"]
 
 config.init.name: base