feat: generate solar profiles during HIFLD grid-building

[danielolsen]

Purpose

• Generate solar profiles at the same time as we build the HIFLD grid, so that we can leverage the plant-specific attributes of each solar farm to generate better profiles (partial fulfillment of Add scripts to generate profiles for HIFLD grid #227).
• Add a module to generate profiles for concentrating solar power (CSP) plants, which represent about 5% of total solar capacity (closes Add a module to create profiles for solar thermal generators #248).
• Simplify solar profile generation process, now that SAM can handle leap years. EDIT: this no longer appears possible, see Update solar profile generation to reflect that nrel-pysam can now handle leap years #254 (comment)
• Update the version of NREL's PySAM used, for compatibility with Python 3.10, and test using Python 3.10.

What the code is doing

• Within Pipfile, Pipfile.lock, and requirements.txt: updating the version of PySAM. Then, we add Python 3.10 to the test matrix in .github/workflows/test.yml.
• Within const.py, we add references to the tables of 'extra' EIA data in the blob storage.
• Within generators.py, we make sure that the plant index gets named (unrelated to main objectives).
• We add a new module prereise.gather.griddata.hifld.data_process.profiles, which takes the 'extra' data table for solar, joins it with the solar plants from the plant dataframe, and passes this (with some pre-processing) to individualized profile generation function introduced in refactor/feat: decouple solar profile generation from Grid object, add individual calculation #247. Most of this new code is copy-and-pasted from the usage examples in that PR.
• Within orchestration.py, we call the new function from prereise.gather.griddata.hifld.data_process.profiles, and then save the resulting profiles to CSV at the same time as we save the rest of the Grid CSVs.
• Within nrel_api.py:
  • we add a method that converts the Psm3Data data structure back to the file format expected by SAM. For whatever reason, the way that weather data are attached to a PySAM.PVWatts object is different than how they're attached to a PySAM.TcsgenericSolar object: TcsgenericSolar needs a filename for a file formatted in a particular way.
  • we make the downloading more resilient to server-side errors.
• Within sam.py:
  • we remove the leap-year-specific logic, now that SAM handles lear years properly.
  • we query the full year's weather data during leap years, since SAM can take it as input, but we still need to manually copy the Feb 28th solar data to leap days, since SAM doesn't produce power output for Feb 29th.
  • we rename the PV-profile-generating function from calculate_power to calculate_power_pv, and add a parallel calculate_power_csp function, which saves weather data into a temporary location and then tells the CSP model to load it from this location (it's ugly, but I don't know if there's a better way).
  • within retrieve_data_individual, we look at each plant's prime mover code to determine whether to calculate profiles using the PV model or the CSP model.
  • miscellaneous docstring fixes.

Testing

CSP profile generation code has been manually tested. Manual testing of profiles for the full solar fleet is in progress, should be done by this afternoon (data are being re-downloaded from NREL with leap days). EDIT: tested manually, the solar profiles are generated and saved and match the number of solar plants in the plant CSV table.

Usage Example/Visuals

from prereise.gather.griddata.hifld import create_csvs
create_csvs(
    DESIRED_OUTPUT_CSV_DIRECTORY,
    YOUR_NREL_EMAIL,
    YOUR_NREL_API_KEY,
    {"year": 2020, "cache_dir": YOUR_CACHE_DIR},
)

Time estimate

1 hour. None of the individual pieces are too complex, but there are a lot of moving parts here. Assuming that y'all approve of the overall logic flow, I'll submit a follow-up PR to add wind profile generation to the orchestration script as well (or I can add it to this PR).

[danielolsen]

From the user perspective, do we want to enable the user to generate the grid without generating the profiles? If so, we could add a flag input, and add default None values for the profile-specific input parameters.

[danielolsen]

Alternatively, we may want the user to be able to create the full Grid CSVs, and then use these for multiple years. We could break create_csvs into a grid-step and a profiles step, or allow the user to pass a list of years to have everything handled automatically.

[jenhagg]

Agree it makes sense to be able to create the grid and profiles separately. Assuming this isn't something that would be used often, I might leave it at something like create_grid and create_profiles, or make create_csvs a wrapper around those if it's convenient.

[danielolsen]

Done. EDIT: The refactor has also been tested and behaves as expected when calling the create_csvs wrapper function.

[danielolsen]

There's another branch daniel/hifld_wind_profiles which adds the wind profile creation step as well, see daniel/hifld_solar_profiles...daniel/hifld_wind_profiles.

[danielolsen]

There's another branch daniel/hifld_wind_profiles which adds the wind profile creation step as well, see daniel/hifld_solar_profiles...daniel/hifld_wind_profiles.

Similarly, there's a branch to add hydro profiles, see daniel/hifld_solar_profiles...daniel/hifld_hydro_profiles.

[jenhagg]

Technically I think metadata_values is a tuple, but would expect that's fine.

[kasparm]

Logic looks good.
Solar profile and grid-building works.
Thank you!