Add scripts to generate profiles for HIFLD grid

[danielolsen]

🚀

• Is your feature request essential for your project?

Describe the workflow you want to enable

Similar to what was done for the TAMU Grid, we want scripts that can ingest data from external sources and produce profiles formatted for intake by PowerSimData/REISE.jl.

[danielolsen]

Using some work-in-progress branches, I've got a workflow that appears to be working at least for the generation of wind profiles:

1. Begin with a pipenv virtual environment that contains both the daniel/read_hifld_grid branch of PowerSimData and the daniel/decouple_profiles_from_grid branch of PreREISE, and with a folder of grid CSVs created by the hifld branch of PreREISE (we'll call this folder hifld_csvs for this example).
2. Within bash, copy the HIFLD CSVs into the appropriate location within the installed version of PowerSimData so that they can be read to create an HIFLD Grid object:

mkdir $(pipenv --venv)/src/powersimdata/powersimdata/network/hifld/data
cp -r test_csvs/. $(pipenv --venv)/src/powersimdata/powersimdata/network/hifld/data/

3. Start a python session and follow the HRRR demo to download wind speed data and convert it to power profiles:

from datetime import datetime
from powersimdata import Grid
from prereise.gather.winddata.hrrr.hrrr import retrieve_data
from prereise.gather.winddata.hrrr.calculations import calculate_pout

start_dt = datetime.fromisoformat("2020-01-01")
end_dt = datetime.fromisoformat("2021-01-01")
directory = "./"

grid = Grid("USA", "hifld")
wind_farms = grid.plant.query("type == 'wind' or type == 'wind_offshore'").copy()
wind_farms["state_abv"] = wind_farms.zone_id.map(grid.model_immutables.zones["id2abv"])
retrieve_data(start_dt=start_dt, end_dt=end_dt, directory=directory)
df = calculate_pout(wind_farms=wind_farms, start_dt=start_dt, end_dt=end_dt, directory=directory)
df.to_csv("wind.csv")

Downloading the wind data will take several hours, I'll update this post once it's done and I can confirm whether the method worked all the way through. EDIT: using the daniel/decouple_profiles_from_grid branch and the branch from #245 (rebased together), I was able to successfully download wind speeds and convert them to power profiles for the HIFLD grid.

[danielolsen]

Getting solar data can follow a similar pattern. Repeat steps 1 and 2 from wind data procedure above, make sure you have the SAM binaries installed, then:

from powersimdata import Grid
from prereise.gather.solardata.nsrdb import sam
from prereise.gather.solardata.helpers import to_reise
grid = Grid("USA", "hifld")
solar_plant = grid.plant.query("type == 'solar'").copy()
solar_plant.index.name = "plant_id"
data = sam.retrieve_data(YOUR_EMAIL_HERE, YOUR_NREL_API_KEY_HERE, solar_plant=solar_plant, grid_model="hifld", year=2020)

Note that the call signature for sam.retrieve_data has changed, so that it doesn't depend on a hard-coded reference to the usa_tamu grid constants, but can either take a grid (which will contain both the solar data and the model immutables) or the solar data and a grid model string which is used to instantiate a new ModelImmutables object.

[rouille]

Getting solar data can follow a similar pattern. Repeat steps 1 and 2 from wind data procedure above, then:

from powersimdata import Grid
from prereise.gather.solardata.nsrdb import sam
from prereise.gather.solardata.helpers import to_reise
grid = Grid("USA", "hifld")
solar_plant = grid.plant.query("type == 'solar'").copy()
solar_plant.index.name = "plant_id"
data = sam.retrieve_data(YOUR_EMAIL_HERE, YOUR_NREL_API_KEY_HERE, solar_plant=solar_plant, grid_model="hifld", year=2020)

Note that the call signature for sam.retrieve_data has changed, so that it doesn't depend on a hard-coded reference to the usa_tamu grid constants, but can either take a grid (which will contain both the solar data and the model immutables) or the solar data and a grid model string which is used to instantiate a new ModelImmutables object.

Don't forget to update the notebook!

[danielolsen]

Looking at the wind and solar code, there seem to be two different 'levels' of making new profiles for the HIFLD grid:

1. Use the existing functions within PreREISE, which create 'blended' attributes by state (or by interconnection, if no plants of a given type in the state), based on CSVs of plant data from 2016.
2. Add more recent EIA data tables to the blob storage, which match the generator tables we're using, and use the plant-specific attributes to convert wind speed and irradiance to power output (ideally avoiding the need for any 'blended' attribute fallback profiles).

[danielolsen]

Now that #247 decouples solar profile generation from the Grid object, we could generate plant-specific profiles as part of the top-level grid-CSV-creation step, while we still have all the 'extra' data columns that are required to associate the main Form 860 records with the 'ancillary' table for solar-specific data. Once #249 is integrated, we could do the same for wind profiles. See #242 (comment) for more context.

[danielolsen]

I believe this will be satisfied upon completion of #256, #260, and #261.

[danielolsen]

Now that I'm coming back to this, I'm realizing that we still need scripts for the creation of demand profiles. If our demand zones match the hourly data available from the EIA (https://www.eia.gov/opendata/qb.php?category=2122628), then we can either download and clean their data, or download pre-cleaned data from https://zenodo.org/record/4116342. If our demand zones don't match the EIA, then we will need another step of processing to re-align them.