Code

Feat/boundary dataloader#90

Code

Open

sadamov wants to merge 88 commits intomllam:mainmllam/neural-lam:mainfrom sadamov:feat/boundary_dataloadersadamov/neural-lam:feat/boundary_dataloaderCopy head branch name to clipboard

+1,240-433

Commits

Commits on Nov 21, 2024

• add datastore_boundary to neural_lam

  sadamov

  committed
  5df1bff

• complete integration of boundary in weatherDataset

  sadamov

  committed
  46590ef

• Add test to check timestep length and spacing

  sadamov

  committed
  b990f49

• setting default mdp boundary to 0 gridcells

  sadamov

  committed
  3fd1d6b

• implement time-based slicing

  Show description for 1f2499c
  sadamov

  committed
  1f2499c

• remove all interior_mask and boundary_mask

  sadamov

  committed
  1af1481

• added gcsfs dependency for era5 weatherbench download

  sadamov

  committed
  d545cb7

• added new era5 datastore config for boundary

  sadamov

  committed
  5c1a7d7

• removed left-over boundary-mask references

  sadamov

  committed
  30e4f05

• make check for existing category in datastore more flexible (for boundary)

  sadamov

  committed
  6a8c593

• implement xarray based (mostly) time slicing and windowing

  sadamov

  committed
  17c920d

• cleanup analysis based time-slicing

  sadamov

  committed
  7919995

• implement datastore_boundary in existing tests

  sadamov

  committed
  9bafcee

• allow for grid shape retrieval from forcing data

  sadamov

  committed
  ce06bbc

• rearrange time slicing, boundary first

  sadamov

  committed
  884b5c6

Commits on Nov 25, 2024

• identified issue, cleanup next

  leifdenby

  committed
  5904cbe

Commits on Nov 26, 2024

• use xarray plot only

  leifdenby

  committed
  efe0302

• don't reraise

  leifdenby

  committed
  a489c2e

• remove debug plot

  leifdenby

  committed
  242d08b

• remove extent calc used in diagnosing issue

  leifdenby

  committed
  c1f706c

Commits on Nov 29, 2024

• add type annotation

  leifdenby

  committed
  cf8e3e4

• ensure tensor copy to cpu mem before data-array creation

  leifdenby

  committed
  85160ce

• apply time-indexing to support ar_steps_val > 1

  leifdenby

  committed
  52c4528

Commits on Nov 30, 2024

• renaming test datastores

  sadamov

  committed
  b96d8eb

• adding num_past/future_boundary_step args

  sadamov

  committed
  72da25f

• using combined config file

  sadamov

  committed
  244f1cc

• proper handling of state/forcing/boundary in dataset

  sadamov

  committed
  a9cc36e

• datastore_boundars=None introduced

  sadamov

  committed
  dcc0b46

• bug fix for file retrieval per member

  sadamov

  committed
  a3b3bde

• rename datastore for tests

  sadamov

  committed
  3ffc413

• aligned time with danra for easier boundary testing

  sadamov

  committed
  85aad66

• Fixed test for temporal embedding

  sadamov

  committed
  64f057f

Commits on Dec 2, 2024

• pin dataclass-wizard <0.31.0 to avoid bug in dataclass-wizard

  leifdenby

  committed
  6205dbd

• allow boundary as input to ar_model.common_step

  sadamov

  committed
  551cd26

• linting

  sadamov

  committed
  fc95350

• improved docstrings and added some assertions

  sadamov

  committed
  01fa807

• update mdp dependency

  sadamov

  committed
  5a749f3

• remove boundary datastore from tests that don't need it

  sadamov

  committed
  45ba607

• fix scope of _get_slice_time

  sadamov

  committed
  f36f360

• fix scope of _get_time_step

  sadamov

  committed
  105108e

• Merge branch 'feat/boundary_dataloader' of https://github.com/sadamov/neural-lam into feat/boundary_dataloader

  sadamov

  committed
  d760145

• added information about optional boundary datastore

  sadamov

  committed
  ae0cf76

• add datastore_boundary to neural_lam

  sadamov

  committed
  9af27e0

• complete integration of boundary in weatherDataset

  sadamov

  committed
  c25fb30

• Add test to check timestep length and spacing

  sadamov

  committed
  505ceeb

• setting default mdp boundary to 0 gridcells

  sadamov

  committed
  e733066

• implement time-based slicing

  Show description for d8349a4
  sadamov

  committed
  d8349a4

• remove all interior_mask and boundary_mask

  sadamov

  committed
  fd791bf

• added gcsfs dependency for era5 weatherbench download

  sadamov

  committed
  ae82cdb

• added new era5 datastore config for boundary

  sadamov

  committed
  34a6cc7

• removed left-over boundary-mask references

  sadamov

  committed
  2dc67a0

• make check for existing category in datastore more flexible (for boundary)

  sadamov

  committed
  9f8628e

• implement xarray based (mostly) time slicing and windowing

  sadamov

  committed
  388c79d

• cleanup analysis based time-slicing

  sadamov

  committed
  2529969

• implement datastore_boundary in existing tests

  sadamov

  committed
  179a035

• allow for grid shape retrieval from forcing data

  sadamov

  committed
  2daeb16

• rearrange time slicing, boundary first

  sadamov

  committed
  cbcdcae

• renaming test datastores

  sadamov

  committed
  e6ace27

• adding num_past/future_boundary_step args

  sadamov

  committed
  42818f0

• using combined config file

  sadamov

  committed
  0103b6e

• proper handling of state/forcing/boundary in dataset

  sadamov

  committed
  0896344

• datastore_boundars=None introduced

  sadamov

  committed
  355423c

• bug fix for file retrieval per member

  sadamov

  committed
  121d460

• rename datastore for tests

  sadamov

  committed
  7e82eef

• aligned time with danra for easier boundary testing

  sadamov

  committed
  320d7c4

• Fixed test for temporal embedding

  sadamov

  committed
  f18dcc2

• allow boundary as input to ar_model.common_step

  sadamov

  committed
  e6327d8

• linting

  sadamov

  committed
  1374a19

• improved docstrings and added some assertions

  sadamov

  committed
  779f3e9

• remove boundary datastore from tests that don't need it

  sadamov

  committed
  f126ec2

• fix scope of _get_time_step

  sadamov

  committed
  4b656da

• added information about optional boundary datastore

  sadamov

  committed
  75db4b8

• Merge branch 'feat/boundary_dataloader' of https://github.com/sadamov/neural-lam into feat/boundary_dataloader

  sadamov

  committed
  58b4af6

Commits on Dec 3, 2024

• moved gcsfs to dev group

  sadamov

  committed
  4c17545

• linting

  sadamov

  committed
  a700350

• Fixed issue with temporal encoding dimensions

  Show description for 16d5d04
  sadamov

  committed
  16d5d04

• format docstrings

  sadamov

  committed
  f1f3f73

• introduced time slicing test for forecast type data

  sadamov

  committed
  8fd7a10

• bugfix temporal embedding dimension

  sadamov

  committed
  252a33c

• linting

  sadamov

  committed
  8a9114a

• switched to low-res data

  sadamov

  committed
  8c7709a

• add datastore_boundary as explicit attribute

  sadamov

  committed
  24cbf13

Commits on Dec 5, 2024

• fixing up forecast type data tests,

  Show description for 1d53ce7
  sadamov

  committed
  1d53ce7

• time step can and should be retrieved in __init__

  Show description for cfe1e27
  sadamov

  committed
  cfe1e27

• Fix dataset issue in npy stat script

  

  joeloskarsson

  authored and

  sadamov

  committed
  e4e4e37

• Merge remote-tracking branch 'mllam/main' into feat/boundary_dataloader

  sadamov

  committed
  3df3fcb

• added static feature to era5 boundary test datastore

  sadamov

  committed
  f8613da

Commits on Dec 6, 2024

• Merge remote-tracking branch 'mllam/main' into feat/boundary_dataloader

  sadamov

  committed
  f0a7046