Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hubValidations::validate_pr() fails with generic error [Error: The operation was canceled] #115

Open
M-7th opened this issue Sep 3, 2024 · 1 comment

Comments

@M-7th
Copy link

M-7th commented Sep 3, 2024

With the current quite complex tasks.json configuration, we are experiencing some issues with the validation phase.
After running for a variable time ( from few minutes to even 4/6 hours, depending on the input file) the validation process, that runs inside a github actions, fails with a generic error Error: The operation was canceled. The runner has received a shutdown signal.
Running the same validation process manually on a local machine, we notice a high memory consumption ( > 25 GB).
In a couple of test we also get a more specific error:

── Individual check results ──

✔ ScenarioModellingHub: All hub config files are valid.
✔ 2024_2025_1_FLU-ISI-TestModel.parquet: File exists at path model-output/ISI-TestModel/2024_2025_1_FLU-ISI-TestModel.parquet.
✔ 2024_2025_1_FLU-ISI-TestModel.parquet: File name "2024_2025_1_FLU-ISI-TestModel.parquet" is valid.
✔ 2024_2025_1_FLU-ISI-TestModel.parquet: File directory name matches `model_id` metadata in file name.
✔ 2024_2025_1_FLU-ISI-TestModel.parquet: `round_id` is valid.
✔ 2024_2025_1_FLU-ISI-TestModel.parquet: File is accepted hub format.
✔ 2024_2025_1_FLU-ISI-TestModel.parquet: Metadata file exists at path model-metadata/ISI-TestModel.yml.
✔ 2024_2025_1_FLU-ISI-TestModel.parquet: File could be read successfully.
✔ 2024_2025_1_FLU-ISI-TestModel.parquet: `round_id_col` name is valid.
✔ 2024_2025_1_FLU-ISI-TestModel.parquet: `round_id` column "round_id" contains a single, unique round ID value.
✔ 2024_2025_1_FLU-ISI-TestModel.parquet: All `round_id_col` "round_id" values match submission `round_id` from file name.
✔ 2024_2025_1_FLU-ISI-TestModel.parquet: Column names are consistent with expected round task IDs and std column names.
✔ 2024_2025_1_FLU-ISI-TestModel.parquet: Column data types match hub schema.
✖ 2024_2025_1_FLU-ISI-TestModel.parquet: EXEC ERROR: Error in purrr::map2(task_id_l, output_type_l, ~expand_output_type_grid(task_id_values = .x, : ℹ
  In index: 1. Caused by error: ! vector memory exhausted (limit reached?)
✔ 2024_2025_1_FLU-ISI-TestModel.parquet: Submission time is within accepted submission window for round.

── Overall validation result ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
✖ 2024_2025_1_FLU-ISI-TestModel.parquet: EXEC ERROR: Error in purrr::map2(task_id_l, output_type_l, ~expand_output_type_grid(task_id_values = .x, : ℹ
  In index: 1. Caused by error: ! vector memory exhausted (limit reached?)
Error in `hubValidations::check_for_errors()`:

We created a test repo with a configuration mirroring the one used in Respicompass and the same validation workflow here
https://github.com/Testing-Forecast-Actions/TestingValidations

A file for testing the validation process through Pull Request is available under the folder Testing-Forecast-Actions/TestingValidations//submission-test-files

You can activate the validation process by forking the hub, copying the test file into the model-output/ISI-TestModel folder and contributing with a Pull Request

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Todo
Development

No branches or pull requests

2 participants