Removing/data path #208

emprzy · 2024-04-16T13:09:16Z

Removed all relevant instances of config['data_path'] in .py and .ipynb files.

deleted instance of config["data_path"]

jcblemai · 2024-04-16T15:33:56Z

update gempyor test config not not need datapath
add an error if data_path is present in the config
update the R code
update the R tests

Added a raise ValueError clause to catch configs that still have a data_path element

Added a raise ValueError clause to catch configs that still have a data_path element.

deleted data_path: data from config header

Deleted data_path: data from config header

Deleted data_path: data from config header.

flepimop/gempyor_pkg/tests/interface/data/config_test.yml

Deleted data_path: data from config header.

Deleted data_path: data from config header (though, it was only commented out).

Deleted data_path: data from config header.

Prepended the mobility object with the correct path (mobility.csv -> data/mobility.csv).

Added a raise ValueError clause to catch if the config being used still has a data_path section. Commented out line 579 because that portion contains config["data_path"], but the spatial_base_path variable it is read into is not referenced anywhere else in this file.

jcblemai

All good, I need to think if we need to update the R code as well before merging

First attempt at removing instances of `config$data_path` from R files.

datasetup/build_US_setup.R

datasetup/build_covid_data.R

datasetup/build_flu_data.R

flepimop/main_scripts/inference_slot.R

saraloo

Looks good to me. Re: @jcblemai 's comment on fixing the output directory: if we want to fix the directories just note that any covid or flu data should be in the data folder as we have defined them - and geodata and mobility in the model_input folders.

I'm not sure we need to define the outdir in any case as it should save to the full relative path anyway.

jcblemai · 2024-04-25T12:09:09Z

I'm not sure we need to define the outdir in any case as it should save to the full relative path anyway.

I think we do so we can create the directory if it does not exist. outdire should just be fixed.

@emprzy If sara's agree some changes are:

outdir <- 'data' in build_flu_data and build_covid data, then keeping it throughout the code (undoing the change done in this files)
outdir <- 'model_input' for build US and non US files, and keeping it throughout the code

The rest is all good to me :)

saraloo · 2024-04-25T12:51:34Z

Ah yes I see the create directory part.

Propose the following change:

instead of setting the directory (which in our case is fine because these are just files we use for covid and we have these folders already set) you could replace the lines that define the directory and create it with the following, replacing whatever the file path in the config is. eg
if(!dir.exists(dirname(config$subpop_setup$geodata))){dir.create(dirname(config$subpop_setup$geodata))}
and the equivalent for the mobility or data files

Either method work though, and everything else is also good to me.

jcblemai · 2024-04-25T14:37:07Z

agree with @saraloo sara. this is way to go @emprzy

datasetup/build_US_setup.R

Added a clause to create the appropriate directory (geodata, mobility, or gt_data) in datasetup R files.

… into removing/data-path

saraloo · 2024-04-25T18:31:20Z

datasetup/build_US_setup.R





 # MOBILITY DATA (COMMUTER DATA) ------------------------------------------------------------


-if(state_level & !file.exists(paste0(config$data_path, "/", config$subpop_setup$mobility))){
+if(state_level & !file.exists(paste0("/", config$subpop_setup$mobility))){


This also should be changed in line 172
You'll have to remove the paste0('/' as well as this will look for the file in a different place.
ie
file.exists(config$subpop_setup$mobility)

Removed references to config$data_path and "/" in paste0 functions.

jcblemai

This looks good to me now

Removing/data path

emprzy added 3 commits April 15, 2024 14:08

Update model_info.py

6ed0e66

deleted instance of config["data_path"]

Update inference_job_launcher.py

8bf7412

deleted instance of config["data_path"]

Update integration_benchmark.ipynb

8ce393d

deleted instance of config["data_path"]

emprzy requested review from jcblemai and alsnhll April 16, 2024 13:11

jcblemai linked an issue Apr 16, 2024 that may be closed by this pull request

Remove or Restructucture "data_path" option and use in config #80

Closed

emprzy added 9 commits April 16, 2024 13:45

Update inference_job_launcher.py

4388c62

Added a raise ValueError clause to catch configs that still have a data_path element

Update model_info.py

bdb16d8

Added a raise ValueError clause to catch configs that still have a data_path element.

Update config.yml

beae824

deleted data_path: data from config header

Update config_min_test.yml

be1c0e8

Deleted data_path: data from config header

Update config_minimal.yaml

4bf609c

Deleted data_path: data from config header.

Update config_test.yml

08a3885

Deleted data_path: data from config header.

Update config_minimal.yaml

31486ba

Deleted data_path: data from config header.

Update config_test.yml

bded719

Deleted data_path: data from config header.

Update config_test_spatial_group_npi.yml

af337e2

Deleted data_path: data from config header.

jcblemai reviewed Apr 17, 2024

View reviewed changes

flepimop/gempyor_pkg/tests/interface/data/config_test.yml Outdated Show resolved Hide resolved

jcblemai previously approved these changes Apr 17, 2024

View reviewed changes

Update config.yml

432f6d6

Deleted data_path: data from config header.

emprzy dismissed jcblemai’s stale review via 432f6d6 April 17, 2024 13:36

emprzy added 10 commits April 17, 2024 09:37

Update config_load.yml

4b45869

Deleted data_path: data from config header.

Update config_mc_selection.yml

acfaca4

Deleted data_path: data from config header.

Update config_npi.yml

2b2e797

Deleted data_path: data from config header.

Update config_npi_custom_pnames.yml

9f3fae9

Deleted data_path: data from config header.

Update config_test.yml

6461f55

Deleted data_path: data from config header.

Update config.yml

242ff56

Deleted data_path: data from config header.

Update config_npi.yml

2224b83

Deleted data_path: data from config header.

Update config_compartment.yml

fadee37

Deleted data_path: data from config header (though, it was only commented out).

Update config_compartmental_model_format.yml

fb8b9ef

Deleted data_path: data from config header.

Update config_compartmental_model_format_with_covariates.yml

f92709d

Deleted data_path: data from config header.

emprzy and others added 10 commits April 18, 2024 09:00

Update config_resume.yml

c0ae7ed

Prepended the mobility object with the correct path (mobility.csv -> data/mobility.csv).

Update config_seir.yml

c35ab7f

Prepended the mobility object with the correct path (mobility.csv -> data/mobility.csv).

Update config_seir_integration_method_rk4_1.yml

c4e8f8f

Prepended the mobility object with the correct path (mobility.csv -> data/mobility.csv).

Update config_seir_integration_method_rk4_2.yml

c66077e

Prepended the mobility object with the correct path (mobility.csv -> data/mobility.csv).

Update config_seir_no_dt.yml

62d68ee

Prepended the mobility object with the correct path (mobility.csv -> data/mobility.csv).

Update config_seir_no_integration.yml

568614e

Prepended the mobility object with the correct path (mobility.csv -> data/mobility.csv).

Update config_seir_unknown_integration.yml

92bc61b

Prepended the mobility object with the correct path (mobility.csv -> data/mobility.csv).

Update config_test.yml

75c448b

Prepended the mobility object with the correct path (mobility.csv -> data/mobility.csv).

Merge branch 'main' into removing/data-path

c15fe83

jcblemai previously approved these changes Apr 19, 2024

View reviewed changes

Removing config$data_path

ecb9cb9

First attempt at removing instances of `config$data_path` from R files.

emprzy dismissed jcblemai’s stale review via ecb9cb9 April 24, 2024 14:12

emprzy requested review from jcblemai and saraloo April 24, 2024 14:33

jcblemai reviewed Apr 24, 2024

View reviewed changes

saraloo reviewed Apr 24, 2024

View reviewed changes

Merge branch 'main' into removing/data-path

d038f55

saraloo reviewed Apr 25, 2024

View reviewed changes

datasetup/build_US_setup.R Show resolved Hide resolved

emprzy added 2 commits April 25, 2024 14:22

Creating appropriate directory in datasetup R files

66dba36

Added a clause to create the appropriate directory (geodata, mobility, or gt_data) in datasetup R files.

Merge branch 'removing/data-path' of https://github.com/emprzy/flepiMoP…

550958e

… into removing/data-path

saraloo reviewed Apr 25, 2024

View reviewed changes

Fixed syntax in paste0 functions

6c4b968

Removed references to config$data_path and "/" in paste0 functions.

jcblemai approved these changes Apr 26, 2024

View reviewed changes

saraloo approved these changes Apr 26, 2024

View reviewed changes

saraloo merged commit 40207b7 into HopkinsIDD:main Apr 26, 2024
1 check passed

jcblemai pushed a commit that referenced this pull request Oct 3, 2024

Merge pull request #208 from emprzy/removing/data-path

db69937

Removing/data path

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Removing/data path #208

Removing/data path #208

emprzy commented Apr 16, 2024

jcblemai commented Apr 16, 2024

jcblemai left a comment

saraloo left a comment

jcblemai commented Apr 25, 2024

saraloo commented Apr 25, 2024 •

edited

Loading

jcblemai commented Apr 25, 2024

saraloo Apr 25, 2024

jcblemai left a comment •

edited

Loading

Removing/data path #208

Removing/data path #208

Conversation

emprzy commented Apr 16, 2024

jcblemai commented Apr 16, 2024

jcblemai left a comment

Choose a reason for hiding this comment

saraloo left a comment

Choose a reason for hiding this comment

jcblemai commented Apr 25, 2024

saraloo commented Apr 25, 2024 • edited Loading

jcblemai commented Apr 25, 2024

saraloo Apr 25, 2024

Choose a reason for hiding this comment

jcblemai left a comment • edited Loading

Choose a reason for hiding this comment

saraloo commented Apr 25, 2024 •

edited

Loading

jcblemai left a comment •

edited

Loading