import script that uses json config #612
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
struan
force-pushed
the
imports_as_config
branch
2 times, most recently
from
ed857c8 to
ffe90b8
Compare
|
Note that once the ONS data is fetched (one file per variable) you need to pivot the data to get it into one row per constituency so: |
struan
changed the title
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #612 +/- ##
==========================================
- Coverage 78.93% 78.70% -0.23%
==========================================
Files 115 115
Lines 3921 3940 +19
Branches 410 415 +5
==========================================
+ Hits 3095 3101 +6
- Misses 720 728 +8
- Partials 106 111 +5 ☔ View full report in Codecov by Sentry. |
struan
changed the title
struan
force-pushed
the
imports_as_config
branch
from
b89b7f2 to
c028b68
Compare
Enable different names and labels for the dataset and type to make creating range datasets easier
previously the only way was to override the delete method which is slightly overkill when you can just set a property
This is helpful if you have multiple sheets in an excel file with inconsistent column names, or if there is just no name
Do not always cast to int and hence lose precision
This adds an `import_from_config` command that gets the details of the imports from a JSON file. You supply the location of the file and the name of the config you want and then it pulls in all the information from the config. At the moment it only works with CSV and Excel files where the data is one row per constituency.
Adds the following for post 2024 constituencies: * constituency age distributution (HoC) * constituency housing tenure (HoC) * constituency population (ONS) * constituency commute method (ONS)
Mostly so data where there are multiple rows per constituency can be turned into one row per constituency with multuple columns of data. i.e. turn cons|data_name|data_value CONS1|badgers|12 CONS1|snakes|3 CONS2|badgers|8 CONS2|snakes|4 to cons|badgers|snakes CONS1|12|3 CONS2|8|4 which you would do with ``` ./manage.py pivot_csv --infile in.csv --outfile out.csv --column data_name --value_colmn data_value --index cons ``` This is mostly because the import from config only copes with the second of these and ONS provides data in the first
struan
force-pushed
the
imports_as_config
branch
from
c028b68 to
c446ec2
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This should enable add simple imports by adding to the JSON rather than writing a new script.
Also includes some examples from ONS and Commons Library plus a utility command to pivot CSVs to help with ONS imports.