Merge branch 'master' into null-vs-0-for-value_as_concept_id,-operato…

…r_concept_id

aurum_etl_ndorms/0_postgres_db_conf.py

@@ -3,15 +3,17 @@
     'password': 'PASSWORD',
     'database': 'DATABASE',
     'source_schema': 'SOURCE_SCHEMA',
+	'source_nok_schema': 'source_nok',
     'target_schema': 'TARGET_SCHEMA',
     'vocabulary_schema': 'VOCABULARY_SCHEMA',
+	'dir_processed': '/processed/',
     'chunk_size': 1000,
     'chunk_limit': -1,
-    'log_directory': 'C:FULL\\PATH\\TO\\LOG_DIRECTORY',
     'source_data_directory': 'C:FULL\\PATH\\TO\\SOURCE\\DATA',
     'lookup_data_directory': 'C:FULL\\PATH\\TO\\AURUM\\LOOKUPS',
     'vocabulary_directory': 'C:FULL\\PATH\\TO\\CDM_VOCABULARY',
     'stcm_directory': 'C:FULL\\PATH\\TO\\STCM_DIRECTORY',
+    'log_directory': 'C:FULL\\PATH\\TO\\LOG_DIRECTORY',
     'medical_dictionary_filename': 'MEDICAL_DICTIONARY_FILENAME.txt',
     'product_dictionary_filename': 'PRODUCT_DICTIONARY_FILENAME.txt'
 })

docs/CPRD/CPRD_Additional_STEM.md

@@ -40,7 +40,7 @@ These concatenated source values will then be mapped to standard concepts using
 | unit_concept_id |  | Look up add_int.unit_source_value in the CONCEPT table where vocabulary_id = 'UCUM' and standard_concept = 'S' and invalid_reason is NULL. |  |
 | unit_source_value | add_int.unit_source_value |  | |
 | start_date | add_int.eventdate |  | For the additional table, the adid is used to link back to the clinical table to get the eventdate. |
-| end_date | start_date |  |  |
+| end_date | NULL |  |  |
 | value_as_number | add_int.value_as_number |   | |
 | value_as_string | add_int.value_as_string  | |  |
 | value_as_concept_id |  |  | If the last part of the source value says 'Read code for condition' then map the code in add_int.value_as_string to a standard concept using the SOURCE_TO_STANDARD query with the filters:<br><br>    WHERE source_vocabulary_id = 'Read'  AND standard_concept = 'S'  AND invalid_concept is NULL  <br><br>   If the last part of the source value says 'Drug code' then map the code in add_int.value_as_string to a standard concept using the SOURCE_TO_STANDARD query with the filters:  <br><br>   WHERE source_vocabulary_id = 'Gemscript'  AND standard_concept = 'S'  AND invalid_concept is NULL  <br><br>   Otherwise, if the value in add_int.qualifier_source_value is not null then lookup the values in add_int.qualifier_source_value in the CONCEPT table where domain_id=' Meas Value' and vocabulary_id=' LOINC' and standard_concept = 'S' and invalid_concept is NULL. |

docs/CPRD/CPRD_Person.md

@@ -26,15 +26,19 @@ The patients in the CDM are restricted to the subset of all CPRD patients deemed
 | month_of_birth | mob |  | Mob is only populated for children, if 0 set as NULL |
 | day_of_birth |  |  |  |
 | birth_datetime |  |  |  |
-| race_concept_id |  | 0 |  |
-| ethnicity_concept_id |  | 0 |  It could be possible to write an algorithm in the future using the Read codes that are available for a patient. |
+| race_concept_id |  | STEM.concept_id | populate if STEM.concept_id has domain 'Race'  |
+| ethnicity_concept_id |  | 0 |   |
 | location_id |  |  |  |
 | provider_id |  |  |  |
 | care_site_id | patid | Last 3 digits of patient.patid is the practice identifier and can be used in the CARE_SITE table to lookup the care_site_id. |  |
 | person_source_value | patid |  |  |
 | gender_source_value | gender |  | 'M' if gender = 1, 'F' if gender = 2 |
 | gender_source_concept_id |  | 0 |  |
-| race_source_value |  |  |  |
-| race_source_concept_id |  | 0 |  |
-| ethnicity_source_value |  |  |  |
-| ethnicity_source_concept_id |  | 0 |  |
+| race_source_value |  | STEM.source_value | populate if STEM.concept_id has domain 'Race'  |
+| race_source_concept_id |  | STEM.source_concept_id | populate if STEM.concept_id has domain 'Race' |
+| ethnicity_source_value |  |  |  | 
+
+## Change log
+
+### 08-Aug-2023
+- added rule: populate Race from the STEM table

docs/CPRD_Aurum/CPRD_Aurum_Death.md

@@ -9,6 +9,8 @@ description: "CPRD Aurum Death Logic"
 
 # CDM Table: DEATH
 
+![](images/death.png)
+
 ## Reading from CPRD_Aurum.Patient
 
 If deathdate is populated with a date in the Patient table then the patient will be included in the Death table.  

docs/CPRD_Aurum/CPRD_Aurum_Drug_Issue_STEM.md

@@ -29,7 +29,7 @@ In the below table, only the relevant STEM fields are shown.
 | concept_id | prodcodeid | Lookup the prodcodeid in the productdictionary. Use the [SOURCE_TO_STANDARD](https://github.com/OHDSI/ETL-LambdaBuilder/blob/master/docs/Standard%20Queries/SOURCE_TO_STANDARD.sql) query to map the dm+d code to standard concept(s) with the following filters: <br> <br>  Where source_vocabulary_id = 'dm+d'  and target_standard_concept = 'S'  and target_invalid_reason is NULL<br><br> If there is no mapping available, set concept_id to zero.  |  |
 | source_value | prodcodeid | Lookup prodcodeid and put the dm+d code here|  |
 | source_concept_id | prodcodeid | Lookup the prodcodeid in the productdictionary.     Use the [SOURCE_TO_SOURCE](https://github.com/OHDSI/ETL-LambdaBuilder/blob/master/docs/Standard%20Queries/SOURCE_TO_SOURCE.sql) query to map the dm+d code to a source concept id with the following filters:<br><br> Where source_vocabulary_id = 'dm+d' <br><br> If there is no mapping available set source_concept_id to zero. | |
-| type_concept_id |  | Set all from drug issue as **38000177** - prescription written.  |  |
+| type_concept_id |  | Set all from drug issue as **32838** - EHR prescription. |  |
 | dose_unit_source_value | quantunitid | Lookup in quantunit and put english version of unit here.|
 | quantity | quantity | |
 

docs/IBM_CCAE_MDCR/CCAE_MDCR_stem_inpatient_admissions.md

@@ -58,7 +58,7 @@ For every record in STEM there should be 1 row record in VISIT_DETAIL (n:1 join)
 | UNIQUE_DEVICE_ID | - | NULL | - |
 | UNIT_CONCEPT_ID | - | 0 | - |
 | UNIT_SOURCE_VALUE | - | NULL | - |
-| VALUE_AS_CONCEPT_ID | - | 0 | - |
+| VALUE_AS_CONCEPT_ID | PDX<br>DX1-15<br>PPROC<br>PROC1-15 | TARGET_VALUE_AS_CONCEPT_ID from the Source_to_Standard Query (resulting from  **'Maps to value'** relationsip);<br> See SOURCE_CONCEPT_ID field logic of how to define SOURCE_CONCEPT_ID being mapped| - |
 | VALUE_AS_NUMBER | - | NULL | - |
 | VALUE_AS_STRING | - | NULL | - |
 | VALUE_SOURCE_VALUE | - | NULL | - |
@@ -91,4 +91,7 @@ For every record in STEM there should be 1 row record in VISIT_DETAIL (n:1 join)
     * Remove any duplicate records 
     * Set VISIT_PROVID and VISIT_PROVSTD as PPROV and PROVCAT
   * As data is being assigned a CONCEPT_ID check the domain, this tells you what domain table the data should land.<br><br>If the CONCEPT = 0 the PDX, DX1-15 rows should land in **CONDITION_OCCURRENCE** and the PPROC, PROC1-15 should land in the **PROCEDURE_OCCURRENCE** table. 
-    * Instead any code without a mapping is assigned a CONCEPT_ID of zero and a DOMAIN_ID of OBSERVATION.
+    * Instead any code without a mapping is assigned a CONCEPT_ID of zero and a DOMAIN_ID of OBSERVATION.
+
+### Aug-01-2023
+ - Added 'Maps to value' logic

docs/IBM_CCAE_MDCR/CCAE_MDCR_stem_inpatient_services.md

@@ -38,8 +38,8 @@ Records will be written from the INPATIENT_SERVICES table mapping the field REVC
 | PROVIDER_ID | **VISIT_DETAIL**<br>PROVIDER_ID | - | - |
 | ID | - | System generated. | - |
 | CONCEPT_ID | PDX<br>DX1-5<br>PPROC<br>PROC1<br>REVCODE |  Use the <a href="https://ohdsi.github.io/CommonDataModel/sqlScripts.html">Source-to-Standard Query</a>.<br><br>  If DXVER does not have a value, review to the "Key Conventions" under the "STEM Key Conventions and Lookup Files" page.  If no map is made, assign CONCEPT_ID to 0 and set DOMAIN_ID as OBSERVATION.<br/><br/>**[PDX, DX1-5]**<br/>If DXVER=9 use the filter:<br/> `WHERE SOURCE_VOCABULARY_ID IN (‘ICD9CM’)`<br>`AND TARGET_STANDARD_CONCEPT = 'S'`<br>`AND TARGET_INVALID_REASON IS NULL`<br/><br/>If DXVER=0 use the filter:<br/>`WHERE SOURCE_VOCABULARY_ID IN (’ICD10CM’)`<br>`AND TARGET_STANDARD_CONCEPT = 'S'`<br>`AND TARGET_INVALID_REASON IS NULL` <br /><br />**[PPROC, PROC1]**<br/>When PROCTYP <> 0:<br />  `WHERE SOURCE_VOCABULARY_ID IN ('ICD9Proc','HCPCS','CPT4',’ICD10PCS’)`<br>`AND TARGET_STANDARD_CONCEPT = 'S'`<br>`AND TARGET_INVALID_REASON IS NULL`<br /><br />**[REVCODE]**<br/>  `WHERE SOURCE_VOCABULARY_ID IN ('Revenue Code’)`<br>`AND TARGET_STANDARD_CONCEPT = 'S'`<br>`AND TARGET_INVALID_REASON IS NULL`  | The concepts in the Revenue Code vocabulary all have the domain “Revenue Code”. These should go to the OBSERVATION table.	  |
-| SOURCE_VALUE | PDX<br>DX1-5<br>PPROC<br>PROC1 | - | - |
-| SOURCE_CONCEPT_ID | PDX<br>DX1-5<br>PPROC<br>PROC1 |  Use the <a href="https://ohdsi.github.io/CommonDataModel/sqlScripts.html">Source-to-Source Query</a>.<br><br>  If DXVER does not have a value, review to the "Key Conventions" under the "STEM Key Conventions and Lookup Files" page.  If no map is made, assign to 0.<br/><br/>**[PDX, DX1-5]**<br/>If DXVER=9 use the filter:<br/> `WHERE SOURCE_VOCABULARY_ID IN (‘ICD9CM’)`<br>`AND TARGET_VOCABULARY_ID IN (‘ICD9CM’)`<br/><br/>If DXVER=0 use the filter:<br/>`WHERE SOURCE_VOCABULARY_ID IN (’ICD10CM’)`<br>`AND TARGET_VOCABULARY_ID IN (‘ICD10CM’)` <br /><br />**[PPROC, PROC1]**<br/>When PROCTYP <> 0:<br />  `WHERE SOURCE_VOCABULARY_ID IN ('ICD9Proc','HCPCS','CPT4',’ICD10PCS’)`<br>`AND TARGET_VOCABULARY_ID IN ('ICD9Proc','HCPCS','CPT4',’ICD10PCS’)` | - |
+| SOURCE_VALUE | PDX<br>DX1-5<br>PPROC<br>PROC1<br>REVCODE | - | - |
+| SOURCE_CONCEPT_ID | PDX<br>DX1-5<br>PPROC<br>PROC1<br>REVCODE |  Use the <a href="https://ohdsi.github.io/CommonDataModel/sqlScripts.html">Source-to-Source Query</a>.<br><br>  If DXVER does not have a value, review to the "Key Conventions" under the "STEM Key Conventions and Lookup Files" page.  If no map is made, assign to 0.<br/><br/>**[PDX, DX1-5]**<br/>If DXVER=9 use the filter:<br/> `WHERE SOURCE_VOCABULARY_ID IN (‘ICD9CM’)`<br>`AND TARGET_VOCABULARY_ID IN (‘ICD9CM’)`<br/><br/>If DXVER=0 use the filter:<br/>`WHERE SOURCE_VOCABULARY_ID IN (’ICD10CM’)`<br>`AND TARGET_VOCABULARY_ID IN (‘ICD10CM’)` <br /><br />**[PPROC, PROC1]**<br/>When PROCTYP <> 0:<br />  `WHERE SOURCE_VOCABULARY_ID IN ('ICD9Proc','HCPCS','CPT4',’ICD10PCS’)`<br>`AND TARGET_VOCABULARY_ID IN ('ICD9Proc','HCPCS','CPT4',’ICD10PCS’)`<br /><br />**[REVCODE]**<br/>When PROCTYP <> 0:<br />  `WHERE SOURCE_VOCABULARY_ID IN ('Revenue Code’)`<br>`AND TARGET_VOCABULARY_ID IN ('Revenue Code’)` | - |
 | TYPE_CONCEPT_ID | - | Set all to `32854` (Inpatient claim detail) | - |
 | START_DATE | **VISIT_DETAIL**<br>VISIT_DETAIL_START_DATE | - | - |
 | START_DATETIME | - | START_DATE + midnight | - |
@@ -63,7 +63,7 @@ Records will be written from the INPATIENT_SERVICES table mapping the field REVC
 | UNIQUE_DEVICE_ID | - | NULL | - |
 | UNIT_CONCEPT_ID | - | 0 | - |
 | UNIT_SOURCE_VALUE | - | NULL | - |
-| VALUE_AS_CONCEPT_ID | - | 0 | - |
+| VALUE_AS_CONCEPT_ID | PDX<br>DX1-5<br>PPROC<br>PROC1 | TARGET_VALUE_AS_CONCEPT_ID from the Source_to_Standard Query (resulting from  **'Maps to value'** relationsip);<br> See SOURCE_CONCEPT_ID field logic of how to define SOURCE_CONCEPT_ID being mapped| - |
 | VALUE_AS_NUMBER | - | NULL | - |
 | VALUE_AS_STRING | - | NULL | - |
 | VALUE_SOURCE_VALUE | - | NULL | - |
@@ -82,6 +82,9 @@ Records will be written from the INPATIENT_SERVICES table mapping the field REVC
 
 ## Change Log
 
+### January 18, 2023
+* Updated revenue code logic
+
 ### November 1, 2021
 * Update CONDITION_STATUS_CONCEPT_ID so that only primary diagnoses and not DX1 are set to `32902`
 
@@ -94,3 +97,6 @@ Records will be written from the INPATIENT_SERVICES table mapping the field REVC
 * Added CONDITION_STATUS_CONCEPT_ID information
 
 * Added information on how to map revenue codes
+
+ ### Aug-01-2023
+ - Added 'Maps to value' logic

docs/IBM_CCAE_MDCR/CCAE_MDCR_stem_outpatient_services.md

@@ -36,8 +36,8 @@ Records will be written from the OUTPATIENT_SERVICES table mapping the field REV
 | PROVIDER_ID | **VISIT_DETAIL**<br>PROVIDER_ID  | - | -|
 | ID | - | System generated. | - |
 | CONCEPT_ID | DX1-5<br>PROC1 | Use the <a href="https://ohdsi.github.io/CommonDataModel/sqlScripts.html">Source-to-Standard Query</a>.<br><br>  If DXVER does not have a value, review to the "Key Conventions" under the "STEM Key Conventions and Lookup Files" page.  If no map is made, assign CONCEPT_ID to 0 and DOMAIN_ID to OBSERVATION.<br/><br/>**[DX1-5]**<br/>If DXVER=9 use the filter:<br/> `WHERE SOURCE_VOCABULARY_ID IN (‘ICD9CM’)`<br>`AND TARGET_STANDARD_CONCEPT = 'S'`<br>`AND TARGET_INVALID_REASON IS NULL`<br/><br/>If DXVER=0 use the filter:<br/>`WHERE SOURCE_VOCABULARY_ID IN (’ICD10CM’)`<br>`AND TARGET_STANDARD_CONCEPT = 'S'`<br>`AND TARGET_INVALID_REASON IS NULL` <br /><br />**[PROC1]**<br/>When PROCTYP <> 0:<br />  `WHERE SOURCE_VOCABULARY_ID IN ('ICD9Proc','HCPCS','CPT4',’ICD10PCS’)`<br>`AND TARGET_STANDARD_CONCEPT = 'S'`<br>`AND TARGET_INVALID_REASON IS NULL` <br /><br />**[REVCODE]**<br/>  `WHERE SOURCE_VOCABULARY_ID IN ('Revenue Code’)`<br>`AND TARGET_STANDARD_CONCEPT = 'S'`<br>`AND TARGET_INVALID_REASON IS NULL`  | The concepts in the Revenue Code vocabulary all have the domain “Revenue Code”. These should go to the OBSERVATION table.	  |
-| SOURCE_VALUE | DX1-5<br>PROC1 | - | - |
-| SOURCE_CONCEPT_ID | DX1-5<br>PROC1 | Use the <a href="https://ohdsi.github.io/CommonDataModel/sqlScripts.html">Source-to-Source Query</a>.<br><br>  If DXVER does not have a value, review to the "Key Conventions" under the "STEM Key Conventions and Lookup Files" page.  If no map is made, assign to 0.<br/><br/>**[DX1-5]**<br/>If DXVER=9 use the filter:<br/> `WHERE SOURCE_VOCABULARY_ID IN (‘ICD9CM’)`<br>`AND TARGET_VOCABULARY_ID IN (‘ICD9CM’)`<br/><br/>If DXVER=0 use the filter:<br/>`WHERE SOURCE_VOCABULARY_ID IN (’ICD10CM’)`<br>`AND TARGET_VOCABULARY_ID IN (’ICD10CM’)` <br /><br />**[PROC1]**<br/>When PROCTYP <> 0:<br />  `WHERE SOURCE_VOCABULARY_ID IN ('ICD9Proc','HCPCS','CPT4',’ICD10PCS’)`<br>`AND TARGET_VOCABULARY_ID IN ('ICD9Proc','HCPCS','CPT4',’ICD10PCS’)`| - |
+| SOURCE_VALUE | DX1-5<br>PROC1<br>REVCODE | - | - |
+| SOURCE_CONCEPT_ID | DX1-5<br>PROC1<br>REVCODE | Use the <a href="https://ohdsi.github.io/CommonDataModel/sqlScripts.html">Source-to-Source Query</a>.<br><br>  If DXVER does not have a value, review to the "Key Conventions" under the "STEM Key Conventions and Lookup Files" page.  If no map is made, assign to 0.<br/><br/>**[DX1-5]**<br/>If DXVER=9 use the filter:<br/> `WHERE SOURCE_VOCABULARY_ID IN (‘ICD9CM’)`<br>`AND TARGET_VOCABULARY_ID IN (‘ICD9CM’)`<br/><br/>If DXVER=0 use the filter:<br/>`WHERE SOURCE_VOCABULARY_ID IN (’ICD10CM’)`<br>`AND TARGET_VOCABULARY_ID IN (’ICD10CM’)` <br /><br />**[PROC1]**<br/>When PROCTYP <> 0:<br />  `WHERE SOURCE_VOCABULARY_ID IN ('ICD9Proc','HCPCS','CPT4',’ICD10PCS’)`<br>`AND TARGET_VOCABULARY_ID IN ('ICD9Proc','HCPCS','CPT4',’ICD10PCS’)`<br /><br />**[REVCODE]**<br />  `WHERE SOURCE_VOCABULARY_ID IN ('Revenue Code’)`<br>`AND TARGET_VOCABULARY_ID IN ('Revenue Code’)`| - |
 | TYPE_CONCEPT_ID | - | Set all to `32860` (Outpatient claim detail)| - |
 | START_DATE | **VISIT_DETAIL**<br>VISIT_DETAIL_START_DATE | - | - |
 | START_DATETIME | - | START_DATE + midnight | - |
@@ -80,6 +80,9 @@ Records will be written from the OUTPATIENT_SERVICES table mapping the field REV
 
 ## Change Log
 
+### January 18, 2023
+* Updated Revenue code mapping
+
 ### June 11, 2021
 * Added QTY to table
 

docs/IBM_MDCD/MDCD_stem_inpatient_admissions.md

@@ -58,7 +58,7 @@ For every record in STEM there should be 1 row record in VISIT_DETAIL (n:1 join)
 | UNIQUE_DEVICE_ID | - | NULL | - |
 | UNIT_CONCEPT_ID | - | 0 | - |
 | UNIT_SOURCE_VALUE | - | NULL | - |
-| VALUE_AS_CONCEPT_ID | - | 0 | - |
+| VALUE_AS_CONCEPT_ID | PDX<br>DX1-15<br>PPROC<br>PROC1-15 | TARGET_VALUE_AS_CONCEPT_ID from the Source_to_Standard Query (resulting from  **'Maps to value'** relationsip);<br> See SOURCE_CONCEPT_ID field logic of how to define SOURCE_CONCEPT_ID being mapped  | - |
 | VALUE_AS_NUMBER | - | NULL | - |
 | VALUE_AS_STRING | - | NULL | - |
 | VALUE_SOURCE_VALUE | - | NULL | - |
@@ -91,4 +91,7 @@ For every record in STEM there should be 1 row record in VISIT_DETAIL (n:1 join)
     * Remove any duplicate records 
     * Set VISIT_PROVID and VISIT_PROVSTD as PPROV and PROVCAT
   * As data is being assigned a CONCEPT_ID check the domain, this tells you what domain table the data should land.<br><br>If the CONCEPT = 0 the PDX, DX1-15 rows should land in **CONDITION_OCCURRENCE** and the PPROC, PROC1-15 should land in the **PROCEDURE_OCCURRENCE** table. 
-    * Instead any code without a mapping is assigned a CONCEPT_ID of zero and a DOMAIN_ID of OBSERVATION.
+    * Instead any code without a mapping is assigned a CONCEPT_ID of zero and a DOMAIN_ID of OBSERVATION.
+
+ ### Aug-01-2023
+ - Added 'Maps to value' logic