Bdanek/halo main 2

docs/api/datasets.rst

@@ -10,6 +10,7 @@ Datasets
     datasets/pyhealth.datasets.SampleEHRDataset
     datasets/pyhealth.datasets.SampleSignalDataset
     datasets/pyhealth.datasets.MIMIC3Dataset
+    datasets/pyhealth.datasets.MIMICExtractDataset
     datasets/pyhealth.datasets.MIMIC4Dataset
     datasets/pyhealth.datasets.eICUDataset
     datasets/pyhealth.datasets.OMOPDataset

docs/api/datasets/pyhealth.datasets.MIMICExtractDataset.rst

@@ -0,0 +1,15 @@
+pyhealth.datasets.MIMICExtractDataset
+===================================
+
+The open Medical Information Mart for Intensive Care (MIMIC-III) database, refer to `doc <https://mimic.mit.edu/>`_ for more information. We process this database into well-structured dataset object and give user the **best flexibility and convenience** for supporting modeling and analysis.
+
+.. autoclass:: pyhealth.datasets.MIMICExtractDataset
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+
+
+
+
+

docs/api/datasets/pyhealth.datasets.OMOPDataset.rst

@@ -1,7 +1,7 @@
 pyhealth.datasets.OMOPDataset
 ===================================
 
-We can process any OMOP-CDM formatted database, refer to `doc <https://www.ohdsi.org/data-standardization/the-common-data-model/>`_ for more information. We it into well-structured dataset object and give user the **best flexibility and convenience** for supporting modeling and analysis.
+We can process any OMOP-CDM formatted database, refer to `doc <https://www.ohdsi.org/data-standardization/the-common-data-model/>`_ for more information. The raw data is processed into well-structured dataset object and give user the **best flexibility and convenience** for supporting modeling and analysis.
 
 .. autoclass:: pyhealth.datasets.OMOPDataset
     :members:

docs/api/models.rst

@@ -15,6 +15,7 @@ We implement the following models for supporting multiple healthcare predictive
     models/pyhealth.models.GAMENet
     models/pyhealth.models.MICRON
     models/pyhealth.models.SafeDrug
+    models/pyhealth.models.MoleRec
     models/pyhealth.models.Deepr
     models/pyhealth.models.ContraWR
     models/pyhealth.models.SparcNet

docs/api/models/pyhealth.models.MoleRec.rst

@@ -0,0 +1,14 @@
+pyhealth.models.MoleRec
+===================================
+
+The separate callable MoleRecLayer and the complete MoleRec model.
+
+.. autoclass:: pyhealth.models.MoleRecLayer
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+.. autoclass:: pyhealth.models.MoleRec
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/log.rst

@@ -4,121 +4,121 @@ We track the new development here:
 
 **May 9, 2023**
 
-.. code-block:: bash
+.. code-block:: rst
 
-    1. add MIMIC-Extract dataset  `#136 <https://github.com/sunlabuiuc/PyHealth/pull/136>`_
+    1. add MIMIC-Extract dataset  `#136`
     2. add new maintainer members for pyhealth: Junyi Gao and Benjamin Danek
 
 **May 6, 2023**
 
-.. code-block:: bash
+.. code-block:: rst
 
-    1. add new parser functions (admissionDx, diagnosisStrings) and prediction tasks for eICU dataset  `#148 <https://github.com/sunlabuiuc/PyHealth/pull/148>`_
+    1. add new parser functions (admissionDx, diagnosisStrings) and prediction tasks for eICU dataset `#148`
 
 **Apr 27, 2023**
 
-.. code-block:: bash
+.. code-block:: rst
 
-    1. add MoleRec model (WWW'23) for drug recommendation `#122 <https://github.com/sunlabuiuc/PyHealth/pull/122>`_
+    1. add MoleRec model (WWW'23) for drug recommendation `#122`
 
 **Apr 26, 2023**
 
-.. code-block:: bash
+.. code-block:: rst
 
-    1. fix bugs in GRASP model `#141 <https://github.com/sunlabuiuc/PyHealth/pull/141>`_
-    2. add pandas install <2 constraints `#135 <https://github.com/sunlabuiuc/PyHealth/pull/135>`_
-    3. add hcpcsevents table process in MIMIC4 dataset `#134 <https://github.com/sunlabuiuc/PyHealth/pull/134>`_
+    1. fix bugs in GRASP model `#141`
+    2. add pandas install <2 constraints `#135` 
+    3. add hcpcsevents table process in MIMIC4 dataset `#134`
 
 **Apr 10, 2023**
 
-.. code-block:: bash
+.. code-block:: rst
 
     1. fix Ambiguous datetime usage in eICU (https://github.com/sunlabuiuc/PyHealth/pull/132)
 
 **Mar 26, 2023**    
 
-.. code-block:: bash
+.. code-block:: rst
 
     1. add the entire uncertainty quantification module (https://github.com/sunlabuiuc/PyHealth/pull/111)
 
 **Feb 26, 2023**
 
-.. code-block:: bash
+.. code-block:: rst
 
     1. add 6 EHR predictiom model: Adacare, Concare, Stagenet, TCN, Grasp, Agent
 
 **Feb 24, 2023**
 
-.. code-block:: bash
+.. code-block:: rst
 
     1. add unittest for omop dataset
-    2. add github action triggered manually, check #104
+    2. add github action triggered manually, check `#104`
 
 **Feb 19, 2023**
 
-.. code-block:: bash
+.. code-block:: rst
 
     1. add unittest for eicu dataset
     2. add ISRUC dataset (and task function) for signal learning
 
 **Feb 12, 2023**
 
-.. code-block:: bash
+.. code-block:: rst
 
     1. add unittest for mimiciii, mimiciv
     2. add SHHS datasets for sleep staging task
     3. add SparcNet model for signal classification task
 
 **Feb 08, 2023**
 
-.. code-block:: bash
+.. code-block:: rst
 
     1. complete the biosignal data support, add ContraWR [1] model for general purpose biosignal classification task ([1] Yang, Chaoqi, Danica Xiao, M. Brandon Westover, and Jimeng Sun. 
         "Self-supervised eeg representation learning for automatic sleep staging."
         arXiv preprint arXiv:2110.15278 (2021).)
 
 **Feb 07, 2023**
 
-.. code-block:: bash
+.. code-block:: rst
 
     1. Support signal dataset processing and split: add SampleSignalDataset, BaseSignalDataset. Use SleepEDFcassette dataset as the first signal dataset. Use example/sleep_staging_sleepEDF_contrawr.py
     2. rename the dataset/ parts: previous BaseDataset becomes BaseEHRDataset and SampleDatast becomes SampleEHRDataset. Right now, BaseDataset will be inherited by BaseEHRDataset and BaseSignalDataset. SampleBaseDataset will be inherited by SampleEHRDataset and SampleSignalDataset.
 
 **Feb 06, 2023**
 
-.. code-block:: bash
+.. code-block:: rst
 
     1. improve readme style
     2. add the pyhealth live 06 and 07 link to pyhealth live
 
 **Feb 01, 2023**
 
-.. code-block:: bash
+.. code-block:: rst
 
     1. add unittest of PyHealth MedCode and Tokenizer
 
 **Jan 26, 2023**
 
-.. code-block:: bash
+.. code-block:: rst
 
     1. accelerate MIMIC-IV, eICU and OMOP data loading by using multiprocessing (pandarallel)
 
 **Jan 25, 2023**
 
-.. code-block:: bash
+.. code-block:: rst
 
     1. accelerate the MIMIC-III data loading process by using multiprocessing (pandarallel)
 
 **Jan 24, 2023**
 
-.. code-block:: bash
+.. code-block:: rst
 
-    1. Fix the code typo in pyhealth/tasks/drug_recommendation.py for issue #71.
+    1. Fix the code typo in pyhealth/tasks/drug_recommendation.py for issue `#71`.
     2. update the pyhealth live schedule 
 
 **Jan 22, 2023**
 
-.. code-block:: bash
+.. code-block:: rst
 
     1. Fix the list of list of vector problem in RNN, Transformer, RETAIN, and CNN
     2. Add initialization examples for RNN, Transformer, RETAIN, CNN, and Deepr
@@ -128,42 +128,42 @@ We track the new development here:
 
 **Jan 21, 2023**
 
-.. code-block:: bash
+.. code-block:: rst
 
     1. Added a new model, Deepr (models.Deepr)
 
 **Jan 20, 2023**
 
-.. code-block:: bash
+.. code-block:: rst
 
     1. add the pyhealth live 05
     2. add slack channel invitation in pyhealth live page
 
 **Jan 13, 2023**
 
-.. code-block:: bash
+.. code-block:: rst
 
     1. add the pyhealth live 03 and 04 video link to the nagivation
     2. add future pyhealth live schedule
 
 **Jan 8, 2023**
 
-.. code-block:: bash
+.. code-block:: rst
 
     1. Changed BaseModel.add_feature_transform_layer in models/base_model.py so that it accepts special_tokens if necessary
     2. fix an int/float bug in dataset checking (transform int to float and then process them uniformly)
 
 **Dec 26, 2022**
 
-.. code-block:: bash
+.. code-block:: rst
 
     1. add examples to pyhealth.data, pyhealth.datasets
     2. improve jupyter notebook tutorials 0, 1, 2
 
 
 **Dec 21, 2022**
 
-.. code-block:: bash
+.. code-block:: rst
 
     1. add the development logs to the navigation
     2. add the pyhealth live schedule to the nagivation

docs/requirements.txt

@@ -1,5 +1,5 @@
 Sphinx==5.2.3
-sphinx-automodapi>
+sphinx-automodapi
 sphinx-autodoc-annotation
 sphinx_last_updated_by_git
 sphinxcontrib-spelling

pyhealth/datasets/base_ehr_dataset.py

@@ -135,6 +135,25 @@ def __init__(
             logger.debug(f"Saved {self.dataset_name} base dataset to {self.filepath}")
             save_pickle(self.patients, self.filepath)
 
+        self.patient_ids = list(self.patients.keys()) # precompute for use in __getitem__
+
+    def __len__(self):
+        return len(self.patients)
+
+    """
+    access dataset like dataset[10], or convert dataset to list via list(dataset)
+    """
+    def __getitem__(self, index: slice):
+        if (isinstance(index, slice)):
+            patient_ids = self.patient_ids[index]
+            res = []
+            for p in patient_ids:
+                res.append (self.patients[p])
+            return res
+        else:
+            patient_id = self.patient_ids[index]
+            return self.patients[patient_id]
+
     def _load_code_mapping_tools(self) -> Dict[str, CrossMap]:
         """Helper function which loads code mapping tools CrossMap for code mapping.
 
@@ -172,7 +191,7 @@ def parse_tables(self) -> Dict[str, Patient]:
         Returns:
            A dict mapping patient_id to `Patient` object.
         """
-        pandarallel.initialize(progress_bar=False)
+        pandarallel.initialize(progress_bar=False)#, shm_size_mb=10000)
 
         # patients is a dict of Patient objects indexed by patient_id
         patients: Dict[str, Patient] = dict()