4. Upload

Data is uploaded to the LIMS database using the AnalysisDataBaseUploader class in the db_uploader module.

The database models are defined using sqlalchemy in the db_models.py module. These essentially mirror those defined in the LIMS repository (Covid_19_Serology/app/models.py).

The general workflow: (where dataset, indexfiles, normalised_data, final_results, failures, model_parameters are from their associated Experiment methods.)

# workflow_id and variant from ingest
# dataframes from Experiment

lims_db = AnalysisDatabaseUploader(session)

if lims_db.already_uploaded(workflow_id, variant):
    logging.error("error message")
    return None

lims_db.upload_plate_results(dataset)
lims_db.upload_indexfiles(indexfiles)
lims_db.upload_normalised_results(normalised_data)
lims_db.upload_final_results(final_results)
lims_db.upload_failures(failures)
lims_db.upload_model_parameters(model_parameters)
lims_db.upload_reporter_plate_status(workflow_id, variant)

if lims_db.is_final_upload(workflow_id):
    lims_db.update_workflow_tracking(workflow_id)

lims_db.commit()

These methods basically rename columns, unpad well labels (e.g A01 -> A1), and use sqlalchemy with dataframes and bulk_insert_mappings

Checks to stop duplication or over-writing.

The AnalysisDatabaseUploader.already_uploaded(workflow_id, variant) method checks for the presence of the current workflow and variant in the NE_final_results table. If it's already in there then we want to exit to avoid over-writing or duplicating data.

Marking workflows as complete

Workflows are created with an expected number of variants. When the final variant is uploaded the workflow is marked as complete in the NE_workflow_tracking table. So we have AnalysisDatabaseUploader.is_final_upload(workflow_id) which checks in the results table for the current number of uploaded results, and if we're variant short (therefore our current analysis would complete it), we mark it as complete.

Atomic commit

We only commit the data at the end if everything was successful. This stops partially uploaded results if something goes wrong half-way through.