meta: sr2silo for Loculus Architecture

[gordonkoehn]

Output from meeting with Alexander and Chaoran on 25.11.2024 – the final architecture of V-pipe & Loculus:

The difficulty: The crux is that V-Pipe's input and output data are much larger than the consensus sequences that Loculus was designed for. 14kb << 600 Mb. Thus, there are some caveats to how Loculus can handle V-Pipe data. Mainly, big data files are not passed through the Loculus backend but through S3 bucket references.

Below is the currently envisioned final setup:

V-Pipe and Loculus:

1. User-uploads raw data via web interface / AWS S3 uploader into an S3 bucket and metadata.
2. Loculus Backend writes the incoming metadata and S3 raw references into the PostgreSQL
3. Loclulus backend dispatches the metadata and S3 raw references to the Pipeline: V-Pipe - GET request
4. V-Pipe fetches the raw data from that S3 bucket and processes them
5. V-Pipe final step: sr2silo activates to process V-Pipe's output.bam to .ndjson of merged, paired reads enriched with the metadata
6. sr2silo uploads the .ndjson to another S3 bucket / and POST request the metadata and S3 URL to the Loculus Backend
7. Loculus Backend sends the metadata .ndjson and S3 URLs to SILO preprocessing.
8. Silo preprocessing enriches the ndjson with the full sequences fetching the correct reads from the S3 output in 6)
9. Silo pre-processing indexes on all files in that massive ndjson, as it needs all sequences

V-Pipe outputs to Silo*

This is step 5-6). In the final setup some wrapper code will implement a GET request in V-Pipe to be received from the Loculus backend upon which it will run V-Pipe and return the request with a POST to the Loculus backend and read and write all data but meta, from an S3.

On the output side of this stands the below s2silo. It will be triggered upon completion of V-Pipe. Probably as a docker-compose, to take a single .bam and metadata. It will do:

0. [temporary] GET request (to be received from Loculus Backend with nothing but an index)

1. read-processing (bam->sam->pair & merge -> align & translate i.e. nextclade like)

2. nextclade-like output to ndjson with Rust code from Fabian

3. enrich ndjson with metadata per line

4. upload ndjson to S3

5. POST to Loculus backend with S3 URLs of processed ndjson and metadata

6. will later be implemented with by the program that prepared inputs and directories for V-pipe, for now we'll artificially import a batch of data with sr2Silo alone.

Sub-Issues:

• Bind silo-input-transformer into sr2silo transform to ndjson #46
• Enrich ndjson with Metadata #47
• Implement S3 uploads #48
• Implement GET / POST with S3 Urls to Loculus #49

Open Questions

• Do users want to download only SNVs? Do users want to download BAMs as well? If so the above would need modification.

[gordonkoehn]

Doubt: What was the reason for processing the .bam to .ndjson in one place? As compared to having the nextclade-like output stored and processed in front of SILO.

So that only at once place we need to store one file with data and metadata once. Special for wastewater.

[gordonkoehn]

Sync with Alexander:

Probably the silo_input_transformer will later move to the SILO Preprocessing and we will upload BAMs and metadata to the Loculus backend, yet the BAMs for nucleotides and amino acids already read paired and merged to some S3.

For now this should be all in sr2silo to get going.

[gordonkoehn]

See the discussion here:

• Raw data sharing loculus-project/loculus#3344

[gordonkoehn]

I shall remove this from the board for cleanness.