Configure structure of HiGlass ingress directory #82
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
muffato
reviewed
Sep 25, 2023
modules/local/upload_higlass_data.nf
Outdated
Comment on lines
38
to
48
| cp -f $mcool $upload_dir | ||
| cp -f $genome $upload_dir/${genome.baseName}.genome | ||
| mkdir -p $upload_dir${higlass_data_basedir}/${species.replaceAll("\\s","_")}/${assembly} | ||
| cp -f $mcool $upload_dir${higlass_data_basedir}/${species.replaceAll("\\s","_")}/${assembly}/${assembly}.mcool | ||
| cp -f $genome $upload_dir/${higlass_data_basedir}/${species.replaceAll("\\s","_")}/${assembly}/${assembly}.genome | ||
|
|
||
| # Load them in Kubernetes | ||
| echo "Loading .mcool file" | ||
| kubectl exec \$pod_name -- python /home/higlass/projects/higlass-server/manage.py ingest_tileset --filename /higlass-temp/$mcool.name --filetype cooler --datatype matrix --project-name $assembly --name ${assembly}_map | ||
| kubectl exec \$pod_name -- python /home/higlass/projects/higlass-server/manage.py ingest_tileset --filename /higlass-temp/${higlass_data_basedir}/${species.replaceAll("\\s","_")}/${assembly}/${assembly}.mcool --filetype cooler --datatype matrix --project-name ${higlass_data_basedir}/${species.replaceAll("\\s","_")}/$assembly --name ${assembly}_map | ||
| echo "Loading .genome file" | ||
| kubectl exec \$pod_name -- python /home/higlass/projects/higlass-server/manage.py ingest_tileset --filename /higlass-temp/${genome.baseName}.genome --filetype chromsizes.tsv --datatype chromsizes --coordSystem ${assembly}_assembly --project-name $assembly --name ${assembly}_grid | ||
| kubectl exec \$pod_name -- python /home/higlass/projects/higlass-server/manage.py ingest_tileset --filename /higlass-temp/${higlass_data_basedir}/${species.replaceAll("\\s","_")}/${assembly}/${assembly}.genome --filetype chromsizes.tsv --datatype chromsizes --coordSystem ${assembly}_assembly --project-name ${higlass_data_basedir}/${species.replaceAll("\\s","_")}/$assembly --name ${assembly}_grid |
There was a problem hiding this comment.
Introduce a variable named something like assembly_path with the value "${higlass_data_basedir}/${species.replaceAll("\\s","_")}/${assembly}" and use it throughout. It'll be easier to read and maintain.
There was a problem hiding this comment.
Follow up from some recent discussions with @muffato and @gq1:
Would it be possible to use the same file name in both the contact_maps output folder as well as the data_to_upload folder. This is mostly if we decide to update certain entries in the future, it keeps things simple. Also makes sure the names are unique.
| @@ -27,6 +27,7 @@ params { | |||
|
|
|||
| // Input data for genome_metadata subworkflow | |||
| assembly = 'GCA_946965045.2' | |||
| species = 'Epithemia_sp._CRS-2021b' | |||
There was a problem hiding this comment.
The actual species name is Epithemia pelagica.
That is just the name given in our local file structure and might break when going to NCBI or other external sources.
There was a problem hiding this comment.
Yes, I noticed the difference in names. Do you know why we don't use the official species name in our local file structure? I used this as the param because currently the only time this parameter is used is when creating the directory structure for higlass and I was trying to replicate what we use elsewhere as closely as possible. Would you suggest I change this to the official species name?
There was a problem hiding this comment.
If you are not using the species name for any queries, it will not matter either way.
There was a problem hiding this comment.
Do you know why we don't use the official species name in our local file structure?
Probably because when we detected the name Epithemia pelagica, it had gone too far into the pipeline. That's the whole species-name change workflow that Paul is coordinating: we don't try to align everything to the NCBI taxonomy, because it's changing all the time.
This seems sensible, the naming format I've used in the |
From what I understand, you are planning to name the |
muffato
approved these changes
Oct 9, 2023
Comment on lines
+27
to
+28
| def project_name = "${higlass_data_project_dir}/${species.replaceAll('\\s','_')}/${assembly}" | ||
| def file_name = "${assembly}_${meta.id}" |
There was a problem hiding this comment.
These two variables make the script below very clean 👍🏼
Co-authored-by: Matthieu Muffato <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR checklist
nf-core lint).nextflow run . -profile test,docker --outdir <OUTDIR>).docs/usage.mdis updated.docs/output.mdis updated.CHANGELOG.mdis updated.README.mdis updated (including new tool citations and authors/contributors).