User's feedback

[tbrown91]

Hi Bogdan, hi Michael,

I just wanted to document some of my experiences installing/running this and the make_lastz_chains directory for some "invaluable" customer feedback.

I downloaded the Release version of TOGA (version 1.1.6) and this does not include the CESAR2.0 and postoga directories, I guess as these are forked from another location. The install files also required that I had run git init . after download, which perhaps would not be an issue if I had used git clone to download TOGA originally, but was still an issue I ran into.

The pip3 install did not work for me at all, this is likely an issue with the way python/conda is setup on our cluster. In the end I installed every dependency using conda inside a toga environment. I have attached the output from conda list here in case it is useful for others.

toga_conda.txt

Inside 'make_lastz_chains` I had to hard-code the slurm arguments inside the script for nextflow. This is due to a number of particulars of our cluster here, such as the main queue not being called "main", etc. I wonder if there is a possibility to point to a config file like in TOGA.

I think this has been mentioned elsewhere, but my 'bigmem' jobs did not run, and as a result I had about 800 orthologs which were missing due to requiring too much memory. Unfortunately if I change the max_mem for the cesar jobs, this changes the max mem for all jobs and I get a very long waiting time for each job to run. In my case I had to set this to 320GB, which was complete overkill for most jobs. Am I supposed to run the bigmem jobs separately and combine in some way? I see there is a nextflow config for bigmem jobs, but don't think this was ever called.

Many thanks for getting the release version working smoothly. It's great to be able to play around with our genomes.

All the best,

Tom

[kirilenkobm]

Hi Tom,

thanks a ton for the feedback!

I downloaded the Release version of TOGA (version 1.1.6) and this does not include the CESAR2.0 and postoga directories

Fair enough, I will need to adjust configure.sh accordingly to cover this scenario too. :)

I have attached the output from conda list here in case it is useful for others.

Indeed, pip3 can be a bit hard to work with sometimes... Thanks for sharing the list; I’ll make sure to update the documentation to reflect this.

Inside 'make_lastz_chains` I had to hard-code the slurm arguments inside the script for nextflow.

Thx, I created an issue specifically for this.
hillerlab/make_lastz_chains#38

I think this has been mentioned elsewhere, but my 'bigmem' jobs did not run, and as a result I had about 800 orthologs which were missing due to requiring too much memory.

This bigmem argument part was quite underdeveloped... I'd recommend using --cb parameter, like --cb 8,16,32,128 ->This will split the CESAR jobs into buckets based on memory requirements (8, 16... etc Gb) and run them separately.

BR,
Bogdan

[MichaelHiller]

Thanks for the feedback Tom, always good to hear from users what didn't work smoothly.

@bogdan: For the bigmem parameter, do we document and explain this aspect clearly (having several 'buckets')?

[kirilenkobm]

@MichaelHiller, there are two distinct facets here:

• Firstly, the 'buckets' are noted in the argument list, yet I think the README could highlight this aspect of running TOGA more. I just looked over it— it's not immediately obvious despite being documented, and the importance might not be clear.
• Secondly, 'bigmem', (essentially shifting the most demanding CESAR tasks to a different partition), has become less crucial in the lab as all nodes now have 1Tb of RAM, so this branch is rather in some "semi-abandoned step", rather confusing for users.

Once Yuri completes his single-exon-CESAR approach, this memory issue should become obsolete (hopefully, we'll manage to contain the RAM usage within 32-64Gb for all genes).
Additionally, @shjenkins94 (big thanks to him!) suggested some significant rearrangements concerning the management of parallel jobs (without altering the scientific aspect). I've stored these suggestions in a separate branch for review and testing. This might simplify partition management somewhat, I suspect

[kirilenkobm]

Hi @tbrown91

Regarding the cluster args in the make_lastz_chains function: I recently realized that the pipeline has a --cluster_queue parameter, which defaults to batch, and controls the cluster's queue.

[tbrown91]

This is also a quirk of our cluster, but I have to specify the --qos argument whenever I run a job, so it is always helpful to have the option to add general arguments to the submission element