Benchmarking #30
Replies: 4 comments 22 replies
-
|
@alkaZeltser do you want to optimize resource allocations yet? It looks like none of the processes are using much more than 1 GB and 1 cpu, but they are currently all running serially since the allocations aren't being controlled. I assume several of the steps could be run in parallel? At least, target and off-target? |
Beta Was this translation helpful? Give feedback.
-
|
I'm still operating on "bare minimum for integration with metapipeline" which means completing all the partition-specific configs. They all exist, but since they were pretty much arbitrarily allocated when I first started the pipeline, I wanted to have some formal data to reference, hence these benchmarks. And then I noticed that the Picard processes are very greedy compared to the others. Over 30G peak v_mem? Does that seem excessive? |
Beta Was this translation helpful? Give feedback.
-
|
I think you can ignore that v_mem value. Those processes all had 30+ GB of memory allocated, but the rss memory used was still less than 1 GB. So if I understand this correctly, the process will always spill memory over 1 GB or so to disk (v_mem) regardless of what's available. |
Beta Was this translation helpful? Give feedback.
-
Is it standard to allocate resources for every process? Or are specific ones chosen? I would expect this would be most relevant for the ones that take the longest runtime, the most resources, and/or can be run parallel to others? Given these criteria, I think I should specifically allocate
|
Beta Was this translation helpful? Give feedback.
-
|
I don't see vast improvements in runtime, so should I just give the same allocations in all partitions? |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
sorry, by partitions I mean partition-specific configs (e.g. F2.config vs F16.config). Since all partitions exceed 1G of available memory, every partition-specific config can be identical - is my interpretation |
Beta Was this translation helpful? Give feedback.
-
|
So here is something I'm confused about. If all processes require less than 1G, then how come run_CollectHsMetrics_picard errors out with an OutOfMemory error when I run it in an F2 (not having allocated any resources, meaning it should have max available on an F2). This is something I ran into with the SWKIGLMP WXS dataset. |
Beta Was this translation helpful? Give feedback.
-
|
This is the entire log: /hot/user/nzeltser/pipeline-targeted-coverage/test/output/F2/mem-test-BZ.log I'm not sure if i'm interpreting right, but it looks like multiple processes are running in parallel? There's more than one |
Beta Was this translation helpful? Give feedback.
-
|
Ahh! you need to define the memory for the java processes, -Xmx. This will probably substantially change the benchmarking. You could start with a maximum of task.memory minus 2GB for the processes that used a lot of v_mem. Then adjust after seeing what they use. Try to keep things in parallel even if they take a little longer individually. See https://github.com/uclahs-cds/pipeline-call-sSNV/blob/90e8960c77aa9bb8a06f53c64c368174628f636b/module/mutect2-processes.nf#L99. |
Beta Was this translation helpful? Give feedback.
-
|
You can look at |
Beta Was this translation helpful? Give feedback.
-
|
Damn, I knew I was missing something! |
Beta Was this translation helpful? Give feedback.
-
|
How do you view these html files on the cluster? I have an extension that I thought was working but all I see is from |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, I usually end up transferring the file to my laptop and opening it with a browser. |
Beta Was this translation helpful? Give feedback.
-
Picard Processes OnlyUsing java allocation
sample: SWKIGLMP F2
F16
F72
|
Beta Was this translation helpful? Give feedback.
-
|
What is the purpose of subtracting the |
Beta Was this translation helpful? Give feedback.
-
|
There's overhead memory needed besides what's allocated to the java memory, so if you assign all of the available memory to the java process it will crash. Subtracting 2G is what Broad recommends. I'm not sure the memory allocation is working? Are the peak usages all still close to 1GB and not much different to previous? Do you need to use -jarg or something like that to pass java arguments through the picard command line? |
Beta Was this translation helpful? Give feedback.
-
|
The command should be right, I copied it from other picard processes: https://github.com/uclahs-cds/pipeline-align-DNA/blob/88ab6e0eb15d89598f4b4cc2d679121c3e66d035/module/mark_duplicate_picardtools.nf#L46 |
Beta Was this translation helpful? Give feedback.
-
|
The peak_vmem correspond to the partition-specific memory allocations I gave for those. e.g. specifically 80G in F72.config |
Beta Was this translation helpful? Give feedback.
-
|
OK I think this means it's not using the extra allocated memory. The runtime difference for |
Beta Was this translation helpful? Give feedback.
-
Trace file results from several sizes of inputs:
Input file stats
Benchmarks
a-mini
BZPRGPT1 - F16
BZPRGPT1 - F72
SWKIGLMP - F16
SWKIGLMP - F72
CPTAC3
Beta Was this translation helpful? Give feedback.
All reactions