Adding hostile v1.1.0 #1111

taylorpaisie · 2024-11-22T20:39:39Z

Pull Request (PR) checklist:

Include a description of what is in this pull request in this message.
The dockerfile successfully builds to a test target for the user creating the PR. (i.e. docker build --tag samtools:1.15test --target test docker-builds/samtools/1.15 )
Directory structure as name of the tool in lower case with special characters removed with a subdirectory of the version number (i.e. spades/3.12.0/Dockerfile)
- (optional) All test files are located in same directory as the Dockerfile (i.e. shigatyper/2.0.1/test.sh)
Create a simple container-specific README.md in the same directory as the Dockerfile (i.e. spades/3.12.0/README.md)
- If this README is longer than 30 lines, there is an explanation as to why more detail was needed
Dockerfile includes the recommended LABELS
Main README.md has been updated to include the tool and/or version of the dockerfile(s) in this PR
Program_Licenses.md contains the tool(s) used in this PR and has been updated for any missing

erinyoung · 2024-12-03T20:04:30Z

The tests appeared to have worked:

#14 [test 2/3] RUN wget https://github.com/bacterial-genomics/test-datasets/raw/assembly/test_data/test_miniburk_R1.fastq.gz     --no-check-certificate &&     wget https://github.com/bacterial-genomics/test-datasets/raw/assembly/test_data/test_miniburk_R2.fastq.gz     --no-check-certificate &&     hostile clean --index /data/test/sars-cov-2/sars-cov-2 --fastq1 test_miniburk_R1.fastq.gz --fastq2 test_miniburk_R2.fastq.gz
#14 0.061 --2024-12-03 19:40:25--  https://github.com/bacterial-genomics/test-datasets/raw/assembly/test_data/test_miniburk_R1.fastq.gz
#14 0.062 Resolving github.com (github.com)... 140.82.116.3
#14 0.065 Connecting to github.com (github.com)|140.82.116.3|:443... connected.
#14 0.105 HTTP request sent, awaiting response... 302 Found
#14 0.445 Location: https://raw.githubusercontent.com/bacterial-genomics/test-datasets/assembly/test_data/test_miniburk_R1.fastq.gz [following]
#14 0.446 --2024-12-03 19:40:26--  https://raw.githubusercontent.com/bacterial-genomics/test-datasets/assembly/test_data/test_miniburk_R1.fastq.gz
#14 0.446 Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.111.133, 185.199.109.133, ...
#14 0.455 Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.
#14 0.461 HTTP request sent, awaiting response... 200 OK
#14 1.246 Length: 18238964 (17M) [application/octet-stream]
#14 1.246 Saving to: 'test_miniburk_R1.fastq.gz'
#14 1.246 
#14 1.246      0K .......... .......... .......... .......... ..........  0% 30.2M 1s
#14 1.248     50K .......... .......... .......... .......... ..........  0% 12.3M 1s
#14 1.252    100K .......... .......... .......... .......... ..........  0% 11.9M 1s
#14 1.256    150K .......... .......... .......... .......... ..........  1% 9.41M 1s
#14 1.261    200K .......... .......... .......... .......... ..........  1% 15.3M 1s
#14 1.264    250K .......... .......... .......... .......... ..........  1%  137M 1s
#14 1.265    300K .......... .......... .......... .......... ..........  1% 17.4M 1s
#14 1.268    350K .......... .......... .......... .......... ..........  2% 24.2M 1s
#14 1.270    400K .......... .......... .......... .......... ..........  2% 27.9M 1s
#14 1.271    450K .......... .......... .......... .......... ..........  2% 28.6M 1s
#14 1.273    500K .......... .......... .......... .......... ..........  3%  156M 1s
#14 1.275    550K .......... .......... .......... .......... ..........  3% 21.2M 1s
#14 1.276    600K .......... .......... .......... .......... ..........  3% 42.6M 1s
#14 1.277    650K .......... .......... .......... .......... ..........  3%  129M 1s
#14 1.277    700K .......... .......... .......... .......... ..........  4%  102M 1s
#14 1.278    750K .......... .......... .......... .......... ..........  4% 79.1M 1s
#14 1.278    800K .......... .......... .......... .......... ..........  4%  103M 1s
#14 1.279    850K .......... .......... .......... .......... ..........  5% 52.3M 1s
#14 1.280    900K .......... .......... .......... .......... ..........  5% 48.3M 1s
#14 1.281    950K .......... .......... .......... .......... ..........  5% 94.2M 1s
#14 1.281   1000K .......... .......... .......... .......... ..........  5% 56.0M 1s
#14 1.282   1050K .......... .......... .......... .......... ..........  6%  156M 1s
#14 1.282   1100K .......... .......... .......... .......... ..........  6% 65.7M 1s
#14 1.283   1150K .......... .......... .......... .......... ..........  6% 67.6M 1s
#14 1.284   1200K .......... .......... .......... .......... ..........  7%  128M 1s
#14 1.284   1250K .......... .......... .......... .......... ..........  7%  162M 0s
#14 1.285   1300K .......... .......... .......... .......... ..........  7%  126M 0s
#14 1.285   1350K .......... .......... .......... .......... ..........  7%  152M 0s
#14 1.286   1400K .......... .......... .......... .......... ..........  8%  148M 0s
#14 1.286   1450K .......... .......... .......... .......... ..........  8%  155M 0s
#14 1.286   1500K .......... .......... .......... .......... ..........  8%  129M 0s
#14 1.286   1550K .......... .......... .......... .......... ..........  8%  142M 0s
#14 1.287   1600K .......... .......... .......... .......... ..........  9%  146M 0s
#14 1.287   1650K .......... .......... .......... .......... ..........  9%  127M 0s
#14 1.287   1700K .......... .......... .......... .......... ..........  9%  153M 0s
#14 1.288   1750K .......... .......... .......... .......... .......... 10%  136M 0s
#14 1.288   1800K .......... .......... .......... .......... .......... 10%  141M 0s
#14 1.289   1850K .......... .......... .......... .......... .......... 10%  156M 0s
#14 1.289   1900K .......... .......... .......... .......... .......... 10%  124M 0s
#14 1.289   1950K .......... .......... .......... .......... .......... 11%  149M 0s
#14 1.290   2000K .......... .......... .......... .......... .......... 11%  161M 0s
#14 1.290   2050K .......... .......... .......... .......... .......... 11%  151M 0s
#14 1.290   2100K .......... .......... .......... .......... .......... 12%  166M 0s
#14 1.291   2150K .......... .......... .......... .......... .......... 12%  153M 0s
#14 1.291   2200K .......... .......... .......... .......... .......... 12%  130M 0s
#14 1.291   2250K .......... .......... .......... .......... .......... 12%  145M 0s
#14 1.292   2300K .......... .......... .......... .......... .......... 13%  146M 0s
#14 1.292   2350K .......... .......... .......... .......... .......... 13%  156M 0s
#14 1.292   2400K .......... .......... .......... .......... .......... 13%  140M 0s
#14 1.293   2450K .......... .......... .......... .......... .......... 14%  144M 0s
#14 1.293   2500K .......... .......... .......... .......... .......... 14%  160M 0s
#14 1.293   2550K .......... .......... .......... .......... .......... 14%  154M 0s
#14 1.293   2600K .......... .......... .......... .......... .......... 14%  150M 0s
#14 1.294   2650K .......... .......... .......... .......... .......... 15%  145M 0s
#14 1.294   2700K .......... .......... .......... .......... .......... 15%  149M 0s
#14 1.294   2750K .......... .......... .......... .......... .......... 15%  158M 0s
#14 1.295   2800K .......... .......... .......... .......... .......... 16%  162M 0s
#14 1.295   2850K .......... .......... .......... .......... .......... 16%  129M 0s
#14 1.296   2900K .......... .......... .......... .......... .......... 16%  149M 0s
#14 1.296   2950K .......... .......... .......... .......... .......... 16%  137M 0s
#14 1.296   3000K .......... .......... .......... .......... .......... 17%  160M 0s
#14 1.297   3050K .......... .......... .......... .......... .......... 17%  138M 0s
#14 1.297   3100K .......... .......... .......... .......... .......... 17%  148M 0s
#14 1.297   3150K .......... .......... .......... .......... .......... 17%  163M 0s
#14 2.750  17100K .......... .......... .......... .......... .......... 92%  140M 0s
#14 2.751  17150K .......... .......... .......... .......... .......... 92% 81.0M 0s
#14 2.751  17200K .......... .......... .......... .......... .......... 92%  152M 0s
#14 2.752  17250K .......... .......... .......... .......... .......... 93% 81.8M 0s
#14 2.752  17300K .......... .......... .......... .......... .......... 93%  120M 0s
#14 2.753  17350K .......... .......... .......... .......... .......... 93%  120M 0s
#14 2.753  17400K .......... .......... .......... .......... .......... 93%  127M 0s
#14 2.754  17450K .......... .......... .......... .......... .......... 94% 68.9M 0s
#14 2.754  17500K .......... .......... .......... .......... .......... 94%  117M 0s
#14 2.755  17550K .......... .......... .......... .......... .......... 94%  118M 0s
#14 2.755  17600K .......... .......... .......... .......... .......... 94%  108M 0s
#14 2.755  17650K .......... .......... .......... .......... .......... 95% 97.1M 0s
#14 2.756  17700K .......... .......... .......... .......... .......... 95% 97.4M 0s
#14 2.756  17750K .......... .......... .......... .......... .......... 95%  130M 0s
#14 2.757  17800K .......... .......... .......... .......... .......... 95%  138M 0s
#14 2.757  17850K .......... .......... .......... .......... .......... 96%  149M 0s
#14 2.757  17900K .......... .......... .......... .......... .......... 96%  157M 0s
#14 2.758  17950K .......... .......... .......... .......... .......... 96%  142M 0s
#14 2.758  18000K .......... .......... .......... .......... .......... 97%  128M 0s
#14 2.759  18050K .......... .......... .......... .......... .......... 97%  160M 0s
#14 2.759  18100K .......... .......... .......... .......... .......... 97%  150M 0s
#14 2.759  18150K .......... .......... .......... .......... .......... 97%  162M 0s
#14 2.759  18200K .......... .......... .......... .......... .......... 98%  140M 0s
#14 2.760  18250K .......... .......... .......... .......... .......... 98%  158M 0s
#14 2.760  18300K .......... .......... .......... .......... .......... 98%  155M 0s
#14 2.760  18350K .......... .......... .......... .......... .......... 98%  160M 0s
#14 2.761  18400K .......... .......... .......... .......... .......... 99%  124M 0s
#14 2.761  18450K .......... .......... .......... .......... .......... 99%  158M 0s
#14 2.761  18500K .......... .......... .......... .......... .......... 99%  157M 0s
#14 2.762  18550K .......... .......... .......... .......... ......... 100%  150M=0.2s
#14 2.762 
#14 2.762 2024-12-03 19:40:28 (111 MB/s) - 'test_miniburk_R2.fastq.gz' saved [19045452/19045452]
#14 2.762 
#14 2.916 19:40:28 INFO: Hostile version 1.1.0. Mode: paired short read (Bowtie2)
#14 2.951 19:40:28 INFO: Found custom index /data/test/sars-cov-2/sars-cov-2
#14 2.952 19:40:28 INFO: Cleaning…
#14 6.789 19:40:32 INFO: Cleaning complete
#14 6.789 [
#14 6.789     {
#14 6.789         "version": "1.1.0",
#14 6.789         "aligner": "bowtie2",
#14 6.789         "index": "/data/test/sars-cov-2/sars-cov-2",
#14 6.789         "options": [],
#14 6.789         "fastq1_in_name": "test_miniburk_R1.fastq.gz",
#14 6.789         "fastq1_in_path": "/data/test/test_miniburk_R1.fastq.gz",
#14 6.789         "fastq1_out_name": "test_miniburk_R1.clean_1.fastq.gz",
#14 6.789         "fastq1_out_path": "/data/test/test_miniburk_R1.clean_1.fastq.gz",
#14 6.789         "reads_in": 500000,
#14 6.789         "reads_out": 500000,
#14 6.789         "reads_removed": 0,
#14 6.789         "reads_removed_proportion": 0.0,
#14 6.789         "fastq2_in_name": "test_miniburk_R2.fastq.gz",
#14 6.789         "fastq2_in_path": "/data/test/test_miniburk_R2.fastq.gz",
#14 6.789         "fastq2_out_name": "test_miniburk_R2.clean_2.fastq.gz",
#14 6.789         "fastq2_out_path": "/data/test/test_miniburk_R2.clean_2.fastq.gz"
#14 6.789     }
#14 6.789 ]
#14 DONE 6.9s

erinyoung · 2024-12-03T20:04:36Z

This is looking great!

Some changes:

The LABELs need to move from the builder stage to the app stage. They get lost when the builder stage gets lost.
It's helpful when all versions are put in ARGs because it helps update the dockerfiles later.
It might be helpful to use jammy instead of focal, because I think the default version of jammy might simplify things.

erinyoung · 2024-12-03T20:05:13Z

hostile/1.1.0/Dockerfile

+ARG HOSTILE_VER="1.1.0"
+
+# Stage 1: Build Dockerfile
+FROM ubuntu:focal AS builder


setting this as ubuntu:jammy might be helpful for you

Made all the changes you suggested besides this one. For some reason i was getting a different error when using ubuntu:jammy, so just switched it back but if its a good switch you think I'm more than willing to go through and fix the error, but wasn't sure if it was worth it or not in terms for the docker build.

hostile/1.1.0/Dockerfile

erinyoung · 2024-12-06T17:19:01Z

Thank you for looking into changing the base. I was hopeful it would be helpful.

It looks like the tests still work

#15 1.939 19:11:21 INFO: Hostile version 1.1.0. Mode: paired short read (Bowtie2)
#15 1.974 19:11:21 INFO: Found custom index /data/test/sars-cov-2/sars-cov-2
#15 1.974 19:11:21 INFO: Cleaning…
#15 6.059 19:11:25 INFO: Cleaning complete
#15 6.060 [
#15 6.060     {
#15 6.060         "version": "1.1.0",
#15 6.060         "aligner": "bowtie2",
#15 6.060         "index": "/data/test/sars-cov-2/sars-cov-2",
#15 6.060         "options": [],
#15 6.060         "fastq1_in_name": "test_miniburk_R1.fastq.gz",
#15 6.060         "fastq1_in_path": "/data/test/test_miniburk_R1.fastq.gz",
#15 6.060         "fastq1_out_name": "test_miniburk_R1.clean_1.fastq.gz",
#15 6.060         "fastq1_out_path": "/data/test/test_miniburk_R1.clean_1.fastq.gz",
#15 6.060         "reads_in": 500000,
#15 6.060         "reads_out": 500000,
#15 6.060         "reads_removed": 0,
#15 6.060         "reads_removed_proportion": 0.0,
#15 6.060         "fastq2_in_name": "test_miniburk_R2.fastq.gz",
#15 6.060         "fastq2_in_path": "/data/test/test_miniburk_R2.fastq.gz",
#15 6.060         "fastq2_out_name": "test_miniburk_R2.clean_2.fastq.gz",
#15 6.060         "fastq2_out_path": "/data/test/test_miniburk_R2.clean_2.fastq.gz"
#15 6.060     }
#15 6.060 ]
#15 DONE 6.2s

erinyoung · 2024-12-06T17:34:48Z

hostile/1.1.0/Dockerfile

+LABEL maintainer.email="[email protected]"
+
+COPY --from=builder /usr/ /usr/
+COPY --from=builder /hostile-1.1.0/tests/data/sars-cov-2/ /data/test/sars-cov-2/


Are these files needed at runtime or are they used for testing/demo-ing?

Those files I had to do the testing.

Are they beneficial to keep in the final image?

no reason to keep them in the final image, just using for test. This program was a bit difficult (at least for me) to build a container for. So any suggestions on what you think i should include or modify for the final image would be appreciated!

If these files aren't used at runtime, then we can drop this line to decrease the image size.

If they're helpful, though, they can stay in.

erinyoung · 2024-12-06T17:47:29Z

hostile/1.1.0/Dockerfile

+LABEL base.image="ubuntu:focal" 
+LABEL dockerfile.version="1" 
+LABEL software="hostile"
+LABEL software.version="${HOSTILE_VER}"


Can you add the following to line 41-ish (a line after the app stage starts, but before the LABEL lines).

ARG HOSTILE_VER

Right now, the software.version label isn't registering and is causing a GA action test to fail.

yeah must have missed that, fixed now!

erinyoung · 2024-12-06T17:48:14Z

hostile/1.1.0/Dockerfile

+LABEL maintainer.email="[email protected]"
+
+COPY --from=builder /usr/ /usr/
+COPY --from=builder /hostile-1.1.0/tests/data/sars-cov-2/ /data/test/sars-cov-2/


Can you change

COPY --from=builder /hostile-1.1.0/tests/data/sars-cov-2/ /data/test/sars-cov-2/

to something like the following

COPY --from=builder /hostile-${HOSTILE_VER}/tests/data/sars-cov-2/ /data/test/sars-cov-2/

yep, missed that change as well, now fixed.

erinyoung · 2024-12-09T21:23:06Z

Test still work:

#15 3.106 2024-12-06 19:44:20 (102 MB/s) - 'test_miniburk_R2.fastq.gz' saved [19045452/19045452]
#15 3.106 
#15 3.263 19:44:20 INFO: Hostile version 1.1.0. Mode: paired short read (Bowtie2)
#15 3.297 19:44:20 INFO: Found custom index /data/test/sars-cov-2/sars-cov-2
#15 3.297 19:44:20 INFO: Cleaning…
#15 7.270 19:44:24 INFO: Cleaning complete
#15 7.270 [
#15 7.270     {
#15 7.270         "version": "1.1.0",
#15 7.270         "aligner": "bowtie2",
#15 7.270         "index": "/data/test/sars-cov-2/sars-cov-2",
#15 7.270         "options": [],
#15 7.270         "fastq1_in_name": "test_miniburk_R1.fastq.gz",
#15 7.270         "fastq1_in_path": "/data/test/test_miniburk_R1.fastq.gz",
#15 7.270         "fastq1_out_name": "test_miniburk_R1.clean_1.fastq.gz",
#15 7.270         "fastq1_out_path": "/data/test/test_miniburk_R1.clean_1.fastq.gz",
#15 7.270         "reads_in": 500000,
#15 7.270         "reads_out": 500000,
#15 7.270         "reads_removed": 0,
#15 7.270         "reads_removed_proportion": 0.0,
#15 7.270         "fastq2_in_name": "test_miniburk_R2.fastq.gz",
#15 7.270         "fastq2_in_path": "/data/test/test_miniburk_R2.fastq.gz",
#15 7.270         "fastq2_out_name": "test_miniburk_R2.clean_2.fastq.gz",
#15 7.270         "fastq2_out_path": "/data/test/test_miniburk_R2.clean_2.fastq.gz"
#15 7.270     }
#15 7.270 ]

erinyoung

I have no additional changes to recommend.

Thank you for putting this together!!!

erinyoung · 2024-12-09T21:25:02Z

I've started the GA to build and deploy this image to dockerhub and quay. You can check the status of the deployment here : https://github.com/StaPH-B/docker-builds/actions/runs/12244563176

Let us know if you run into problems!!!

taylorpaisie and others added 11 commits November 21, 2024 10:41

adding hostile to new branch

766cb82

editing hostile readme and test in dockerfile

80e04d8

making edits to readme and hostile dockerfile

8d625d1

adding hostile tag to readme

e454d82

edited hostile dockerfile

b74548b

minor edits to dockerfile and hostile readme

0ecd884

updaing workflows

3afa298

making changes on branch to match master branch

bcc09f2

updated program licenses with hostile

ba84ae9

Merge branch 'master' into tkp-hostile

9be5401

Added hyperlink

a1ea237

erinyoung reviewed Dec 3, 2024

View reviewed changes

hostile/1.1.0/Dockerfile Outdated Show resolved Hide resolved

erinyoung reviewed Dec 3, 2024

View reviewed changes

hostile/1.1.0/Dockerfile Outdated Show resolved Hide resolved

erinyoung reviewed Dec 3, 2024

View reviewed changes

hostile/1.1.0/Dockerfile Show resolved Hide resolved

taylorpaisie added 4 commits December 4, 2024 11:22

moving labels and hostile help command to app stage in dockerfile

f0b0fcb

adding bowtie2 version arg to dockerfile

84ac86e

Updated Dockerfile with ubuntu:jammy

fed5b82

changed dockerfile back to ubuntu:focal

f2e2542

erinyoung reviewed Dec 6, 2024

View reviewed changes

taylorpaisie and others added 5 commits December 6, 2024 14:42

updated dockerfile with hostile ver arg and hostile test paths

f25b3b2

Update update_pango_aliasor.yml

920cc16

Update update_pango_aliasor.yml

1548ab2

Update update_pango_aliasor.yml

d4de033

Delete .github/workflows/update_pango_aliasor.yml

940a168

erinyoung approved these changes Dec 9, 2024

View reviewed changes

erinyoung merged commit c165457 into StaPH-B:master Dec 9, 2024
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding hostile v1.1.0 #1111

Adding hostile v1.1.0 #1111

taylorpaisie commented Nov 22, 2024

erinyoung commented Dec 3, 2024

erinyoung commented Dec 3, 2024

erinyoung Dec 3, 2024

taylorpaisie Dec 5, 2024

erinyoung commented Dec 6, 2024

erinyoung Dec 6, 2024

taylorpaisie Dec 6, 2024

erinyoung Dec 6, 2024

taylorpaisie Dec 6, 2024

erinyoung Dec 6, 2024

erinyoung Dec 6, 2024

taylorpaisie Dec 6, 2024

erinyoung Dec 6, 2024

taylorpaisie Dec 6, 2024

erinyoung commented Dec 9, 2024

erinyoung left a comment

erinyoung commented Dec 9, 2024

Adding hostile v1.1.0 #1111

Adding hostile v1.1.0 #1111

Conversation

taylorpaisie commented Nov 22, 2024

erinyoung commented Dec 3, 2024

erinyoung commented Dec 3, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erinyoung commented Dec 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erinyoung commented Dec 9, 2024

erinyoung left a comment

Choose a reason for hiding this comment

erinyoung commented Dec 9, 2024