Cache docker when using Github actions #2196

causten · 2023-09-16T18:21:40Z

This PR is to reduce the load on CI servers by building docker images once and then uploading to the dockerhub registry. Future PR to handle Jenkins.

codecov · 2023-09-16T18:30:29Z

Codecov Report

Merging #2196 (88147b0) into develop (0bb8508) will not change coverage.
The diff coverage is n/a.

❗ Current head 88147b0 differs from pull request most recent head 1f6ac2b. Consider uploading reports for the commit 1f6ac2b to get more accurate results

@@           Coverage Diff            @@
##           develop    #2196   +/-   ##
========================================
  Coverage    91.49%   91.49%           
========================================
  Files          427      427           
  Lines        15953    15953           
========================================
  Hits         14596    14596           
  Misses        1357     1357

migraphx-bot · 2023-09-16T20:02:37Z

Test	Batch	Rate new 1f6ac2	Rate old 340012	Diff	Compare
torchvision-resnet50	64	2,287.20	2,282.79	0.19%	✅
torchvision-resnet50_fp16	64	5,350.95	5,367.78	-0.31%	✅
torchvision-densenet121	32	1,822.81	1,835.07	-0.67%	✅
torchvision-densenet121_fp16	32	3,393.31	3,391.29	0.06%	✅
torchvision-inceptionv3	32	1,344.13	1,336.41	0.58%	✅
torchvision-inceptionv3_fp16	32	2,597.91	2,588.08	0.38%	✅
cadene-inceptionv4	16	677.34	678.73	-0.20%	✅
cadene-resnext64x4	16	591.47	589.72	0.30%	✅
slim-mobilenet	64	7,223.13	7,213.88	0.13%	✅
slim-nasnetalarge	64	237.00	237.05	-0.02%	✅
slim-resnet50v2	64	2,530.33	2,528.69	0.06%	✅
bert-mrpc-onnx	8	721.79	721.09	0.10%	✅
bert-mrpc-tf	1	390.55	390.79	-0.06%	✅
pytorch-examples-wlang-gru	1	301.01	310.08	-2.93%	✅
pytorch-examples-wlang-lstm	1	309.47	310.72	-0.40%	✅
torchvision-resnet50_1	1	557.95	556.07	0.34%	✅
torchvision-inceptionv3_1	1	305.80	307.56	-0.57%	✅
cadene-dpn92_1	1	345.98	352.44	-1.83%	✅
cadene-resnext101_1	1	221.06	220.50	0.25%	✅
slim-vgg16_1	1	224.67	224.86	-0.08%	✅
slim-mobilenet_1	1	1,470.99	1,483.04	-0.81%	✅
slim-inceptionv4_1	1	219.87	221.72	-0.84%	✅
onnx-taau-downsample	1	322.88	322.55	0.10%	✅
dlrm-criteoterabyte	1	21.69	21.68	0.03%	✅
dlrm-criteoterabyte_fp16	1	40.66	40.62	0.10%	✅
agentmodel	1	5,946.36	5,836.42	1.88%	✅
unet_fp16	2	55.16	55.08	0.15%	✅

This build is OK for merge ✅

migraphx-bot · 2023-09-16T20:02:38Z

:white_check_mark:bert-mrpc-onnx: PASSED: MIGraphX meets tolerance

:white_check_mark:bert-mrpc-tf: PASSED: MIGraphX meets tolerance

:white_check_mark:pytorch-examples-wlang-gru: PASSED: MIGraphX meets tolerance

:white_check_mark:pytorch-examples-wlang-lstm: PASSED: MIGraphX meets tolerance

:white_check_mark:torchvision-resnet50_1: PASSED: MIGraphX meets tolerance

🔴torchvision-inceptionv3_1: FAILED: MIGraphX is not within tolerance - check verbose output

🔴cadene-dpn92_1: FAILED: MIGraphX is not within tolerance - check verbose output

:white_check_mark:cadene-resnext101_1: PASSED: MIGraphX meets tolerance

:white_check_mark:slim-vgg16_1: PASSED: MIGraphX meets tolerance

:white_check_mark:slim-mobilenet_1: PASSED: MIGraphX meets tolerance

🔴slim-inceptionv4_1: FAILED: MIGraphX is not within tolerance - check verbose output

:white_check_mark:dlrm-criteoterabyte: PASSED: MIGraphX meets tolerance

:white_check_mark:agentmodel: PASSED: MIGraphX meets tolerance

:white_check_mark:unet: PASSED: MIGraphX meets tolerance

causten · 2023-09-16T20:49:19Z

Internal server built the docker image in 43 minutes, The free 2 core server available from Github took 2 hours

causten · 2023-09-16T23:06:36Z

With no changes to any of the docker affecting files and Tidy/Format/cppcheck already cached, time on the larger runners dropped from 170 to 6.35 minutes

.github/workflows/ci.yaml

pfultz2 · 2023-09-19T15:51:15Z

.github/workflows/ci.yaml

+        DOCKERIMAGE: ${{ needs.check_image.outputs.imagetag }}
+      run: |
+        echo $DOCKER_TOKEN | docker login -u $DOCKER_USER --password-stdin
+        docker build . --file hip-clang.docker --tag $DOCKERIMAGE;


You should use --cache-from so we can reuse caches from previous builds, instead of rebuilding all layers again.

I have a code change to do that. I'd like to first get this in "as is" and than can create a PR that doubles down on the caching.

pfultz2 · 2023-09-19T15:52:58Z

.github/workflows/ci.yaml

+      run: |
+        echo $DOCKER_TOKEN | docker login -u $DOCKER_USER --password-stdin
+        docker build . --file hip-clang.docker --tag $DOCKERIMAGE;
+        docker push $DOCKERIMAGE;


This should also push rocm/migraphx-private:latest and rocm/migraphx-private:${BRANCH_NAME} so we can use these images in --cache-from.

yup will do exactly that in next PR

pfultz2 · 2023-09-19T15:53:25Z

.github/workflows/ci.yaml

+      - name: Create Image Tag
+        id: image_hash
+        run: |
+          echo "imagetag=rocm/migraphx-private:hip-clang-${{hashFiles('**/hip-clang.docker', '**/*requirements.txt', '**/install_prereqs.sh', '**/rbuild.ini')}}" >> $GITHUB_OUTPUT


You should have one variable for image name and then another for the tag.

I'll do that as part of the --cache-from update in another PR

pfultz2 · 2023-09-19T15:54:34Z

.github/workflows/ci.yaml

+        id: image_hash
+        run: |
+          echo "imagetag=rocm/migraphx-private:hip-clang-${{hashFiles('**/hip-clang.docker', '**/*requirements.txt', '**/install_prereqs.sh', '**/rbuild.ini')}}" >> $GITHUB_OUTPUT
+          echo "imagetag_sles=rocm/migraphx-private:hip-clang-sles-${{hashFiles('**/tools/docker/sles.docker', '**/*requirements.txt', '**/install_prereqs.sh', '**/rbuild.ini')}}" >> $GITHUB_OUTPUT


This should use a different image name such as rocm/migraphx-sles-private.

Fixed, I created another repo rocm/migraphx-sles-private

Initial code for docker caching

ab1da75

Chris Austen added 2 commits September 16, 2023 18:51

Fix bugs and add caching

8fcbbb3

password vs token rename

f4beade

add support for SLES

efc701d

causten requested a review from pfultz2 September 17, 2023 02:08

causten marked this pull request as ready for review September 17, 2023 02:12

causten requested a review from TedThemistokleous September 17, 2023 02:13

causten changed the title ~~Initial code for docker caching~~ Cache docker when using Github actions Sep 17, 2023

TedThemistokleous approved these changes Sep 18, 2023

View reviewed changes

TedThemistokleous added Continous Integration Pull request updates parts of continous integration pipeline high priority A PR with high priority for review and merging. labels Sep 18, 2023

pfultz2 reviewed Sep 19, 2023

View reviewed changes

.github/workflows/ci.yaml Outdated Show resolved Hide resolved

switch format to free job

081224c

umangyadav reviewed Sep 19, 2023

View reviewed changes

.github/workflows/ci.yaml Show resolved Hide resolved

.github/workflows/ci.yaml Show resolved Hide resolved

.github/workflows/ci.yaml Outdated Show resolved Hide resolved

.github/workflows/ci.yaml Outdated Show resolved Hide resolved

rename SLES image variable

0dd227c

umangyadav approved these changes Sep 19, 2023

View reviewed changes

pfultz2 reviewed Sep 19, 2023

View reviewed changes

Chris Austen and others added 3 commits September 19, 2023 13:27

rename SLES image variable

5f87605

Fix SLES test skipping

b25b3d9

Merge branch 'develop' into efficient_docker

1f6ac2b

causten merged commit 482e8d6 into develop Sep 20, 2023
15 checks passed

causten deleted the efficient_docker branch September 20, 2023 12:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cache docker when using Github actions #2196

Cache docker when using Github actions #2196

causten commented Sep 16, 2023

codecov bot commented Sep 16, 2023 •

edited

Loading

migraphx-bot commented Sep 16, 2023 •

edited

Loading

migraphx-bot commented Sep 16, 2023 •

edited

Loading

causten commented Sep 16, 2023

causten commented Sep 16, 2023

pfultz2 Sep 19, 2023

causten Sep 19, 2023

pfultz2 Sep 19, 2023

causten Sep 19, 2023

pfultz2 Sep 19, 2023

causten Sep 19, 2023

pfultz2 Sep 19, 2023

causten Sep 19, 2023

Cache docker when using Github actions #2196

Cache docker when using Github actions #2196

Conversation

causten commented Sep 16, 2023

codecov bot commented Sep 16, 2023 • edited Loading

Codecov Report

migraphx-bot commented Sep 16, 2023 • edited Loading

migraphx-bot commented Sep 16, 2023 • edited Loading

causten commented Sep 16, 2023

causten commented Sep 16, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Sep 16, 2023 •

edited

Loading

migraphx-bot commented Sep 16, 2023 •

edited

Loading

migraphx-bot commented Sep 16, 2023 •

edited

Loading