added multiarch build

[AndrewChubatiuk]

What type of PR is this?

Build multiarch docker images

• Refactor
• Feature
• Bug Fix
• New Query Runner (Data Source)
• New Alert Destination
• Other

Description

Added ability to create multiarch images

How is this tested?

• Unit tests (pytest, jest)
• E2E Tests (Cypress)
• [] Manually
• N/A

[guidopetri]

Hey @AndrewChubatiuk ! Thanks for the PR. We've tried a arm64 build on GHA recently and it runs super long - so long, in fact, that it hits some hidden 6 hour limit for GHA and does not finish. Would you mind testing this on your own repo and potentially making changes there before we consider merging this in?

[AndrewChubatiuk]

this change will run arm64 and amd64 image builds on separate machines, so it should improve build speed. okay, will test it in my fork

[guidopetri]

Here's an example action that we already ran: https://github.com/getredash/redash/actions/runs/6546702269

I'm not sure how that was running so maybe we just did it wrong. :)

[guidopetri]

Could you explain the changes you're making to the Dockerfile here?

[AndrewChubatiuk]

Removed frontend build from Dockerfile as there are to many places, where same node dependencies are installed. Also there's no sense to build same platform independent frontend in many places

[guidopetri]

I don't understand any of the JS dependency related changes; @eradman would you mind taking a look? Are we able to change these deps to enable arm64 images?

[AndrewChubatiuk]

reverted frontend changes, as they are built separately now

[AndrewChubatiuk]

Here's an example action that we already ran: https://github.com/getredash/redash/actions/runs/6546702269

I'm not sure how that was running so maybe we just did it wrong. :)

@guidopetri Finished doing my changes. you can check build time here
https://github.com/AndrewChubatiuk/redash/actions/runs/6786331640

There's lot's of work still, lot's of python packages which are either not supported or outdated, do not have arm64 wheels. Making them up to date can reduce build time as well

Also node dependencies for cypress can be optimized, cause it also takes several minutes to install them inside container

[AndrewChubatiuk]

@guidopetri @eradman
Are there any additional changes needed?

[eradman]

This change seems to check out for me. Triggering CI tests

[codecov]

Codecov Report

Merging #6577 (1dda186) into master (8bfc574) will not change coverage.
The diff coverage is n/a.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #6577   +/-   ##
=======================================
  Coverage   61.89%   61.89%           
=======================================
  Files         158      158           
  Lines       12992    12992           
  Branches     1774     1774           
=======================================
  Hits         8042     8042           
  Misses       4673     4673           
  Partials      277      277           

[eradman]

The code coverage report is not relavent.

@guidopetri can you re-test a build with these changes on arm64?

[guidopetri]

I'm not 100% clear where this frontend gets used in the build. Would you mind explaining? Does this also get pushed to Dockerhub as part of the preview image?

[AndrewChubatiuk]

@guidopetri frontend is added to docker image during a build for each target arch and then first pushed to redash/preview and then copied from redash/preview to redash/redash

[guidopetri]

How do we build this locally, then? docker build -t redash_test . seems to not build the frontend

[guidopetri]

I'm unclear on what these changes are doing, and I think we need better documentation around this. How do we build things locally? What exactly is happening in the GHA build process? Why do we need to split up the frontend build?

[AndrewChubatiuk]

I'm unclear on what these changes are doing, and I think we need better documentation around this. How do we build things locally? What exactly is happening in the GHA build process? Why do we need to split up the frontend build?

• github actions is using amd64 runners even for arm64 builds, so moving frontend out of docker images makes build faster
• to build an image locally is a two step procedure, which requires to build frontend and then build a docker image (i can make a separate dockerfile for CI or add a make action to run a single action for a build)
• during a build process frontend is built outside a docker images, cause it's platform independent and then docker copies it inside in image during a build of docker image for each platform

[guidopetri]

Hmm, alright. I think we should separate this into two PRs then:

• one enabling ARM64 builds (which may be slow if they're building the frontend);
• one separating the frontend build out, and making it clear how to build this locally. We should be able to run a docker command for this, and not depend on e.g. yarn being installed locally.

Does this make sense to you too? Or how do you think we should proceed?

[AndrewChubatiuk]

merged alternative PR in #6674