Add a job that runsmake full-source-dist, but mark it as "allow fail" for now
#252
Conversation
DilumAluthge
force-pushed
the
dpa/full-source-dist
branch
from
13edb9b
to
976b938
Compare
DilumAluthge
changed the title
make full-source-dist doesn't regressmake full-source-dist, but mark it as "allow fail" for now
|
I'm getting: @staticfloat @ararslan Any ideas what's going on? |
pipelines/main/misc/fullsrcdist.yml
Outdated
| # Drop default "registries" directory, so it is not persisted from execution to execution | ||
| persist_depot_dirs: packages,artifacts,compiled | ||
| version: '1.6' | ||
| timeout_in_minutes: 10 |
There was a problem hiding this comment.
I think full-source-dist may take more than 10 minutes... I've never timed it but anecdotally it feels like an eternity. It does a source build of all dependencies then Julia then all documentation.
There was a problem hiding this comment.
Wait, it actually builds Julia? I thought it just downloads the source code of the dependencies.
There was a problem hiding this comment.
Yeah haha. It builds Julia so that it can use the built binary to build the documentation.
There was a problem hiding this comment.
Hmmm. A full from-source build of the dependencies and Julia takes more than 30 minutes on our fastest CI machines. See e.g. https://buildkite.com/julialang/julia-master-scheduled/builds/159#01847a53-374d-441c-90cc-c4dcfef6ae22
Maybe we should move the full-source-dist job to the once-daily scheduled pipeline. That's where the existing USE_BINARYBUILDER=0 job lives.
There was a problem hiding this comment.
I'm now downloading a pre-built Julia, but it seems to insist on building everything (including LLVM) from source.
There was a problem hiding this comment.
@ararslan What's the difference between make full-source-dist and make full-source-dist USE_BINARYBUILDER=0? Shouldn't they both produce the same output? Presumably we shouldn't have to build LLVM from source just to be able to build the docs and download the source tarballs of all dependencies, right?
There was a problem hiding this comment.
USE_BINARYBUILDER=0 ensures that the source code of the dependencies is included in the resulting tarball alongside Julia's source code. It's meant to be platform-agnostic and buildable without network access, which IIUC isn't possible with the default JLL-based dependencies.
There was a problem hiding this comment.
Sure, but shouldn't make full-source-dist do that regardless of the value of USE_BINARYBUILDER?
I guess what I'm getting at is, do we need to actually do a full from-source build of LLVM etc just to build this tarball? AFAICT, the tarball needs the following contents:
- Source code of Julia
- Source code of all of Julia's dependencies
- HTML documentation
It seems to me that we can use a regular Julia build to build the HTML docs, and then download the source code of all the dependencies, and then stick everything together.
There was a problem hiding this comment.
We definitely could do that, the Makefile just isn't set up that way
Co-authored-by: Alex Arslan <[email protected]>
None at all from me. Where are you seeing that? I still haven't gotten used to navigating the buildkite UI... |
Here's an example log: https://buildkite.com/julialang/julia-buildkite/builds/1144#01847d9b-3335-4515-8457-3e87976b0331 |
|
Try adding julia-buildkite/pipelines/main/platforms/build_linux.yml Lines 17 to 18 in 709d226 I also suggest using a newer package rootfs image; we've made lots of improvements since the |
In a follow-up PR, we'll take this output and upload it to S3. But for now, let's just get this PR in, so that we can make sure that the full source dist doesn't regress again.
Refs #190 (but does not fix it, because we still need to actually upload them to S3, which we'll do in a follow-up PR).