Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to use ccache anymore #1869

Open
jbleonesio opened this issue Sep 9, 2023 · 11 comments
Open

Unable to use ccache anymore #1869

jbleonesio opened this issue Sep 9, 2023 · 11 comments

Comments

@jbleonesio
Copy link

jbleonesio commented Sep 9, 2023

Since migrating from BuildStream 1.6.8 to 2.0.1 files created during a "bst build" step are not kept, even using a workspace.

As a result builds using ccache with a CCACHE_DIR contained inside the workspace directory are silently slowed down because they keep caching compilation objects without being able to take advantage of them in a later compilation.

I created a fully reproducible example of this behavior difference between bst 1.6.8 and bst 2.0.1 here : https://github.com/jbleonesio/bst_build_in_workspace_tests

flatpak-builder has a --ccache option flag dedicated to this purpose. Is there any way to achieve a similar goal with BuildStream and it is documented ?

For information, the CI/CD pipeline duration of an open-source project that I maintain went from ~8min to ~20min for that reason.

@jbleonesio
Copy link
Author

I ended up using recc by following @gtristan POC (see implementation here).

@nanonyme
Copy link
Contributor

@gtristan this now feels not great if downstreams have already started patching BuildStream to get casd socket in. Please resume #1772 and let's formalize sandbox configuration that allows the socket in.

@nanonyme
Copy link
Contributor

@jbleonesio do you happen to have any performance metrics after switching to recc?

@jbleonesio
Copy link
Author

@nanonyme After switching to recc I managed to get the pipeline to complete in ~4min30 (but in the meantime I migrated from BuildStream 1.6.8 to 2.0.1, from Freedesktop SDK 22.08 to 23.08 and I also simplified some build scripts).

@nanonyme
Copy link
Contributor

I am expecting we will start providing recc as a freedesktop-sdk element as soon as BuildStream lands support for this socket handling properly.

@nanonyme
Copy link
Contributor

Well, we did land recc in freedesktop-sdk 24.08 (not yet released). The BuildStream-side change is still disappointingly absent.

@jbleonesio
Copy link
Author

I'm now using recc from freedesktop-sdk 24.08 instead of building it on my own.

Since BuildStream 2.3.0, the patching needs to be updated to :

sed '135 i \            buildbox_command.append("--bind-mount={}:/tmp/casd.sock".format(casd._socket_path))\n' -i /path/to/_sandboxbuildboxrun.py

Still looking forward to have all of this officially supported.

@nanonyme
Copy link
Contributor

We had some discussion about this and @juergbi was hesitant towards the approach since it may expose entire LocalCAS to sandbox.

@jbleonesio
Copy link
Author

jbleonesio commented Oct 23, 2024

I do not know enough BuildStream internals to suggest nor appreciate a better design but there should be a supported and documented way to cache previously compiled objects IMHO.

Having a compilation time being 4x longer than what it should be (and that's just my own figures) could sadly turn users away from this really powerful and flexible project.

@nanonyme
Copy link
Contributor

Some relevant discussion here #1945 (comment)

@nanonyme
Copy link
Contributor

I do not know enough BuildStream internals to suggest nor appreciate a better design but there should be a supported and documented way to cache previously compiled objects IMHO.

Having a compilation time being 4x longer than what it should be (and that's just my own figures) could sadly turn users away from this really powerful and flexible project.

I hear you. Just thought to mention it looks like it's not as simple as just merging the proposed change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants