[question] Incremental builds

[tylerjw]

This is partially related to colcon/colcon-package-selection#25. I am trying to understand when a we wouldn't be able to depend on cmake/make/ninja to handle the situation of the incremental building and would need more control over when to clean (start over) on specific targets and their dependencies. I'd like to be able to use incremental builds in CI, caching the whole workspace.

[mikepurvis]

colcon has no visibility into what must be rebuilt— its policy is simply to call the appropriate build command for every workspace package in the proper topological order according to the build-dependency information available to it. So the baseline behaviour is that packages which are unchanged have their build command be a no-op based on whatever the internal logic of the buildsystem is (so in the make/ninja case, it's running all over the filesystem checking a bunch of timestamps). This will be much faster than an actual build, but obviously not zero time.

In a CI setting, it's possible you have more information than colcon does, and you may know (from your vcs diffs or whatever else) that particular packages have experienced no change at all since your last build. In that case, you could use colcon's various package selection capabilities to tell it to skip those ones, and then they're a true no-op.

We use a scheme like this at Clearpath to get fast incremental builds, but unfortunately it's pretty tied to a bunch of internal details on our end, so it's not really something we can open up in its present state.

[tylerjw]

How would the approach of updating the repos and running colcon build work if the dependencies changed or repos were added or removed from the workspace?

I'm wondering if the case where a repo is removed from a workspace would be particularly bad because any packages it contained would still be in the build folder and other packages could be built on top of those build products when you instead intend them to be built on Debian installed packages.

[mikepurvis]

Adding packages to the workspace may require manually forcing CMake to rerun (--cmake-clean-cache/--cmake-force-configure) as CMake will otherwise have cached that its dep location is in the underlay rather than the workspace.

Changed deps will be picked up automatically as the timestamp on the package.xml file moves forward, which I believe signals to the CMake logic that it needs to reconfigure.

Removing packages is definitely the trickiest case; I would imagine that your safest path there would be to a clean build of any package that's down the dependency tree from a removed one.

[tylerjw]

I'm curious if this would be something colcon should try to handle nicely (through some sort of statefulness from build/tests) or if we should look to build external tooling for deciding when to do what. This is related to knowing when things have changed in repositories. In this case, since the last build. In my other use-case, since the last test run.