Merge pull request #4467 from acmorrow/parallel-cache-push-strawman

Parallel cache push strawman

CHANGES.txt

@@ -65,6 +65,9 @@ RELEASE  VERSION/DATE TO BE FILLED IN LATER
       Python to 3.6, and it was not until Python 3.7 where `threading` became
       default supported. In practice, we expect most real world Python 3.6 deployments
       will have `threading` support enabled, so this will not be an issue.
+    - CacheDir writes no longer happen within the taskmaster critical section,
+      and therefore can run in parallel with both other CacheDir writes and the
+      taskmaster DAG walk.
 
   From Mats Wichmann:
     - Add support for Python 3.13 (as of alpha 2). So far only affects

RELEASE.txt

@@ -63,6 +63,8 @@ IMPROVEMENTS
   the new scheduler is now used for -j1 builds as well.
   NOTE: This should significantly improve SCons performance for larger parallel builds
         (Larger -j values)
+- CacheDir writes no longer happen within the taskmaster critical section, and therefore
+  can run in parallel with both other CacheDir writes and the taskmaster DAG walk.
 
 
 PACKAGING

SCons/Taskmaster/__init__.py

@@ -244,6 +244,8 @@ def execute(self):
                         SCons.Warnings.warn(SCons.Warnings.CacheCleanupErrorWarning,
                             "Failed copying all target files from cache, Error while attempting to remove file %s retrieved from cache: %s" % (t.get_internal_path(), e))
                 self.targets[0].build()
+                for t in self.targets:
+                    t.push_to_cache()
             else:
                 for t in cached_targets:
                     t.cached = 1
@@ -299,8 +301,6 @@ def executed_with_callbacks(self) -> None:
                 for side_effect in t.side_effects:
                     side_effect.set_state(NODE_NO_STATE)
                 t.set_state(NODE_EXECUTED)
-                if not t.cached:
-                    t.push_to_cache()
                 t.built()
                 t.visited()
                 if (not print_prepare and