[Bug]: JavaUsingPython pipline create large tmp files quickly causing CI drain disk space

[Abacn]

What happened?

It is found that Jenkins often run of of disk space even inventory job run twice per day. Checking the filesystem it is found there are many tmp file named after /tmp/beam-artifact.... hundreds MB per file. see https://ci-beam.apache.org/view/Inventory/job/beam_Inventory_apache-beam-jenkins-7/

This is from

https://github.com/apache/beam/blob/master/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/External.java#L393

Done at pipeline expansion time for external transform.

This suggests real issue for users as tmp file left are large. We should be able to clean up these files after pipeline run.

Interestingly it seems only Jenkins 7 (or a few other workers) are affected. It is known that Jenkins scheduler tend to run same postcommit job on same machine. Likely due to this job:

https://ci-beam.apache.org/job/beam_PostCommit_XVR_JavaUsingPython_Dataflow/

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner