failed builds in build folder

[puggan]

Expected behaviour

When build is done, or removed, the data in the build folder gets removed.

Actual behaviour

There is alot of folders for old build, not just one per build, multiple:

28_54ad7 33_09d07 33_193e1 33_28b2d 33_3878d 33_46faf 33_54880 33_63593 33_75f01 33_8f167 33_b45b4 33_c350e 33_cddb1 33_d8850 33_ed1f5 33_fab9a 34_0e938 34_384e1 34_50d8e 34_6b538 34_80ba4 34_9d411 34_c2842 34_d42ba 34_e7581 34_f5fd8
28_9502a 33_0a29c 33_19624 33_2c3b2 33_3aeb3 33_49abb 33_54ac6 33_682b4 33_81041 33_942b9 33_b4a71 33_c44ef 33_ce4c5 33_df81e 33_ee61b 33_fb42e 34_11203 34_3ac63 34_52aa4 34_6ded7 34_83d19 34_9eea5 34_c7220 34_d7cfc 34_e7f8a 34_fa29c
29_4d2d3 33_0a49d 33_1aafd 33_30bbd 33_3c014 33_4addf 33_5759f 33_6d83a 33_818e3 33_96ba7 33_b5d99 33_c53d8 33_ce67d 33_e2ec1 33_ef8ef 33_fc2a3 34_14494 34_3b47a 34_54547 34_6e3e1 34_869e2 34_aac42 34_c761a 34_d7faa 34_e87b2 34_fca04
33_0125e 33_0c902 33_1ce06 33_32a88 33_3c0bd 33_4ba64 33_58e5f 33_6ebce 33_8769b 33_9c43a 33_b87a4 33_c6776 33_d1499 33_e4415 33_f0182 33_fe4d3 34_156de 34_3bca6 34_59a32 34_6f514 34_86f79 34_b1a4d 34_ca819 34_dadc2 34_e9450
33_0359f 33_0cd09 33_1feeb 33_34efe 33_3c8ea 33_4c894 33_590ad 33_7023c 33_88a8d 33_9cc0d 33_bbcb6 33_c6ab6 33_d5094 33_e4668 33_f0c2a 34_013a9 34_1765d 34_3bf12 34_5c050 34_70a10 34_91ac7 34_b7782 34_cb01f 34_e0a7c 34_e9e72
33_04be8 33_18595 33_207b2 33_35a30 33_3fb91 33_52f18 33_5a0c1 33_73ce7 33_8bb21 33_a66f9 33_bf39a 33_c83b8 33_d71e7 33_e4da8 33_f3a86 34_03dc8 34_1b05b 34_3e797 34_5d614 34_72d01 34_98c4e 34_b8ce7 34_cb528 34_e2c4a 34_ec07d
33_0610a 33_189ed 33_249cf 33_3760f 33_41004 33_53e02 33_5c671 33_7404d 33_8d310 33_a99d2 33_c08ba 33_cd769 33_d8460 33_e7193 33_f4e6e 34_051ec 34_21944 34_48d57 34_6459c 34_776fd 34_9a3e4 34_bad5b 34_cd6ee 34_e3089 34_ede48
33_066cf 33_18b6c 33_285c8 33_37ec4 33_41a3b 33_5400e 33_5de93 33_75674 33_8d979 33_b06ec 33_c245c 33_cd7c6 33_d86a7 33_e8594 33_f8e41 34_0e23a 34_294db 34_4d27b 34_69213 34_7c2e4 34_9b5d5 34_bfbdb 34_d13eb 34_e3280 34_f1ef8

Steps to reproduce

1. Fail the build, (in my case, old phpcs with bug: squizlabs/PHP_CodeSniffer@9a70ae2)
2. wait
3. remove the build
4. notice you get an "disk full"-warning
5. find alot of folders in PHPCI/build/, like over 100 for a build you already removed.

Environment info

Operating System: Ubuntu
PHPCI version: tag/1.7.1
PHP Version: 7.0.8-0ubuntu0.16.04.2
MySQL Version: 5.7.15-0ubuntu0.16.04.1
Supervisor Version: 3.2.0-2

Logs or other output that would be helpful

(If logs are large, please upload as attachment).

[puggan]

Looks like then restarts the build every 2minutes.
phpci.yml.txt
phpci_build_35.log.txt

[dmecke]

I have the same problem. It always starts another build with the same build number. It also happens with successful builds in my case.

[uLow]

Its a huge issue in fact. If you run acceptance tests, its going crazy every n minutes. It can run even foreign project in current build, just because it thinks that queue is ready for new one... And yeah, build id is staying the same.

[jscrobinson]

This is an issue for us as well. Not sure if this is related but occasionally the build will restart (do another git clone and begin the process again) but with the same build number with a different hash. This leaves the older build folder in place which causes the build to restart as described above.

[jscrobinson]

I think this is due to the 'time to run' setting when reserving the Beanstalk job. http://beanstalkc.readthedocs.io/en/latest/tutorial.html#basic-operation

In BuildWorker https://github.com/Block8/PHPCI/blob/master/PHPCI/Worker/BuildWorker.php#L101 reserve() is called without a timeout set so it uses the default of 120 seconds.

$job = $this->pheanstalk->reserve();

This causes beanstalk to re-queue the job and it's picked up again by another worker.

I think this should be set to whatever the build.failed_after setting is. Alternatively configuring beanstalk so it has a much higher TTR value might be a temporary solution.