closure_js_test - Passed tests sometimes hang

[robfig]

We are just starting to use rules_closure for real, and the second test suite that I've migrated shows flaky behavior on our CI system. I was hoping for some advice on how to diagnose this.

We use teamcity, and when running the suite 3 times in a row, one test finished in TIMEOUT the first run, a different test the second, and all tests passed on the third. It ran on 3 different hosts, but they are identically configured with ansible.

The tests don't take long to run, and the runner outputs "all tests passed" long before the timeout. They do not do anything that could account for a long delay.

First run:

[13:16:46]	==================== Test output for //js/corp/consumerfeedbackstorm/invites/templates:addtemplatedialog_test:
[13:16:46]	Listening http://localhost:35355/index.html
[13:16:46]	Serving /index.html
[13:16:46]	Serving /filez/corp/js/corp/consumerfeedbackstorm/invites/templates/addtemplatedialog_test_bin.js
[13:16:46]	Serving /filez/corp/js/vendor/jquery-ui-1.8.24.min.js
[13:16:46]	Serving /filez/corp/js/vendor/jquery-1.10.2.min.js
[13:16:46]	-> 13:11:33.964 : Starting tests: Untitled Test Case
[13:16:46]	-> 13:11:33.965 : Running test: testErrorRendering
[13:16:46]	-> 13:11:33.975 : testErrorRendering : PASSED
[13:16:46]	-> 13:11:33.975 : Running test: testUserInput
[13:16:46]	-> 13:11:34.003 : testUserInput : PASSED
[13:16:46]	-> 13:11:34.003 : 2 of 2 tests run in 38.78055200000017ms.
[13:16:46]	-> 2 passed, 0 failed.
[13:16:46]	19 ms/test. 4 files loaded.
[13:16:46]	-> 13:11:34.003 : Tests complete
[13:16:46]	Terminated
[13:16:46]	-- Test timed out at 2019-07-21 13:16:30 EDT --
[13:16:46]	================================================================================
[13:16:46]	//js/corp/consumerfeedbackstorm/accountsettings:customlegal_test         PASSED in 1.7s
[13:16:46]	//js/corp/consumerfeedbackstorm/accountsettings:emailinvites_test        PASSED in 1.8s
[13:16:46]	//js/corp/consumerfeedbackstorm/accountsettings:maxcontactfrequency_test PASSED in 2.3s
[13:16:46]	//js/corp/consumerfeedbackstorm/accountsettings:quarantine_test          PASSED in 1.5s
[13:16:46]	//js/corp/consumerfeedbackstorm/accountsettings:settingsrow_test         PASSED in 1.8s
[13:16:46]	//js/corp/consumerfeedbackstorm/accountsettings:surveypageenabled_test   PASSED in 1.3s
[13:16:46]	//js/corp/consumerfeedbackstorm/accountsettings:textinvitesettingsvalidator_test PASSED in 0.8s
[13:16:46]	//js/corp/consumerfeedbackstorm/accountsettings:textinvitesettingsview_test PASSED in 1.9s
[13:16:46]	//js/corp/consumerfeedbackstorm/invites:reviewinvitationvalidator_test   PASSED in 1.2s
[13:16:46]	//js/corp/consumerfeedbackstorm/invites/templates:templatevalidator_test PASSED in 1.1s
[13:16:46]	//test/com/corp/consumerfeedback:unit                                    PASSED in 48.1s
[13:16:46]	//js/corp/consumerfeedbackstorm/invites/templates:addtemplatedialog_test TIMEOUT in 315.1s
[13:16:46]	  /home/cdbuilder/.cache/bazel/_bazel_TeamCity/df2b12f83f809abace199d393a19f0d3/execroot/corp/bazel-out/k8-fastbuild/testlogs/js/corp/consumerfeedbackstorm/invites/templates/addtemplatedialog_test/test.log
[13:16:46]	
[13:16:46]	Executed 12 out of 12 tests: 11 tests pass and 1 fails locally.
[13:16:46]	

Second run

[14:46:17]	==================== Test output for //js/corp/consumerfeedbackstorm/accountsettings:settingsrow_test:
[14:46:17]	Listening http://localhost:54287/index.html
[14:46:17]	Serving /index.html
[14:46:17]	Serving /filez/corp/js/corp/consumerfeedbackstorm/accountsettings/settingsrow_test_bin.js
[14:46:17]	Serving /filez/corp/js/vendor/jquery-ui-1.8.24.min.js
Serving /filez/corp/js/vendor/jquery-1.10.2.min.js
[14:46:17]	-> 14:41:04.476 : Starting tests: Untitled Test Case
[14:46:17]	-> 14:41:04.476 : Running test: testCancelReset
[14:46:17]	-> 14:41:04.497 : testCancelReset : PASSED
[14:46:17]	-> 14:41:04.497 : Running test: testRenderErrors
[14:46:17]	-> 14:41:04.500 : testRenderErrors : PASSED
[14:46:17]	-> 14:41:04.500 : Running test: testSubmitToggle
[14:46:17]	-> 14:41:04.502 : testSubmitToggle : PASSED
[14:46:17]	-> 14:41:04.502 : Running test: testToggleEdit
[14:46:17]	-> 14:41:04.513 : testToggleEdit : PASSED
[14:46:17]	-> 14:41:04.514 : 4 of 4 tests run in 38.01679799999988ms.
[14:46:17]	-> 4 passed, 0 failed.
[14:46:17]	10 ms/test. 4 files loaded.
[14:46:17]	Loading: 
[14:46:17]	-> 14:41:04.514 : Tests complete
[14:46:17]	Terminated
[14:46:17]	Loading: 0 packages loaded
[14:46:17]	-- Test timed out at 2019-07-21 14:46:02 EDT --
[14:46:17]	================================================================================
[14:46:17]	//js/corp/consumerfeedbackstorm/accountsettings:customlegal_test         PASSED in 0.9s
[14:46:17]	//js/corp/consumerfeedbackstorm/accountsettings:emailinvites_test        PASSED in 1.0s
[14:46:17]	//js/corp/consumerfeedbackstorm/accountsettings:maxcontactfrequency_test PASSED in 1.1s
[14:46:17]	//js/corp/consumerfeedbackstorm/accountsettings:quarantine_test          PASSED in 1.1s
[14:46:17]	//js/corp/consumerfeedbackstorm/accountsettings:surveypageenabled_test   PASSED in 1.1s
[14:46:17]	//js/corp/consumerfeedbackstorm/accountsettings:textinvitesettingsvalidator_test PASSED in 1.1s
[14:46:17]	//js/corp/consumerfeedbackstorm/accountsettings:textinvitesettingsview_test PASSED in 1.2s
[14:46:17]	//js/corp/consumerfeedbackstorm/invites:reviewinvitationvalidator_test   PASSED in 0.9s
[14:46:17]	//js/corp/consumerfeedbackstorm/invites/templates:addtemplatedialog_test PASSED in 0.9s
[14:46:17]	//js/corp/consumerfeedbackstorm/invites/templates:templatevalidator_test PASSED in 0.9s
[14:46:17]	//test/com/corp/consumerfeedback:unit                                    PASSED in 19.2s
[14:46:17]	//js/corp/consumerfeedbackstorm/accountsettings:settingsrow_test        TIMEOUT in 315.0s

Third run didn't print anything because it was successful, taking 1m44s to build/run the java and js test suites.

This seems very much like the JS test suites succeeds, but for whatever reason Bazel doesn't get that memo and waits until the 5 minute limit expires, after which Bazel declares the test timed out.

Have you seen anything like this before?