xGDBvm scripts can be fooled by false FAILED message from Agave

[jduvick]

There is currently an issue with job status reporting via the Agave API that can cause the xGDBvm workflow to stop looking for output data from a remote job. This happens when a false FAILED status is returned to xGDBvm's webhook.php script, during a job that actually has not failed. In a pipeline-integrated job, this status when passed back to xGDB_Procedure.sh results in exiting a loop that would normally monitor for outputs in the user's job archive. Or, if a standalone job, it will simply result a false outcome displayed in the jobs.php listing. This 'false' FAILED status tends to happen to my jobs after SUBMITTING and before QUEUED status. Identical submissions may or may not 'falsely' FAIL. It's been reported to the Agave team, but in the meantime xGDBvm may report a job failure that is false. The current end user workaround would be to check for output from a FAILED run at /iplant/home/[username]/archive/jobs/[job-id]/ and if present, manually copy it to the input directory if further processing with a genome annotation is desired. If this Agave bug is not easily fixable, a technical workaround would be to modify xGDB_Procedure.sh to not react to a FAILED message unless it is not followed by any additional status messages over a specified amount of time. But hopefully this is a short-lived issue...!