Misidentified workflow entrypoint from WorkflowHub entry

[jmfernandez]

(originally submitted by @kbarrass1252)

Hi @jmfernandez ! TY so much!

However - we still have an issue as follows:

"(.pyWEenv) @ip----:~/wfexs/WfExS-backend$ python3 WfExS-backend.py -L config/local_config_admin.yaml execute -W omopworkflow/omop-workflow.stage
2024-10-16 08:33:14,835 - [INFO] Setting up docker to /usr/bin/docker (derived from docker) (core: True)
2024-10-16 08:33:14,836 - [INFO] Setting up git to /usr/bin/git (derived from git) (core: False)
2024-10-16 08:33:14,836 - [INFO] Setting up java to /usr/bin/java (derived from java) (core: True)
2024-10-16 08:33:14,836 - [INFO] Setting up singularity to /usr/local/bin/singularity (derived from singularity) (core: True)
2024-10-16 08:33:14,836 - [INFO] Setting up staticBash to /usr/bin/bash-linux-x86_64 (derived from bash-linux-x86_64) (core: False)

• Command "execute".
  - Working directory will be /tmp/WfExS-workdirch23x2cubackend/53ce3c92-673a-4250-ab58-9a6e03b7ab31
  Instance 53ce3c92-673a-4250-ab58-9a6e03b7ab31 (nickname 'OmopWorkersubnatural summeriness') (to be used with -J)
  2024-10-16 08:33:16,421 - [INFO] downloading workflow input: file:///home/admin/wfexs/WfExS-backend/jobs/work/data/querycurrent.json
  2024-10-16 08:33:16,425 - [INFO] downloaded workflow input: file:///home/admin/wfexs/WfExS-backend/jobs/work/data/querycurrent.json => /home/admin/wfexs_tmpdir/wf-inputs/sha256Et7jAhr7ebK9hh5H4P7h1T-6Ore8kXeEPcwV-6N8hbY=
  2024-10-16 08:33:16,425 - [INFO] downloaded workflow input chain: file:///home/admin/wfexs/WfExS-backend/jobs/work/data/querycurrent.json => /home/admin/wfexs_tmpdir/wf-inputs/sha256Et7jAhr7ebK9hh5H4P7h1T-6Ore8kXeEPcwV-6N8hbY=
  2024-10-16 08:33:16,953 - [INFO] downloaded putative workflow: https://workflowhub.eu/ga4gh/trs/v2/tools/471/versions/3/CWL/files?format=zip -> /home/admin/wfexs_tmpdir/wf-inputs/sha256~M0pCvIkfBp8aE972mNTiAxYr1SL_2OGNBBGjGxWRq_w=
  2024-10-16 08:33:16,972 - [INFO] putative workflow https://workflowhub.eu/ga4gh/trs/v2/tools/471/versions/3/CWL/files?format=zip seems to be a packed RO-Crate
  2024-10-16 08:33:18,327 - [INFO] materialized workflow repository (checkout 724775ab48bff4ae6da6c39da4ad1619abcf8518): /tmp/WfExS-workdirch23x2cubackend/53ce3c92-673a-4250-ab58-9a6e03b7ab31/workflow
  2024-10-16 08:33:18,853 - [WARNING] CWL entrypoint cannot be a directory
  Traceback (most recent call last):
  File "/home/admin/wfexs/WfExS-backend/WfExS-backend.py", line 22, in
  main()
  File "/home/admin/wfexs/WfExS-backend/wfexs_backend/main.py", line 1666, in main
  stagedSetup = wfInstance.stageWorkDir()
  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/admin/wfexs/WfExS-backend/wfexs_backend/workflow.py", line 3611, in stageWorkDir
  self.materializeWorkflowAndContainers(
  File "/home/admin/wfexs/WfExS-backend/wfexs_backend/workflow.py", line 2241, in materializeWorkflowAndContainers
  self.setupEngine(
  File "/home/admin/wfexs/WfExS-backend/wfexs_backend/workflow.py", line 2150, in setupEngine
  self.fetchWorkflow(
  File "/home/admin/wfexs/WfExS-backend/wfexs_backend/workflow.py", line 2104, in fetchWorkflow
  raise WFException(
  wfexs_backend.workflow.WFException: Engine cwl did not recognize a workflow at https://github.com/HDRUK/rquest-omop-worker-workflows"

However - our fallback test which is to run the CWL through cwltool through conda runs the CWL without any error :(

Originally posted by @kbarrass1252 in #124 (comment)

[jmfernandez]

@kbarrass1252 This issue is happening because although old heuristics are obtaining the workflow repo from the RO-Crate, they are not realising about the workflow entrypoint within the repo.

I have performed an inspection of the RO-Crate, and it has all what it is needed for that, so implementation and heuristics are incomplete.

I'm going to write a fix for these cases.

[kbarrass1252]

Amazing! - TY @jmfernandez !

[jmfernandez]

@kbarrass1252 I have merged a commit with a partial fix to main branch.

It is partial, because it works only when the relative internal structure of the RO-Crate (where the mainEntity of the workflow is declared) matches the relative internal structure of the original repository, and the default branch of the repo contains it. Details about the specific commit, branch or tag used when the WorkflowHub entry was created or updated do not appear in the RO-Crate (at least, not yet).

Once this issue is solved, I'm tagging 1.0.0rc2

[beforan]

@jmfernandez just to add to this, this is something I also encountered when testing my TRS implementation as a possible solution to airgapped executions (sourcing workflows from a local TRS service rather than public internet).

Happy to test changes to this going forwards in our regular collaboration.

[jmfernandez]

@beforan I would love to test it with a local TRS service, so I can stress the codepath at

https://github.com/inab/WfExS-backend/blob/main/wfexs_backend/wfexs_backend.py#L2448-L2495

I wrote that code ages ago in order to support "pure" TRS implementations like Dockstore and Sapporo, but I have not tested it recently. The only prerequisite is having either remote_workflow_entrypoint declared in its metadata, or all the involved workflow files plus the workflow_entrypoint in its metadata.