Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Workflow Hub Crate Error #124

Closed
kbarrass1252 opened this issue Oct 8, 2024 · 6 comments
Closed

Workflow Hub Crate Error #124

kbarrass1252 opened this issue Oct 8, 2024 · 6 comments
Assignees
Milestone

Comments

@kbarrass1252
Copy link

When pulling from workflowhub.eu (both samples and our own) we encounter the following error: "JSON-LD from /home/admin/wfexs_tmpdir/ro-crate-cache/***.crate.zip is not a Workflow RO-Crate"

A dump of the error is shown below.

All pre-requisites are shown as valid. Config and stage file are taken from a working instance from another user who have patched a local copy to run from local git - but this scenario does not work for us.

Error is coming from 2686 to 2689 of this file:
[WfExS-backend/wfexs_backend/wfexs_backend.py at main · inab/WfExS-backend · GitHub]
(https://github.com/inab/WfExS-backend/blob/main/wfexs_backend/wfexs_backend.py#L2686C1-L2689C14)

(.pyWEenv) xxxxx@ip-xxx-xxx-xxx-xxx:~/wfexs/WfExS-backend$ python3 WfExS-backend.py -L config/local_config_admin.yaml execute -W omopworkflow/omop-workflow.stage

2024-10-08 13:33:13,543 - [INFO] Setting up docker to /usr/bin/docker (derived from docker) (core: True)
2024-10-08 13:33:13,543 - [INFO] Setting up git to /usr/bin/git (derived from git) (core: False)
2024-10-08 13:33:13,543 - [INFO] Setting up java to /usr/bin/java (derived from java) (core: True)
2024-10-08 13:33:13,543 - [INFO] Setting up singularity to /usr/local/bin/singularity (derived from singularity) (core: True)
2024-10-08 13:33:13,544 - [INFO] Setting up staticBash to /usr/bin/bash-linux-x86_64 (derived from bash-linux-x86_64) (core: False)
* Command "execute".
        - Working directory will be /tmp/WfExS-workdirmn8phbeobackend/6834f973-3fb1-4912-abce-412f007b80b9
          Instance 6834f973-3fb1-4912-abce-412f007b80b9 (nickname 'OmopWorkerhypercoagulable Newell teapot') (to be used with -J)
2024-10-08 13:33:15,706 - [INFO] downloaded putative workflow: https://workflowhub.eu/ga4gh/trs/v2/tools/471/versions/3/CWL/files?format=zip -> /home/admin/wfexs_tmpdir/wf-inputs/sha256~M0pCvIkfBp8aE972mNTiAxYr1SL_2OGNBBGjGxWRq_w=
2024-10-08 13:33:15,726 - [INFO] putative workflow https://workflowhub.eu/ga4gh/trs/v2/tools/471/versions/3/CWL/files?format=zip seems to be a packed RO-Crate
Traceback (most recent call last):
  File "/home/admin/wfexs/WfExS-backend/WfExS-backend.py", line 22, in <module>
    main()
  File "/home/admin/wfexs/WfExS-backend/wfexs_backend/__main__.py", line 1619, in main
    stagedSetup = wfInstance.stageWorkDir()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/admin/wfexs/WfExS-backend/wfexs_backend/workflow.py", line 3530, in stageWorkDir
    self.materializeWorkflowAndContainers(
  File "/home/admin/wfexs/WfExS-backend/wfexs_backend/workflow.py", line 2223, in materializeWorkflowAndContainers
    self.setupEngine(
  File "/home/admin/wfexs/WfExS-backend/wfexs_backend/workflow.py", line 2134, in setupEngine
    self.fetchWorkflow(
  File "/home/admin/wfexs/WfExS-backend/wfexs_backend/workflow.py", line 1968, in fetchWorkflow
    ) = self.wfexs.cacheWorkflow(
        ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/admin/wfexs/WfExS-backend/wfexs_backend/wfexs_backend.py", line 2094, in cacheWorkflow
    i_workflow, repoDir = self.getWorkflowRepoFromTRS(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/admin/wfexs/WfExS-backend/wfexs_backend/wfexs_backend.py", line 2414, in getWorkflowRepoFromTRS
    ) = self.getWorkflowBundleFromURI(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/admin/wfexs/WfExS-backend/wfexs_backend/wfexs_backend.py", line 2646, in getWorkflowBundleFromURI
    self.getWorkflowRepoFromROCrateFile(roCrateFile, expectedEngineDesc),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/admin/wfexs/WfExS-backend/wfexs_backend/wfexs_backend.py", line 2683, in getWorkflowRepoFromROCrateFile
    raise WfExSBackendException(
wfexs_backend.wfexs_backend.WfExSBackendException: JSON-LD from /home/admin/wfexs_tmpdir/ro-crate-cache/d37a7cf3a801c5622900e0bfc103d744abd70f30.crate.zip is not a Workflow RO-Crate
@jmfernandez
Copy link
Member

Dear @kbarrass1252 , first of all, thanks for the feedback! I have been analysing the generated RO-Crate from WorkflowHub, and in this case, strictly following the specification, the code is complaining properly. The documentation about profile https://about.workflowhub.eu/Workflow-RO-Crate/1.0/ specifies that :

The Metadata File Descriptor conformsTo SHOULD be an array that contains at least https://w3id.org/ro/crate/1.1 and https://w3id.org/workflowhub/workflow-ro-crate/1.0

In the RO-Crate obtained from WorkflowHub, the conformsTo predicate declaring https://w3id.org/workflowhub/workflow-ro-crate/1.0 profile is tied to the root dataset instead of the Metadata File Descriptor:

{
  "@context": "https://w3id.org/ro/crate/1.1/context",
  "@graph": [
    {
      "@id": "ro-crate-metadata.json",
      "@type": "CreativeWork",
      "about": {
        "@id": "./"
      },
      "conformsTo": {
        "@id": "https://w3id.org/ro/crate/1.1"
      }
    },
    {
      "@id": "ro-crate-preview.html",
      "@type": "CreativeWork",
      "about": {
        "@id": "./"
      }
    },
    {
      "@id": "./",
      "@type": "Dataset",
      "conformsTo": [
        {
          "@id": "https://w3id.org/ro/crate/1.1"
        },
        {
          "@id": "https://w3id.org/workflowhub/workflow-ro-crate/1.0"
        }
      ],
      "mainEntity": {
        "@id": "x86/workflow-entrypoint.cwl"
      },
      "author": [
        {
          "@id": "#person-476"
        }
      ],

On the other hand, RO-Crate 1.2 draft specification declares that any RO-Crate profile should be referred from the conformsTo tied to the root dataset https://www.researchobject.org/ro-crate/specification/1.2-DRAFT/profiles.html#declaring-conformance-of-an-ro-crate-profile (as it is right now WorkflowHub doing).

@stain @fbacall I have a question, as this issue might partially fall on your side. RO-Crates obtained from WorkflowHub with URLs similar to https://workflowhub.eu/ga4gh/trs/v2/tools/471/versions/3/CWL/files?format=zip , are generated on the fly? If it is so, then either the Workflow RO-Crate profile specification should be updated in order to be conformant to RO-Crate 1.2 draft, or WorkflowHub should fix the issue about the location of conformsTo referring to Workflow RO-Crate.

@jmfernandez
Copy link
Member

In any case @kbarrass1252 , I'm going to relax the detection of Workflow RO-Crate

@jmfernandez
Copy link
Member

@kbarrass1252 I have tagged WfExS 1.0.0rc1, where commit 61fd06a relaxes Workflow RO-Crate profile detection .

Could you try using it in your usage scenario, please?

@kbarrass1252
Copy link
Author

Hi @jmfernandez ! TY so much!

However - we still have an issue as follows:

"(.pyWEenv) @ip----:~/wfexs/WfExS-backend$ python3 WfExS-backend.py -L config/local_config_admin.yaml execute -W omopworkflow/omop-workflow.stage
2024-10-16 08:33:14,835 - [INFO] Setting up docker to /usr/bin/docker (derived from docker) (core: True)
2024-10-16 08:33:14,836 - [INFO] Setting up git to /usr/bin/git (derived from git) (core: False)
2024-10-16 08:33:14,836 - [INFO] Setting up java to /usr/bin/java (derived from java) (core: True)
2024-10-16 08:33:14,836 - [INFO] Setting up singularity to /usr/local/bin/singularity (derived from singularity) (core: True)
2024-10-16 08:33:14,836 - [INFO] Setting up staticBash to /usr/bin/bash-linux-x86_64 (derived from bash-linux-x86_64) (core: False)

  • Command "execute".
    - Working directory will be /tmp/WfExS-workdirch23x2cubackend/53ce3c92-673a-4250-ab58-9a6e03b7ab31
    Instance 53ce3c92-673a-4250-ab58-9a6e03b7ab31 (nickname 'OmopWorkersubnatural summeriness') (to be used with -J)
    2024-10-16 08:33:16,421 - [INFO] downloading workflow input: file:///home/admin/wfexs/WfExS-backend/jobs/work/data/querycurrent.json
    2024-10-16 08:33:16,425 - [INFO] downloaded workflow input: file:///home/admin/wfexs/WfExS-backend/jobs/work/data/querycurrent.json => /home/admin/wfexs_tmpdir/wf-inputs/sha256Et7jAhr7ebK9hh5H4P7h1T-6Ore8kXeEPcwV-6N8hbY=
    2024-10-16 08:33:16,425 - [INFO] downloaded workflow input chain: file:///home/admin/wfexs/WfExS-backend/jobs/work/data/querycurrent.json => /home/admin/wfexs_tmpdir/wf-inputs/sha256
    Et7jAhr7ebK9hh5H4P7h1T-6Ore8kXeEPcwV-6N8hbY=
    2024-10-16 08:33:16,953 - [INFO] downloaded putative workflow: https://workflowhub.eu/ga4gh/trs/v2/tools/471/versions/3/CWL/files?format=zip -> /home/admin/wfexs_tmpdir/wf-inputs/sha256~M0pCvIkfBp8aE972mNTiAxYr1SL_2OGNBBGjGxWRq_w=
    2024-10-16 08:33:16,972 - [INFO] putative workflow https://workflowhub.eu/ga4gh/trs/v2/tools/471/versions/3/CWL/files?format=zip seems to be a packed RO-Crate
    2024-10-16 08:33:18,327 - [INFO] materialized workflow repository (checkout 724775ab48bff4ae6da6c39da4ad1619abcf8518): /tmp/WfExS-workdirch23x2cubackend/53ce3c92-673a-4250-ab58-9a6e03b7ab31/workflow
    2024-10-16 08:33:18,853 - [WARNING] CWL entrypoint cannot be a directory
    Traceback (most recent call last):
    File "/home/admin/wfexs/WfExS-backend/WfExS-backend.py", line 22, in
    main()
    File "/home/admin/wfexs/WfExS-backend/wfexs_backend/main.py", line 1666, in main
    stagedSetup = wfInstance.stageWorkDir()
    ^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/admin/wfexs/WfExS-backend/wfexs_backend/workflow.py", line 3611, in stageWorkDir
    self.materializeWorkflowAndContainers(
    File "/home/admin/wfexs/WfExS-backend/wfexs_backend/workflow.py", line 2241, in materializeWorkflowAndContainers
    self.setupEngine(
    File "/home/admin/wfexs/WfExS-backend/wfexs_backend/workflow.py", line 2150, in setupEngine
    self.fetchWorkflow(
    File "/home/admin/wfexs/WfExS-backend/wfexs_backend/workflow.py", line 2104, in fetchWorkflow
    raise WFException(
    wfexs_backend.workflow.WFException: Engine cwl did not recognize a workflow at https://github.com/HDRUK/rquest-omop-worker-workflows"

However - our fallback test which is to run the CWL through cwltool through conda runs the CWL without any error :(

@jmfernandez
Copy link
Member

Mmmmmm, I see @kbarrass1252 , thanks for the tests!! As you have uncovered a different bug, I have transferred your last report to the new issue #128

@kbarrass1252
Copy link
Author

Perfect! TTY!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants