track down transfer errors when there are spaces in transfer paths

[blaiszik]

No description provided.

[blaiszik]

There seems to be Foundry transfer errors when using globus=True flag (i.e., transferring via the Forge client using Globus SDK). The error likely traces to the permalink below. Perhaps we should just use the Globus SDK directly to make the transfer instead of using forge globus_download() function which is quite heavy

https://github.com/MLMI2-CSSI/foundry/blob/efc7f68e33618aca741c543bc287032ef38304f7/foundry/foundry.py#LL543C16-L543C16

Image and err via @isaac-darling

  "code": "EndpointNotFound",
  "message": "No such endpoint with legacy name 'c:\\Users\\Isaac Darling\\Desktop\\Programs\\globus\\seedlings\\foundry_example\\data'",
  "request_id": "UCS7NJOuD",
  "resource": "/operation/endpoint/c%3A%5CUsers%5CIsaac%20Darling%5CDesktop%5CPrograms%5Cglobus%5Cseedlings%5Cfoundry_example%5Cdata/ls"
}
Shared in

[ascourtas]

Getting this issue in MDF as well when they publish things with space in the path -- must be a globus change

[ascourtas]

TODO: ask someone on transfer team

[ericblau]

After looking into this for just a few minutes here are my thoughts:

The string 'c:\Users\Isaac Darling\Desktop\Programs\globus\seedlings\foundry_example\data' is being used as the endpoint_id of the local GlobusConnectPersonal endpoint. This is almost definitely wrong.

Tracing through the code to see where this comes from:
in mdf_forge: Forge.globus_download() has all the logic.

Line 720 of forge.py: https://github.com/materials-data-facility/forge/blob/a7f545170faab4ff312ffb3149b2346ee4b059c7/mdf_forge/forge.py#L720

assembles g_link as:
g_link = res.get("data", {}).get("endpoint_path", None)

which is looking for something like the endpoint_path field from:

example_dataset = {
"mdf": {
"resource_type": "dataset",
"source_id": "foobar_v1"
},
"data": {
"endpoint_path": ("globus://e38ee745-6d04-11e5-ba46-22000b92c6ec"
"/MDF/mdf_connect/test_files/")
}
}

The error looks to me like the dataset that is being attempted to transfer probably has an endpoint_path defined as 'c:\Users\Isaac Darling\Desktop\Programs\globus\seedlings\foundry_example\data'
or possibly something similar that is confusing urlparse at lines 738 and 739.

In any case, I don't think this is a simple "error on a globus transfer because of spaces in path", I think something else is going on, and the first thing to check would be the dataset.

[ascourtas]

Eric had some thoughts on this one!