Authentication timeout when requesting cmip6 data from "cil-gdpcir-cc0"

[kbonney]

Hello,

When I attempt to fetch cmip6 data from the "cil-gdpcir-cc0" collection, the download is abnormally slow and often ends up in an authentication timeout. For example, fetching "tasmin" for a single point for a single model/ssp over the entire time period (~85 years of daily time points) takes over 20 minutes to fetch and ultimately gives the following error:

Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:29b00994-201e-003a-052f-2a8982000000
Time:2024-10-29T18:25:28.1373289Z
ErrorCode:AuthenticationFailed
authenticationerrordetail:Signature not valid in the specified time frame: Start [Mon, 28 Oct 2024 17:40:28 GMT] - Expiry [Tue, 29 Oct 2024 18:25:28 GMT] - Current [Tue, 29 Oct 2024 18:25:28 GMT]
Content: <?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:29b00994-201e-003a-052f-2a8982000000
Time:2024-10-29T18:25:28.1373289Z</Message><AuthenticationErrorDetail>Signature not valid in the specified time frame: Start [Mon, 28 Oct 2024 17:40:28 GMT] - Expiry [Tue, 29 Oct 2024 18:25:28 GMT] - Current [Tue, 29 Oct 2024 18:25:28 GMT]</AuthenticationErrorDetail></Error>
azure.core.exceptions.HttpResponseError: Operation returned an invalid status 'Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.'
Content: <?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:29b00994-201e-003a-052f-2a8982000000
Time:2024-10-29T18:25:28.1373289Z</Message><AuthenticationErrorDetail>Signature not valid in the specified time frame: Start [Mon, 28 Oct 2024 17:40:28 GMT] - Expiry [Tue, 29 Oct 2024 18:25:28 GMT] - Current [Tue, 29 Oct 2024 18:25:28 GMT]</AuthenticationErrorDetail></Error>

To confirm this was not an issue with my code, I attempted to run the ensemble.ipynb from the planetary computer examples. I edited cell 10 to:

subset = all_datasets.pr.sel(
    lon=slice(129.408463169, 145.543137242),
    lat=slice(31.0295791692, 45.5514834662),
    time=slice("2020-01-01", "2030-01-05"),
)

which extends the time period to 10 years and I obtained a similar error to what I saw in my own code.

In both examples, the data I am fetching is pretty small so it surprises me that they are extending past the authentication time period.
I saw that #109 brought up some similar concerns and the suggestion was to use a subscription key or work on Planetary Computer Hub. I have not been able to find a way to obtain a subscription key and the Hub has been discontinued, so I am looking for other routes to address this timeout issue.

[777arc]

Out of curiosity, when that last step is running, what network download speed do you see? Eg if you open task manager and go to Performance then Ethernet.

[kbonney]

@777arc, during the execution of cell 11 (using the original ensemble.ipynb) I saw network speeds between 4,000-17,000 KB/sec for the associated Python process, which took 260 seconds to complete. After it completed, I looked at subset.nbytes which returned 1484800, or ~1.5 MB. The network rates I saw and the time it took to download don't seem to match up with the data I end up obtaining.

Also, I noticed that in the cell outputs saved to the version of the notebook online, cell 11 only took 9 seconds to complete, which is significantly faster than what I am experiencing.

[777arc]

This one took about 80s for me, but yeah let me try to figure out why it's downloading GBs of data to only produce a few MB of an xarray

subset = all_datasets.tasmax.sel(
    lon=slice(129.408463169, 145.543137242),
    lat=slice(31.0295791692, 45.5514834662),
    time=slice("2020-01-01", "2020-01-05"),
)

[777arc]

Turns out there are some hints as to why this is the case in the output of the 2nd cell- "The data is chunked at each interval of 365 days and 90 degree interval of latitude and longitude. Therefore, each chunk is (365, 360, 360), with each chunk occupying approximately 180MB in memory" so relatively large chunks, and even if you look at a tiny piece of the world, you'll need one of those 360*360 chunks, and that's for each time step. So I think this isn't really a bug or issue, it's just unfortunate that the zarr store isn't chunked into smaller chunks (which we don't have control over).

[kbonney]

Thanks @777arc, that provides some clarity on the download size discrepancy. I understand that the data fetching cannot be sped up, but is there a way to prevent the SAS token from expiring during long downloads? My original issue is that my data fetches were resulting in Authentication errors due to downloads takings longer that the token lifetime.

[777arc]

Ah ok, yeah I think it's set to 45m expiration right now. We can't increase that, so what I would do is break up the time range so that you grab a new token between each time step, or every N time steps, since you're interested in such a huge span of time.

[kbonney]

Okay that makes sense. Do you have a code snippet for generating and using new SAS tokens on the fly that you could share?

[kbonney]

And to confirm, there is currently no option to use a permanent API key with planetary computer, correct?

[777arc]

There is no permanent API key because there is no API key needed for the planetary computer, it's totally public. It's the blob storage SAS key that only lasts 45m, it gets generated when you do a search for items to go along with each asset so that you can download it freely from our blob storage (for 45m).

So what you can do is do your initial search to get the list of item IDs,

search = catalog.search(
    collections=["cil-gdpcir-cc0", "cil-gdpcir-cc-by"],
    query={"cmip6:experiment_id": {"eq": "ssp370"}},
)
items = search.item_collection()
ids = [item.id for item in items]
print(ids)

then loop through each item and do a search on it to make sure the token is fresh, then do whatever processing you want

from urllib.parse import unquote
first_id = ids[0] # you'll loop through instead, im just showing one as an example
search = catalog.search(
    collections=["cil-gdpcir-cc0", "cil-gdpcir-cc-by"],
    ids=[first_id],
)
item0 = search.item_collection()[0]
cred = item0.assets["pr"].to_dict()["xarray:open_kwargs"]["storage_options"]["credential"] # this is where the SAS token is stored
cred = unquote(cred)
print(cred.split('&')[1]) # this is the expiration date portion of the token

It's possible there's some sort of permanent key we can arrange though, email [email protected] about the details of what dataset you're using.

Also the last section here might help https://planetarycomputer.microsoft.com/docs/concepts/sas/

[kbonney]

If I am following correctly, the expiration date should update each time you run catalog.search. However, when I run the following snippet built from what you shared I do not see the expiration date change:

catalog = pystac_client.Client.open(
    "https://planetarycomputer.microsoft.com/api/stac/v1/",
    modifier=planetary_computer.sign_inplace,
)
search = catalog.search(
collections=["cil-gdpcir-cc0", "cil-gdpcir-cc-by"],
query={"cmip6:experiment_id": {"eq": "ssp370"}},
)
items = search.item_collection()
ids = [item.id for item in items]
print(ids)

from urllib.parse import unquote
for id in ids:
    search = catalog.search(
        collections=["cil-gdpcir-cc0", "cil-gdpcir-cc-by"],
        ids=[id],
    )
    item0 = search.item_collection()[0]
    cred = item0.assets["pr"].to_dict()["xarray:open_kwargs"]["storage_options"]["credential"] # this is where the SAS token is stored
    cred = unquote(cred)
    print(cred.split('&')[1]) # this is the expiration date portion of the token
    time.sleep(10)

produces

se=2024-11-05T19:51:48Z
se=2024-11-05T19:51:48Z
se=2024-11-05T19:51:48Z
se=2024-11-05T19:51:48Z
....
se=2024-11-05T19:51:48Z
se=2024-11-05T19:51:48Z
se=2024-11-05T19:51:48Z
se=2024-11-05T19:51:48Z

[ghidalgo3]

The planetary_computer python module caches tokens internally. You would receive a new token closer to its expiration.

Alternatively if you can run your code in the Azure West Europe region, we will issue longer-lived tokens and by collocating your compute with the data region you will get faster data reads.