Error with krigR() Pipeline usage when dowloading GMTED2010

[jimoreira]

hi,
had been watching and trying your workshop, great material!

When i use the 3 steps everything works fine, but when i try to use krigR() i found troubles.
I have tried,
add cores, change Source to "Drive", change nmax to lesser value.
Still the process of download fails with different percentage
Downloading GMTED2010 covariate data. |======================= | 34%Error in curl::curl_fetch_disk(url, x$path, handle = handle) : transfer closed with 182325606 bytes remaining to read

I'm using a small Extent so shouldn't be that, also this error shows rapidly, some 2-3 minutes after execution.

my code
Pipe_Krig <- krigR( Variable = "evaporation_from_vegetation_transpiration", Type = "reanalysis", DataSet = "era5-land", DateStart = "2019-04-15", DateStop = "2019-04-20", TResolution = "day", TStep = 1, Extent = Extent, API_User = API_User, API_Key = API_Key, Target_res = .08, Cores = 4, FileName = "papa_evapo_Pipe.nc", Dir = Dir.papa, Keep_Temporary = FALSE, nmax=12, Source = "Drive" )

perhaps you con find something to help me,
thanks!!

Javier

[ErikKusch]

Hi Javier,
could I ask you to provide the Extent object (just the extent numbers will be fine, too) so I can troubleshoot this issue on my end? I have run your code with Extent <- extent(c(7.5,8.5, 61, 62)) for my trouble shooting and could not reproduce the error you are experiencing.

That being said, you made me aware of an oversight. Source is currently not supported by the pipeline function call. It was not being handed to the covariate download call. I have fixed this with this merge. You can simply reinstall KrigR to obtain this fix.

A few things I have noticed in your code that I think I should clarify:

• Cores controls how many parallel downloads of ERA5(-Land) data to make at any given time. Download chunks are split by months. Since the data you query doesn't cover more than one calendar month, changing Cores won't to anything
• nmax controls the kriging localisation and does not come into play when covariates are prepared so changing it also won't affect your experience here

Lastly, just as a personal note, I somewhat doubt that interpolation of evaporation from vegetation transpiration will be aided greatly by considering elevation for interpolation. You might want to use other covariates and follow the three-step process with KrigR to do so.

[jimoreira]

HI Erik,
thanks for your help.

I have updated the package and tried it again. You have to make sure to delete the incomplete created folder (GMTED2010) otherwise the process fail again.
But once i did it, it shows the next error

proj_as_proj_string: C:\Program Aggregating to temporal resolution of choice
  |=========================================================================| 100%

Downloading GMTED2010 covariate data.
  |=========================================================================| 100%
Error in .local(x, y, ...) : extents do not overlap

as you request, my extent object is:

Extent <- extent(c(-56.6, -56.4,-34.8, -34.7)) 

And, you are right about the selected variables, i was just trying to make it work to then worry about real life. But thanks for the comment.

thanks!

[ErikKusch]

Hey Javier,

I can reproduce this issue - thanks for providing your extent specification.

I am doing some troubleshooting, but CDS is a bit slow today in providing the actual downloads. I'll get back to you. Just wanted to let you know that I am working on it. I have tracked down the problem to CDS NETCDF, despite being queried with the correct extent, once downloaded reports the x-vlaues as 360-x-extent, respectively. Thus, in the download, -56.6 turns into 303.4. I have no idea why that is happening and I have never encountered this before.

Again, sorry for not having more of a solution at this point, but rest assured that I'm on it and will come back to you as soon as I find a solution. Have you gotten the downloads for your chosen extent to work with other variables?

EDIT: For clarification, I could not reproduce this error with the extent I used above so I highly suspect this to be an Extent-specific issue. Still troubleshooting.

Cheers,
Erik

[ErikKusch]

Hej Javier,

It took quite a bit of testing on my end, but I believe I have figured it out.

The problem you were experiencing is one that occurs only for select variables like "evaporation_from_vegetation_transpiration" hence why we didn't find it yet. Moreover, this issue does not stem from KrigR, but from ECMWF CDS. Even when going through their web portal and querying the same data you are after (see attached screenshots) the issue persists.

So what is the issue? For a select few variables, including "evaporation_from_vegetation_transpiration", CDS reports any westerly extent as 360-abs(queried x Extent). See below:

The data is correct. Only the extents are muddled - I conclude that from testing with areas that have recognisable coastlines and overlaying products where this issue is present with products where it isn't present.

This confuses the KrigR pipeline workflow as it expects the extent you provided be matched at each step.

Here's what you came for, though:

I fixed it

To do so, download_ERA() now compares queried extent to downloaded extent and assigns queried extent when the issue I described above is detected. I have made this a part of KrigR with merged pull request #14.

I have also submitted a support ticket to ECMWF CDS to make them aware and hopefully resolve this server-side. The way my solution is coded, a fix server-side will not break the new download_ERA() and you are good to go.

Simply reinstall and you are good to go. Thanks for bringing this up (it was actually a lot of fun figuring this one out and implementing a fix).

Cheers,
Erik

[ErikKusch]

Implement the fix found here also for the new CDownloadS()function