Various issues after upgrading to 0.14.0, probably related to end_date

[guidocioni]

Hi,
I was a happy user of wetterdienst until yesterday when I decided to update to 0.14.0. Since then code that was working well before now gives me many different issues appearing randomly. here are some examples

Daily data
Trying this code on station ids 1975, 1691, both stations that tested positive for this set of time periods/parameters with the method DWDObservationStations, it fails on 1975 (with UnpicklingError: invalid load key, ' '.) but not on 1691.

observations = DWDObservationValues(
    station_id=[1691],
    parameter=[DWDObservationParameterSet.CLIMATE_SUMMARY],
    resolution=DWDObservationResolution.DAILY,
    start_date="1937-01-01",
    end_date="2021-02-17",
    tidy_data=False,
    humanize_parameters=True,
)

Moreover, even though the method DWDObservationStations reports that the TO_DATE is 2021-02-18, I have to put 2021-02-17 otherwise I get an empty dataframe as response. I believe if end_date > TO_DATE the method should just take end_date == TO_DATE. I'm pretty sure this was the behaviour in the previous version.

Hourly Data
Here I have a similar problem.
First of all, the TO_DATE returned by DWDObservationStations is 2021-02-18, although if you download the data from the server you can see that the last timestamp is actually 2021-02-18 23:00, so I should be able to get the data up to this date.

Unfortunately, I can only get the data if I use end_date='2021-02-17', otherwise with end_date='2021-02-18' I get an empty dataframe. The difference from daily data is that here it doesn't fail with an error but returns an empty dataframe. Which means that at the end I only get the data up to 2021-02-17 00:00, almost 2 days before the last timestamp on server.

10 Minutes data
Here I get a completely different behaviour. If end_date > TO_DATE the response correctly contains the data up to TO_DATE and the rest is filled with NaN (as expected) BUT testing it on station_id 1975 (which should have the data according to the response of DWDObservationStations with resolution=Resolution.MINUTE_10) returns an error error: cannot add item to database, while it works on station_id 1981, which should have exactly the same data...

Again, I don't know if I'm not using the API in the correct way, but I'm pretty sure I didn't have all these problems with the previous version. I'm thinking whether specifying the end_date is the problem, but unfortunately omitting this parameter is not allowed...would be good to put end_date = None and obtaining the data up to the most recent timestamp.

[amotl]

Hi Guido,

thank you very much for your kind words and for giving us so detailed feedback about the issues you have been observing, this is really important to us. At the same time I want to deeply apologize for the hassle you are experiencing after upgrading to version 0.14.0.

Wetterdienst is still very much in flux as we are aiming at supporting more data sources beyond DWD. @gutzbenj does a tremendous job on that and I want to salute him for this.

I believe he will like your feedback very much and will take it into consideration when working on a followup release. In the meanwhile, can I humbly ask you to downgrade to version 0.13.0 again?

With kind regards,
Andreas.

[amotl]

Regarding errors like UnpicklingError: invalid load key, ' ', well, this is also sometimes a pain for us. We will have to get rid of the dbmfile-based cache which proved to be very brittle, see also #243 ff. On that matter, may I humbly ask you to clear your cache folder before trying again? It might be located at:

• ~/Library/Caches/wetterdienst (macOS)
• ~/.local/share/wetterdienst (Linux)
• C:\Users\{username}\AppData\Local\Acme\wetterdienst\Cache (Windows)

-- https://pypi.org/project/appdirs/

[gutzbenj]

Dear @guidocioni ,

I originally decided that if either one of start_date and end_date is left blank, it would be set equal to the other value - for the courtesy of querying only a value for one timestamp. I would leave it this way as for other requests one simply could define the date with datetime.datetime.utcnow() or similar ones to get data up to the latest point.

Otherwise, related to your issue with the empty dataframe I fixed this with the latest patch at #357 . Other problems should be related to the cache so removing that should do well to your seen issues. If other problems occur, please come back to us.

If you are still a happy user, we may ask you to leave us a star and/or report your success story at #220 .

[amotl]

Hi Benjamin,

thanks for fixing this with #357. Can you issue a 0.14.1 bugfix release?

With kind regards,
Andreas.

[gutzbenj]

Already included ;)

[guidocioni]

Hey guys, sorry for the late reply. I just quickly tried to clear the cache and implement #357 and it seems that all issues are resolved now ;) Thanks for the help and keep up the good work!

Don't hesitate to write me if you decide in the future to implement also forecast data from DWD in your API, I have some experience with that and could be a really interesting application :)

[gutzbenj]

Dear @guidocioni ,

I just drafted a new release so the fix should be available on PyPi now.