BOM restricting direct access to data files

[jonocarroll]

To begin with:

• Can anyone get in touch with BOM directly to see if
  a) they are aware that bomrang is blocked by the current changes? It may be a case of someone else's behaviour, not bomrang directly which has precipitated (ba dum tiss) this new policy enforcement. On the other hand, maybe it's bomrang specifically. Who knows?
  b) there is a way to support bomrang's current behaviour, either via a specific HTTPUserAgent or other query method
  c) there is a mandate to support publically funded data to be publically available
• Should the functions which rely on the now blocked access be temporarily removed? Can the authors of the relevant functions prioritize this?
• Should some effort be made towards leveraging the available FTP resources?

[adamhsparks]

I’ve had some conversations with other researchers writing packages that access BOM data. This is not aimed just at bomrang. We’re working on a way forward with this but it’s likely to be slow.

In the meantime, I just can’t. I’m done jumping through CRAN’s ever-moving hoops. I’m electing to just leave everything as-is and let it be archived on CRAN. I’ve too many other things to worry about without fussing about something that isn’t actually broken in our package.

CRAN’s interest or disinterest in FTP recently also has me rethinking the viability bomrang if BOM won’t make a proper API and won’t allow HTTP access. CRAN is requesting any examples using FTP be removed. While this is just examples right now, I can see this easily becoming any FTP functionality in the future.

[jonocarroll]

I entirely sympathise, and from what I hear you're not alone in this view of CRAN.

I'll be happy to see if I can build something against the FTP resources, but it will take some time. If CRAN isn't allowing FTP examples but demands examples and FTP is the only way bomrang can access the data, then it sounds like the only option would be having CRAN archive the package and for bomrang to be hosted on Github / r-universe.

I hope the discussions lead to a reasonable outcome - the BOM data should be available more easily.

[VidarParry]

Can I just jump in to this discussion. I'm an amateur programmer in Brisbane. I am looking for code that retrieves radar images from BOM, which BOMRang seems to do very nicely. But I get an error when I try and connect to the FTP BOM site (same as your example executions on GitHub). I tried a bit of ftp code using PHP and then cURL to try and connect and BOM blocks them as well. I'm guessing the FTP block is a general one by BOM that you are referring to above.

However utilities like WinFTP/WinSCP seem to work fine.

So my question is whether this 'block' is an intentional block by BOM (even though they say it is open for anonymous FTP use), and if so, do you know how the WinSCP etc apps get through (and can we exploit this as a workaround)?

Thanks folks!

Cheers
...Steve

[adamhsparks]

I've not even bothered checking lately TBH. If BOM is now blocking even anonymous FTP access, I can't see any good reason for keeping this package alive. I think it's high time to archive it. BOM has shown that they are very disinterested in supporting the scientific community with these sorts of efforts so keeping this alive seems pointless. I've not had any fruitful engagement in spite of my (and others) trying.

[VidarParry]

Understand the frustration.

[jonocarroll]

I was browsing the BOM site recently and was able to manually grab some all-years historical data for temperature, rainfall, etc... and it looks like the filename patterns are still amenable to some programming, but the terms of use still state that this is not allowed, regardless of whether or not we are able.

I'm tempted to extract what processing we have of these files into a smaller, more focussed package for unpacking and rectangling the (manually downloaded) data, but I don't see any point in maintaining code that tries to fetch the data.

[VidarParry]

Thanks both. I might just try BOM as a novice and see what I get. Don't hold out much hope though. Still puzzled why WinFTP works but FTP from PHP/cURL can't even open the connection. Will let you know if I get any success.
Cheers...Steve