Page Loading Animation Trouble

[modem158]

Hi All,

I created a pull request for Harrogate UK today that is based off of the Harrogate Borough website. This seems to be working appropriately.

As Harrogate has been absorbed in to North Yorkshire I'd like to pull from the North Yorkshire website directly (since I'm afraid they will kill the Harrogate one) but the site has a "Loading" animation then loads the full page. Any thoughts on how I skip past the loading animation and wait for the full page before scraping the HTML? I've tried all I can think of but keep getting the loading page and not the final calendar page.

https://www.northyorks.gov.uk/bin-calendar/Harrogate/results/100050389710

[5ila5]

I would POST request the url (like the website does) and get the html from the response.

This should work:

import requests

URL = "https://www.northyorks.gov.uk/bin-calendar/Harrogate/results/{uprn}/ajax"

r = requests.post(URL.format(uprn=100050389710), params={"_wrapper_format":"drupal_ajax"})
r.raise_for_status()
html = r.json()[0]["data"]

[modem158]

Thank you. I'll work on this and create another pull request once I have it working.