Inquiry: Reasons for choosing XPath over CSS selectors

[DenDen047]

I’ve been studying your paper “AutoCrawler: A Progressive Understanding Web Agent for Web Crawler Generation” and noticed that you’ve chosen to use XPath for element selection rather than CSS selectors. As both methods are commonly used in web scraping, I’m curious about the reasoning behind this decision.

Could you please elaborate on why XPath was preferred for AutoCrawler/AutoScraper? Specifically, I’m interested in understanding:

1. Were there specific advantages of XPath that made it more suitable for your progressive understanding approach?
2. Did you encounter any limitations with CSS selectors that XPath addressed?
3. How does the choice of XPath align with AutoCrawler’s goal of generating web crawlers through progressive understanding?

Your insights would be valuable for those of us working on similar projects and trying to make informed decisions about selector methods in web scraping applications.
Thank you for your time and for sharing your research with the community.