How does it work and parse etc, costs?

[jay377]

New to this as far as i can tell this is using the LLM to parse the questions right, does that mean the LLM is getting passed the full page of raw page code and parsing it and could this be very expensive per page as its a lot of tokens (unless you have fast good local llm)? What tokens or costs for a standard page are you seeing, what llms are you using?

[VinciGit00]

look smart_scraper_openai.py inside the examples folder, it calculate the cost