WIP: Add Scraper/ScraperRun classes #7
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
First stab at factoring out a Scraper class and a ScraperRun class. The IndexToMemebers class is probably overkill for now — we can find the higher level abstractions like that after we nail down the basics — but it's probably useful to include in this intial WIP which is more for feedback than expected to actually merge yet.
tmtmtmtm
force-pushed
the
scraper-class
branch
from
407b363 to
598e59b
Compare
|
Part of everypolitician/everypolitician#572. |
|
The |
This gives us the full stack trace and ensures that the program actually exits after an error.
This was trying to call `#keys` on an array, which was raising an error. For now just use the first item from the data array.
Not all records seem to have a start date, so removing this from the list of default index fields for now.
This doesn't need to be a public method as it's only used internally.
Rather than trying to find the correct generic abstraction this forces the scrapers to have a class that encapsulates navigating a website to get the correct content. In the future we can potentially wrap these up in `IndexToMembers` type classes, but for now it seems easier to just do scraper specific ones until we find the correct abstraction.
|
Thinking about this a little more, I fear that we got a key underlying concept back to front here. Here we say that a I now think this should all be the other way around: a |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
First stab at factoring out a Scraper class and a ScraperRun class.
The IndexToMemebers class is probably overkill for now — we can find the
higher level abstractions like that after we nail down the basics — but
it's probably useful to include in this intial WIP which is more for
feedback than expected to actually merge yet.