User lookup #30
Open
User lookup #30
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zijwang
reviewed
Mar 28, 2022
| @@ -265,3 +353,54 @@ def process_twitter(self, data): | |||
| "output": pred[id] | |||
| } | |||
| return output | |||
|
|
|||
|
|
|||
| def process_twitter_batch(self, data_list, batch_size, num_workers): | |||
There was a problem hiding this comment.
This function seems largely overlaps with the non-batched version. Could you call the non-batched version and then aggregate the result?
zijwang
reviewed
Mar 28, 2022
| for id in id_list: | ||
| # If a json file exists, we'll use that. Otherwise go get the data. | ||
| try: | ||
| with open("{}/{}.json".format(self.cache_dir, id), "r") as fh: |
zijwang
reviewed
Mar 28, 2022
| if len(id_list) > 0: | ||
| # the twitter API handles a maximum of 100 user IDs per request. Chunk up the user | ||
| # id list into batches of max 100 IDs and run them through the pipeline sequentially | ||
| API_batch_size = 100 |
There was a problem hiding this comment.
This could be defined as a global variable for easy tracking?
zijwang
reviewed
Mar 28, 2022
| @@ -193,6 +193,34 @@ def _twitter_api(self,id=None,screen_name=None): | |||
| return self.process_twitter(r.json()) | |||
|
|
|||
|
|
|||
| def _twitter_api_lookup(self, ids=None, batch_size=16, num_workers=4): | |||
There was a problem hiding this comment.
We probably could reuse/extend the twitter_api function by extending id to ids?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added a new function "infer_ids" and corresponding helper functions that uses the GET /users/loopup endpoint to request information for up to 100 user IDs at the same time.