Keep running into _ is not a submission, and it won't download more than half of my saved posts #877
-
|
As the title describes. I keep running into this message when verbose is enabled, which seems like it might be the issue: I submitted a data request, and I have the links to all of my saved posts. Would it be possible to just use that instead to specify what I would like to download? It'd be easier to modify that list to stop and start places, as well as get my above 1k saves downloaded. |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 5 replies
-
|
We have a maximum that cannot be bypassed. That's Reddit's limit, not ours. You're using the download function of the BDFR. That applies only to the contents of a post, that is either the text or the image or the link. Comments are none of those and are ignored. If you want to get both or just comments, then the cloning or archive functions actually do that. They scrape the actual metadata of the comments and submissions. |
Beta Was this translation helpful? Give feedback.
-
|
What is the limit? It isn't going beyond half of the 1000 post limit, and I'm wondering where it stops. BDFR says it stops at 1000 as that is reddit's limit, I think? |
Beta Was this translation helpful? Give feedback.
-
|
It depends. Generally it's a thousand but Reddit isn't very clear on how this works and, right now, the API is somewhat in flux. They may be reducing it, and rate limiting is being put into effect. Not every request we make returns 1000 posts but that is the hard limit. I know from experience when writing and testing the code that sometimes the 'maximum' is like 900 posts. Why I couldn't be sure. The BDFR also removes duplicate post IDs being downloaded so cross-linked submissions are automatically removed from the download process. |
Beta Was this translation helpful? Give feedback.
-
|
Oh okay, thank you. Do you know of a way to download posts/comments based on links in a csv? That would be better suited for my use case then. Otherwise I'd have to find a way to download all the other posts that it is not getting with this. |
Beta Was this translation helpful? Give feedback.
-
|
You can use --include-id-file to pull the links from a text file with one link per line, sadly this only works for full links if you want the comments too, as stated in #839. There are plenty of methods to extract the column with the links and create the .txt file, a csv file is basically a text file with commas. |
Beta Was this translation helpful? Give feedback.
-
|
That is exactly what I am looking for, thank you. This is perfect. I didn't know that's what that did, haha. Downloaded my user data and just removed the link column (and the headers) and saved it as a txt, then input it as an argument for --include-id-file, and it worked. Thank you very much for your help! |
Beta Was this translation helpful? Give feedback.
-
|
You're welcome! |
Beta Was this translation helpful? Give feedback.
You can use --include-id-file to pull the links from a text file with one link per line, sadly this only works for full links if you want the comments too, as stated in #839. There are plenty of methods to extract the column with the links and create the .txt file, a csv file is basically a text file with commas.