workflow for Erin's refdb builder

[btupper]

Hi All,

I started to convert the recent @egreyavis script to a more workflow friendly form. I forked this repos and will eventually make a pull request after I am done messing around. The workflow is in a subfolder in the fork here. I'm hoping that we can post questions/answers here so we have a more reliable place to track communications than my inbox (trust me, you know it will end up being eaten by the dog or going through the laundry there.)

The workflow moves most of the user defined values into a YAML, includes a set up script the will install and or load packages, and moves reusable code into a suite of functions. I haven't made it to the steps where queries are made, but I hope to get to that today.

Cheers,
Ben

[btupper]

Hi,

I have pulled a section of the script that tries to capture information where order is missing. It can be found here I think I have it close to the desired behavior, but it is a complex decision making step and it's hard to know for sure.

I do encounter this error from NCBI "HTTP failure: 502, bad gateway. This error code is often returned when trying to download many records in a single request. Try using web history as described in the rentrez tutorial". So I have added use_history = TRUE to each call to rentrez::entrez_search() It raises the question about the maximum number of returns when searching for targets rather than mitogenomes; former is set to 999999 which seems like a lot compared to 9999 for the latter. What is the motivation for setting the return max so high?

[egreyavis]

Thanks Ben. It would be fine to reduce the number of returns - there is no good reason why it was set so high in the first place. I've got a busy Friday and weekend, but hope to take a look at this early next week.

…

On Thu, Feb 22, 2024, 3:49 PM Ben Tupper ***@***.***> wrote: Hi, I have pulled a section of the script that tries to capture information where order is missing. It can be found here <https://github.com/BigelowLab/mednaTaxaRef/blob/main/egrey/functions/ncbi.R> I think I have it close to the desired behavior, but it is a complex decision making step and it's hard to know for sure. I do encounter this error from NCBI "HTTP failure: 502, bad gateway. This error code is often returned when trying to download many records in a single request. Try using web history as described in the rentrez tutorial". So I have added use_history = TRUE to each call to rentrez::entrez_search() It raises the question about the maximum number of returns when searching for targets rather than mitogenomes; former is set to 999999 which seems like a lot compared to 9999 for the latter. What is the motivation for setting the return max so high? — Reply to this email directly, view it on GitHub <#11 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI3FEDOBKLDJBOBORFG34ADYU6VNVAVCNFSM6AAAAABDTBXUUKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRQGI4TANBQGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>