Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change target-bigquery default variant to one of the SDK based ones #1147

Open
DouweM opened this issue Feb 2, 2023 · 2 comments
Open

Change target-bigquery default variant to one of the SDK based ones #1147

DouweM opened this issue Feb 2, 2023 · 2 comments
Labels
kind/Feature New feature or request valuestream/Hub

Comments

@DouweM
Copy link
Contributor

DouweM commented Feb 2, 2023

I think we should make https://hub.meltano.com/loaders/target-bigquery--youcruit/ (showcased in https://meltano.com/blog/6x-more-speed-for-your-data-pipelines-with-batch-messages/) or https://hub.meltano.com/loaders/target-bigquery--z3z1ma/ (By Alex @ Harness) the default target-bigquery variant as it’s built with the SDK and supports BATCH.

@tayloramurphy tayloramurphy added kind/Feature New feature or request valuestream/Hub labels Feb 2, 2023
@pnadolny13
Copy link
Contributor

This is a good idea especially because we've seen a fair amount of users hit hard failures in the adswerve variant due to data types and it doesnt look like theyre prioritizing supporting a fail safe type adswerve/target-bigquery#35.

I'm not sure how we decide which variant becomes default out of the 2 SDK based versions. It seems like Alex has tried to build a more feature rich/optimized version so from a surface level exploration I'd probably vote for that one but clearly the youcruit version is good as well based on the blog post. I wonder if we should ask them directly if either want to consolidate effort on one of the variants, or volunteer to not be default. We could also ask the community to weigh in on this if they have experience with either.

@tayloramurphy
Copy link
Collaborator

I'm generally supportive - hard to say if there are any critical missing features from an SDK-based one. But if folks are feeling confident we can make the switch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/Feature New feature or request valuestream/Hub
Projects
None yet
Development

No branches or pull requests

3 participants