Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sink to BigQuery with google pubsub sink #15306

Closed
docteurklein opened this issue Feb 27, 2024 · 1 comment · Fixed by #16363
Closed

Sink to BigQuery with google pubsub sink #15306

docteurklein opened this issue Feb 27, 2024 · 1 comment · Fixed by #16363

Comments

@docteurklein
Copy link

Is your feature request related to a problem? Please describe.

Given the level of integration with pubsub between the different GCP components (cloud run, eventarc, dataflow, bigquery, workflows, ...), having a pubsub sink would be highly beneficial.
I'm currently missing this to take advantage of the pubsub bigquery CDC feature as documented here: https://cloud.google.com/pubsub/docs/bigquery.

Describe the solution you'd like

I'd like a sink wose definition would look like:

CREATE SINK sink1 FROM mv1
WITH (
   connector='google_pubsub',
   topic='topic1'
)
FORMAT PLAIN ENCODE JSON; -- possibly AVRO  [1]

[1] https://cloud.google.com/pubsub/docs/samples/pubsub-publish-avro-records

Describe alternatives you've considered

I tried to use the native RW bigquery sink, but it's using the legacy tabledata.insertAll method, which is not compatible with the CDC message ingestion.

Additional context

opening the issue here as discussed in https://github.com/orgs/risingwavelabs/discussions/15237

@github-actions github-actions bot added this to the release-1.7 milestone Feb 27, 2024
@fuyufjh fuyufjh changed the title google pubsub sink Sink to BigQuery with google pubsub sink Feb 28, 2024
@neverchanje
Copy link
Contributor

#16363 will resolve this issue.

@neverchanje neverchanje linked a pull request Apr 23, 2024 that will close this issue
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants