feat(sink): Exploiting BigQuery CDC feature

[docteurklein]

Hey there :)

Just discovered that bigquery supports CDC tables in GA since Oct. 2023 (https://cloud.google.com/bigquery/docs/release-notes#October_31_2023)

I'm trying to avoid google dataflow/datastream solutions, and risingwave is a serious candidate.

I just noticed that RW's bigquery sink becomes unnecessarily conservative on refusing Op different than Insert (https://github.com/risingwavelabs/risingwave/blob/main/src/connector/src/sink/big_query.rs#L328-L332),
if we append a _CHANGE_TYPE column to a CDC-enabled BQ table.

It would lift the append-only restriction, and simplify the whole stack! (see https://cloud.google.com/bigquery/docs/change-data-capture#upsert-delete-example)

Technically, this would still be an append-only sink (i.e, we only INSERT new rows, but those rows represent upstream UPDATES or DELETES).

EDIT: this would require to change the write API from tabledata.insertAll to using the storage write API (https://gkkachi.github.io/gapi-grpc-rs/gapi_grpc/google/cloud/bigquery/storage/v1beta2/big_query_write_client/struct.BigQueryWriteClient.html#method.append_rows)

[github-actions]

This issue has been open for 60 days with no activity. Could you please update the status? Feel free to continue discussion or close as not planned.

[neverchanje]

Hi, @docteurklein Please be aware that StorageWrite API is supported now. We'll include it in the upcoming 1.9 version.