[Feature Request] Expose some LoadJobConfig attributes for configuration/Allow truncate write mode

[GrgDev]

Right now, there is no way to customize how writes are done because of the exclusive use of the LoadConfigJob with default attributes.

target-bigquery/target_bigquery/__init__.py

Line 188 in 80b167c

load_config = LoadJobConfig()

Some of these attributes could be exposed for user configuration.

In my particular use case, I would like to change the write_disposition attribute to work in truncate mode instead of append mode. This way, when using Singer with this target to pipe data to BigQuery in a time partitioned table, if something goes wrong during one of the time periods, I can just rerun the job and overwrite the partitions in question instead of worrying about duplicate data.

https://googleapis.dev/python/bigquery/latest/generated/google.cloud.bigquery.job.LoadJobConfig.html#google.cloud.bigquery.job.LoadJobConfig.write_disposition

[daigotanaka]

Just a note to show where API doc is: https://googleapis.dev/python/bigquery/latest/generated/google.cloud.bigquery.job.LoadJobConfig.html

[daigotanaka]

Probably implement this by creating a config section called
LoadJobConfig and let user populate the key-value as in the API doc then let target-biguqery to pass it as kwargs.

[daigotanaka]

^ It seems possible https://github.com/googleapis/python-bigquery/blob/80944f080bcc4fda870a6daf1d884de616d39ae7/google/cloud/bigquery/job/base.py#L668

[daigotanaka]

@GrgDev I drafted in #5, but I have not tested this yet.

[daigotanaka]

I finally got a chance to test PR #5

@GrgDev LoadJobConfig params can be passed, but it would not work well with auto table creation. For example, if you
specify clustering_fields with some column name(s) and etc, the upload assumes the table to be created with compatible settings. To make use of LoadJobConfig, you need to make sure the default table created by target-bigquery-partition when the table does not exist is compatible to the LoadJobConfig params you pass on. Otherwise, you will need to prepare the compatible table before running target-bigquery-partition.

Which parameters are you trying to customize?
https://googleapis.dev/python/bigquery/latest/generated/google.cloud.bigquery.job.LoadJobConfig.html

[daigotanaka]

oops, you mentioned in the original post. write_disposition, correct?

[daigotanaka]

Personal Note:

• write_deposition
• WriteDeposition values