Records including datetime to be saved as DATETIME or TIMESTAMP in BigQuery

[kennknowles]

When trying to write some row in BigQuery that include a python datetime object, the marshaling used to save a row in BigQuery is impossible.

File "/home/pascal/Wks/GitHub/PEAT-AI/Albatros/venv/local/lib/python2.7/site-packages/apache_beam/internal/gcp/json_value.py",
line 124, in to_json_value
    raise TypeError('Cannot convert %s to a JSON value.' % repr(obj))
TypeError:
Cannot convert datetime.datetime(2018, 9, 25, 18, 57, 18, 108579) to a JSON value. [while running 'save/WriteToBigQuery']

However, this is something perfectly feasible, as google-cloud-python supports it since this issue has been solved: googleapis/google-cloud-python#2957

thanks to this pull request: https://github.com/GoogleCloudPlatform/google-cloud-python/pull/3426/files

As similar approach could be taken for the json_value.py helper.

Is there any workaround that can be applied to solve this issue?

 

Imported from Jira BEAM-5510. Original Jira may contain additional context.
Reported by: GeoloeG_IsT.

[lazarillo]

It seemed to me that this was more of a Google issue than an Apache Beam issue.

I created an issue here that summarizes the problem, and my humble opinion as to a means to resolve it that is hopefully flexible.

https://issuetracker.google.com/issues/318478318

[lazarillo]

If some core beam devs agree, then maybe we can close this issue. 😄

[kennknowles]

Sounds good to me! It is very old and a fresh report would be probably needed anyhow.