Improve BQ <-> Avro conversions #32482
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
RustedBones
force-pushed
the
bq-avro-schema
branch
2 times, most recently
from
c2055eb to
55017e6
Compare
|
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
RustedBones
force-pushed
the
bq-avro-schema
branch
from
55017e6 to
2312dbe
Compare
|
Assigning reviewers. If you would like to opt out of this review, comment R: @robertwb for label java. Available commands:
The PR bot will only process comments in the main thread (not review comments). |
Abacn
reviewed
Sep 19, 2024
There was a problem hiding this comment.
Thanks for this effort, went through once and had comments. For BQIO changes, @ahmedabu98 could you please also take a look? Thanks!
...-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtilsTest.java
Show resolved
Hide resolved
...ogle-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java
Show resolved
Hide resolved
...ogle-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java
Show resolved
Hide resolved
...ogle-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java
Show resolved
Hide resolved
...ud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageSourceBase.java
Outdated
Show resolved
Hide resolved
...o/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java
Outdated
Show resolved
Hide resolved
...a/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
Outdated
Show resolved
Hide resolved
...ogle-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySourceDef.java
Outdated
Show resolved
Hide resolved
...ogle-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java
Show resolved
Hide resolved
...ud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOStorageReadTest.java
Outdated
Show resolved
Hide resolved
RustedBones
force-pushed
the
bq-avro-schema
branch
2 times, most recently
from
e22368a to
a4ad8d3
Compare
RustedBones
force-pushed
the
bq-avro-schema
branch
from
a4ad8d3 to
e1f6d5a
Compare
ahmedabu98
reviewed
Sep 20, 2024
...ogle-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java
Outdated
Show resolved
Hide resolved
...ogle-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java
Outdated
Show resolved
Hide resolved
...ogle-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java
Show resolved
Hide resolved
...ogle-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java
Outdated
Show resolved
Hide resolved
...ogle-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java
Outdated
Show resolved
Hide resolved
RustedBones
commented
Sep 23, 2024
Comment on lines
+104
to
+106
| // in Extract Jobs, it always uses the Avro logical type | ||
| // we may have to change this if we move to EXPORT DATA | ||
| return LogicalTypes.timestampMicros().addToSchema(SchemaBuilder.builder().longType()); |
There was a problem hiding this comment.
useAvroLogicalTypes has a different behavior when used in extract job vs export data for TIMESTAMP columns.
Currently, beam's implementation of query is relying on temp table + extract job, so we don't have to handle this discrepancy.
|
Failing test is unrelated, merging now |
17 tasks
reeba212
pushed a commit
to reeba212/beam
that referenced
this pull request
Dec 4, 2024
* Improve BQ avro handling * Include minimal changeset * Apply review comments * Streamline avro type conversion with beam schema
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Extracted from #32360
This change-set focuses on conversions between BQ
TableSchema/TableRowand AvroSchema/GenericRecordFix #20677