Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[JdbcIO] Allow fetchSize to be set for partitioned reads #28999

Merged
merged 1 commit into from
Oct 16, 2023

Conversation

bvolpato
Copy link
Contributor

The default for fetchSize is 50000 (https://github.com/apache/beam/blob/master/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java#L372), which is not even supported by some databases.

For example, DB2 limits to 32767 (source: https://knowledge.informatica.com/s/article/000205468?language=en_US).

This change will not only allow performance tuning when needed, but will enable usage of partitions on specific databases.

@github-actions
Copy link
Contributor

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @Abacn for label java.
R: @johnjcasey for label io.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@Abacn
Copy link
Contributor

Abacn commented Oct 16, 2023

LGTM. thanks for the change. A followup could be expose this in xlang wrapper also

@Abacn Abacn merged commit 6a57d0d into apache:master Oct 16, 2023
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants