-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[pipeline-connector][paimon] add paimon pipeline data sink connector. #2916
Conversation
f8d62e5
to
40517a7
Compare
40517a7
to
767b1cc
Compare
1c9357e
to
bb75217
Compare
@PatrickRen PTAL. |
bb75217
to
0eff9c6
Compare
4208998
to
2e7fe85
Compare
Thanks @yanghuaiGit for pointing out this, address it. |
address it. |
Support reading data from multiple table messages written to the same topic? |
You can do this by using |
paimon latest version is 0.7,we should update paimon version from 0.6 to 0.7 |
@lvyanquan Could you take a look at this one? I prefer to catch up with the latest version as well. Also could you rebase the latest master? Thanks |
dbbdbc3
to
6eaadc2
Compare
Done and rebased to master. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for @lvyanquan's great work! I left some comments & doubts here, please kindly check.
...nector-paimon/src/main/java/org/apache/flink/cdc/connectors/paimon/sink/v2/PaimonWriter.java
Outdated
Show resolved
Hide resolved
...-paimon/src/main/java/org/apache/flink/cdc/connectors/paimon/sink/PaimonDataSinkFactory.java
Show resolved
Hide resolved
...-paimon/src/main/java/org/apache/flink/cdc/connectors/paimon/sink/PaimonDataSinkFactory.java
Outdated
Show resolved
Hide resolved
flink-cdc-common/src/main/java/org/apache/flink/cdc/common/types/utils/DataTypeUtils.java
Outdated
Show resolved
Hide resolved
...-paimon/src/main/java/org/apache/flink/cdc/connectors/paimon/sink/PaimonDataSinkOptions.java
Outdated
Show resolved
Hide resolved
Does kafka header set constant values? For example, if data from multiple data centers is written to the same kafka topic, add a region key to the kafka header. |
a635bd6
to
907f2cd
Compare
Thanks @yuxiqian for those comments, I've addressed it and resubmitted. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for @lvyanquan's kindly clarification!
910b495
to
776dc57
Compare
…geEvent after receiving SchemaChangeEvent.
776dc57
to
e1e054a
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@lvyanquan Thanks for the PR! LGTM.
This close #2856.
Some codes are inspired by FlinkCdcMultiTableSink in Paimon repo, and add a sinkV2 implement.