Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The connector using Debezium and Kafka does not create tables and does not report any errors. #930

Open
DmitryTuryshev opened this issue Nov 22, 2024 · 0 comments
Milestone

Comments

@DmitryTuryshev
Copy link

DmitryTuryshev commented Nov 22, 2024

I'm trying to write messages from Kafka to ClickHouse. The messages are being received via Debezium, and I am using the image altinity/clickhouse-sink-connector:2.4.0-kafka.

Problem: The connector shows zero lag, and offsets exist, but no logs are available in the container, and data is not being written to ClickHouse.

Kafka Offsets
The offsets exist, but no processing is happening.
image

Configuration File
Here is my configuration for the connector:
{
"name": "clickhouse-sink-connector-10",
"config": {
"connector.class": "com.altinity.clickhouse.sink.connector.ClickHouseSinkConnector",
"tasks.max": "1",
"topics": "pg-source-1.public.t1",
"topics.regex" : "pg-source-1.*",
"clickhouse.server.url": "http://clickhouse",
"clickhouse.server.user": "default",
"clickhouse.server.password": "",
"clickhouse.topic2table.map": "pg-source-1.public.t1:default.t1",
"metrics.enable": "false",
"auto.create.tables": "true",
"auto.create.tables.replicated": "false",
"schema.evolution": "true",
"snowflake.id": "false",
"replacingmergetree.delete.column": "id",
"enable.kafka.offset": "true",
"errors.log.enable": "true",
"errors.log.include.messages": "true",
"key.converter": "org.apache.kafka.connect.json.JsonConverter",
"key.converter.schemas.enable": "true",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter.schemas.enable": "true"
}
}

Observations
The messages from Debezium are being received successfully:
image

Troubleshooting Steps Already Tried

  1. Verified that messages are present in Kafka topics.
  2. Checked for container logs, but no output is present.
  3. Confirmed that offsets and lags are being updated correctly.

Let me know if additional details are needed, or if you have any suggestions!

@subkanthi subkanthi added this to the 2.5.0 milestone Nov 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants