-
Notifications
You must be signed in to change notification settings - Fork 71
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
bug: target table is being reset after a successful sync, then completes without logging an error #2117
Comments
Hi @gkarugi! Thanks for reporting. One thing that'd be extremely useful is to see which queries are being executed. The following setup should help you get SQLAlchemy logs from target-clickhouse:
version: 1
disable_existing_loggers: false
formatters:
sqlalchemy:
format: "{asctime} {message}"
style: "{"
handlers:
sqlalchemy:
class: logging.FileHandler
formatter: sqlalchemy
filename: sqlalchemy.log
loggers:
sqlalchemy.engine:
level: DEBUG
handlers: [ sqlalchemy ]
propagate: false
|
(See also shaped-ai/target-clickhouse#89) |
Hi @edgarrmondragon . Thanks for the sql alchemy log idea. Didn't know this existed. The issue might be coming from the following line. It appears twice. the second time is when I think it truncates.
The column SQLAlchemy logs below
|
@gkarugi Gotcha, thanks for sharing the debug log! That confirms the culprit in https://github.com/shaped-ai/target-clickhouse/blob/f70e4a7f92162922dae0d7936639c794a837d41e/target_clickhouse/sinks.py#L114-L122. The target copied the reference implementation in the SDK: Lines 381 to 389 in 40b6333
In the meantime you can try passing And fixing this probably involves:
PS I've transferred the issue to meltano/sdk |
Hello @edgarrmondragon Thanks for your help with this. I am just starting out with Meltano, so didn't have an idea where I should start debugging.
|
@gkarugi I see, yeah all records seem to have a Some debugging steps that might be useful (might try them myself at a later time if this is isn't fixed:
|
I debugged as per your suggestion and the first option(Run You can find the full log here
|
Hi @edgarrmondragon. Thanks for your assistance so far. Is there any way I can assist with progressing this? |
@gkarugi The Lines 375 to 379 in c8e2461
is creating a column with a 0 value as default. |
Meltano Version
3.1.0
Python Version
3.10
Bug scope
Other
Operating System
Linux Ubuntu 22
Description
sync reset's table after completing successfully and it seems to want to start afresh but it completes without any error.
It initially synced over 500k records before resetting and adding some 9k records before completing without logging any error.
I suspect this is happening on the near end of the log attached where it says " Cleaning up db_schema-locations"
Code
The text was updated successfully, but these errors were encountered: