Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Datetime parser in HTTP driver transport does not support datetime with nanoseconds #339

Open
jordanauge opened this issue Oct 1, 2024 · 0 comments

Comments

@jordanauge
Copy link

Describe the bug
I have OpenTelemetry data stored in Clickhouse. Timestamps have nanosecond granularity.

To Reproduce
Any SQL query retrieving those timestamps.

Expected behavior
Timestamps should be properly parsed (eventually losing precision), but an exception is returned. The call to strptime fails when timestamps have a nanosecond component with a ValueError (unconverted data remains) :

https://github.com/xzkostyan/clickhouse-sqlalchemy/blob/master/clickhouse_sqlalchemy/drivers/http/transport.py#L29

This simple workaround where the argument to strptime is truncated to 26 chars max clears the issue:

def datetime_converter(x):
    if x == DATETIME_NULL:
        return None
    elif len(x) > 19:
        return datetime.strptime(x[:26], '%Y-%m-%d %H:%M:%S.%f')
    else:
        return datetime.strptime(x, '%Y-%m-%d %H:%M:%S')

Versions

  • Version of package with the problem : 0.3.2
  • Python version : 3.12.4

Sample run

File /opt/homebrew/lib/python3.12/site-packages/clickhouse_sqlalchemy/drivers/http/transport.py:36, in datetime_converter(x)
     32 #    elif len(x) > 26:
     33 #        x = x[:26]
     34 #        return datetime.strptime(x, '%Y-%m-%d %H:%M:%S.%f')
     35     elif len(x) > 19:
---> 36         return datetime.strptime(x, '%Y-%m-%d %H:%M:%S.%f')
     37     else:
     38         return datetime.strptime(x, '%Y-%m-%d %H:%M:%S')

File /opt/homebrew/Cellar/[email protected]/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/_strptime.py:554, in _strptime_datetime(cls, data_string, format)
    551 def _strptime_datetime(cls, data_string, format="%a %b %d %H:%M:%S %Y"):
    552     """Return a class cls instance based on the input string and the
    553     format string."""
--> 554     tt, fraction, gmtoff_fraction = _strptime(data_string, format)
    555     tzname, gmtoff = tt[-2:]
    556     args = tt[:6] + (fraction,)

File /opt/homebrew/Cellar/[email protected]/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/_strptime.py:336, in _strptime(data_string, format)
    333     raise ValueError("time data %r does not match format %r" %
    334                      (data_string, format))
    335 if len(data_string) != found.end():
--> 336     raise ValueError("unconverted data remains: %s" %
    337                       data_string[found.end():])
    339 iso_year = year = None
    340 month = day = 1

ValueError: unconverted data remains: 264
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant