Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SQLAlchemy dialect support more SQL datatypes #107

Open
1 of 8 tasks
dungdm93 opened this issue Sep 13, 2021 · 3 comments
Open
1 of 8 tasks

SQLAlchemy dialect support more SQL datatypes #107

dungdm93 opened this issue Sep 13, 2021 · 3 comments

Comments

@dungdm93
Copy link
Member

dungdm93 commented Sep 13, 2021

This is a meta issue that tracks the support of SQL datatypes after #81 merged.

@MichaelTiemannOSC
Copy link

MichaelTiemannOSC commented May 7, 2022

TIMESTAMP(6) definitely needs to be supported for pd.to_sql to work with datetype.datetime types (e.g. dtype='datetime64[ns]'). Test case:

# The variable `engine` is a SQLAlchemy engine connected to a Trino/Iceberg database with a `sandbox` schema available for table creation by the test user.
# The variable `connection ` is the engine's connection interface

import math
from datetime import datetime
drop_table = engine.execute(f"drop table if exists sandbox.test_timezone6")
drop_table.fetchall()
test_df = pd.DataFrame(
    {"A": [4.5], "B'C": [math.nan], None: [math.inf], "D": [-math.inf], "E": [datetime(2022, 1, 1)], ":F": [1.0]}
).convert_dtypes()
assert (test_df.dtypes == ['Float64', 'Int64', 'Float64', 'Float64', 'datetime64[ns]', 'Int64']).all()
xdf.to_sql("test_timezone6",
    con=connection,
    schema="sandbox",
    if_exists="append",
    index=False,
    method=osc.TrinoBatchInsert(batch_size=5, verbose=True),
)

This presently fails because Pandas converts the datatime64 data type to TIMESTAMP, which gets converted by SQLAlchemy/DBZPI to TIMESTAMP(3) instead of TIMESTAMP(6).

@erikerlandson
Copy link

@MichaelTiemannOSC does #181 resolve this?

@MichaelTiemannOSC
Copy link

There are changes to the correct functions in the code, so will test as soon as I clear out some other nuisances that are impeding my daily tech work. I'm optimistic the answer is yes...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

3 participants