Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix BUG: read_sql tries to convert blob/varbinary to string with pyarrow backend #60105

Open
wants to merge 22 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
2244869
Add fix for #59242
kastkeepitjumpinlikekangaroos Oct 25, 2024
bd00fc5
add skip import
kastkeepitjumpinlikekangaroos Oct 25, 2024
6a23f05
address comment
kastkeepitjumpinlikekangaroos Nov 12, 2024
2f261c8
merged main
kastkeepitjumpinlikekangaroos Nov 12, 2024
8f900b8
also fix for dtype_backend=numpy_nullable
kastkeepitjumpinlikekangaroos Nov 13, 2024
536b1ed
Merge branch 'main' into fix-59242
kastkeepitjumpinlikekangaroos Nov 14, 2024
8965a1d
merge main
kastkeepitjumpinlikekangaroos Nov 17, 2024
a32b4a6
fix
kastkeepitjumpinlikekangaroos Nov 17, 2024
f412c72
Merge branch 'main' into fix-59242
kastkeepitjumpinlikekangaroos Nov 20, 2024
a0200d0
address comment
kastkeepitjumpinlikekangaroos Nov 20, 2024
10ca030
Merge branch 'main' into fix-59242
kastkeepitjumpinlikekangaroos Nov 21, 2024
ba2e82d
address comment
kastkeepitjumpinlikekangaroos Dec 4, 2024
8dd45ca
Merge branch 'fix-59242' of github.com:kastkeepitjumpinlikekangaroos/…
kastkeepitjumpinlikekangaroos Dec 4, 2024
602194a
remove cast
kastkeepitjumpinlikekangaroos Dec 4, 2024
f8ae285
fix test
kastkeepitjumpinlikekangaroos Dec 4, 2024
02514e0
fix logic
kastkeepitjumpinlikekangaroos Dec 4, 2024
ef5c2ed
fix
kastkeepitjumpinlikekangaroos Dec 7, 2024
da7817a
fix
kastkeepitjumpinlikekangaroos Dec 9, 2024
de5c604
also include pa.OpaqueType
kastkeepitjumpinlikekangaroos Dec 9, 2024
e025d84
fix
kastkeepitjumpinlikekangaroos Dec 9, 2024
3d83bab
Merge branch 'main' into fix-59242
kastkeepitjumpinlikekangaroos Dec 9, 2024
6ea5785
Merge branch 'main' into fix-59242
kastkeepitjumpinlikekangaroos Dec 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v2.3.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,7 @@ MultiIndex
I/O
^^^
- :meth:`DataFrame.to_excel` was storing decimals as strings instead of numbers (:issue:`49598`)
- Bug in :func:`read_sql` causing an unintended exception when byte data was being converted to string when using the pyarrow dtype_backend (:issue:`59242`)
-

Period
Expand Down
4 changes: 0 additions & 4 deletions pandas/core/internals/construction.py
Original file line number Diff line number Diff line change
Expand Up @@ -968,10 +968,6 @@ def convert(arr):
# i.e. maybe_convert_objects didn't convert
convert_to_nullable_dtype = dtype_backend != "numpy"
arr = maybe_infer_to_datetimelike(arr, convert_to_nullable_dtype)
if convert_to_nullable_dtype and arr.dtype == np.dtype("O"):
new_dtype = StringDtype()
arr_cls = new_dtype.construct_array_type()
arr = arr_cls._from_sequence(arr, dtype=new_dtype)
elif dtype_backend != "numpy" and isinstance(arr, np.ndarray):
if arr.dtype.kind in "iufb":
arr = pd_array(arr, copy=False)
Expand Down
32 changes: 32 additions & 0 deletions pandas/tests/io/test_sql.py
Original file line number Diff line number Diff line change
Expand Up @@ -4358,3 +4358,35 @@ def test_xsqlite_if_exists(sqlite_buildin):
(5, "E"),
]
drop_table(table_name, sqlite_buildin)


@pytest.mark.parametrize("con", all_connectable)
def test_bytes_column(con, request):
# GitHub Issue #59242
conn = request.getfixturevalue(con)
pa = pytest.importorskip("pyarrow")
for dtype_backend in ["pyarrow", "numpy_nullable", lib.no_default]:
query = """
select x'0123456789abcdef0123456789abcdef' a
"""
df = pd.read_sql(query, conn, dtype_backend=dtype_backend)

dtype = "O"
if dtype_backend == "pyarrow":
# sqlite3 + mysql both return a binary type
# for the binary literal
dtype = pd.ArrowDtype(pa.binary())
elif dtype_backend == "pyarrow" and "postgres" in con:
# postgres column is a bit type
# but converts to a string when returned
dtype = pd.ArrowDtype(pa.string())

expected = DataFrame(
[
{
"a": b"\x01#Eg\x89\xab\xcd\xef\x01#Eg\x89\xab\xcd\xef",
}
],
dtype=dtype,
)
tm.assert_frame_equal(df, expected)
kastkeepitjumpinlikekangaroos marked this conversation as resolved.
Show resolved Hide resolved
Loading