Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PERF: Improve conversion in read_csv when string option is set #56046

Merged
merged 4 commits into from
Nov 30, 2023
Merged

PERF: Improve conversion in read_csv when string option is set #56046

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f6d84b9
PERF: Improve conversion in read_csv when string option is set
phofl Nov 18, 2023
e47d8f6
Merge branch 'main' into csv_arrow_conversion
phofl Nov 19, 2023
acd59e5
Merge branch 'main' into csv_arrow_conversion
phofl Nov 21, 2023
b051e8c
Merge branch 'main' into csv_arrow_conversion
phofl Nov 29, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion pandas/io/parsers/arrow_parser_wrapper.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -295,7 +295,15 @@ def read(self) -> DataFrame:
dtype_mapping[pa.null()] = pd.Int64Dtype()
frame = table.to_pandas(types_mapper=dtype_mapping.get)
elif using_pyarrow_string_dtype():
frame = table.to_pandas(types_mapper=arrow_string_types_mapper())

def types_mapper(dtype):
dtype_dict = self.kwds["dtype"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is dtype guaranteed to be in self.kwds at this point?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep

if dtype_dict is not None and dtype_dict.get(dtype, None) is not None:
return dtype_dict.get(dtype)
return arrow_string_types_mapper()(dtype)

frame = table.to_pandas(types_mapper=types_mapper)

else:
if isinstance(self.kwds.get("dtype"), dict):
frame = table.to_pandas(types_mapper=self.kwds["dtype"].get)
Expand Down
Loading