BUG: read_csv not respecting object dtype when option is set #56047
Conversation
|
Can we get this one in? |
| @@ -1846,7 +1851,29 @@ def read(self, nrows: int | None = None) -> DataFrame: | |||
| else: | |||
| new_rows = len(index) | |||
|
|
|||
| df = DataFrame(col_dict, columns=columns, index=index) | |||
| if hasattr(self, "orig_options"): | |||
| dtype_arg = self.orig_options.get("dtype", None) | |||
There was a problem hiding this comment.
Is the dtype option normally applied in _engine.read? Just curious why it needs to be done here
There was a problem hiding this comment.
Yes, but the DataFrame constructor infers object to string again if the option is set, which would discard the original dtype
There was a problem hiding this comment.
OK makes sense.
Could we defer looping over col_dict if dtype isn't specified to be object-like?
There was a problem hiding this comment.
Updated, only doing this now if we have a dict or object dtype
| @@ -295,18 +295,8 @@ def read(self) -> DataFrame: | |||
| dtype_mapping[pa.null()] = pd.Int64Dtype() | |||
| frame = table.to_pandas(types_mapper=dtype_mapping.get) | |||
| elif using_pyarrow_string_dtype(): | |||
|
|
|||
There was a problem hiding this comment.
These mappers don't work, arrow supports type -> type not column -> type
|
cc @mroeschke gentle ping |
|
Thanks @phofl |
| @@ -1846,7 +1853,40 @@ def read(self, nrows: int | None = None) -> DataFrame: | |||
| else: | |||
| new_rows = len(index) | |||
|
|
|||
| df = DataFrame(col_dict, columns=columns, index=index) | |||
| if hasattr(self, "orig_options"): | |||
There was a problem hiding this comment.
can we do something more explicit than a hasattr check?
There was a problem hiding this comment.
no, you can subclass the reader, so we don't have any control over it
There was a problem hiding this comment.
does anybody actually do this? i judge those people, their ethics, and their hygiene.
There was a problem hiding this comment.
That's something I can't answer, we might want to deprecate maybe, but we are stuck with hasattr here until then
doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.we are not honouring object dtype here, thoughts on performance @jbrockmendel ?