BUG: DataFrame.to_json OverflowError with np.long* dtypes

[gupta-paras]

• closes BUG: DataFrame.to_json OverflowError with np.long* dtypes #55403 (Replace xxxx with the GitHub issue number)
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

[WillAyd]

We don't support long double as a data type and I don't think upcasting double to long double is appropriate. We should probably just raise here

[gupta-paras]

@WillAyd, Thanks for identifying the issue with commit. I have updated the commit with changes suggested in above comment.
Thanks

[WillAyd]

This works, but I think the lines directly following it are supposed to be a catchall for unsupported types. Do you know why that isn't being hit? I would rather we keep this generic instead of having to specify an error message for every type that we don't serialize

[gupta-paras]

There is a code section at the end of this function:

    pc->iterBegin = Dir_iterBegin;
    pc->iterEnd = Dir_iterEnd;
    pc->iterNext = Dir_iterNext;
    pc->iterGetValue = Dir_iterGetValue;
    pc->iterGetName = Dir_iterGetName;
    return;

By default everything falls to this section and this causes infinite loop.

[WillAyd]

Right but I'm asking about the next branch after what you've added.

} else if (PyArray_Check(obj) && PyArray_CheckScalar(obj)) {

Do you know what hits that currently? From reading the function I think the intent of that was to generically catch the issue you've described, but its possible the invariant is incorrect. Would something pass both PyArray_Check and PyArray_CheckScalar? Maybe the PyArray_Check call is incorrect and removing that alone would fix your issue?

[gupta-paras]

Any value like: np.array(1) [0d array] evaluates true for both of them, so that check was intended for handling this case only. Also for any numpy scalar type, second will be true. So, simply removing pyArray_check handles both cases. Only thing we need to make sure, that all numpy scalar types are handled before this if block. I will add similar comment in code as well.
Thanks

[nmacholl]

We don't support long double as a data type and I don't think upcasting double to long double is appropriate. We should probably just raise here

I'm curious, if pandas doesn't support these numpy types in a data frame, would it make sense to raise an exception at the time of creation or dtype assignment? There are maybe other places in the code these unsupported types would break?

[WillAyd]

@nmacholl you might want to check existing issues for any discussion, but I don't think its quite that simple. For starters assignment/creation isn't the only way those types might make there way into a DataFrame. There also exists a subset of users who may not use pandas for I/O / computations (where a lot of those limitations are imposed) versus just using it as an easy way to get labeled access to NumPy arrays. As an example, you'll see quite a few use cases in the issue log where users contain NumPy unicode dtypes in DataFrames even though pandas has almost no support for those; in such cases a DataFrame is just a labeled indexer, and I don't think we want to break that type of workflow

[WillAyd]

@gupta-paras this looks good to me aside from outstanding comments / code suggestions. @mroeschke any thoughts on your end?

[mroeschke]

Thanks @gupta-paras