BUG: replace na_rep for pd.NA values in to_string

[rsm-23]

• xref BUG: na_rep not respected in to_* methods with pd.NA #54872
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

[rsm-23]

@mroeschke Need some help here. This test will fail pandas/tests/arrays/string_/test_string.py::test_repr. Should I update the test or will that break some behavior?

[rsm-23]

@mroeschke The documentation says, that the default value for na_rep is 'NaN' so in pandas/io/formats/format.py, the replace never happens.

    def _format(x):
            if self.na_rep is not None and is_scalar(x) and isna(x):
                try:
                    # try block for np.isnat specifically
                    # determine na_rep if x is None or NaT-like
                    if x is None:
                        return "None"
                    elif x is NA:
                        return str(NA)
                    elif x is NaT or np.isnat(x):
                        return "NaT"
                except (TypeError, ValueError):
                    # np.isnat only handles datetime or timedelta objects
                    pass
                return self.na_rep

and if I replace elif x is NA: return self.na_rep then it breaks tests because the expectation is to get <NA>

[mroeschke]

Which tests need fixing?

[rsm-23]

@mroeschke Some tests that are failing are below. This is not an exhaustive list. The expected value in these tests is <NA> but according to docs it should be NaN since that's the default value for na_rep. Somehow this behaviour is not in place.

• pandas/tests/arrays/boolean/test_repr.py::test_repr
• pandas/tests/arrays/floating/test_repr.py::test_frame_repr[Float32Dtype]
• pandas/tests/arrays/integer/test_repr.py::test_frame_repr[Int8Dtype]
• pandas/tests/arrays/string_/test_string.py::test_repr[pyarrow]
• pandas/tests/io/formats/test_to_string.py::test_nullable_float_to_string[Float32]

[rsm-23]

@mroeschke pinging for visibility. Please let me know if the above message was clear or not.

[rsm-23]

@mroeschke need some inputs here :)

[mroeschke]

Yeah I think the tests should be updated for this change

[rsm-23]

@mroeschke can you please review this PR?

[rsm-23]

@mroeschke can you please review this? I have updated the test cases.

[rsm-23]

@mroeschke pinging on green.

[rsm-23]

@mroeschke @MarcoGorelli can one of you please review this?

[MarcoGorelli]

Thanks for your PR!

@mroeschke @MarcoGorelli can one of you please review this?

Just as a heads up, I'm not familiar with strings - I can take a look, but I'm afraid I'll need to prioritise other things first

[mroeschke]

Sorry for the delay on my end

[mroeschke]

All the usages that were changed should still use <NA> rather than NaN. This should only change if na_rep was specified as something else

[rsm-23]

The expected value in these tests is <NA> but according to docs it should be NaN since that's the default value for na_rep. Somehow this behaviour is not in place.

@mroeschke

[mroeschke]

I think since the current behavior has been the case for a few releases now, the default of na_rep="NaN" should be changed to na_rep=lib.no_default so that NaN, NA, and None can continue to use their reprs here

[rsm-23]

@mroeschke the problem is if I set na_rep=lib.no_default then the string representation will not contain either of NaN, NA or None. I have tried setting the default as <NA> and reverted most of the test cases, however, there's issue in a couple of them.

[mroeschke]

Using na_rep=lib.no_default would mean making NaN, NA and None return their string defaults like

if self.na_rep is lib.no_default:
    ...existing logic
else:
    ...use na_rap

[mroeschke]

Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.