CI: Pin matplotlib < 3.8

[lithomas1]

• closes #xxxx (Replace xxxx with the GitHub issue number)
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

[mroeschke]

cc @twoertwein I believe the new matplotlib version created new typing issues if you have any familiarity

[twoertwein]

cc @twoertwein I believe the new matplotlib version created new typing issues if you have any familiarity

I will look into it (I can't promise how soon). Is the only reason to pin it for typing reasons?

We also had a few issues in pandas-stubs: previously, mypy/pyright resolved any matplotlib types as Any/Unknown since it wasn't declared a py.typed library (I believe it is now). Technically, the new matplotlib release isn't causing new type issues but just exposing existing issues that previously weren't caught ;)

[twoertwein]

I think two main sources triggering these many typing errors are:

• pandas might be using private methods
• the type annotations of matplotlib are sometimes too strict, might not yet cover the entire public API

Here are some type errors that seem to indicate that pandas is doing something fishy:

pandas/plotting/_matplotlib/core.py:485: error: "Axes" has no attribute "_get_lines"  [attr-defined]
pandas/plotting/_matplotlib/core.py:486: error: "Axes" has no attribute "_get_patches_for_fill"  [attr-defined]
pandas/plotting/_matplotlib/core.py:487: error: "Axes" has no attribute "right_ax"  [attr-defined]
pandas/plotting/_matplotlib/core.py:487: error: "_AxesBase" has no attribute "left_ax"  [attr-defined]
pandas/plotting/_matplotlib/core.py:1438: error: "Axes" has no attribute "_plot_data"; maybe "plot_date"?  [attr-defined]
pandas/plotting/_matplotlib/core.py:1586: error: "Axes" has no attribute "_stacker_pos_prior"  [attr-defined]
pandas/plotting/_matplotlib/core.py:1588: error: "Axes" has no attribute "_stacker_neg_prior"  [attr-defined]

[ksunden]

Here are my annotations on the type errors I was able to look at:

Unrelated to mpl/maybe need a config/mypy plugin/other optional install?:

- pandas/core/arrays/masked.py:1010: error: Slice index must be an integer or None  [misc]
- pandas/core/generic.py:4259: error: Slice index must be an integer or None  [misc]
- pandas/core/generic.py:11816: error: Slice index must be an integer or None  [misc]
- pandas/core/indexes/multi.py:3035: error: Slice index must be an integer or None  [misc]
- pandas/io/excel/_calamine.py:78: error: Unused "type: ignore" comment
- pandas/conftest.py:199: error: "Type[HealthCheck]" has no attribute "differing_executors"  [attr-defined]

Legitimate type conflict in usage of mpl (though perhaps rare cases):

- pandas/plotting/_matplotlib/tools.py:55: error: "Text" has no attribute "set_ha"  [attr-defined]

• Uses dynamic alias behavior-- resolve by switching to cannonical version, set_horizontalalignment
• Since these are dynamically generated, we don't statically type hint them
  • technically we probably could, but would be painting ourselves into a corner regarding if we ever want to inline type hints instead of stubs
  • dynamic behavior also affects e.g. docstrings and such

- pandas/plotting/_matplotlib/tools.py:58: error: Argument 1 to "maybe_adjust_figure" has incompatible type "Optional[Figure]"; expected "Figure"  [arg-type]

• technically, ax.get_figure() can return None, though admittedly only for axes that cannot actually be drawn, so?
• Threading through the | None to maybe adjust_figure/do_adjust_figure should work, may want to explicitly handle in do_adjust_figure to keep type checker happy (it'll work with the hasattr, but type won't narrow)

- pandas/plotting/_matplotlib/tools.py:372: error: Incompatible types in assignment (expression has type "List[Any]", variable has type "GrouperView[Any]")  [assignment]
- pandas/plotting/_matplotlib/tools.py:378: error: "List[Any]" has no attribute "get_position"  [attr-defined]

• Probably should be resolved by using separate variables for the get_shared_[xy]_axes calls (which return GrouperView) and the get_siblings call on the GrouperView (which returns list[Axes])

- pandas/plotting/_matplotlib/converter.py:189: error: Argument 2 of "__call__" is incompatible with supertype "Formatter"; supertype defines the argument type as "Optional[int]"  [override]
- pandas/plotting/_matplotlib/converter.py:1068: error: Argument 2 of "__call__" is incompatible with supertype "Formatter"; supertype defines the argument type as "Optional[int]"  [override]
- pandas/plotting/_matplotlib/converter.py:1102: error: Argument 2 of "__call__" is incompatible with supertype "Formatter"; supertype defines the argument type as "Optional[int]"  [override]

• arg pos is unused anyway, for consistency with mpl, should probably default to None and be typed as int | None

- pandas/tests/plotting/common.py:125: error: Item "Axes" of "Union[Axes, Sequence[Axes]]" has no attribute "__iter__" (not iterable)  [union-attr]

• pandas _flatten_visible/flatten_axes method is actually type narrowing there, but type checker doesn't know that

- pandas/plotting/_matplotlib/timeseries.py:119: error: Incompatible types in assignment (expression has type "None", variable has type "str")  [assignment]

• changing the type of a variable
• should declare the variable with expanded type

- pandas/plotting/_matplotlib/core.py:1251: error: "None" not callable  [misc]
- pandas/plotting/_matplotlib/core.py:1251: error: Item "None" of "Optional[Colormap]" has no attribute "N"  [union-attr]

• if you know cmap can't be None, you need to inform the type checker here, I don't see anything preventing it
• or handle the None case explicitly

New mpl typing provides better way of handling this, though backwards compatibility is a potential problem:

- pandas/plotting/_matplotlib/style.py:272: error: Argument 1 to "to_rgba" of "ColorConverter" has incompatible type "Union[str, Sequence[float]]"; expected "Union[Union[Tuple[float, float, float], str], Union[str, Tuple[float, float, float, float], Tuple[Union[Tuple[float, float, float], str], float], Tuple[Tuple[float, float, float, float], float]]]"  [arg-type]

• We have mpl.typing.ColorType which specifies all the things that we actually mean when we say Color, and is more precise than Sequence[float] and also includes the new tuple[Color, alpha] form

Pandas dynamically adding things to MPL objects that aren't there by default:

- pandas/plotting/_matplotlib/timeseries.py:127: error: "Axes" has no attribute "_plot_data"; maybe "plot_date"?  [attr-defined]
- pandas/plotting/_matplotlib/timeseries.py:139: error: "Axes" has no attribute "_plot_data"; maybe "plot_date"?  [attr-defined]
- pandas/plotting/_matplotlib/timeseries.py:156: error: "Axes" has no attribute "_plot_data"; maybe "plot_date"?  [attr-defined]
- pandas/plotting/_matplotlib/timeseries.py:158: error: "Axes" has no attribute "freq"  [attr-defined]
- pandas/plotting/_matplotlib/timeseries.py:160: error: "XAxis" has no attribute "freq"  [attr-defined]
- pandas/plotting/_matplotlib/timeseries.py:162: error: "Axes" has no attribute "legendlabels"  [attr-defined]
- pandas/plotting/_matplotlib/timeseries.py:166: error: "Axes" has no attribute "date_axis_info"  [attr-defined]
- pandas/plotting/_matplotlib/core.py:487: error: "Axes" has no attribute "right_ax"  [attr-defined]
- pandas/plotting/_matplotlib/core.py:487: error: "_AxesBase" has no attribute "left_ax"  [attr-defined]
- pandas/plotting/_matplotlib/core.py:1438: error: "Axes" has no attribute "_plot_data"; maybe "plot_date"?  [attr-defined]
- pandas/plotting/_matplotlib/core.py:1442: error: "Axes" has no attribute "freq"  [attr-defined]
- pandas/plotting/_matplotlib/core.py:1456: error: "Axes" has no attribute "_stacker_pos_prior"  [attr-defined]
- pandas/plotting/_matplotlib/core.py:1458: error: "Axes" has no attribute "_stacker_neg_prior"  [attr-defined]
- pandas/plotting/_matplotlib/core.py:1459: error: "Axes" has no attribute "_stacker_pos_prior"  [attr-defined]
- pandas/plotting/_matplotlib/core.py:1460: error: "Axes" has no attribute "_stacker_neg_prior"  [attr-defined]
- pandas/plotting/_matplotlib/core.py:1471: error: "Axes" has no attribute "_stacker_pos_prior"  [attr-defined]
- pandas/plotting/_matplotlib/core.py:1473: error: "Axes" has no attribute "_stacker_neg_prior"  [attr-defined]
- pandas/plotting/_matplotlib/core.py:1486: error: "Axes" has no attribute "_stacker_pos_prior"  [attr-defined]
- pandas/plotting/_matplotlib/core.py:1488: error: "Axes" has no attribute "_stacker_neg_prior"  [attr-defined]
- pandas/plotting/_matplotlib/core.py:1586: error: "Axes" has no attribute "_stacker_pos_prior"  [attr-defined]
- pandas/plotting/_matplotlib/core.py:1588: error: "Axes" has no attribute "_stacker_neg_prior"  [attr-defined]

• As these are purely pandas, up to you how you work around these, but I'm personally a little skeptical of attaching things to upstream objects...

- pandas/plotting/_matplotlib/timeseries.py:165: error: "Axes" has no attribute "view_interval"  [attr-defined]

• This one there is some reference to in mpl, though never as an instance attribute currently, there are a get_view_interval and set_view_interval pair though
• there is a deprecation note on MPL 3.6 for v_interval as an attribute on Axes3D, with replacment being the getter/setter pair

Pandas using private (and thus untyped) functionality:

- pandas/plotting/_matplotlib/core.py:485: error: "_AxesBase" has no attribute "_get_lines"; maybe "get_lines"?  [attr-defined]
- pandas/plotting/_matplotlib/core.py:485: error: "Axes" has no attribute "_get_lines"  [attr-defined]
- pandas/plotting/_matplotlib/core.py:486: error: "_AxesBase" has no attribute "_get_patches_for_fill"  [attr-defined]
- pandas/plotting/_matplotlib/core.py:486: error: "Axes" has no attribute "_get_patches_for_fill"  [attr-defined]

• There is some recent talk about actually getting public API for these, probably worth voicing support for it:
  - [ISSUE]: The iterator ''prop_cycler'' is removed from ax._get_lines after v3.8 matplotlib/matplotlib#26831 is the recent one, but was closed as duplicate
  - Support sharing cycler instances across axes matplotlib/matplotlib#19479 is the older request which remains open

May need more thought on mpl side...:

- pandas/plotting/_matplotlib/converter.py:353: error: Argument 1 to "set_axis" of "TickHelper" has incompatible type "Union[None, Axis, _DummyAxis, _AxisWrapper]"; expected "Union[Axis, _DummyAxis, None]"  [arg-type]
- pandas/plotting/_matplotlib/converter.py:355: error: Item "None" of "Union[None, Axis, _DummyAxis, _AxisWrapper]" has no attribute "set_view_interval"  [union-attr]
- pandas/plotting/_matplotlib/converter.py:355: error: Item "None" of "Union[None, Axis, _DummyAxis, _AxisWrapper]" has no attribute "get_view_interval"  [union-attr]
- pandas/plotting/_matplotlib/converter.py:356: error: Item "None" of "Union[None, Axis, _DummyAxis, _AxisWrapper]" has no attribute "set_data_interval"  [union-attr]
- pandas/plotting/_matplotlib/converter.py:356: error: Item "None" of "Union[None, Axis, _DummyAxis, _AxisWrapper]" has no attribute "get_data_interval"  [union-attr]
- pandas/plotting/_matplotlib/converter.py:970: error: Item "None" of "Union[None, Axis, _DummyAxis, _AxisWrapper]" has no attribute "get_view_interval"  [union-attr]
- pandas/plotting/_matplotlib/converter.py:992: error: Item "None" of "Union[None, Axis, _DummyAxis, _AxisWrapper]" has no attribute "get_data_interval"  [union-attr]
- pandas/plotting/_matplotlib/converter.py:1060: error: Item "None" of "Union[None, Axis, _DummyAxis, _AxisWrapper]" has no attribute "get_view_interval"  [union-attr]
- pandas/plotting/_matplotlib/converter.py:1103: error: Item "None" of "Union[None, Axis, _DummyAxis, _AxisWrapper]" has no attribute "get_view_interval"  [union-attr]

• MPL is leaking private behavior
• the "May be None" might be legitimate, though also could be "yes, technically, this field is set to None, but only during instantiation, so maybe we should report what it ends up as"
• We are leaking private classes as type hints here, perhaps making a Protocol is the way to go? (i.e. actually encode the methods we expect to be available and let the private shims ducktype as working)

- pandas/tests/plotting/common.py:127: error: Item "Artist" of "Union[Any, Artist]" has no attribute "fill"  [union-attr]

• We did not make ArtistList generic in the first go around, so it lacks the type narrowing that it probably should have
• Can probably make this better on our end
• ArtistList is pseudo temporary to transition from a mutable list-like to an immutable tuple-like, and may just go away, not sure yet/what timescale
• short term, can probably cast/ignore

- pandas/plotting/_matplotlib/core.py:468: error: "Axes" has no attribute "containers"  [attr-defined]

• Actually does seem to exist, but not set directly in __init__ (actually set in __clear) so it was missed, may be others as well

- pandas/plotting/_matplotlib/core.py:1118: error: Module "matplotlib.axes" does not explicitly export attribute "Subplot"  [attr-defined]

• Subplot is now just an alias for Axes
• runtime uses import Axes as Subplot, but that is insufficient for mypy to see it as exported
• probably the play is to add __all__ to axes/__init__.py
• also have a star import in there, which could potentially mess some things up?

Unclear where to land on what needs fixing:

- pandas/plotting/_matplotlib/core.py:1172: error: Argument 1 to "set_xlabel" of "_AxesBase" has incompatible type "Hashable"; expected "str"  [arg-type]
- pandas/plotting/_matplotlib/core.py:1173: error: Argument 1 to "set_ylabel" of "_AxesBase" has incompatible type "Hashable"; expected "str"  [arg-type]
- pandas/plotting/_matplotlib/core.py:1820: error: Argument 1 to "set_xlabel" of "_AxesBase" has incompatible type "Optional[Hashable]"; expected "str"  [arg-type]
- pandas/plotting/_matplotlib/hist.py:184: error: Argument 1 to "set_xlabel" of "_AxesBase" has incompatible type "Hashable"; expected "str"  [arg-type]
- pandas/plotting/_matplotlib/hist.py:185: error: Argument 1 to "set_ylabel" of "_AxesBase" has incompatible type "Optional[Hashable]"; expected "str"  [arg-type]
- pandas/plotting/_matplotlib/hist.py:187: error: Argument 1 to "set_xlabel" of "_AxesBase" has incompatible type "Optional[Hashable]"; expected "str"  [arg-type]
- pandas/plotting/_matplotlib/hist.py:188: error: Argument 1 to "set_ylabel" of "_AxesBase" has incompatible type "Hashable"; expected "str"  [arg-type]

• Docstring supports current mpl type hints, but could arguably be expanded (the underlying implementation doesn't actually rely on str specifically, and in fact ends up at Any, even Hashable is not actually a requirement)

- pandas/plotting/_matplotlib/core.py:1507: error: Argument 1 to "FixedLocator" has incompatible type "ndarray[Any, Any]"; expected "Sequence[float]"  [arg-type]
- pandas/plotting/_matplotlib/hist.py:434: error: Argument 1 to "set_ticks_props" has incompatible type "ndarray[Any, dtype[Any]]"; expected "Union[Axes, Sequence[Axes]]"  [arg-type]

• NDArray doesn't qualify as Sequence
  • NDArray doesn't have an index or count method, which is expected by collections.ABC.Sequence
• but Sequence is the most accurate expectation we have
• there are probably a few more of these, because I was trying an experiment when I sat down to run these (that's actually why I started...)
  • My experiment is to use a Protocol class for Sequence that omits index and count, and thus is a strict subset of collections.ABC.Sequence but includes NDArray
  • It seems to work really well, but still not convinced its a good idea, at the very least may want to actually change the name...

Most of the fixes are fairly straightforward (especially to pandas itself), with the most sticky things probably being on our end (mpl). May require a few extra ignores in the interim, for those.

Sorry to have missed these when I did run against downstreams, I think I ran against the stubs repo thinking that mypy would report inconsistencies there, but it didn't find anything, so I moved on.

[mroeschke]

I will look into it (I can't promise how soon). Is the only reason to pin it for typing reasons?

I believe code checks were the only reason why this was pinned. @lithomas1 might be able to confirm

[lithomas1]

No, there were some other failing tests too IIRC.

I don't remember, though. Sorry about that, I think I was rushing to get CI green for the release and forgot to open an issue for this.

[twoertwein]

I opened #55253 to take care of most of the typing errors - thank you very much @ksunden! There is still more to do, see comments in the PR.