Rollout pandas version 2

[victorlin]

Pandas version 2 is supported by Augur and it's likely that our runtimes will eventually resolve to this new version with potentially breaking changes.

Tasks

• Merge Support pandas v2 augur#1671
• 26.2.0 release augur#1684
• Wait for pandas to resolve to version 2 in conda-base and docker-base. Pending at least
  • Support pandas version 2 blab/evofr#42
  • Support pandas version 2 fauna#169 or Use custom runtime fauna#170:
• Be on the watch for potential unexpected pandas upgrade related changes in our ecosystem
  • ncov-ingest (pinned to v1, installed via pip in custom Dockerfile layered atop docker-base)

Related issues

• test failure beyond pandas 2.0 augur#1303
• Numpy deprecation warnings for pandas imports augur#1316
• parse: Support pandas 2.x augur#1436
• augur/parse.py: fix test failure with pandas 2.2. augur#1471

[corneliusroemer]

The latest pyarrow no longer supports pandas <2 it seems so this is another good reason to allow pandas 2.

[corneliusroemer]

An even bigger motivation: we can't support python 3.12 until we support pandas v2

Would be really cool if you could figure this out @victorlin 😄

[corneliusroemer]

Any progress on this? We'd like to use augur in some parts of Loculus and we've been using some Python 3.12 features (f-strings, type hints) that make downgrading a pain. Would be really cool if we could get Python 3.12 support done, especially now that it's not even the latest anymore (since Python 3.13 release).

[victorlin]

@corneliusroemer thanks for the bump – I've been busy with some Auspice stuff but will try to take a look next week. If anyone else wants to look in the meantime, feel free to do so!

[victorlin]

I just tested locally and Augur+pandas are installable on Python 3.12 as-is. The lack of 3.12 support on pandas 1.x just means that it has to be built from source instead of installing pre-built wheels. I haven't done any actual testing, but maybe this is good enough to get things going on your end.

[corneliusroemer]

Thanks @victorlin - we'll have a look, this means we have to pip-install augur in the conda environment.yaml. Seems to work as a workaround for now, good idea! Though we'll also have to downgrade pandas from 2->1 then.

[corneliusroemer]

I've just had another look at supporting pandas v2, tests pass up to latest pandas v2.2.3

I think we should just go and officially sort it, all the tests pass: unit and functional. If something does come up, we can fix it. Would be good to run pathogen-ci with it put that's about it.

Update: pathogen-CI works with v2.2.3 of pandas 🎉
nextstrain/augur#1672 (comment)

[victorlin]

@corneliusroemer thanks for looking! Did you mean to create 2 PRs for this (nextstrain/augur#1671, nextstrain/augur#1672)? The only difference between the two is that one supports pandas v1 while the other doesn't. I think it would be good to keep support for v1 if trivial. The tests are passing so it seems fine?

I think pandas changes should be good as long as tests are passing. But I'll want to review the Cram stuff in more detail. Could you rebase the pandas PR so that it's not based on the changes from nextstrain/augur#1667?

[corneliusroemer]

Yes the two PRs are absolutely on purpose. I agree we should keep v1 compatibility for now. The reason I made the second PR excluding V1 is that pathogen CI will still use pandas v1 in tests unless you exclude v1. Does that make sense?

The reason is that we use the conda base and then pip install augur into it in CI. Since conda base already has pandas v1 and v1 is allowed pip keeps it around and doesn't upgrade to V2. So the actual V2 PR doesn't actually check V2 in pathogen CI. Hence the need for the test PR.

I expect that we can upgrade pandas to V2 in conda base once augur supports it so excluding V1 is just a temporary test PR

[corneliusroemer]

@victorlin I've cleaned up the v2 pandas PR nextstrain/augur#1671 - I think we're pretty much good to go, let's get this done it unblocks Python 3.12 and other stuff :)

[corneliusroemer]

Progress, I merged the PR! Should we wait a few days for any potential 26.1.0 bugs to surface first so we know bugs are not from pandas 2? Then release in say a week.

[victorlin]

@joverlee521 re: your edits to issue description (thanks for adding!)

• fauna (pinned to v1, installed via pip in docker-base before augur)
• ncov-ingest (pinned to v1, installed via pip in custom Dockerfile layered atop docker-base)

fauna: Can we remove the ==1.* ceiling? From the commit that added it, I don't see a reason why pandas v2 can't be supported.

ncov-ingest: there is no ceiling pin. From a quick skim, none of the pandas usage sticks out as potentially breaking with v2, but the easiest way to know for sure is to just let it run.

[joverlee521]

Wanted to flag these two that specifically mentioned pandas v1 so we don't have any surprises.

fauna: I think we can just remove the ceiling.

For context, the download scripts that are used seasonal-flu within the Nextstrain docker runtime don't even use pandas. Our manual upload scripts that do use pandas also use xlrd v1.2.0 which errors with Python >=3.8 so I actually have a separate pyenv with Python 3.7 for that process. The use of Python 3.7 automatically limits pandas to v1. (It's definitely a bit of a mess that I need to sort out eventually...)

ncov-ingest: If we want to be cautious, we could create a branch-* image then use it fora a trial run of the workflow. But as you said, I'm not super concerned and we could just let it run to tackle any breakages.

[victorlin]

• Ensure the pins are propagated to conda-base and docker-base

The runtimes still resolve to pandas v1 due to the version constraints from evofr and fauna (this is only somewhat obvious in logs from the docker-base rebuild). I'll make separate issues in those repos.

[victorlin]

The work in Augur is done. I think it's still beneficial to keep this open to be aware of if/when Nextstrain runtimes resolve to Pandas v2, so I'll transfer this to the nextstrain/public repo.