Usage of boto (not boto3) prevents toil from working with Python3.12

[tillea]

Hi,
Debian is currently transitioning to Python3.12. Inside setup.py of toil I've found the usage of imp and the comment that it is known that this modules is deprecated but usage of its successor importlib would not be possible. To work around this I wrote a small patch where I'm simply using a shell command to obtain the version. I'd recommend to replace imp somehow (more?) sensibly to follow Python3 progress sooner or later.
The more concerning issue is the usage of boto. In issue #3951 of boto authors wrote:

This repo, `boto` was deprecated in 2020 and the last supported Python 3 version was 3.4. It's unlikely this package will ever work with Python 3.12.
 
The current AWS SDK for Python, boto3, can be found [here](https://github.com/boto/boto3) and does support Python 3.12.

Thus I strongly recommend to switch to boto3 before your next release.

Kind regards, Andreas.

┆Issue is synchronized with this Jira Story
┆Issue Number: TOIL-1468

[mr-c]

@adamnovak @DailyDreaming any advice here?

[adamnovak]

We could probably take the approach taken by that patch to eliminate imp, but we'd want to cut the functions it is just bypassing.

We'd like to switch away from boto, but we don't actually use the AWS logic so much here at UCSC anymore and so we don't have an internal fire we need to put out to motivate doing it in house. @DailyDreaming: you're revising the S3 integration, right? How much ripping out of boto 2 is happening in that?

[adamnovak]

boto is am optional dependency; Toil can be packaged and work without it, it just won't be able to do AWS-related things.

[stxue1]

I think I changed the setup.py file some time ago to properly use importlib, but left the comment even though it doesn't seem to apply anymore.

With the ported code to boto3 though Python 3.12 should be pretty easy to do, and the development branch builds with python 3.12 locally. I can add it to CI later.

[tillea]

Am Wed, Apr 10, 2024 at 08:01:00AM -0700 schrieb stxue1:

With the ported code to boto3 though Python 3.12 should be pretty easy to do, and the [development branch](https://github.com/DataBiosphere/toil/tree/issues/4718-boto-python-312-all) builds with python 3.12 locally. I can add it to CI later.

Sounds great. I hope this branch will make it into the next release. Thanks a lot, Andreas.

[mr-c]

Thank you @stxue1 @adamnovak !