RFC: Drop Python 2 Support

[schrockn]

The first line of code of Dagster was written about two and a half years ago. At that juncture, we decided to support Python 2. Python 2 had not been sunsetted, and there were large extant codebases out there that were python 2. We did not know how quickly python 3 usage would overtake python 2, we did not know if the “sunset” date for Python 2 would actually hold.

Fast forward two years: python 2 has been sunsetted, major libraries are aggressively dropping support for python 3 (see https://python3statement.org/), and that major user has not materialized. The dropoff in usage is very real and is more acute than I would have anticipated. If you go to pypistats and examine other projects in the data domain, the drop off in usage is acute. For major projects in the data domain, we have seen 2x-4x fewer downloads for python 2 in the last six months alone.

Why Now?

There are real costs to supporting python 2:

• All the incidental complexity in dagster.seven and the bugs and development time it represents.
• Development friction, slower dev cycles.
• Python 2 only bugs that require non-trivial amounts of engineering time to fix.
• Cannot use python 3 features for our own code, such as type hints
• We cannot update to or depend on libraries that only support python 3.
• Our public APIs are not “python-3-native”.

I believe the first four points are not controversial and fairly obvious. I consider them issues that increase incidental complexity and friction in our code base, but that could be managed. It is the last two points that are the most persuasive in my mind.

For example, dagit is limited to python 2 right now, and that limits what libraries we can bring it, especially in the graphql/web-serving front. It is actually a single-threaded, synchronous webserver right now, and can only field on request at a time. This is a real problem, and moving to Python 3 opens up the solution space for us quite a bit.

We are also starting to feel the pain of python 3 only dependencies. We just moved the slack integration to be python 3 only because the underlying client became python-3-only.

The second point is that because nearly all of our test cases are written to support python 2, we do not habitually use python 3 idioms when exercising the system ourselves. I think this has subtle but profound effects, as the way we develop diverges from our user base. A particularly good example is our use of type hints to express typing in the dagster domain, and its awkward relationship with mypy typing. If we were python 3 from the beginning, I would bet that we would have ended up at a different spot, with a different set of tradeoffs that is more intuitive for python 3 users.

Proposal:

Drop support for Python 2 at our next major release. This is a fast timeframe that could be adjusted pending community feedback. But in my ideal world, we would be developing Dagster in Python 3 only starting January 2021.

[mgasner]

I think moving to Python 3 only opens up all kinds of interface possibilities that we haven't even considered because the backcompat story is too complex. For example, Çaǧatay suggested that dataclasses (3.7+, backport to 3.6) might be a natural way to define config schemas, but this is something that we can't explore as long as we have 2/3 compat.

[schrockn]

We've decided to move forward with dropping Python 2. Starting with the 0.10.0 release, scheduled on January 7th 2021, we will not support Python 2.