[Feature Request]: Upgrade or provide ETA on dill

[eddie-scio]

What would you like to happen?

2.41.0 has dill<0.3.2, which is now more than 2 years old. We're trying to upgrade one of our core ML libraries that we run on Beam, but we are blocked on a dill>=0.3.4 requirement there. I do see the note about dill incompat in setup.py, and I do see a rollback PR for the upgrade -- I'm wondering if there is any ETA or communication with dill that can be shared.

Issue Priority

Priority: 2

Issue Component

Component: sdk-py-core

[tvalentyn]

@ryanthompson591 could you provide your input here please?

[eddie-scio]

Hey @ryanthompson591 , just a friendly ping!

[ryanthompson591]

There are two efforts underway. The first was to upgrade our dill version, which got blocked by some failed tests.

See:
uqfoundation/dill#517
python/cpython#94245

In dill's defense it's a python error - that gets resolved in 3.10.

The second is to vendor dill, which there is some agreement to do.

Either would solve this problem.

As to the question of when, I am not yet sure. Can you give some context as to why allen NLP requires this version of dill? Maybe it can help us prioritize.

[tvalentyn]

In dill's defense it's a python error - that gets resolved in 3.10.

Is this fixed on dill master?

[ryanthompson591]

I just downloaded dill master, and tried this code out and it is fixed.

Dill added a workaround for the python error here:
https://github.com/uqfoundation/dill/blob/25a7e450ed76c7a0820834a3a91134476b1b8253/dill/_dill.py#L746d.

[eddie-scio]

That's awesome. So we just need dill to release 0.3.6 (https://github.com/uqfoundation/dill/milestones), and then release a new version of Beam that changes the dependency constraint (e.g.dill>=0.3.6)?

[eddie-scio]

Good news, dill just released 0.3.6 (https://pypi.org/project/dill/0.3.6/) two days ago, so in 2.42.1 we should be able to take care of this?

[ryanthompson591]

Thanks for this eddie. I will be looking into doing this import and testing it throughout our code bases. I was able to validate that the new version of dill did fix this problem.

[eddie-scio]

Thanks so much, Ryan! Really excited to get this working.

[jmelot]

Just chiming in to say this is blocking us deploying a pipeline that runs an mlflow model (we have to use mlflow==1.24.0 and it has a dill==0.3.5.1 requirement). Please let us know if there's anything we can do to help!

[tvalentyn]

Thanks all, here is an update.

• Dill has made several breaking changes between dill version 0.3.1.1 and dill version 0.3.6, which affect what gets pickled and how: https://github.com/uqfoundation/dill/issues?q=is%3Aissue+regression
• Switching Apache Beam to newer version of dill will very likely negatively affect some group of users, while it may be a non-issue for some other group of users.
• Dill has also made changes to internal code, which Beam had some assumptions about. Beam 2.47.0 will include code changes to be able to work with dill==0.3.6: Fix compatibility with dill 0.3.6. #26086, however the default and required version of dill still remains dill==0.3.1.1 for now.
• dill==0.3.1.1. doesn't support Python 3.11. For Beam, one of the primary motivation to upgrade dill is to support Python 3.11, in addition to concerns in this bug. To unblock Python 3.11 support we went with monkey-patching dill 0.3.1.1 at runtime. The patch is applied only if dill version is 0.3.1.1 and Python version 3.11 or higher: 3d0ee7b (#26121) . The alternative we have considered is to vendor dill. I decided against vendoring at this time at last minute because: dill makes changes to the global state, for example modifies global dispatch table used by standard pickler. Given the demand for this issue it is clear that newer versions of dill will be installed in addition to vendored version. The vendored version and a stock version installed at runtime may potentially modify the global state differently. I didn't have enough time before 2.47.0 release cut to properly evaluate and address a risk of such concurrent modification.
• I have also evaluated setting cloudpickle as default pickler, and have encountered one issue that warranted additional investigation: [Bug]: cloudpickle appears to incorrectly unpickle cloned combiners  #26209 .
• I plan to have a conversation with dill maintainers to see if we can mitigate the impact of the breaking changes they have introduced to be able to update smoothly to a newer version of dill. Update: [Discuss] Recent breaking changes in new dill releases  uqfoundation/dill#589

In the meantime, with Beam 2.47.0, users can try to update to newer version of dill if they must, even though beam requires dill 0.3.1.1. Users can force-install newer version of dill in their submission environment as long as they install the same version of dill in the runtime environment. As I mentioned above, some users may not be affected by dill's breaking changes while some other users may be. Dill's breaking changes are not something Beam controls, but as mentioned above, Beam did make code changes to work with newer versions of dill. I will also continue to work on a better solution for this issue.

[jacek-jablonski]

Hello,
any updates on this? The old dil is still present in beam version 2.51.0.

[tvalentyn]

No update yet but we have prioritzed resolving blockers for cloudpickle adoption. In the meantime, the recommendations above still apply. Specifically

You can force-install a different version of dill than required by Beam under the following conditions:

• You install the same version in submission and in the runtime environment.
• The chosen version works for your pipeline

See also: https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/#make-the-launch-environment-compatible-with-the-runtime-environment

[siddharthab]

The suggestion to force a version of dill may not work in many situations, and so a proper remediation will be very much appreciated.

Many people are using pip-tools as their dependency manager these days, which does not allow a user to force install a specific version. Build systems like Bazel rules for Python also rely on pip-tools to build the dependency tree with resolved versions. I was unable to override the version constraints from my dependencies in my devops setup.

Environment compatibility checks between submission and runtime environments are actually supposed to be part of distributed compute systems. Dask and newer versions of Beam SDK have them. Should we not rely on these checks and remove the narrow version constraints in the requirements? If a user wants to use a provided (instead of custom) Beam docker image, then maybe each image can come with suggested job submission environment setup scripts.

[tvalentyn]

If a package manager prevents you from force-installing a different version of dill in your environment, you could create a custom distribution of beam or other package that uses dill, and change the version constraints for dill in setup.py to your liking and at your own risk. Then, install the package from a local copy instead of from pypi.

The suggested version of dill is still 0.3.1.1, because we are not confident that newer versions of dill won't break someone if we upgrade it across the board. Switching to cloudpickle is still the plan.

[siddharthab]

@tvalentyn Thank you for your suggestion. I think a forked source archive of beam could be a reasonable fallback. I also filed an issue for updating numpy - #30098.

[poffdeluxe]

Now that #26086 has been merged in, can we expect an upgrade to BEAM's dill dep soon?