Explicit installation of poetry export plugin breaks deployments with Poetry <1.6

[bfontaine]

Hello,
I use the latest version of this buildpack and since #66 it installs poetry-plugin-export, but this plugin requires Poetry 1.6 and so the deployment doesn’t work if one uses an older version.

Enumerating objects: 32, done.
Counting objects: 100% (32/32), done.
Delta compression using up to 8 threads
Compressing objects: 100% (13/13), done.
Writing objects: 100% (17/17), 2.48 KiB | 636.00 KiB/s, done.
Total 17 (delta 9), reused 0 (delta 0), pack-reused 0
-----> Cleaning up...
-----> Building myapp from herokuish
-----> Adding BUILD_ENV to build environment...
       BUILD_ENV added successfully
-----> Multipack app detected
=====> Downloading Buildpack: https://github.com/moneymeets/python-poetry-buildpack.git
=====> Detected Framework: Python Poetry
-----> Read Poetry version 1.5.1 from poetry.lock
-----> Generate requirements.txt with Poetry
-----> Install Poetry
       Retrieving Poetry metadata
       
       # Welcome to Poetry!
       
       This will download and install the latest version of Poetry,
       a dependency and package manager for Python.
       
       It will add the `poetry` command to Poetry's bin directory, located at:
       
       /app/.local/bin
       
       You can uninstall at any time by executing this script with the --uninstall option,
       and these changes will be reverted.
       
       Installing Poetry (1.5.1)
       Installing Poetry (1.5.1): Creating environment
       Installing Poetry (1.5.1): Installing Poetry
       Installing Poetry (1.5.1): Creating script
       Installing Poetry (1.5.1): Done
       
       Poetry (1.5.1) is installed now. Great!
       
       To get started you need Poetry's bin directory (/app/.local/bin) in your `PATH`
       environment variable.
       
       Add `export PATH="/app/.local/bin:$PATH"` to your shell configuration file.
       
       Alternatively, you can call Poetry explicitly with `/app/.local/bin/poetry`.
       
       You can test that everything is set up by executing:
       
       `poetry --version`
       
-----> Add Poetry to the PATH
-----> Install poetry-plugin-export
remote: 
remote: Because no versions of poetry-plugin-export match >1.6.0,<2.0.0
remote:  and poetry-plugin-export (1.6.0) depends on poetry (>=1.6.0,<2.0.0), poetry-plugin-export (>=1.6.0,<2.0.0) requires poetry (>=1.6.0,<2.0.0).
remote: So, because poetry-instance depends on both poetry (1.5.1) and poetry-plugin-export (^1.6.0), version solving failed.
       Using version ^1.6.0 for poetry-plugin-export
       
       Updating dependencies
       Resolving dependencies...
remote:  !     Failure during app build
remote:  !     Retagging old image 793abcde as myremote/myapp:latest
remote:  !     App build failed

Would it be possible to install the plugin only if the Poetry version is >1.6?

[zyv]

This is extremely weird, actually there are versions of poetry-plugin-export almost for any version of poetry:

https://pypi.org/project/poetry-plugin-export/#history

... and with 1.5.1 it is shipped inside Poetry, so this command should be a no-op. I wonder why it is trying to install the latest version of plugin in the first place. Maybe there is a different command to install the plugin and we are not using the right one?

We could of course only run this for Poetry 1.6+, but version comparison in Bash is annoying...

/cc @mm-felixh

[bfontaine]

poetry-plugin-export@* seems to work:

% poetry init -n
% poetry add poetry==1.5.1

Updating dependencies
Resolving dependencies... Downloading
[…]

Package operations: 48 installs, 0 updates, 0 removals

 […]
  • Installing poetry-plugin-export (1.5.0)
[…]
  • Installing poetry (1.5.1)

Writing lock file

% poetry add poetry-plugin-export
Using version ^1.6.0 for poetry-plugin-export

Updating dependencies
Resolving dependencies... (0.0s)

Because no versions of poetry-plugin-export match >1.6.0,<2.0.0
 and poetry-plugin-export (1.6.0) depends on poetry (>=1.6.0,<2.0.0), poetry-plugin-export (>=1.6.0,<2.0.0) requires poetry (>=1.6.0,<2.0.0).
So, because toto depends on both poetry (1.5.1) and poetry-plugin-export (^1.6.0), version solving failed.

% poetry add poetry-plugin-export@*

Updating dependencies
Resolving dependencies... (0.2s)

No dependencies to install or update

Writing lock file

[mm-matthias]

Maybe there is a different command to install the plugin and we are not using the right one?

Here are the poetry docs. This section explicitly advises against using poetry self add so maybe we should just resort to using a plain pip install (need to check if that works)?

As a side-note the export plugin seems to have a troubled history with circular dependencies and such. Not sure if this factors into the problem.

[mm-matthias]

@bfontaine Does it work because of the poetry-plugin-export@* or because you use poetry add instead of poetry self add?

[bfontaine]

@mm-matthias I used poetry add so it uses the local runtime with Poetry 1.5.1 and not my user-installed Poetry 1.6.1. I just tested by downgrading to 1.5.1 and it works the same:

% poetry self update 1.5.1
[…]
  • Downgrading poetry (1.6.1 /…/lib/python3.11/site-packages -> 1.5.1)

Writing lock file

% poetry self add poetry-plugin-export
[…]
Using version ^1.6.0 for poetry-plugin-export

Updating dependencies
Resolving dependencies... (0.0s)

Because no versions of poetry-plugin-export match >1.6.0,<2.0.0
 and poetry-plugin-export (1.6.0) depends on poetry (>=1.6.0,<2.0.0), poetry-plugin-export (>=1.6.0,<2.0.0) requires poetry (>=1.6.0,<2.0.0).
So, because poetry-instance depends on both poetry (1.5.1) and poetry-plugin-export (^1.6.0), version solving failed.

% poetry self add poetry-plugin-export@*

Updating dependencies
Resolving dependencies... (0.2s)

Package operations: 0 installs, 46 updates, 0 removals

[…]
  • Updating poetry (1.5.1 /home/linuxbrew/.linuxbrew/opt/[email protected]/lib/python3.11/site-packages -> 1.5.1)
  • Updating poetry-plugin-export (1.5.0 /home/linuxbrew/.linuxbrew/opt/[email protected]/lib/python3.11/site-packages -> 1.5.0)

Writing lock file

[mzdatawars]

Same issue here. Heroku deployments stopped working since the mentioned update.

[bfontaine]

@mzdatawars you should be able to circumvent the issue by explicitely specifying the hash of the commit before the problematic one:

https://github.com/moneymeets/python-poetry-buildpack.git#0bbaf48423f0caac527e185b1517abac1610dc46

See https://devcenter.heroku.com/articles/buildpacks#buildpack-references

[mzdatawars]

Yes, that temporarily solves the issue. I will keep an eye for a longer term solution. Thank you!

[mm-matthias]

@bfontaine Interesting, here are my outputs (on Mac):

% poetry self update 1.5.1

  • Downgrading poetry-plugin-export (1.6.0 -> 1.5.0)
  • Downgrading poetry (1.7.0 -> 1.5.1)

% poetry self add poetry-plugin-export
Using version ^1.6.0 for poetry-plugin-export

Updating dependencies
Resolving dependencies... (0.0s)

Because no versions of poetry-plugin-export match >1.6.0,<2.0.0
 and poetry-plugin-export (1.6.0) depends on poetry (>=1.6.0,<2.0.0), poetry-plugin-export (>=1.6.0,<2.0.0) requires poetry (>=1.6.0,<2.0.0).
So, because poetry-instance depends on both poetry (1.5.1) and poetry-plugin-export (^1.6.0), version solving failed.

poetry self add poetry-plugin-export@*

Updating dependencies
Resolving dependencies... (0.2s)

No dependencies to install or update

Writing lock file

So it appears that I had already poetry-plugin-export installed before (but I never did this manually AFAIK) and it was downgraded appropriately.

It's also interesting that in your output there is this line

Updating poetry (1.5.1 /home/linuxbrew/.linuxbrew/opt/[email protected]/lib/python3.11/site-packages -> 1.5.1)

I wonder why poetry itself should be upgraded and why it is listed as an upgrade from 1.5.1 to 1.5.1 (which would not qualify as an update in my book). But that might be another artifact.

Then I re-installed poetry completely to make sure the output that I get is the same as with the poetry self update 1.5.1 method:

% curl -sSL https://install.python-poetry.org | python3 - --uninstall
Removing Poetry (1.7.0)
% poetry
[...] No such file or directory
% curl -sSL https://install.python-poetry.org | python3 - --version 1.5.1
[...] Poetry (1.5.1) is installed now. Great!
% poetry --version
Poetry (version 1.5.1)
% poetry self add poetry-plugin-export
Using version ^1.6.0 for poetry-plugin-export

Updating dependencies
Resolving dependencies... (0.1s)

Because no versions of poetry-plugin-export match >1.6.0,<2.0.0
 and poetry-plugin-export (1.6.0) depends on poetry (>=1.6.0,<2.0.0), poetry-plugin-export (>=1.6.0,<2.0.0) requires poetry (>=1.6.0,<2.0.0).
So, because poetry-instance depends on both poetry (1.5.1) and poetry-plugin-export (^1.6.0), version solving failed.
% poetry self add poetry-plugin-export@*

Updating dependencies
Resolving dependencies... (3.0s)

No dependencies to install or update

Writing lock file

Wondering why the @* part worked I tried this:

% poetry self add poetry-plugin-export@latest
Using version ^1.6.0 for poetry-plugin-export

Updating dependencies
Resolving dependencies... (0.0s)

Because no versions of poetry-plugin-export match >1.6.0,<2.0.0
 and poetry-plugin-export (1.6.0) depends on poetry (>=1.6.0,<2.0.0), poetry-plugin-export (>=1.6.0,<2.0.0) requires poetry (>=1.6.0,<2.0.0).
So, because poetry-instance depends on both poetry (1.5.1) and poetry-plugin-export (^1.6.0), version solving failed.

% poetry self add poetry-plugin-export@^1.5.0

Updating dependencies
Resolving dependencies... (0.2s)

No dependencies to install or update

Writing lock file

So it seems that poetry tries to just install the latest version of the package. It probably does not take into account that its own version limits the version of the package that can be installed thus yielding the error. Why the wildcard suddenly makes everything work is something I do not really understand.

So we could introduce the @* for now to solve the problems of everyone running into this issue, but I wonder if this needs a follow-up with either the poetry or the poetry-plugin-export guys. WDYT @zyv @marns93?

[mm-matthias]

On a freshly installed poetry 1.7.0 I get this:

$ poetry self add poetry-plugin-export@*

Updating dependencies
Resolving dependencies... Downloading https://files.pythonhosted.org/packages/a9/58/3feea94f12f25714f54a1cc14f3760977631d62c70952de3ab4bd0c6
Resolving dependencies... (2.5s)

No dependencies to install or update

Writing lock file
$ poetry self add poetry-plugin-export
The following packages are already present in the pyproject.toml and will be skipped:

  • poetry-plugin-export

If you want to update it to the latest compatible version, you can use `poetry self update`.
If you prefer to upgrade it to the latest available version, you can use `poetry self add package@latest`.

Nothing to add.
$ poetry self add poetry-plugin-export@latest
Using version ^1.6.0 for poetry-plugin-export

Updating dependencies
Resolving dependencies... (0.2s)

No dependencies to install or update

Writing lock file

Interestingly, there is no pyproject.toml file present at all in the directory where I am running the command...

[mm-matthias]

Ok, I understand the reason now. Quoting python-poetry/poetry#707

Your issue is not the resolver but the semantics of poetry add:
IIRC poetry add foo is the same as poetry add foo@latest. What you expect is probably poetry add foo@* (or something more sophisticated).

This seems to already have confused quite a few other people, e.g. python-poetry/poetry#6812 .

So the @* thing is probably the right fix, because this matches the way poetry-plugin-export does its versioning (basically in lock-step with poetry, c.f. https://github.com/python-poetry/poetry-plugin-export/blob/main/pyproject.toml).

[mm-matthias]

@bfontaine @mzdatawars The fix from #67 is on master now. Can you check if this works for you?

@bfontaine Thank you for the suggested fix and your console outputs!

[bfontaine]

Thank you for the quick update; it works for me!