poetry install subtly fails to install python-ldap

[axelboc]

I'm setting up mxcubeweb on a new machine again, and I'm hitting the same problem I hit every time, and which always takes me ages to figure out. Here are the steps I follow, as per the documentation (except with micromamba instead of conda):

1. Clone mxcubeweb and cd into it.

2. Run:

   micromamba env create --file conda-environment.yml
   micromamba activate mxcubeweb
   poetry install
   pnpm --prefix ui install
   pnpm --prefix ui build
   

3. Start the server:

   mxcubeweb-server -r $(pwd)/mxcubeweb/demo/ --static-folder $(pwd)/mxcubeweb/ui/build/ -L debug
   > mxcubeweb-server: command not found
   

After a while, it all comes back to me: I remember to run poetry install again, which reveals an error with python-ldap:

- Updating python-ldap (3.4.3 /home/conda/feedstock_root/build_artifacts/python-ldap_1698435803608/work -> 3.4.4): Failed
ChefBuildError
Backend subprocess exited when trying to invoke build_wheel
...
error: command 'gcc' failed: No such file or directory

The fix is easy:

sudo apt install build-essential
poetry install

---

From what I understand, there's no way around having to compile python-ldap. So to save me (and maybe others) some hair pulling next time, should we maybe update the documentation of MXCuBE-Web to recommend:

• installing build-essential as a prerequisite;
• running poetry install twice in a row to reveal any errors that may have occurred the first time around?

That being said, I'd be curious to know why poetry doesn't bring out the error more clearly the first time around... It really looks like the command completes normally. 😑

[fabcor-maxiv]

I already discussed a (different, but related?) "python-ldap+conda+poetry" issue at mxcube/mxcubecore#849 (comment) (and probably other PRs or issues as well). The short of it, is that if python-ldap is installed with Conda, it will be ignored by Poetry and reinstalled anyway. Poetry is right to ignore it. The reason lies in the way a bunch of conda packages are created, including the one for python-ldap. I reported it for the python-ldap conda package: conda-forge/python-ldap-feedstock#28. I am tempted to recommend removing python-ldap from our conda-environment.yml file, but as far as I remember, pip behaves a bit differently and does take the python-ldap installed via conda into account and does not try to reinstall it, so it might be worth keeping python-ldap in conda-environment.yml after all.

---

The fix is easy:

sudo apt install build-essential
poetry install

I had in mind that it was also necessary to install (some of) these as well (on Debian/Ubuntu):

ldap-utils
libldap2-dev
libsasl2-dev
python3-dev

---

I'd be curious to know why poetry doesn't bring out the error more clearly the first time around

Yes, I might recall this behaviour, I do not know the reason, and I agree that Poetry should fail with a clear error message.

---

From what I understand, there's no way around having to compile python-ldap. So to save me (and maybe others) some hair pulling next time, should we maybe update the documentation of MXCuBE-Web to recommend:

* installing `build-essential` as a prerequisite;

* running `poetry install` twice in a row to reveal any errors that may have occurred the first time around?

That seems to make sense.

And then again, I wonder what is the point of conda-environment.yml if it does not actually make our lives easier and we still have to jump through a bunch hoops anyway? Maybe different hoops, but not obviously easier hoops. : D

Maybe if it was possible to have the build dependencies of python-ldap in conda-environment.yml there would be a stronger point for having and maintaining a conda-environment.yml.

---

Also poetry.lock has python-ldpap==3.4.4 but conda-environment.yml still has 3.4.3. : D

[fabcor-maxiv]

#1515

Well... that is not going well... :D

[axelboc]

Thanks for catching me up.

Stupid question: could build-essential (and whatever else) be installed with Conda? 🤣

In the end, all I want is a development set-up of MXCuBE-Web that Just Works ™️, with commands I can copy-paste and as few surprises as possible. 😂

[fabcor-maxiv]

Stupid question: could build-essential (and whatever else) be installed with Conda? 🤣

Yes that was what I wondered here as well:

Maybe if it was possible to have the build dependencies of python-ldap in conda-environment.yml there would be a stronger point for having and maintaining a conda-environment.yml.

---

In the end, all I want is a development set-up of MXCuBE-Web that Just Works ™️

Yes, of course, me too. But, from my point of view there is no such thing. Or at least, I have not found out how.

as few surprises as possible

Yes... we thought we had something reasonable, but as you just demonstrated here, we were wrong... :'(

... I am not giving up yet :D