Formulae names & dependencies

[luispuerto]

Here are some ideas we should keep in mind about the renaming of the formulae and if we should bring dependencies into the tap. This issue is highly related to #769.

Any opinion is highly welcomed. @fjperini and specially @miguelangelperg that seems that it's suffering from "name issues".

---

• We need to keep the number of formulae in the tap as low as possible.
  More formulae → more to maintain → more work → more difficulty to keep everything in shape.
• We should try to rely as much as possible into core formulae.
  • We don't have to maintain it.
  • Core formulae is more wide available to other people so it's easier to detect bugs.
• We are the Hombrew tap of the OSGeo project, so should try to host just the formulae related to OSGeo projects. In the same way, shouldn't we host all the OSGeo projects related formulae instead of host them in core?
• Any other package in the tap should fall into these two categories:
  • It's a dependency or it enhances the software in some way and it isn't in the core.
    • Can we add it to Homebrew core?
    • Do we really need to bring that formula to our tap?
  • It's a dependency or it enhances the software in some way and it is in the core, but it doesn't fulfill our needs.
    • Do we really need to bring that formula to our tap?
    • Can't we really use core formula as it's?
    • Can we propose a PR over that formulae that will fulfill our necessities while while keeping the current requirements of Homebrew core?
• There are currently some OSGeo projects that currently have formulae in the core tap. For example gdal, pdal, postgis... We have some of those formulae cloned in this tap with a different name so it doesn't shadowed the core formulae. What we should do?
  • Does it have any sense that there are two formulas that do the same?
  • Since it's OSGeo software, shouldn't we ask for deletion the formula in core and keep it here instead?
  • Can we propose a PR over that formulae that will fulfill our necessities while while keeping the current requirements of Homebrew core?
• Whether it's a OSGeo project package formulae or not, we need to keep in mind the following if we decide to duplicate:
  • Core is going to have always preference, since depends_on: in any core or other tap formulae unless otherwise is specify is going to use core formulae. Probably they are going to be linked too first.
  • They are thinking to deprecate pinned taps.
  • We probably need to keep our formula keg only so it doesn't interfere with other formulae that could use the core version, but this is something we need to decide.
• Name of formulae should be consistent:
  • If we duplicate a core formulae we should name the formula something like osgeo-<formula_name>.
  • Try to avoid numbers, even less if they are related to the version. The formula always is the last version of the software, any other version is <formula-name>@<version-number>
• We probably can't figure part of this on our own and we need to contact people on Homebrew and in OSGeo.
  • In Homebrew: to know their opinion about all of these and how they can help, and if they are willing to.
  • In OSGeo: I really don't know who has create gdal, pdal and other formulae relate to OSGeo projects in core, but I guess OSGeo project have some opinion about if formulae for their project should be in core or in osgeo4mac taps.

Thoughts?

[fjperini]

To take into consideration: Homebrew/homebrew-core#31510

[luispuerto]

I think we should also check what are they take into duplicate formulae and so. Either we ask in the issue tracker of core or here: https://discourse.brew.sh

[miguelangelperg]

Very timely reflection.

Beyond the difficulties involved, I especially agree with

• Less is more

"We need to keep the number of formulae in the tap as low as possible."

Now, if this is the official tap of the Osgeo project, this tap must be the host of the formulas related to it

On the duplicity of formulas.

What options are not available in the core formulas? Why are they not available?
What advantages do these options offer?
It is worth having the formulas dulicadas to have these options, or can we wait for the problems that now prevent them to be active in the nucleus to be solved?

[fjperini]

Sorry for the inconvenience. But know understand that we decided to rename formulas to make other things simpler.
The process took time due to the restrictions in CI and the delay of Homebrew Core in updating many of the dependencies, for example Qt and PROJ (a few hours ago). This leads to rebuild all the bottles.
For this reason and another is that we were working on optimizing the tap for the future.

p/d: I will be rebuilding everything for PROJ 6.

[fjperini]

we are working to migrate the bottles to osgeo, bintray already caused many problems.

[grischard]

We need to keep the number of formulae in the tap as low as possible.

We should try to rely as much as possible into core formulae.

Yes please! I currently use osgeo4mac to install qgis. Its 294 dependencies include:

osgeo-gdal
osgeo-gdal-python
osgeo-hdf4
osgeo-hexer
osgeo-laszip@2
osgeo-laz-perf
osgeo-libgeotiff
osgeo-libkml
osgeo-liblas
osgeo-libspatialite
osgeo-matplotlib
osgeo-netcdf
osgeo-opencollada
osgeo-openscenegraph
osgeo-pcl
osgeo-pdal
osgeo-postgresql
osgeo-proj
osgeo-pyqt
osgeo-qt-webkit
osgeo-sip
osgeo-vtk

This is quite a lot. Why does qgis want to overwrite the core postgresql, and why does qgis have a hard requirement on postgresql anyway?

Installing qgis also installs lighttpd, mysql, openldap, gtk, unbound and e2fsprogs. I don't understand why, especially the last one.

[fjperini]

@grischard For several reasons. Mainly to not have as many flags in the formulas and to provide more supports. In this way the user will not have to build on his machine if he requires optional supports (but required by many).

Also to have more control of when to update, there are many dependencies that are updated very late and inopportunely. And that they do not provide all their functions.

For example, for PostgreSQL the versions of homebrew-core do not work for a migration process.

We are having problems with Bintray, that's why the work stopped. Tonight we will remate everything.

p/d: yes, e2fsprogs can be removed, since it is for linux.

[grischard]

I'm really thankful for the qgis packages, but I feel misunderstood as an user. I don't get control of when to update, I'm married to osgeo4mac's postgresql and proj and gdal and so on. I'm currently sitting on a broken osgeo4mac qgis that can't find libproj.13.dylib in osgeo4mac's proj, and wondering if installing all these osgeo formulae won't just break something else.

Yeah, I got bitten by that postgresql migration thing myself too, but isn't the right way to go about this to fix the core?

Can qgis be built with, e.g., mysql support but not include the whole server as a dependency?

[fjperini]

All secondary dependencies will be on the tap.
For this reason, everything will work correctly.

I'm already working to load everything again http://bottle.download.osgeo.org/

Once updated you will realize that everything works very well.

[fjperini]

to temporarily fix proj: ln -s libproj.15.dylib libproj.13.dylib

[grischard]

Yup, that gets qgis to load, thanks.

I see what you're trying to do. But the way I had to symlink libproj.15.dylib to libproj.13.dylib, the way PyQt5.QtCore seems to be missing as a dependency which produces an error when opening, the way python produces a second error when opening because it can't find the module called 'qgis', scares me.

[fjperini]

Just wait, it will be a few hours and everything will return to normal.

I really regret the inconvenience, but we had problems with Bintray in the middle of the tap update process.

[grischard]

I'll be patient then. Thank you!

[fjperini]

I have finished configuring everything! I will start to migrate the bottles from Bintray and update the missing ones.

[luispuerto]

@grischard I really don't know what is your specific problem, but as thing are right now, you can make it work qgis with a little bit of tweaking. At least it's working for me.

Anyhow.... wait a little bit and probably in the next hours everything is going to back to normal.

[fjperini]

During the morning I will transfer the bottles that are in bintray to osgeo.org.

After eating I will sit down to complete the updates.

Everything will return to normal, in the future we will not have these problems.

[fjperini]

I have already built qgis on my machine, I just need to fix something in orfeo. Later I apply all the changes.

[fjperini]

already building qgis! #1063

[fjperini]

the bottles for qgis are ready!

[chinigo]

Thank you so much for your hard work!

I just nuked my previous installation of qgis + dependencies and reinstalled from scratch, everything looks good.

[grischard]

Thank you. So I can now nuke any unlinked foo package where qgis required I install osgeo-foo?

[fjperini]

@grischard yes, we will use our versions for some dependencies.

[luispuerto]

Yeah!!! @fjperini thanks a lot for all the work these last weeks, we have a really great version of QGIS up and running thanks to you!

[grischard]

I'm not sure how things are supposed to be installed cleanly now. QGIS depends on ffmpeg (why?) which depends on openjpeg, which conflicts with osgeo-insighttoolkit on /usr/local/lib/pkgconfig/libopenjp2.pc. My osm2pgsql requires, directly or indirectly, postgres, gdal and proj, and qgis installs its own version. All this for QGIS. Hmm.

Is the long-term plan to have qgis in the main homebrew repository?

[luispuerto]

Is the long-term plan to have qgis in the main homebrew repository?

No, this is in anyway possible now with the current Homebrew core policy... QGIS is an app so there is no place for it in Core. It has to be build and has several dependencies... so it doesn't fit either in Cask.

I know that it's annoying but it's what it's

I'm not sure how things are supposed to be installed cleanly now. QGIS depends on ffmpeg (why?) which depends on openjpeg, which conflicts with osgeo-insighttoolkit on /usr/local/lib/pkgconfig/libopenjp2.pc. My osm2pgsql requires, directly or indirectly, postgres, gdal and proj, and qgis installs its own version. All this for QGIS. Hmm.

I think, @fjperini can give you a more comprehensive answer about it since he has engineered and knitted the formulae... However, I can give you some insights:

• The dependency tree is quite vast, so it's quite difficult to keep things under control. You just have to type brew deps --tree osgeo-qgis to see it.
• The idea of bring some formulae in-house was to try to have more control about how QGIS was build and try to easy things for you all and for the maintainers.
• Core doens't allow options anymore so... if you want to build with some capabilities you need to have your own formulae.
• Some of us have in mind to continue doing changes, talk with core and osgeo so we can bring more order and sense to the mess all of this is.
• As always this is a volunteering work... we do what we can do in our free time.

We are really open so suggestions, feedback and even more to people to participate in the tap. In other words, help is wanted!

[luispuerto]

PS/ in the opinion of some of us, we have to best QGIS in years using Homebrew, and I would say that it's even better than the one you can install from the official installer. Thanks to @fjperini we don't have to build anymore in our machines to have most the capabilities. This doesn't mean that it's perfect, just better. We want to make it even better 😉 and easier, but QGIS is complicated.

[grischard]

It's definitely a lot better than all the other methods we had before!

[fjperini]

@LeonardAukea you can start by uninstalling these dependencies:

  proj
  libgeotiff
  libspatialite
  netcdf
  open-scene-graph

and install these:

 osgeo-pyqt-webkit
 osgeo-sip
 osgeo-pyqt

[LeonardAukea]

@fjperini Ok, thanks. this is what sort of happens:

~ » brew uninstall proj                                                                                   00:03:24

Error: Refusing to uninstall /usr/local/Cellar/proj/6.0.0
because it is required by gdal, libgeotiff, liblas, libspatialite, osgeo-gdal-python, osgeo-grass, osgeo-liblas, osgeo-orfeo, osgeo-ossim, osgeo-pyspatialite, osgeo-qgis-res, osgeo-saga-lts, osgeo-taudem and pdal, which are currently installed.
You can override this and force removal with:
  brew uninstall --ignore-dependencies proj
~ »                                                                                                       00:03:36
~ » brew uninstall --ignore-dependencies proj                                                             00:03:48
Uninstalling /usr/local/Cellar/proj/6.0.0... (56 files, 13MB)
~ » brew uninstall --ignore-dependencies libgeotiff                                                       00:04:01
Uninstalling /usr/local/Cellar/libgeotiff/1.5.1... (39 files, 576.9KB)
~ » brew uninstall --ignore-dependencies libspatialite                                                    00:04:11
Uninstalling /usr/local/Cellar/libspatialite/4.3.0a_7... (31 files, 18.6MB)
~ » brew uninstall --ignore-dependencies netcdf                                                           00:04:23
Uninstalling /usr/local/Cellar/netcdf/4.6.3... (85 files, 6.2MB)
~ » brew uninstall --ignore-dependencies open-scene-graph                                                 00:04:49
Uninstalling /usr/local/Cellar/open-scene-graph/3.6.3... (8,893 files, 229.2MB)
~ » brew install osgeo-pyqt-webkit                                                                        00:06:20
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 1 tap (brewsci/bio).
==> Updated Formulae
brewsci/bio/minimap2

Warning: osgeo/osgeo4mac/osgeo-pyqt-webkit 5.12.1 is already installed and up-to-date
To reinstall 5.12.1, run `brew reinstall osgeo-pyqt-webkit`
~ » brew reinstall osgeo-pyqt-webkit                                                                      00:06:56
==> Reinstalling osgeo/osgeo4mac/osgeo-pyqt-webkit
==> Installing dependencies for osgeo/osgeo4mac/osgeo-pyqt-webkit: osgeo-sip and osgeo-pyqt
==> Installing osgeo/osgeo4mac/osgeo-pyqt-webkit dependency: osgeo-sip
==> Downloading https://bottle.download.osgeo.org/osgeo-sip-4.19.15.mojave.bottle.1.tar.gz

curl: (22) The requested URL returned error: 502 Bad Gateway
Error: Failed to download resource "osgeo-sip"
Download failed: https://bottle.download.osgeo.org/osgeo-sip-4.19.15.mojave.bottle.1.tar.gz
Warning: Bottle installation failed: building from source.
==> Downloading https://www.riverbankcomputing.com/static/Downloads/sip/sip-4.19.15.tar.gz

curl: (22) The requested URL returned error: 404 Not Found
Error: An exception occurred within a child process:
  DownloadError: Failed to download resource "osgeo-sip"
Download failed: https://www.riverbankcomputing.com/static/Downloads/sip/sip-4.19.15.tar.gz

I've got similar url error codes earlier.

[fjperini]

@LeonardAukea solved the problem with osgeo-sip #1102

[LeonardAukea]

@fjperini Thanks. but something seems to be messy still. Looks like the packages without the osgeo-prefix are still a dependency to the new naming convention. Or am I mistaken?

Warning: You have unlinked kegs in your Cellar.
Leaving kegs unlinked can lead to build-trouble and cause brews that depend on
those kegs to fail to run properly once built. Run `brew link` on these:
  osgeo-sip
  osgeo-pyqt

Warning: Some installed formulae are missing dependencies.
You should `brew install` the missing dependencies:
  brew install libgeotiff libspatialite netcdf open-scene-graph proj

Run `brew missing` for more details.
~/ » brew missing                                                                                                                                                                                                23:11:47
gdal: proj libgeotiff libspatialite netcdf
liblas: proj libgeotiff libspatialite netcdf
osgeo-gdal-python: proj libgeotiff libspatialite netcdf
osgeo-grass: proj libgeotiff netcdf libspatialite
osgeo-liblas: proj libgeotiff libspatialite netcdf
osgeo-orfeo: netcdf proj libgeotiff open-scene-graph libspatialite
osgeo-ossim: open-scene-graph proj libgeotiff libspatialite netcdf
osgeo-pyspatialite: proj libspatialite
osgeo-qgis-res: proj libgeotiff libspatialite netcdf
osgeo-saga-lts: proj netcdf libgeotiff libspatialite
osgeo-taudem: proj libgeotiff libspatialite netcdf
pcl: netcdf
pdal: proj libgeotiff libspatialite netcdf
vtk: netcdf

Linking the packages leads to error sins e.g sip is present: should I overwrite?

And if I want to e.g reinstall osgeo-grass:

~/ » brew reinstall osgeo-grass                                                                                                                                                                                  23:18:27
osgeo-postgresql: You have other linked versions!

Unlink with brew unlink postgresql or remove with brew uninstall --ignore-dependencies postgresql

osgeo-gdal: You have other linked versions!

Unlink with brew unlink gdal or remove with brew uninstall --ignore-dependencies gdal

Error: Unsatisfied requirements failed this build.

[luispuerto]

@LeonardAukea

@fjperini Thanks. but something seems to be messy still. Looks like the packages without the osgeo-prefix are still a dependency to the new naming convention. Or am I mistaken?

there are tons of packages without the osgeo-* prefix that are still a dependency. We just brought to the tap the most problematic and important ones.

Warning: You have unlinked kegs in your Cellar.
Leaving kegs unlinked can lead to build-trouble and cause brews that depend on
those kegs to fail to run properly once built. Run `brew link` on these:
  osgeo-sip
  osgeo-pyqt

Warning: Some installed formulae are missing dependencies.
You should `brew install` the missing dependencies:
  brew install libgeotiff libspatialite netcdf open-scene-graph proj

Run `brew missing` for more details.
~/ » brew missing                                                                                                                                                                                                23:11:47
gdal: proj libgeotiff libspatialite netcdf
liblas: proj libgeotiff libspatialite netcdf
osgeo-gdal-python: proj libgeotiff libspatialite netcdf
osgeo-grass: proj libgeotiff netcdf libspatialite
osgeo-liblas: proj libgeotiff libspatialite netcdf
osgeo-orfeo: netcdf proj libgeotiff open-scene-graph libspatialite
osgeo-ossim: open-scene-graph proj libgeotiff libspatialite netcdf
osgeo-pyspatialite: proj libspatialite
osgeo-qgis-res: proj libgeotiff libspatialite netcdf
osgeo-saga-lts: proj netcdf libgeotiff libspatialite
osgeo-taudem: proj libgeotiff libspatialite netcdf
pcl: netcdf
pdal: proj libgeotiff libspatialite netcdf
vtk: netcdf

Linking the packages leads to error sins e.g sip is present: should I overwrite?

You can have both sips if you want, but our recommendation is to have just one... of course you can have linked just one of them and if you update osgeo-qgis is going to install, link and so on and so forth the one from this tap.

If you want to have both sips you are probably going to need to juggle a little bit with them, in other words decide which one you one linked and/or over the other and cope with the consequences.

I'm doing the same with r since I don't have the homebrew-core install on my computer. So each time I reinstall/update osgeo-qgis I have to rename that keg so osgeo-qgis install r from core, and then delete that keg and replace with the one I like.

And if I want to e.g reinstall osgeo-grass:

~/ » brew reinstall osgeo-grass                                                                                                                                                                                  23:18:27
osgeo-postgresql: You have other linked versions!

Unlink with brew unlink postgresql or remove with brew uninstall --ignore-dependencies postgresql

osgeo-gdal: You have other linked versions!

Unlink with brew unlink gdal or remove with brew uninstall --ignore-dependencies gdal

Error: Unsatisfied requirements failed this build.

With this is the same... you still have the other versions installed...

If you are using those formulae just with QGIS the recommendation is to uninstall all the other and let osgeo-qgis do it's business during the install.

[Emic37]

The idea is to have a QGIS as complete as possible. For this reason so many dependencies are installed, ffmpeg corresponds to osgeo-grass.

@fjperini, a bit like @grischard I don't understand what are all these dependencies for ? I understand the idea but maybe that these formulae install a Qgis version which is too complete ? On my computer, qgis-ltr and his dependencies use 11 Go of disk space! It's considerably more than the 2Go of data installed by official advanced installer for a base qgis-ltr installation on windows computer.

Of course, it's a base installation without many add-ons but that's enough for me. For me, less is better. I would prefer can choose to install or not osgeo-saga-lts, osgeo-grass, osgeo-lastools, osgeo-taudem, r, and surely more.

Maybe formulae should be base on the official installer with similar options for more simplicity ?

And if the advanced user want use a specific add-on, he can install it using an option or simply install "optional" package.

[fjperini]

Ok! In the course of the day I will be doing some tests.

[fjperini]

We will be working, some inconveniences with proj6 and gdal3 arose.
Do not update at the moment.

[fjperini]

@grischard is almost decided, we will use gdal 2.4.1 and proj 5.0.0.
Just wait until I apply all the changes.

[fjperini]

Keep in mind that for osgeo-proj, osgeo-gdal, osgeo-gdal-* and osgeo-libgeotiff you will have to uninstall and reinstall. And then continue with the other updates.

In case homebrew does not update these packages

[kevinstadler]

While I really appreciate the ability to install QGIS from Homebrew and all the work that must have gone into this, I just want to add an enormous +1 to the request for cutting down the gigantic dependencies tree.

I understand that all the dependencies are required for building the QGIS Bottle with support for all the modules on the server, but is it really necessary to specify them as dependencies when installing the Formula on the user side? Even if QGIS was built with support for certain modules, it's still possible to run the build without most of the current dependencies (as I'm currently doing), and it would only throw up appropriate error messages once the users attempts to use a functionality that requires other libraries. You say that a large part of the community uses many of the tools, but I doubt that even a power user ever makes use of even half of those dependencies. Things like the full postgresql and mysql servers aren't actually required by anyone who doesn't know how to install them themselves (apart from the fact that the dependency should really only be some (client) header files and not the full server formula, no?).

I gave up full installation about half way through the dependencies, then just installed with --ignore-dependencies and now have a seemingly fully functional QGIS running. If you want I can try to figure out where in the dependency tree I aborted the installs so that other users can also make use of a much smaller installation this way.

[fjperini]

@kevinstadler we will create an option for those that decide a simpler installation, first we are solving other problems with proj and gdal.

Just wait for the process to complete.

[grischard]

Hurray to the option for a simpler installation with less dependencies. Thank you @fjperini for all your hard work!

[kevinstadler]

Thanks so much! Just one more slightly unrelated question, is there a specific formula for installing QGIS with Python plugin support? For me the plugin menu just says "no Python support detected" even though I'm pretty sure I have all necessary dependencies, and I've also manually put a plugin in the profile/default/python/plugins folder but it's not showing up. I'm using the osgeo-qgis (not LTR) formula right now.

[fjperini]

@kevinstadler It should work. You could share the plugin that you put so I try it.

[kevinstadler]

@fjperini thanks for your help! It's the profiletool plugin: https://plugins.qgis.org/plugins/profiletool/
I've put the extracted profiletool/ folder into what I think is the appropriate subfolder in the active QGIS profile:
/Users/kevin/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/

The python/ folder already existed but only had an expressions/ subdirectory, so I had to create the plugins/ one myself.

[fjperini]

@kevinstadler After updating QGIS I will check this.

[fjperini]

@kevinstadler I have checked this and the plugin works correctly.

cd /usr/local/Cellar/osgeo-qgis/3.6.3/QGIS.app/Contents/Resources/python/plugins/

git clone https://github.com/PANOimagen/profiletool

[kevinstadler]

I just updated to 3.6.3 as well and cloned the profiletool plugin into the same directory as you, but I can still only see core plugins in the plugin manager.

Just to check, does the "Settings" tab also look like this for you?

During installation I also get the following message, maybe this has something to do with it?

==> Pouring osgeo-qgis-3.6.3.high_sierra.bottle.tar.gz
Warning: osgeo/osgeo4mac/osgeo-qgis dependency gcc was built with a different C++ standard
library (libstdc++ from clang). This may cause problems at runtime.
GRASS 7 GrassUtils.py already updated
Whitebox Tools whiteboxProvider.py already updated
TauDEM taudemProvider.py already updated
LAStools LAStoolsProvider.py already updated
Warning: The post-install step did not complete successfully
You can try again using `brew postinstall osgeo/osgeo4mac/osgeo-qgis`

Even with --verbose I don't get any more information about why exactly the step "did not complete successfully"

[fjperini]

@kevinstadler The message is just a message, it is not an error.
As for the plugins window, try to delete the QGIS configuration ~/Library/Application\ Support/QGIS, it may be due to that.

If you can share the result of brew doctor
to check if you have a problem with any dependency.

[kevinstadler]

Nevermind, I think I got it:

../src/app/qgisapp.cpp:10483 : (loadPythonSupport) [101ms] load library /usr/local/opt/osgeo-qgis/QGIS.app/Contents/MacOS/lib/qgispython (3.6.3)
../src/core/qgsmessagelog.cpp:29 : (logMessage) [10ms] 2019-05-23T19:47:40 [1] Couldn't load Python support library: Cannot load library /usr/local/opt/osgeo-qgis/QGIS.app/Contents/MacOS/lib/qgispython: (dlopen(/usr/local/opt/osgeo-qgis/QGIS.app/Contents/MacOS/lib/libqgispython.dylib, 10): Library not loaded: /usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/Python
  Referenced from: /usr/local/opt/osgeo-qgis/QGIS.app/Contents/MacOS/lib/libqgispython.dylib
  Reason: image not found)

With so many different Pythons installed I was sure there would be a 3.7 among them but apparently not (and it doesn't seem to be among the osgeo-qgis dependencies?). I'm just running brew install python37 right now, I hope that will do the trick!

Edit: I actually had a 3.7 installed in /usr/local/Cellar/python3/3.7.3 but for some reason this wasn't linked anywhere, the hard-coded /usr/local/opt/python/Frameworks/Python.framework/Versions/ path only had a 3.6 subdirectory.

[fjperini]

@kevinstadler Yes, you need to have linked python.

brew link python --force

[zbeekman]

👋 Howdy! Homebrew maintainer here, also in the process of reviewing a submission to JOSS. I haven't read this entire thread, but just wanted to weigh RE: depending on packages in core, duplicating packages, option removal.

Dependencies in homebrew-core

I would highly recommend you try to depend on formulae in home-brew core since it's very well tested, predictable, and plays nice with other brewed software. In addition, when a dependency of a package is updated, all the formula that use it are tested and the linkage is tested.

Option removal

I understand the desire to provide flexibility, but keep in mind that there is a combinatorial explosion of possible builds if a package has options and it's dependencies have options, and their dependencies, etc. Since this change, the rate of errors reported by users has significantly dropped, and the vast majority of software gets installed from a bottle which has been tested in a very controlled and reproducible environment.

If you see a package in homebrew core where you wish there was an option that was previously removed, then there are two possible recourses for regaining the missing functionality:

1. If the missing option does not conflict with the current default build and simply adds additional functionality, you are welcome and encouraged to submit a pull request making it enabled by default. Unless it breaks something or adds a very deep dependency tree it should get accepted
2. For cases where an option would conflict with a default or where it doesn't simply add features but changes the existing functionality, homebrew-core will consider formulae duplicated but built with other options, if they were popular in the past

The ultimate goal is to provide the most functionality possible, while minimizing problems.

Duplicating packages

You're always welcome to duplicate packages in your tap. However, if software is licensed with an open-source license, there's little recourse for removing packages from homebrew-core. The arguments above, IMO, favor trying to contribute as much as you can upstream to core for better testing and leveraging existing infrastructure and expertise.

I definitely don't want to re-litigate the option remove discussion, but wanted to share my $0.02 here, while reporting an issue I encountered.

[grischard]

@zbeekman as far as I understand, the beginning of all this was that a .app could only be installed as a cask, and a cask couldn't have tons of dependencies like osgeo-qgis has. Is our understanding of homebrew-core's policies correct?

[luispuerto]

@zbeekman thanks a lot for your contribution to the discussion, we really appreciate!! However, as @grischard has said, QGIS is a a little bit special —as almost anything in GIS— so it's a little bit more complicated.

We really have in mind to try to make as reliant in Core as possible, mainly for the reasons you stated, but things take time and we at this moment decided to head in this direction because we thought was going to be quicker.

[zbeekman]

Hi all, sorry for my slow response, I've been underwater.

@grischard

@zbeekman as far as I understand, the beginning of all this was that a .app could only be installed as a cask, and a cask couldn't have tons of dependencies like osgeo-qgis has. Is our understanding of homebrew-core's policies correct?

Yes, this is correct. However, the whole point of an app bundle to bundle the required dynamic libraries and other assets within the app bundle. CMake has fairly good and extensive support for doing this. Apple has the install_name_tool that lets you copy dylibs into your app bundle and re-write paths in executables and dylibs to find the bundled libraries. Furthermore, tools exist out there to "freeze" python applications and bundle them and their dependencies as apps. There was a discussion on discourse.brew.sh here about this: https://discourse.brew.sh/t/python-gui-app-formula-core-or-cask/3863. In particular https://github.com/pyinstaller/pyinstaller looks like a promising tool to help with this. Given the rather extreme complexity of the OSGeo codebase/toolchain, a standalone app bundle really makes the most sense to me here. I don't know of any other .app that has external dependencies.

@luispuerto

We really have in mind to try to make as reliant in Core as possible, mainly for the reasons you stated, but things take time and we at this moment decided to head in this direction because we thought was going to be quicker.

Sure, I understand sometimes time pressures require you to implement something as a temporary work around. I would just like to point out that while your implementation does an OK job of not shadowing core formulae, there are a number of problems, like #1198. Opening and parsing every single formula at once is not a good idea, and will cause breakage for many users when they hit the upper limit on max open files. In addition, it's super slow, so just processing the Formula takes a long time, nevermind waiting for a long list of from-source installs.

So my (completely unsolicited) advice is:

1. Do everything in your power to "freeze" and bundle qgis as a stand alone app. (And anything else with a crazy amount of complexity and a GUI.)
2. Try to reduce the number of options in osgeo formulae as much as possible by enabling sane defaults that are as broadly useful as possible, and shipping multiple versions of the same formula, where needed, if there are conflicting options.
3. Try to run brew audit --strict from time to time to attempt to keep up, where possible, with upstream Homebrew best practices and current, non-deprecated or removed functionality.
4. Enabled by 2 and 3, where possible open PRs to port changes in shadowed formulae or alternate versions of formulae back upstream to homebrew-core. I realize this can be difficult because homebrew-core is opinionated about style & maintainability. But where you're able to do this, you shift much of the maintenance burden of formulae to the greater community and the homebrew maintainers, and you're able to leverage the robust Homebrew CI environment and ship bottled, tested binaries to users.

You're welcome to take it or leave it. But, I'm a Homebrew maintainer and if I'm running into problems trying to install your software (see #1198) then users most certainly are. See, also, for example, https://twitter.com/manlius84/status/1139490041791733765. It looks like the default homebrew-core gdal was not useful for the user, but in general from source builds, and builds with options trigger their dependencies to be built from source, and lead to severe system breakage and misconfiguration for some users. And then they often go blaming homebrew.

At any rate, I'm not trying to argue here. I just wanted to point out that there may be ways to improve user experience and your experience maintaining osgeo if you're able to upstream stuff, eliminate those "what if", or "just because it's nice" options, and in general work on reducing complexity if not conforming with typical homebrew-core practices for formulae.

You're welcome to completely ignore my advice, of course, I won't take it personally.

Thanks for your hard work maintaining osgeo, and for taking the time to read my suggestions.

[fjperini]

@zbeekman @grischard @luispuerto The issue goes through experience: use and building of QGIS. As it is currently it solves most of the problems and you should only have a few things in consideration.

I agree to have a smaller version for those who require it.

We could start with this.

[grischard]

I'm very happy with the constructive way this discussion is going. Thank you all!

There are some core casks that do have dependencies - for example mactex or ibm-cloud-cli - but nowhere near as complex as this here.

I did try to look at the differences between the formulas to see how much work @zbeekman's step 4 would potentially be, but it looks like most are quite significant rewrites. @fjperini did you write down somewhere why you forked a particular formula?

[grischard]

http://blog.qgis.org/2019/07/29/introducing-new-qgis-macos-packages/ is pretty big, no?

[Emic37]

@grischard The last brew upgrade broke again my qgis installation. Since a few months, I encountered a lot of problems with these formula, despite all the efforts of @fjperini.
So, when I saw that, I immediatly tried. All functionnalities I use works perfectly and the size of the app is around 1,9 Go !

So, I'm happy ! I save about 10 Go of free space using this package instead of the osgeo formulae ! And these official packages should be follow up for the update.