Why is the Storeman Installer so complicated to use?

[Olf0]

When performing the migration from Storeman 0.2.9 to Storeman 0.2.10 (which resulted in issue #1), I wondered why this whole procedure has to be so complex for the user?
I.e. the usage description at https://openrepos.net/content/osetr/storeman-installer lets the user perform multiple steps, especially when migrating from Storeman ≤ 0.2.9!

Specific questions:

1. Why does one have to uninstall Storeman manually, before installing Storeman Installer?
   IMO you could execute in the %post section of the spec file
   if ! ssu lr | grep -qF mentaljam-obs; then
pkcon -y remove storeman || true
rm -f /var/cache/ssu/features.ini && ssu ur && ssu er mentaljam-obs || true
fi
   Edit (2022-12-23): This can and should be achieved via a Conflicts:, Obsoletes: and Provides: triple in the spec file, as implemented in Storeman Installer v1.3.1.
2. Why is it not possible to integrate / append the two lines of the mini shell script into the %post section of the spec file?
   Then the desktop icons etc. would become superfluous, but most importantly, the user does not have to manually start the Storeman Installer (at the GUI or CLI) any more! Storeman just would be installed by installing the Storeman Installer.

Plus two minor questions for checks & balances:

3. Why does the %postun script not ssu rr mentaljam-obs?
Well, it might be better to use a separate repo then (e.g. mentaljam-storeman), but IMO one should properly (and fully) restore the state before the initial installation after an uninstalling.
4. Do I understand correctly that with %{_datadir}/%{ssu_features_dir}/mentaljam-obs.ini in place plus an ssu er mentaljam-obs that is fully equivalent to ssu ar mentaljam-obs http://repo.merproject.org/obs/home:/mentaljam/%(release)_%(arch)/ && ssu er mentaljam-obs?
Due to issue #1 I was searching for an ssu ar command and wondered, if one can really do without it; i.e. if an ssu ar really does nothing more than creating an ini-file?

But as you are much more experienced in the topic "package management on SFOS" than me, I likely missed something crucial.

[Olf0]

Reading https://forum.sailfishos.org/t/ssu-features-are-not-updated-on-removal-of-the-settings-files/7364
I think the answer to question 4 might be "yes", correct?
Plus removing an ini file seems to be supposed to automatically trigger an ssu rr, but that does not work AFAIU (hence one should plan for it to stay in this state for long, because it is Jolla's task to fix this). So that addresses question 3.

Edit: Ah, this comment section explains it: After the own ini file is removed (by the uninstalling proper), the additional rm -f /var/cache/ssu/features.ini && ssu ur results in an the whole action being equivalent to an ssu rr.

[mentaljam]

Specific questions:
1 ...
2 ...

Because the RPM database is locked while all the SPEC scripts are executed. So you cannot call rpm, zypper, or pkcon to install/remove/update packages in the SPEC scripts. My very first idea of the Storeman installer was fully automatic installation. But I had to switch to the desktop icon approach.

[Olf0]

Because the RPM database is locked while all the SPEC scripts are executed. So you cannot call rpm, zypper, or pkcon to install/remove/update packages in the SPEC scripts.

O.K., so it needs some kind of indirection, but (ab)using the user for this is IMHO a bad approach.

Ideas:

• Starting a background ("detached") script per & in the %post section. But that might also not be allowed.
• Deploying a oneshot unit (or a timer unit, which triggers that oneshot unit), which is triggered in the %post section and starts the installation of the right Storeman RPM in 2 seconds.

Generally, I believe this must be a long solved task to let the installation of an RPM trigger a script, which was deployed by this RPM. _{Though I have not started searching the web for this.}

[mentaljam]

PR is welcome😉 I didn't manage to implement it better myself

[Olf0]

But as you are much more experienced in the topic "package management on SFOS" than me, ...

... still I will keep thinking about this.

[mentaljam]

It's more about RPM SPEC files and shell scripting. I tried to detach the script and tried timers (but not units). I think a timer unit is really a good idea. @Olf0, can you try to implement it?

UPD: If you find a working way of automatic installation we can then use it for the Chum GUI!

[Olf0]

UPD: If you find a working way of automatic installation we can then use it for the Chum GUI!

I know. ;-\

[Olf0]

It's more about RPM SPEC files and shell scripting. I tried to detach the script and tried timers (but not units). I think a timer unit is really a good idea. @Olf0, can you try to implement it?

Not anytime soon (except when using Jolla's definition of "soon"): Maybe still in November, likely before the end of the year.

[Olf0]

Sorry, this is becoming Jolla-style "soon": I will not start with this before February, earliest.

Please do not rely on me: If you find the time (and you already have a head-start, as you were experimenting with this in the past), go ahead.

[Olf0]

I think I discovered a way, the one SSU is using:

• sailfishos/ssu@5bd8a09#diff-0f288bc5faebe340bee7a07dc37c9b0840ad44c2f891d3bacc98e38e4755380eR203
• See also line 42 there.
• The ssu-update-repos script is also displayed at the bottom of the page (i.e. it is the second changed file there).

[poetaster]

I think I discovered a way, the one SSU is using:

* [sailfishos/ssu@5bd8a09#diff-0f288bc5faebe340bee7a07dc37c9b0840ad44c2f891d3bacc98e38e4755380eR203](https://github.com/sailfishos/ssu/commit/5bd8a09f2db5b7ebccb01855b297c6350c900a7a#diff-0f288bc5faebe340bee7a07dc37c9b0840ad44c2f891d3bacc98e38e4755380eR203)

* See also line 42 there.

* The `ssu-update-repos` script is also displayed at the bottom of the page (i.e. it is the second changed file there).

If I read this correctly, the approach ssu takes to upgrading itself and manipulating the rpm database actually gives us a road map for moving away from the installer altogether and just doing the needed rpm db updates directly in the harbour-storeman .spec? Is this the gist?

[Olf0]

If I read this correctly, the approach ssu takes to upgrading itself and manipulating the rpm database actually gives us a road map for moving away from the installer altogether and just doing the needed rpm db updates directly in the harbour-storeman .spec? Is this the gist?

No, this is solely about Storeman Installer. Its job is to install the variant of Storeman (CPU-architecture ("arch") and SFOS release) which fits to the device. For this to work under all circumstances, Storeman Installer must stay a noarch RPM.

The gist is, that this approach would allow upon installing Storeman Installer to automatically remove any old Storeman release currently installed and immediately and automatically (i.e. without user interaction) download and install the appropriate Storeman variant (this is what it already does, but the user needs to start Storeman Installer manually).
_{Additionally (and optionally) Storeman Installer could automatically uninstall itself, but currently this final step is carried out by RPM due to Conflicts: and Obsoletes: statements in the spec file of Storeman ≥ 0.2.9, which appears to be more fail safe, hence we might keep it that way.}

[poetaster]

Hmm. Since the installation of Storeman is 'always' initially 'manual', is it really necessary to have a noarch 'installer'? I believe I see the utility if one assumes that users may not know the %arch they are using but that's about it, isn't it?

On both 3.4 phones I have i only just now noticed that they were still 0.2.3. If I recall correctly, those were installed directly from openrepos rpms. And there is nothing currently keeping a user from directly installing storeman, ist there?

As for the SFOS release, do I understand this correctly: it's just because of how obs is being used? To use a chum example:
https://repo.sailfishos.org/obs/sailfishos:/chum/3.2.1.20_i486/ is a repository url that can be dynamically assembled in the .spec.

[Olf0]

Hmm. Since the installation of Storeman is 'always' initially 'manual', is it really necessary to have a noarch 'installer'? I believe I see the utility if one assumes that users may not know the %arch they are using but that's about it, isn't it?

So you consider sending users to build.sailfishos.org/home:mentaljam with the message "Make sure you pick the right RPM" as appropriate?

On both 3.4 phones I have i only just now noticed that they were still 0.2.3. If I recall correctly, those were installed directly from openrepos rpms. And there is nothing currently keeping a user from directly installing storeman, is[n]t there?

There is:
a. There are no Storeman RPMs at OpenRepos, any more.
b. See above reply WRT OBS.

As for the SFOS release, do I understand this correctly: it's just because of how obs is being used?

No, take a look at the Storeman's GH-repo branches: The code has to differ slightly for various SFOS releases.

To use a chum example: https://repo.sailfishos.org/obs/sailfishos:/chum/3.2.1.20_i486/ is a repository url that can be dynamically assembled in the .spec.

Yes, but not for "installing oneself", i.e. Storeman proper. See, now you have been successfully looping back to why Storeman Installer exists.

[poetaster]

So you consider sending users to build.sailfishos.org:home:mentaljam with the message "Be sure to pick the right RPM" as appropriate?

No, I thought the road map leads us to the use of chum. But that is essentially what is stated on the storeman-installer info page in Storeman itself :)

No, take a look at the GH-repo branches: The code differs and must differ.

I just did. The code does not differ (the c++, I mean). Meta data does. But I'm probably missing something.

[poetaster]

Yes, but not for "installing oneself", i.e. Storeman proper. See, now you have been successfully looping back to why Storeman Installer exists.

I'm assuming that you mean installation of the rpm/ssu metadata? All apps are installed by pkcon/rpm/zypper whether or not they are 'installed' by harbour/storeman/chum, or?

[Olf0]

I'm assuming that you mean installation of the rpm/ssu metadata?

No, see #2 (comment).

All apps are installed by pkcon/rpm/zypper whether or not they are 'installed' by harbour/storeman/chum, or?

Well, almost: All package management tools ultimately use libzypp on SailfishOS, except for calling rpm directly (which one should solely use for querying with rpm -q!).

[poetaster]

In any case, as my own 3.4 devices demonstrate, currently a user might not even KNOW there is an update.

[poetaster]

I'm assuming that you mean installation of the rpm/ssu metadata? All apps are installed by pkcon/rpm/zypper whether or not they are 'installed' by harbour/storeman/chum, or?

See #2 (comment)

Which you have found a solution for with the 'self-installation' of ssu :) Now we are going in cricles. I'll stop and try to find some time to take this apart further. I mentioned it to rinigus last night. But this week I won't find any time, I'm afraid.

[poetaster]

Ok. Lastly, I went through the process of removal on 3.4 and install from openrepos. The noarch installer makes perfect sense here. I get it. I had just never noticed, having always installed manually.

[poetaster]

In the interest of actually working on code, @olf, is it possible for you to create a branch here that includes your changes (as per pulls here). Then I would take a crack at the approach that you discerned in
#2 (comment)

[Olf0]

In any case, as my own 3.4 devices demonstrate, currently a user might not even KNOW there is an update.

So what? This is the slightly unfortunate situation mentaljam created about a year ago, because OpenRepos cannot handle releases for different SFOS release versions.

IMO this has deviated far enough from the topic "Why is the Storeman Installer so complicated to use?" to be called "off topic". The existence of Storeman Installer has proper reasons.

No, I thought the road map leads us to the use of chum.

The planned switch from build.sailfishos.org:home:mentaljam to build.sailfishos.org:sailfishos:chum does not resolve the issue of "bootstrapping", which Storeman Installer addresses.
Actually the plan is to create a SailfishOS:Chum Installer (because the current route of installing the SailfishOS:Chum GUI app is even more awkward), as soon as this issue for the Storeman Installer is resolved, using the scheme developed for it.

No, take a look at the GH-repo branches: The code differs and must differ.

I just did. The code does not differ (the c++, I mean). Meta data does. But I'm probably missing something.

Yes IIRC, AFAIR QML code. Even if it is really only metadata, mentaljam, @rinigus and I have not found a better way of handling this, yet. If you do, please describe it.

[poetaster]

Yes IIRC, AFAIR QML code. Even if it is really only metadata, mentaljam, @rinigus and I have not found a better way of handling this, yet. If you do, please describe it.

So, I went through the branches with meld. There is no substantial difference but a puzzle:

The pro file, in current master contains this file: .github/workflows/build.yml \

I'm unable, even after submodule inits to find this.

[Olf0]

Yes IIRC, AFAIR QML code. Even if it is really only metadata, mentaljam, @rinigus and I have not found a better way of handling this, yet. If you do, please describe it.

So, I went through the branches with meld. There is no substantial difference

See https://github.com/storeman-developers/harbour-storeman/wiki/Delta-between-release-branches-and-master

but a puzzle:
The pro file, in current master contains this file: .github/workflows/build.yml \

???
https://github.com/storeman-developers/harbour-storeman/blob/master/harbour-storeman.pro does not contain the line you mentioned, any more (for quite a while, already).
The line was deleted by storeman-developers/harbour-storeman@ca09209

I'm unable, even after submodule inits to find this.

The single submodule is also on its way out (and exists for a completely different purpose).

[Olf0]

poetaster commented on Mar 7:

On both 3.4 phones I have i only just now noticed that they were still 0.2.3. If I recall correctly, those were installed directly from openrepos rpms.

poetaster commented on Mar 7:

In any case, as my own 3.4 devices demonstrate, currently a user might not even KNOW there is an update.

Thank you for reminding me that automatically triggering the installation of Storeman Installer and removing an old Storeman release is also part of this issue, not only the automatic installation of a current Storeman.

I addressed this with Storeman Installer v1.3.0 and with an additional fix in v1.3.1 Storeman Installer provides these new properties:

• While Storeman Installer is installing (requires SailfishOS ≥ 3.1.0), a formerly installed Storeman < 0.3.0 is automatically removed.
• Storeman Installer ≥ 1.3.0 will be offered as an update to Storeman < 0.3.0 (e.g. in Storeman < 0.3.0) on SailfishOS ≥ 3.1.0.
• Storeman < 0.3.0 will likely be automatically updated to Storeman Installer ≥ 1.3.0 when upgrading to SailfishOS ≥ 3.1.0.

The latter two points require to have an RPM repository enabled, which offers Storeman Installer ≥ 1.3.0.

[Olf0]

mentaljam commented on Nov 1, 2021:

It's more about RPM SPEC files and shell scripting.

So it seems I might be the right one to address this. I finally started analysing and addressing this issue, explore solutions etc., because I had some spare time at the start of December 2022; but not as much it ultimately needed!
And "yes", it was solely about RPM packaging, writing shell scripts and systemd units, respectively UNIX process handling know-how (double forking / "daemonizing").

I tried to detach the script and tried timers (but not units). I think a timer unit is really a good idea. @Olf0, can you try to implement it?

I implemented "2½" variants, all basically working nicely: Via a systemd timer unit (which triggers a systemd service unit), a sole systemd service unit and as a fully detached shell script.

• The systemd.timer variant is in the branch defer-inst-via-systemd-timer, its last release is v2.0.30.
  But a sytemd timer is superfluous, because one can perform any waiting via OnActiveSec= in a timer unit alternatively with ExecStartPre=/bin/sleep statements in a service unit.
• The systemd.unit variant is in the branch defer-inst-via-systemd-unit, its last release is v2.0.31.
  But it can only wait for triggering the installation of the harbour-storeman RPM time-based, not event-based (i.e. asynchronously), and systemd correctly complains that the unit file vanished, while it was executing it, because the triggered installation of harbour-storeman removes harbour-storeman-installer, including this file.
  Furthermore, utilising systemd implies other limitations, though it is elegant on a Linux distribution, which relies on systemd (i.e. it is guaranteed to be installed and running).
• The detached.script variant, which was in the branch defer-inst-via-detached-script which has been merged into the devel branch and its commits will be merged into in the master branch for the next release. Its latest release is v2.1.5.
  For details, how it is implemented and why this is IMO the preferable variant, see the section "Motivation" in "Double-fork ("daemonize") in shell code".

Concurrently to developing Storeman Installer v2, which runs "unattended" (i.e. without any manual steps, after its installation has been triggered, until Storeman is installed), I developed these enhancements for Storeman Installer v1, which now resides in the branch v1.x.y-legacy. Its current (and likely last) release is v1.4.3.

• Storeman Installer 1.3.1 and all later versions are offered as an update candidate for Storeman, if an RPM repository is enabled, which offers the harbour-storeman-installer package and Storeman (harbour-storeman package) < 0.2.99 is installed.
• Installing Storeman Installer 1.3.1 and all later versions also automatically removes an installed Storeman (harbour-storeman package) < 0.2.99.
• Storeman Installer 1.3.8 and all later versions create a persistent log file /var/log/harbour-storeman-installer.log.txt.

Side note

For analysing the hierarchy of the spawned processes, specifically the aspects, in which session group they reside, and who the session leader is (oneself or not), I also created a heavily instrumented variant, which is based on the outdated v2.0.22 of the detached.script variant, and resides in the branch defer-via-script-test. Its only release is v2.0.44.

Things left to do in order to finalise this task

• Rewrite README.md to denote the newly acquired properties of Storeman Installer, so the unattended installation of Storeman does not come as a surprise.
• Roll out the current Storeman Installer 2.x.y.

Done by v2.1.6.

[Olf0]

Basically fixed by MR #139, ultimately fixed by release v2.1.6.

[Olf0]

@mentaljam, please upload the RPM of Storeman Installer 1.4.3 1.3.5 (due to this fresh report) to the Storeman Installer page of your OpenRepos repository.

Why v1.4.3 v1.3.5, the latest (and presumably last) a release of Storeman Installer 1?

• The description does not at all fit to Storeman Installer 2's behaviour.
• v1.4.3 received all fixes and features from the v2-branch up to (and including) v2.1.5, except for running fully automatically, and v2.1.6 added only a reworked README (i.e. no code changes).

Reasoning: Though Storeman Installer v1.3.1 and later is offered as an update-candidate for all Storeman < 0.2.99 (on SailfishOS ≥ 3.1.0), all installations of Storeman ≥ 0.2.9 already should have received newer Storeman releases as update-candidates months ago, and one of these Storeman updates should have added the new Storeman repos the at SailfishOS-OBS. But if a user does not have my repository at OpenRepos enabled, someone who has a Storeman < 0.2.9 installed (on SailfishOS ≥ 3.1.0) will never be notified about an available update. Uploading [a] Storeman Installer 1.4.3 ≥ 1.3.1 to your Open Repos repository will resolve that, because people who have an Storeman < 0.2.9 installed, likely also have your OpenRepos repository enabled.

P.S.: May I remind you of this question, as originally posed in the first real paragraph of this message, and later again as point 2 of this message? I would gladly patch them to run well under recent SailfishOS releases; if my know-how turns out to be insufficient, I can usually count on nephros.

[mentaljam]

@Olf0, I am grateful to you for the all the contribution to Storeman that you have made and continue to do. With great regret, I must say that after 8 years "on board" I still returned to android. Now I cannot continue to participate in the development of Storeman. I'll add a notion on the ORN page about discontinues

[Olf0]

For readers: Storeman Issue 290 and these two threads at FSO [1] & [2] provide the necessary context to understand this conversation.

@Olf0, I am grateful to you for the all the contribution to Storeman that you have made and continue to do.

As usual with Open Source Software developed in one's spare time, this is self-motivated and has been (partially; not the SailfishOS:Chum debate) fun.
I have been concluding my work on Storeman Installer and Storeman since the end of November 2022 up to the releases of Storeman 0.3.2 (three weeks ago) and Storeman Installer 1.4.3 / 2.1.6 (a few days ago). In the future I will be mostly handling contributions (if there are any).

With great regret, I must say that after 8 years "on board" I still returned to android.

That is really sad!

I'll add a notion on the ORN page about discontinues

Please, let us ponder a little bit, how to implement that:

• "Discontinue statements" on project pages at ORN are not at all helpful for users, as long as they do not redirect users to a new location. Hence such a message on your personal ORN page is fine, the field "TMO profile" in your account settings is basically a free text field which can be (ab)used for that.
• On your Storeman Installer project page, I would appreciate, if you copy the statement from my comment (in its original or any altered form, as long the abstract message and link is kept intact) to the project page proper. Done (2022-12-26), thank you!
  … after uploading the Storeman Installer 1.4.3 RPM there! 😉
• What about Recorder, Mashka and Beerware (again)?
  My suggestion:
  1. Transfer these projects to storeman-developers at GitHub.
  2. I would create Recorder, Mashka and Beerware (draft) project pages at ORN, if you agree to proceeding as suggested here.
  3. You can add a "discontinue statement" with a link to the new project pages at ORN.

[Olf0]

@mentaljam, the "discontinue statement" on your Storeman Installer project page is fine.
But still, please do upload the RPM of v1.4.3 v1.3.5 (due to this fresh report) there!

For Recorder, Mashka and Beerware, I quickly created these project page placeholders you can link to:

• Recorder: https://openrepos.net/content/olf/recorder
• Mashka: https://openrepos.net/content/olf/mashka
• Beerware: https://openrepos.net/content/olf/beerware

Please mind to also transfer the three corresponding GitHub projects (Recorder, Mashka and Beerware) to Storeman Developers, because that is much more elegant than cloning them ("fork" in GH-parlance), as transferring also moves the issue tracker, technically redirects HTTP-requests to the old location etc.

Thank you for these nice pieces of software and helping to migrate them, so I can try to keep them alive!