Git scripts repository update fails

[sobuj53]

Expected behavior

After opening PhoenicisPlayOnLinux and moving to app tab, it should download app list for PlayOnLinux.

Actual behavior

Opened the app and waiting for to 30 min. it never really fetch any data from internet and shows the loading animation.

Steps to reproduce

Install phoenicis-playonlinux 5.0_alpha2-3 app from org.manjaro.pamac.manager and move to app tab.

System information

• software version (git revision) of phoenicis and scripts repositories: phoenicis-playonlinux 5.0_alpha2-3,
  Repo:

1. git+https://github.com/PhoenicsOrg/script:master
2. classpath:///org/phoenics/repository

• used operating system: Manjaro
• used Java version: how do i find that out?

[plata]

If you use the current master of the scripts repository, that won't work with the alpha 2 software. Either you must use the alpha 2 scripts version or use the current master of the software.

[sobuj53]

Thank you for your replay. I didn't change anything in the repo. Just went with the default. Would you mind telling me which url to use? I'm bit confused with

you must use the alpha 2 scripts version or use the current master of the software.

didn't fully grasp the meaning.
FYI: I've updated to the latest version but the issue is still there.

[madoar]

FYI: I've updated to the latest version but the issue is still there.

Where did you get the latest version from?

Phoenicis alpha-1 and alpha-2 each use a frozen version of the scripts. The reason for this is that the script format is still work in progress and can change between the alpha versions. Therefore if you want to access the newest version of the scripts you need to build Phoenicis your self. This will allow you to use the newest version of the scripts. If you choose this option please be aware that you need to keep your build up to date manually to ensure that you stay compatible with future script updates. Also please be aware that you this only applies for now. Later on we plan to finalize the script format, which will allow you update the scripts independent of the used Phoenicis version. We are just not that far yet :)

[sobuj53]

Sorry for being late. the issue has been resolved in latest release.

[qparis]

Same problem here with Java 14. Stuck on:
[INFO ] org.phoenicis.repository.types.GitRepository (l.145) - Pulling new commits from git-repository https://github.com/PhoenicisOrg/scripts, local folder: /Users/qparis/Library/Phoenicis/cache/git-331995911, branch: master

[qparis]

In fact, same problem with Java 11.
Jgit seems very unstable to me. I would suggest another alternative to handle script downloading

[plata]

Any suggestions?

[madoar]

@qparis is it possible that there are some kinds of permissions missing? I think it is strange that jgit is "only" stuck but does not throw an exception/error

[qparis]

I have completely wiped out my ~/Library/Phoenicis home directory and the problem is still the same.

Any suggestions?

Could we simply download an artifact archive from github and extract it?

[madoar]

We can do that but it would make updating the scripts more difficult later on.

[qparis]

Why?

[madoar]

In my opinion calling a git pull is easier compared to writing some custom logic to:

1. check if the github archive has changed
2. download the archive
3. extract it
4. replace all non-modified scripts (to allow for customization by the user)

In addition we would need to download all scripts instead of only the scripts that are really out of date.

[qparis]

Git is very slow to perform these operation. We have so many problems that it makes Phoenicis hardy impossible to run.
Git is meant to store source file, whereas we mostly have (is size) binary files. Our script repository is about 25Mb, 90% are images that could be lazy loaded later on.
I think that we should be as stateless as possible, and handle script loading exactly the same way a browser handles asset loading.

Concerning your fourth point, this is why we have a local repository implementation. The user should never update directly a file

[qparis]

Miniatures are not part of the required elements to use Phoenicis, this is why I strongly encourage to lazy load them

[madoar]

I agree, but I don't think that this is something that needs to be tackled right now. There are other areas I think we should prioritize like overhauling the internal repository structure and the way we access the repositories (#2107) and issues like #2104. As far as I understand it the git repository is used now because of its easy setup and its huge functionality. Later on when we are actively thinking about leaving the beta releases (which we have not entered yet) we should add a more sophisticated and performant repository type (i.e. a web repository which would fetch the scripts from a predefined source).

Until we reach this point in time I would recommend that we fix the currently used git repository.

[qparis]

The real problem is for macOS. Legacy version of PlayOnMac do no longer run and users are waiting for something working.

The script directory structure is not something that prevent the software from being used ; whereas the git design seems to be a blocker for many users

[madoar]

The real problem is for macOS. Legacy version of PlayOnMac do no longer run and users are waiting for something working.

I understand this, but in my opinion I don't think that this is a sufficient reason for swapping our current git repository approach with something else. In my opinion it is easier and faster to first search for an error in the current implementation. If we can not find an error we should then open an issue in the jgit bugtracker (maybe we are not the only ones who have a problem with it on macOS?). Only afterwards I believe we should think about changing the git repository approach to something else.

[qparis]

Totally agree.
However, this is not the first time jgit is causing troubles. What about using native git command?

[qparis]

@plata I'd like to have your though here. Java IDE tend to use native git command. I wonder if there's a legitimate reason not to use jgit for them

[plata]

In general I would prefer a native Java solution because it will reduce our system dependencies. However, if this doesn't work properly, we should go for whatever works. I would really be interested what exactly is causing the issue in jgit in this case. I cannot really say much about this issue because I'm always using my own git checkout (local repository).

[plata]

I'm not sure anymore that this issue is really related to git. I rather have the impression that the view update does not work anymore if the content has not been loaded before the view is created. For example, I also cannot see a new category in the library after installing a new app (until I restart). Similarly, I couldn't see any apps after removing repositoryManager.triggerCallbacks() in ContainersController and MainController (log output says that repository is being loaded when opening the apps tab).
@madoar could this be related to the MVC restructuring somehow?

[madoar]

@plata is this still the case after #2149?

The UI components depend on some component telling them what information they should display.
The process of informing the UI components is done by simply adding the information to ObservableLists that are managed by the UI components. There are two ways to "update" the UI:

• trigger a reload of the repositories (this is done after startup to show the apps etc after startup)
• add the information manually (this is done for example when the "install" button is pressed in the "Apps" view to open an installation dialog)

[Zemogiter]

I have this issue using latest commit for both repos

[WARNING] 
java.lang.NoSuchMethodError: 'boolean org.phoenicis.repository.RepositoryManager.isRepositoryLoaded()'
    at org.phoenicis.javafx.controller.MainController.lambda$new$0 (MainController.java:80)
    at com.sun.javafx.binding.ExpressionHelper$Generic.fireValueChangedEvent (ExpressionHelper.java:360)
    at com.sun.javafx.binding.ExpressionHelper.fireValueChangedEvent (ExpressionHelper.java:80)
    at javafx.beans.property.ReadOnlyBooleanPropertyBase.fireValueChangedEvent (ReadOnlyBooleanPropertyBase.java:72)
    at javafx.beans.property.ReadOnlyBooleanWrapper.fireValueChangedEvent (ReadOnlyBooleanWrapper.java:103)
    at javafx.beans.property.BooleanPropertyBase.markInvalid (BooleanPropertyBase.java:111)
    at javafx.beans.property.BooleanPropertyBase.set (BooleanPropertyBase.java:145)
    at javafx.scene.control.Tab.setSelected (Tab.java:189)
    at javafx.scene.control.TabPane$TabPaneSelectionModel.select (TabPane.java:738)
    at javafx.scene.control.TabPane$TabPaneSelectionModel.select (TabPane.java:751)
    at javafx.scene.control.TabPane$TabPaneSelectionModel.select (TabPane.java:672)
    at com.sun.javafx.scene.control.behavior.TabPaneBehavior.selectTab (TabPaneBehavior.java:82)
    at javafx.scene.control.skin.TabPaneSkin$TabHeaderSkin$5.handle (TabPaneSkin.java:1469)
    at javafx.scene.control.skin.TabPaneSkin$TabHeaderSkin$5.handle (TabPaneSkin.java:1447)
    at com.sun.javafx.event.CompositeEventHandler.dispatchBubblingEvent (CompositeEventHandler.java:86)
    at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent (EventHandlerManager.java:238)
    at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent (EventHandlerManager.java:191)
    at com.sun.javafx.event.CompositeEventDispatcher.dispatchBubblingEvent (CompositeEventDispatcher.java:59)
    at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent (BasicEventDispatcher.java:58)
    at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent (EventDispatchChainImpl.java:114)
    at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent (BasicEventDispatcher.java:56)
    at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent (EventDispatchChainImpl.java:114)
    at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent (BasicEventDispatcher.java:56)
    at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent (EventDispatchChainImpl.java:114)
    at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent (BasicEventDispatcher.java:56)
    at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent (EventDispatchChainImpl.java:114)
    at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent (BasicEventDispatcher.java:56)
    at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent (EventDispatchChainImpl.java:114)
    at com.sun.javafx.event.EventUtil.fireEventImpl (EventUtil.java:74)
    at com.sun.javafx.event.EventUtil.fireEvent (EventUtil.java:54)
    at javafx.event.Event.fireEvent (Event.java:198)
    at javafx.scene.Scene$MouseHandler.process (Scene.java:3862)
    at javafx.scene.Scene.processMouseEvent (Scene.java:1849)
    at javafx.scene.Scene$ScenePeerListener.mouseEvent (Scene.java:2590)
    at com.sun.javafx.tk.quantum.GlassViewEventHandler$MouseEventNotification.run (GlassViewEventHandler.java:409)
    at com.sun.javafx.tk.quantum.GlassViewEventHandler$MouseEventNotification.run (GlassViewEventHandler.java:299)
    at java.security.AccessController.doPrivileged (Native Method)
    at com.sun.javafx.tk.quantum.GlassViewEventHandler.lambda$handleMouseEvent$2 (GlassViewEventHandler.java:447)
    at com.sun.javafx.tk.quantum.QuantumToolkit.runWithoutRenderLock (QuantumToolkit.java:412)
    at com.sun.javafx.tk.quantum.GlassViewEventHandler.handleMouseEvent (GlassViewEventHandler.java:446)
    at com.sun.glass.ui.View.handleMouseEvent (View.java:556)
    at com.sun.glass.ui.View.notifyMouse (View.java:942)
    at com.sun.glass.ui.gtk.GtkApplication._runLoop (Native Method)
    at com.sun.glass.ui.gtk.GtkApplication.lambda$runLoop$11 (GtkApplication.java:277)
    at java.lang.Thread.run (Thread.java:834)

[plata]

@Zemogiter this error indicates that you do not use the latest software (only scripts).

[madoar]

I have thought about the problems with our GitRepository a bit more over the last days.
I think that the best way to "fix" the issue is to provide a WebRepository which we can use also after final release.

In my opinion the WebRepository should provide the following:

• a REST interface which allows communication between a server and a Phoenicis client
• a versioning scheme for the scripts and other related files like the images/icons

I think this can be achieved if we provide at least 2 REST endpoints:

• an endpoint to grab a summary of the contents of the repository, e.g. a list of all applications and scripts, images and categories contained in the repository along with their corresponding latest version (maybe represented by an update timestamp or a hash)
• an endpoint to grab a specific version of a script or image (i.e. resource)

The benefit of this approach is that we could easily provide some kind of tags that allow us to freeze a version for the scripts for our alpha releases if this is necessary.

@plata @qparis what do you think?

[plata]

@qparis had planned the WebRepository for release anyways. Maybe we even have an issue about it already.
I'm not sure if REST is the best option or something else or how it works in PoL4.

[madoar]

I can reproduce the issue that @qparis described on Linux Mint 19.3 using a clean installation of Phoenicis using the .deb installer.

[plata]

That's good. Can you debug what exactly is going on?

[madoar]

I will give it a bit more time to fetch the repository because it seems as if Phoenicis is still doing something with around 50% CPU usage.

[madoar]

Ok the issue seems to be related to the .deb creation process. If you start phoenicis from IntelliJ it is working correctly. If you start it from a .deb installation it hangs.

[madoar]

More detailed it hangs on:

Git gitRepository = Git.cloneRepository()
                    .setURI(this.repositoryUri.toString())
                    .setDirectory(this.localFolder)
                    .setBranch(this.branch).call()

[plata]

Is it the command or the thread which executes the command?

[madoar]

What do you mean by the command?

[plata]

The git clone.

[madoar]

The freeze should occur in the call() operation. The operations before simply fill a builder object

[plata]

Yes, my question is: Does the call() cause the freeze? Or does the thread which executes the cal() stop for some reason such that the call() cannot be executed?

[madoar]

I can't tell. I think that it hangs somewhere inside the call() method

[plata]

Mh. How could we figure out what's causing the problem?

[madoar]

Do we have a logging configuration file?
If yes we could maybe add a setting to change the log level for jgit, see https://github.com/eclipse/jgit/blob/master/org.eclipse.jgit.test/tst-rsrc/log4j.properties?

[plata]

There is something in phoenicis-configuration.

[madoar]

@qparis out of interest because you face the jgit problem as well:
How are you starting phoenicis? Do you start it through maven, IntelliJ or by using a packaged installer from our script?

[qparis]

I tried mvn and a packaged installer

[madoar]

@qparis starting phoenicis via mvn exex:java and IntelliJ and work for me as expected, i.e. the git repository is correctly cloned. Only starting phoenicis through the packaged installer hangs during git repository cloning for me.

Do both options fail for you or only the packaged installer?

[qparis]

Both option fail