[AI] Add more uses statements #11788

koppor · 2024-09-18T20:44:35Z

Fixes https://github.com/JabRef/jabref-issue-melting-pot/issues/538

Still not working

An error occurred while building the embedding model
java.lang.RuntimeException: Unable to find the embedding model by the URL djl://ai.djl.huggingface.pytorch/sentence-transformers/all-MiniLM-L12-v2
	at [email protected]/org.jabref.logic.ai.ingestion.model.UpdateEmbeddingModelTask.call(Unknown Source)
	at [email protected]/
...
Caused by: ai.djl.repository.zoo.ModelNotFoundException: ModelZoo doesn't support specified engine: PyTorch
	at [email protected]/ai.djl.repository.zoo.Criteria.resolveModelLoaders(Unknown Source)
	at [email protected]/ai.djl.repository.zoo.Criteria.loadModel(Unknown Source)
	at [email protected]/org.jabref.logic.ai.ingestion.model.DeepJavaEmbeddingModel.<init>(Unknown Source)
	... 10 more

Refs deepjavalibrary/djl#3473

Mandatory checks

Change in CHANGELOG.md described in a way that is understandable for the average user (if applicable)
Tests created for changes (if applicable)
Manually tested changed features in running JabRef (always required)
Screenshots added in PR description (for UI changes)
Checked developer's documentation: Is the information available and up to date? If not, I outlined it in this pull request.
Checked documentation: Is the information available and up to date? If not, I created an issue at https://github.com/JabRef/user-documentation/issues or, even better, I submitted a pull request to the documentation repository.

koppor · 2024-09-18T23:10:20Z

Tested locally:

An error occurred while building the embedding model
java.lang.RuntimeException: Unable to find the embedding model by the URL djl://ai.djl.huggingface.pytorch/sentence-transformers/all-MiniLM-L12-v2
	at [email protected]/org.jabref.logic.ai.ingestion.model.UpdateEmbeddingModelTask.call(Unknown Source)
	at [email protected]/org.jabref.logic.ai.ingestion.model.UpdateEmbeddingModelTask.call(Unknown Source)
	at [email protected]/org.jabref.gui.util.UiTaskExecutor$1.call(Unknown Source)
	at javafx.graphics@23/javafx.concurrent.Task$TaskCallable.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: ai.djl.repository.zoo.ModelNotFoundException: ModelZoo doesn't support specified engine: PyTorch
	at [email protected]/ai.djl.repository.zoo.Criteria.loadModel(Unknown Source)
	at [email protected]/org.jabref.logic.ai.ingestion.model.DeepJavaEmbeddingModel.<init>(Unknown Source)
	... 10 more

Siedlerchr · 2024-09-19T04:37:59Z

The uses statements are not enough, you need to add the provides (for the services loading mechanism) to the build gradle, see my comment in the melting pot. It's about the merged module.
That's what the error message is saying

Use the suggested merge module gradle task to add the correct provides

build.gradle

koppor · 2024-09-19T06:46:30Z

Use the suggested merge module gradle task to add the correct provides

No "real" change:

         uses 'ai.djl.engine.EngineProvider'
         uses 'ai.djl.repository.RepositoryFactory'
         uses 'ai.djl.repository.zoo.ZooProvider'
-        uses 'dev.langchain4j.spi.prompt.PromptTemplateFactory'
         uses 'kong.unirest.core.json.JsonEngine'
+        uses 'org.controlsfx.glyphfont.GlyphFont'
+        uses 'org.eclipse.jgit.lib.GpgSignatureVerifierFactory'
         uses 'org.eclipse.jgit.lib.GpgSigner'
         uses 'org.eclipse.jgit.transport.SshSessionFactory'
+        uses 'org.glassfish.hk2.extension.ServiceLocatorGenerator'
         uses 'org.mariadb.jdbc.LocalInfileInterceptor'
-        uses 'org.mariadb.jdbc.authentication.AuthenticationPlugin'
-        uses 'org.mariadb.jdbc.credential.CredentialPlugin'
-        uses 'org.mariadb.jdbc.tls.TlsSocketPlugin'
+        uses 'org.postgresql.shaded.com.ongres.stringprep.Profile'

+        requires 'com.fasterxml.jackson.annotation'
+        requires 'com.fasterxml.jackson.databind'
+        requires 'com.fasterxml.jackson.core'
+        requires 'com.fasterxml.jackson.datatype.jdk8'
+        requires 'com.google.errorprone.annotations'
         requires 'com.google.gson'
+        requires 'com.googlecode.javaewah'
+        requires 'com.sun.xml.txw2'
+        requires 'jakarta.activation'
+        requires 'jakarta.annotation'
+        requires 'jakarta.inject'
+        requires 'jakarta.validation'
+        requires 'jakarta.ws.rs'
         requires 'jakarta.xml.bind'
         requires 'java.compiler'
         requires 'java.datatransfer'
         requires 'java.desktop'
+        requires 'java.instrument'
         requires 'java.logging'
         requires 'java.management'
         requires 'java.naming'
         requires 'java.net.http'
         requires 'java.rmi'
-        requires 'java.scripting'
         requires 'java.security.jgss'
-        requires 'java.security.sasl'
         requires 'java.sql'
         requires 'java.sql.rowset'
         requires 'java.transaction.xa'
         requires 'java.xml'
         requires 'javafx.base'
         requires 'javafx.controls'
         requires 'javafx.fxml'
         requires 'javafx.graphics'
         requires 'javafx.media'
         requires 'javafx.swing'
-        requires 'jdk.jsobject'
+        requires 'jdk.attach'
+        requires 'jdk.jdi'
+        requires 'jdk.management'
         requires 'jdk.security.jgss'
         requires 'jdk.unsupported'
-        requires 'jdk.unsupported.desktop'
-        requires 'jdk.xml.dom'
+        requires 'kotlin.stdlib'
+        requires 'org.apache.commons.codec'
+        requires 'org.apache.commons.compress'
+        requires 'org.apache.commons.io'
         requires 'org.apache.commons.lang3'
         requires 'org.apache.commons.logging'
         requires 'org.apache.commons.text'
         requires 'org.freedesktop.dbus'
+        requires 'org.glassfish.grizzly'
+        requires 'org.glassfish.grizzly.http'
+        requires 'org.glassfish.grizzly.http.server'
+        requires 'org.jetbrains.annotations'
         requires 'org.jsoup'
         requires 'org.slf4j'
+        requires 'org.yaml.snakeyaml'

Thus, leaving that part as is.

koppor · 2024-09-19T06:55:54Z

Tested locally; works 😅

koppor · 2024-09-19T06:59:34Z

java.lang.IllegalAccessError: superclass access check failed: class io.github.stefanbratanov.jvm.openai.AssistantsToolChoiceSerializer (in module org.jabref.merged.module) cannot access class com.fasterxml.jackson.databind.ser.std.StdSerializer (in module com.fasterxml.jackson.databind) because module org.jabref.merged.module does not read module com.fasterxml.jackson.databind

Siedlerchr · 2024-09-19T07:04:23Z

build.gradle

@@ -738,48 +741,52 @@ jlink {
    // This is based on ./gradlew suggestMergedModuleInfo, sort, strip ";"", comment non-used modules, and include the suggested directives here.
    //   However, we need to fine-tune this manually (non-alphabetic order)
    mergedModule {
-        requires 'javafx.base'


For the xth time! Do not reorder the dependencies here!

see the comment above in the build.gradle!

I wanted to find out the reason, while I am on it. We did not document it back than. We just said: Some local experiments showd that it does not work.

I tried today with alphabetic ordering. It works. - Having alphabetic ordering in place helps to include the output of ./gradlew suggestMergedModuleInfo. Otherwise, it is an endless brainfuck.

If it does not, we have enough time before a release.

build.gradle

github-actions · 2024-09-19T10:33:02Z

The build for this PR is no longer available. Please visit https://builds.jabref.org/main/ for the latest build.

koppor · 2024-09-19T12:12:45Z

Works fine here. @ThiloteE Can you test, too?

ThiloteE · 2024-09-19T12:44:46Z

Doesn't work on my Windows 10 with Nvidia Geforce GTX 1060 3GB with gradlew run.

2024-09-19 14:42:46 [JavaFX Application Thread] org.jabref.logic.ai.ingestion.model.JabRefEmbeddingModel.lambda$startRebuildingTask$1()
2024-09-19 14:42:46 [JavaFX Application Thread] org.jabref.gui.JabRefDialogService.notify()
ERROR: An error occurred while building the embedding model: ai.djl.engine.EngineException: Failed to load PyTorch native library
INFO: An error occurred while building the embedding model
        at [email protected]/ai.djl.pytorch.engine.PtEngine.newInstance(PtEngine.java:90)
        at [email protected]/ai.djl.pytorch.engine.PtEngineProvider.getEngine(PtEngineProvider.java:41)
        at [email protected]/ai.djl.engine.Engine.getEngine(Engine.java:190)
        at [email protected]/ai.djl.Model.newInstance(Model.java:99)
        at [email protected]/ai.djl.repository.zoo.BaseModelLoader.createModel(BaseModelLoader.java:224)
        at [email protected]/ai.djl.repository.zoo.BaseModelLoader.loadModel(BaseModelLoader.java:165)
        at [email protected]/ai.djl.repository.zoo.Criteria.loadModel(Criteria.java:151)
        at [email protected]/org.jabref.logic.ai.ingestion.model.DeepJavaEmbeddingModel.<init>(DeepJavaEmbeddingModel.java:23)
        at [email protected]/org.jabref.logic.ai.ingestion.model.UpdateEmbeddingModelTask.call(UpdateEmbeddingModelTask.java:65)
        at [email protected]/org.jabref.logic.ai.ingestion.model.UpdateEmbeddingModelTask.call(UpdateEmbeddingModelTask.java:20)
        at [email protected]/org.jabref.gui.util.UiTaskExecutor$1.call(UiTaskExecutor.java:188)
        at javafx.graphics@23/javafx.concurrent.Task$TaskCallable.call(Task.java:1401)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.UnsatisfiedLinkError: C:\Users\Thilo\.djl.ai\pytorch\2.4.0-cu124-win-x86_64\torch_cuda.dll: Can't find dependent libraries
        at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2418)
        at java.base/java.lang.Runtime.load0(Runtime.java:852)
        at java.base/java.lang.System.load(System.java:2025)
        at [email protected]/ai.djl.pytorch.jni.LibUtils.loadNativeLibrary(LibUtils.java:379)
        at [email protected]/ai.djl.pytorch.jni.LibUtils.loadLibTorch(LibUtils.java:195)
        at [email protected]/ai.djl.pytorch.jni.LibUtils.loadLibrary(LibUtils.java:82)
        at [email protected]/ai.djl.pytorch.engine.PtEngine.newInstance(PtEngine.java:53)
        ... 17 more

Siedlerchr · 2024-09-19T13:47:13Z

seems like your Caused by: java.lang.UnsatisfiedLinkError: C:\Users\Thilo.djl.ai\pytorch\2.4.0-cu124-win-x86_64\torch_cuda.dll: Can't find dependent libraries is still missing

Did you install the VC++ stuff?
http://djl.ai/engines/pytorch/pytorch-engine/#windows

ThiloteE · 2024-09-19T13:59:34Z

@Siedlerchr yes, indeed. I have opened an issue here: https://github.com/JabRef/jabref-issue-melting-pot/issues/537. This is the third or fourth pull-request I am commenting on :D Yes, I have tried what you suggested already. See https://github.com/JabRef/jabref-issue-melting-pot/issues/537#issuecomment-2357959861

…xArxivHtmlImport * 'fixArxivHtmlImport' of github.com:JabRef/jabref: Fix focus for keywords and crossref fields (#11792) Fix ai chat not on fx thread (#11796) [AI] Add more uses statements (#11788) Update djl api dependency (#11787) Improve pdf content parser for DOIs (#11782) minor refactor to JabRefDialogService (#11767) Add more OS-dependent context to panel freeze dev documentation. (#11781)

koppor added 2 commits September 18, 2024 22:43

Add more uses statements

068be0d

Merge branch 'main' into fix-mp-540

ebce8d9

koppor marked this pull request as draft September 18, 2024 21:01

koppor added 2 commits September 18, 2024 23:09

Make use of BOM

110e1a0

Add requires 'ai.djl.pytorch_model_zoo'

8aaa849

ThiloteE added the AI Related to AI Chat/Summarization label Sep 18, 2024

ThiloteE changed the title ~~Add more uses statements~~ [AI] Add more uses statements Sep 18, 2024

Trying downgrade of djl

abbc2df

koppor mentioned this pull request Sep 18, 2024

pin djl to 0.29.0 #11789

Closed

6 tasks

Siedlerchr reviewed Sep 19, 2024

View reviewed changes

build.gradle Show resolved Hide resolved

koppor added 3 commits September 19, 2024 08:39

Sort "requires" alphabetically

c6afa40

Add org.freedesktop.dbus as requriement

399d00e

Sort "uses" alphabetically

22c21f8

Add "provides"

173cc49

Update djl to 0.30.0

9a7716a

Siedlerchr reviewed Sep 19, 2024

View reviewed changes

Fix jackson requirements

ae5749a

Siedlerchr reviewed Sep 19, 2024

View reviewed changes

build.gradle Show resolved Hide resolved

subhramit added the dependencies Pull requests that update a dependency file label Sep 19, 2024

koppor marked this pull request as ready for review September 19, 2024 12:12

Siedlerchr approved these changes Sep 19, 2024

View reviewed changes

Siedlerchr added this pull request to the merge queue Sep 19, 2024

Merged via the queue into main with commit c5e81a9 Sep 19, 2024
30 of 32 checks passed

Siedlerchr deleted the fix-mp-540 branch September 19, 2024 18:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AI] Add more uses statements #11788

[AI] Add more uses statements #11788

koppor commented Sep 18, 2024 •

edited

Loading

koppor commented Sep 18, 2024

Siedlerchr commented Sep 19, 2024

koppor commented Sep 19, 2024

koppor commented Sep 19, 2024

koppor commented Sep 19, 2024 •

edited

Loading

Siedlerchr Sep 19, 2024

Siedlerchr Sep 19, 2024

koppor Sep 19, 2024

github-actions bot commented Sep 19, 2024 •

edited

Loading

koppor commented Sep 19, 2024

ThiloteE commented Sep 19, 2024 •

edited

Loading

Siedlerchr commented Sep 19, 2024 •

edited

Loading

ThiloteE commented Sep 19, 2024

[AI] Add more uses statements #11788

[AI] Add more uses statements #11788

Conversation

koppor commented Sep 18, 2024 • edited Loading

Mandatory checks

koppor commented Sep 18, 2024

Siedlerchr commented Sep 19, 2024

koppor commented Sep 19, 2024

koppor commented Sep 19, 2024

koppor commented Sep 19, 2024 • edited Loading

Siedlerchr Sep 19, 2024

Choose a reason for hiding this comment

Siedlerchr Sep 19, 2024

Choose a reason for hiding this comment

koppor Sep 19, 2024

Choose a reason for hiding this comment

github-actions bot commented Sep 19, 2024 • edited Loading

koppor commented Sep 19, 2024

ThiloteE commented Sep 19, 2024 • edited Loading

Siedlerchr commented Sep 19, 2024 • edited Loading

ThiloteE commented Sep 19, 2024

koppor commented Sep 18, 2024 •

edited

Loading

koppor commented Sep 19, 2024 •

edited

Loading

github-actions bot commented Sep 19, 2024 •

edited

Loading

ThiloteE commented Sep 19, 2024 •

edited

Loading

Siedlerchr commented Sep 19, 2024 •

edited

Loading