Dependancy issue with pulsar4s and pulsar-functions-local-runner-original/pulsar-client-admin-original

[Haaroon]

Having issues whereby my Local File Reader runner fails to run because of a dependency clash between pulsar4s and the pulsar java libraries.

The code is attempting to load a connector via the soureConfig object

Below is the build.sbt file,

libraryDependencies += "com.clever-cloud.pulsar4s"    % "pulsar4s-core_2.13"               % "2.8.0"
libraryDependencies += "com.clever-cloud.pulsar4s"    % "pulsar4s-circe_2.13"              % "2.8.0"
libraryDependencies += "org.apache.pulsar"            %  "pulsar-client-messagecrypto-bc"  % pulsarVersion
libraryDependencies += "org.apache.pulsar"            %  "pulsar-functions-local-runner-original" % pulsarVersion
libraryDependencies += "org.apache.pulsar"            %  "pulsar-client-admin-original"    % pulsarVersion

Below is the error log

15:19:03.712 [main] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Searching for connectors in /Users/haaroony/Documents/pulsar/pulsar-test/connectors
15:19:03.773 [main] WARN  org.apache.pulsar.functions.utils.io.ConnectorUtils - Failed to load connector from /Users/haaroony/Documents/pulsar/pulsar-test/connectors/pulsar-io-file-2.7.2.nar
java.lang.NoSuchMethodError: 'com.fasterxml.jackson.databind.ObjectMapper org.apache.pulsar.common.util.ObjectMapperFactory.getThreadLocalYaml()'
	at org.apache.pulsar.functions.utils.io.ConnectorUtils.getConnectorDefinition(ConnectorUtils.java:115) ~[pulsar-functions-utils-2.7.2.jar:2.7.2]
	at org.apache.pulsar.functions.utils.io.ConnectorUtils.getConnectorDefinition(ConnectorUtils.java:107) ~[pulsar-functions-utils-2.7.2.jar:2.7.2]
	at org.apache.pulsar.functions.utils.io.ConnectorUtils.searchForConnectors(ConnectorUtils.java:164) [pulsar-functions-utils-2.7.2.jar:2.7.2]
	at org.apache.pulsar.functions.LocalRunner.getConnectors(LocalRunner.java:508) [pulsar-functions-local-runner-original-2.7.2.jar:2.7.2]
	at org.apache.pulsar.functions.LocalRunner.isBuiltInSource(LocalRunner.java:477) [pulsar-functions-local-runner-original-2.7.2.jar:2.7.2]
	at org.apache.pulsar.functions.LocalRunner.start(LocalRunner.java:275) [pulsar-functions-local-runner-original-2.7.2.jar:2.7.2]
	at com.raphtory.FileSpoutLocal$.main(FileSpoutLocal.scala:32) [classes/:?]
	at com.raphtory.FileSpoutLocal.main(FileSpoutLocal.scala) [classes/:?]
Exception in thread "main" java.lang.NoSuchMethodError: 'com.fasterxml.jackson.databind.ObjectMapper org.apache.pulsar.common.util.ObjectMapperFactory.getThreadLocalYaml()'
	at org.apache.pulsar.functions.utils.io.ConnectorUtils.getIOSourceClass(ConnectorUtils.java:59)
	at org.apache.pulsar.functions.utils.SourceConfigUtils.validate(SourceConfigUtils.java:303)
	at org.apache.pulsar.functions.LocalRunner.start(LocalRunner.java:290)
	at com.raphtory.FileSpoutLocal$.main(FileSpoutLocal.scala:32)
	at com.raphtory.FileSpoutLocal.main(FileSpoutLocal.scala)
15:19:03.893 [Thread-0] INFO  org.apache.pulsar.functions.LocalRunner - Shutting down the localrun runtimeSpawner ...

code example

      val sourceConfig = SourceConfig.builder
        .tenant("public")
        .namespace("default")
        .topicName("example-file-io")
        .name("file-source-example")
        .archive("/Users/haaroony/Documents/pulsar/pulsar-test/connectors/pulsar-io-file-2.7.2.nar")
        .configs(configs)
        .build

      val localRunner = LocalRunner.builder
        .brokerServiceUrl("pulsar://localhost:6650")
        .sourceConfig(sourceConfig)
        .build

      localRunner.start(false)

[Haaroon]

fyi if i remove pulsar4s it runs fine

[judu]

Does this fail for every pulsar java client version? Which version are you on?

[Haaroon]

Running the latest 2.8.0 with pulsar 2.9.0, i even tried pulsar 2.8.1 and same issues

[judu]

Out of curiosity: did you try pulsar4s 2.8.0?
It's been released yesterday.
Beware: since the original maintainer gave the project away, the organisation for the artifact is now com.clever-cloud.pulsar4s instead of com.sksamuel.pulsar4s.

[Haaroon]

yes i did and it still doesnt work unfortunately,

[Haaroon]

I have had a similar dependency issue with pulsar before, which is why we add the -original name to the dep which pull the build without the shady deps

[miratepuffin]

I don't think this is a solution solution, but as a quick patch to get this working I excluded the base pulsar libs from pulsar4s in my build and it now runs (I work with Haaroon).

[judu]

That is weird, because pulsar4s 2.8.0 depends on pulsar-client 2.9.0. 🤔

[Haaroon]

It is indeed. I believe client has dependency issues which is why use client-common instead. It gives problems with pulsar functions . Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Julien Durillon ***@***.***> Sent: Thursday, December 9, 2021 5:49:47 PM To: CleverCloud/pulsar4s ***@***.***> Cc: Haaroon Y ***@***.***>; Author ***@***.***> Subject: Re: [CleverCloud/pulsar4s] Dependancy issue with pulsar4s and pulsar-functions-local-runner-original/pulsar-client-admin-original (Issue #371) That is weird, because pulsar4s 2.8.0 depends on pulsar-client 2.9.0. 🤔 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#371 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABWNAOT42RYEJI3JUAHP6ITUQDT3XANCNFSM5JWXUDKQ>.

[judu]

I may investigate this in future versions. If possible, drop the pulsar-client to use the other ones, or add the others with versions pinning.

[judu]

Would you agree to run some tests in local by cloning this repo, mangling the deps to your liking, see if your project works with it and see if the tests still pass? 🙏

[miratepuffin]

Sure, can do. We are getting a wonderful netty issue as well at the moment (below) which also seems to be a dependency problem, so this could help with that.

[judu]

Oh, the much dreaded netty error. Some coworkers got it but we could not find out why. :(

[Haaroon]

we have the netty but it seems to work all fine