[Bug]: beam-sdks-java-io-kafka:2.52.0 - Causes SLF4J Failure : Error loading StaticLoggerBinder: org.slf4j.impl.StaticLoggerBinder

[kuro337]

What happened?

beam-sdks-java-io-kafka:2.52.0 Causing SLF4j Failures

Summary

2.52.0

• When using implementation("org.apache.beam:beam-sdks-java-io-kafka:2.52.0") - JAR has failures for SLF4j Affecting all logging backends

2.51.0

• Downgrading to implementation("org.apache.beam:beam-sdks-java-io-kafka:2.51.0") seems to resolve the issue

• Errors can be misleading and difficult to track the reason

---

Hello!

Wanted to report an observed bug - I realized when building a jar I kept seeing these errors at runtime :

Error loading StaticLoggerBinder: org.slf4j.impl.StaticLoggerBinder                 
Unable to print classpath: jdk.internal.loader.ClassLoaders$AppClassLoader.getURLs()
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

After some digging and isolation - realized this package was causing the Failures

implementation("org.apache.beam:beam-sdks-java-io-kafka:2.52.0") 

I am using logback-classic in my application and creating a JAR - and when I remove this package or downgrade to implementation("org.apache.beam:beam-sdks-java-io-kafka:2.51.0") - my logging works and the ServiceLoader error goes away

Example build.gradle.kts :

plugins {
    id("eventstream.kotlin-application-conventions")
    application
    id("com.github.johnrengelman.shadow") version "8.1.1"
    id("java-library")
}

/*
Below configurations block makes no difference to the outcome - whether commented in or out
*/

configurations {
    all {
        "implementation" {
            exclude(group = "org.slf4j", module = "slf4j-simple")
            exclude(group = "org.slf4j", module = "slf4j-nop")
            exclude(group = "org.slf4j", module = "slf4j-jdk14")
            exclude(group = "org.slf4j", module = "slf4j-log4j12")
        }
    }
}

dependencies {

  /* This Dep is pulled by logback-classic , makes no difference if explicitly included or not (For 2.52.0 Issue) */
    implementation("org.slf4j:slf4j-api:2.0.7")

    implementation("ch.qos.logback:logback-classic:1.4.11")
    implementation("io.github.oshai:kotlin-logging-jvm:5.1.0")
    implementation("org.jetbrains.kotlin:kotlin-reflect:1.9.20")
    implementation("org.apache.commons:commons-text")
    implementation("org.apache.beam:beam-sdks-java-core:2.50.0")
    implementation("org.apache.beam:beam-runners-direct-java:2.50.0")
    implementation("org.apache.beam:beam-runners-flink-1.16:2.50.0")
    implementation("org.apache.beam:beam-sdks-java-io-parquet:2.51.0")
    implementation("org.apache.beam:beam-sdks-java-io-amazon-web-services2:2.51.0")

    /* Problematic Dependency  (2.51.0 Works - 2.52.0 does not work ) */

    implementation("org.apache.beam:beam-sdks-java-io-kafka:2.51.0")

}

application {
    mainClass.set("eventstream.app.AppKt")
}

tasks.shadowJar {
    mergeServiceFiles()

/*
Below Exclude block for shadowjar makes no difference to the outcome - whether commented in or out
*/
    dependencies {
        exclude("org.slf4j:slf4j-simple")
        exclude("org.slf4j:slf4j-nop")
        exclude("org.slf4j:slf4j-jdk14")
        exclude("org.slf4j:slf4j-log4j12")

    }

    duplicatesStrategy = DuplicatesStrategy.EXCLUDE

    transform(com.github.jengelman.gradle.plugins.shadow.transformers.ServiceFileTransformer::class.java)

    isZip64 = true
    archiveFileName.set("KafkaAppLibUsage-all.jar")

    manifest {
        attributes["Main-Class"] = "eventstream.app.AppKt"
    }
}

Logs Working :

28-Nov-23 01:40:43.520 [main] INFO  eventstream.app.App - main - Test Log New 
28-Nov-23 01:40:43.532 [main] INFO  eventstream.app.App - main - Starting Kafka Reader App

So logs work in all scenarios where these things have been tested - because the Error points toslf4j Docs that say this happens due to an issue with the following - which can be misleading

This warning message is reported when the org.slf4j.impl.StaticLoggerBinder class could not be loaded into memory. This happens when no appropriate SLF4J binding could be found on the class path.

Placing one (and only one) of slf4j-nop.jar slf4j-simple.jar, slf4j-log4j12.jar, slf4j-jdk14.jar or logback-classic.jar on the class path should solve the problem.

Note that slf4j-api versions 2.0.x and later use the [ServiceLoader](https://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html) mechanism. Backends such as logback 1.3 and later which target slf4j-api 2.x, do not ship with org.slf4j.impl.StaticLoggerBinder. If you place a logging backend which targets slf4j-api 2.0.x, you need slf4j-api-2.x.jar on the classpath. See also [relevant faq](https://www.slf4j.org/faq.html#changesInVersion18) entry.

Hence have tried all combinations of the following settings -

// Tried with excludes and without - always causes an Error 

    implementation("org.apache.beam:beam-sdks-java-io-kafka:2.52.0"){
        exclude(group = "org.slf4j", module = "slf4j-simple")
            exclude(group = "org.slf4j", module = "slf4j-nop")
            exclude(group = "org.slf4j", module = "slf4j-jdk14")
            exclude(group = "org.slf4j", module = "slf4j-log4j12")
    }
    
tasks.shadowJar {
    mergeServiceFiles()

  // Tried with and Without this 
  
    dependencies {
        exclude("org.slf4j:slf4j-simple")
        exclude("org.slf4j:slf4j-nop")
        exclude("org.slf4j:slf4j-jdk14")
        exclude("org.slf4j:slf4j-log4j12")

    }

    duplicatesStrategy = DuplicatesStrategy.EXCLUDE

    transform(com.github.jengelman.gradle.plugins.shadow.transformers.ServiceFileTransformer::class.java)

    isZip64 = true
    archiveFileName.set("KafkaAppLibUsage-all.jar")

    manifest {
        attributes["Main-Class"] = "eventstream.app.AppKt"
    }
}

// Tried with and Without this

configurations {
    all {
        "implementation" {
            exclude(group = "org.slf4j", module = "slf4j-simple")
            exclude(group = "org.slf4j", module = "slf4j-nop")
            exclude(group = "org.slf4j", module = "slf4j-jdk14")
            exclude(group = "org.slf4j", module = "slf4j-log4j12")
        }
    }
}

So basically the error goes away when downgraded

implementation("org.apache.beam:beam-sdks-java-io-parquet:2.51.0")

But for latest version when used - introduces these failures

implementation("org.apache.beam:beam-sdks-java-io-parquet:2.52.0")

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

---

Other Information:

• Logback.xml

<configuration>
    <!-- ConsoleAppender sends logged messages to the console -->

    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <!-- Pattern layout for logging. %d is the date, %logger is the logger name, %M is the method name, %msg is the message -->
            <pattern>%d{dd-MMM-yy HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %M - %msg%n</pattern>

        </encoder>
    </appender>
    <!-- Root logger configuration -->
    <root level="info">
        <appender-ref ref="STDOUT"/>
    </root>
</configuration>

• Project

Gradle Multi Module Project
Kotlin Compiled to Java 8 and Java 11

Issue Present with ShadowJar and Native Gradle Builds

Tested with Versions of Shadow
-  id("com.github.johnrengelman.shadow") version "8.1.1"
-  id("com.github.johnrengelman.shadow") version "7.1.2"

Gradle and Environment Info - (Also Tested on Linux Ubuntu)

$ ./gradlew --version

------------------------------------------------------------
Gradle 8.3
------------------------------------------------------------

Build time:   2023-08-17 07:06:47 UTC


Kotlin:       1.9.0
Groovy:       3.0.17
Ant:          Apache Ant(TM) version 1.10.13 compiled on January 4 2023
JVM:          17.0.9 (Amazon.com Inc. 17.0.9+8-LTS)
OS:           Windows 10 10.0 amd64

JDK Info:

openjdk version "17.0.9" 2023-10-17 LTS
OpenJDK Runtime Environment Corretto-17.0.9.8.1 (build 17.0.9+8-LTS)
OpenJDK 64-Bit Server VM Corretto-17.0.9.8.1 (build 17.0.9+8-LTS, mixed mode, sharing)

# Used with Java 17 , 19 , and 21 too

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[kuro337]

Feel free to reach out if more info is required , would be glad to help!

[turb]

We also have the same problem updating from 2.51 to 2.52.

It seems slf4j 1.7 shaded in beam-sdks-java-transform-service-launcher jar is suddenly in conflict with slf4j 2 we have. I could not find why it does work with 2.51, that already had those shaded classes.

Our current solution is to depend on both logback-classic 1.4 (what we already had) and slf4j-simple 1.7 with a simplelogger.properties file, just for Apache Beam logs.

[hekk-kazuki-hasegawa]

Is the Beam team aware of this issue?
This is a version upgrade blocker for people using the Beam SDK in a production environment.

[dhs3000]

Looks like this is related or caused by the shadowing reported in #29881

[liferoad]

cc @Abacn

[Abacn]

CC: @chamikaramj