Race condition when initializing ClassAnalyzer with CdiInjecteeSkippingClassAnalyzer #559
Comments
…CdiInjecteeSkippingClassAnalyzer
|
#560 Merge request trying to fix this issue |
|
Looks like something similar: payara/Payara#1160 |
|
It seems it affected much more users, year after year, but I didn't find any explanation except "something uses the container before it finished it's initialization" until now. I believe this is not the issue of HK2, but of the project using Jersey with HK2 where is something wrong. The example project always passed all tests on my computer with JDK17 (except log messages from JaCoCo instrumentation; it passed also in eclipse without these messages). But what finally attracted my eye is the class However I am still convinced that using Simply said - it tries to inject something bound to the The
But then the |
|
Thanks for your follow-up!! I am using Weld 4.0.2 with org.glassfish.jersey.ext.cdi::jersey-weld2-se which is under the hood depends on jersey-cdi1x. It is compatible with Weld 4. I agree it probably comes with the order of initialization which is not deterministic. However it might also come with a concurrency issue with some value in cpu cache not matching the main memory. I already tried to replace hk2 with pure cdi injection but it requires major rework in the way we use them now. Here is all the dependencies | | +- org.jboss.weld.se:weld-se-core:jar:4.0.2.Final:compile
| | | +- org.jboss.weld.environment:weld-environment-common:jar:4.0.2.Final:compile
| | | +- org.jboss.weld.probe:weld-probe-core:jar:4.0.2.Final:compile
| | | \- org.jboss.classfilewriter:jboss-classfilewriter:jar:1.2.5.Final:compile
| | +- org.jboss.weld:weld-core-impl:jar:4.0.2.Final:compile
| | | \- org.jboss.weld:weld-spi:jar:4.0.SP1:compile
| | +- org.jboss.weld:weld-api:jar:4.0.SP1:compile
| | \- jakarta.servlet:jakarta.servlet-api:jar:5.0.0:compile
| \- jakarta.activation:jakarta.activation-api:jar:2.0.1:compile
| +- jakarta.inject:jakarta.inject-api:jar:2.0.0:compile
| +- jakarta.enterprise:jakarta.enterprise.cdi-api:jar:3.0.0:compile
| | \- jakarta.interceptor:jakarta.interceptor-api:jar:2.0.0:compile
| +- jakarta.annotation:jakarta.annotation-api:jar:2.0.0:compile
| \- org.slf4j:slf4j-api:jar:1.7.32:compile
| | +- jakarta.xml.bind:jakarta.xml.bind-api:jar:3.0.1:compile
| | +- com.sun.xml.bind:jaxb-impl:jar:3.0.2:compile
| | | \- com.sun.xml.bind:jaxb-core:jar:3.0.2:compile
| | +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.12.5:compile
| | \- com.fasterxml.jackson.module:jackson-module-parameter-names:jar:2.12.5:compile
| +- io.swagger:swagger-annotations:jar:1.6.2:compile
| +- org.glassfish.jersey.media:jersey-media-json-jackson:jar:3.0.3:compile
| | +- org.glassfish.jersey.core:jersey-common:jar:3.0.3:compile
| | +- org.glassfish.jersey.ext:jersey-entity-filtering:jar:3.0.3:compile
| | +- com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar:2.12.5:compile
| | \- javax.xml.bind:jaxb-api:jar:2.3.1:compile
| \- jakarta.ws.rs:jakarta.ws.rs-api:jar:3.0.0:compile
| | \- org.ehcache:ehcache:jar:3.9.3:compile
| +- org.glassfish.jersey.ext.cdi:jersey-weld2-se:jar:3.0.3:compile
| | +- org.glassfish.jersey.core:jersey-server:jar:3.0.3:compile
| | | \- org.glassfish.jersey.core:jersey-client:jar:3.0.3:compile
| | \- org.glassfish.jersey.ext.cdi:jersey-cdi1x:jar:3.0.3:compile
| +- org.glassfish.jersey.inject:jersey-hk2:jar:3.0.3:compile
| | +- org.glassfish.hk2:hk2-locator:jar:3.0.1:compile
| | | +- org.glassfish.hk2.external:aopalliance-repackaged:jar:3.0.1:compile
| | | +- org.glassfish.hk2:hk2-api:jar:3.0.1:compile
| | | \- org.glassfish.hk2:hk2-utils:jar:3.0.1:compile
| | \- org.javassist:javassist:jar:3.25.0-GA:compile
| +- org.eclipse.jetty:jetty-servlet:jar:11.0.7:compile
| | \- org.eclipse.jetty:jetty-security:jar:11.0.7:compile
| +- org.glassfish.jersey.containers:jersey-container-jetty-servlet:jar:3.0.3:compile
| | +- org.glassfish.jersey.containers:jersey-container-servlet:jar:3.0.3:compile
| | +- org.glassfish.jersey.containers:jersey-container-servlet-core:jar:3.0.3:compile
| | \- org.glassfish.jersey.containers:jersey-container-jetty-http:jar:3.0.3:compile
| +- org.glassfish.jersey.ext:jersey-proxy-client:jar:3.0.3:compile
| +- org.glassfish.jersey.ext:jersey-bean-validation:jar:3.0.3:compile
| +- org.hibernate.validator:hibernate-validator:jar:7.0.1.Final:compile
| | +- org.jboss.logging:jboss-logging:jar:3.4.1.Final:compile
| | \- com.fasterxml:classmate:jar:1.5.1:compile
| +- org.glassfish.jersey.media:jersey-media-multipart:jar:3.0.3:compile
| | \- org.jvnet.mimepull:mimepull:jar:1.9.13:compile
| +- org.eclipse.jetty.http2:http2-server:jar:11.0.7:compile
| | +- org.eclipse.jetty.http2:http2-common:jar:11.0.7:compile
| | | \- org.eclipse.jetty.http2:http2-hpack:jar:11.0.7:compile
| | \- org.eclipse.jetty:jetty-server:jar:11.0.7:compile
| | \- org.eclipse.jetty:jetty-http:jar:11.0.7:compile
| +- org.eclipse.jetty:jetty-alpn-java-server:jar:11.0.7:compile
| | +- org.eclipse.jetty:jetty-io:jar:11.0.7:compile
| | | \- org.eclipse.jetty:jetty-util:jar:11.0.7:compile
| | \- org.eclipse.jetty:jetty-alpn-server:jar:11.0.7:compile
| +- com.github.nhenneaux.jersey.connector.httpclient:jersey-httpclient-connector:jar:1.0.10:compile
| \- com.fasterxml.jackson.core:jackson-annotations:jar:2.12.5:compile
| \- com.github.stefanbirkner:system-lambda:jar:1.1.0:compile
+- org.mockito:mockito-core:jar:3.12.4:compile
| +- net.bytebuddy:byte-buddy:jar:1.11.13:compile
| +- net.bytebuddy:byte-buddy-agent:jar:1.11.13:compile
| \- org.objenesis:objenesis:jar:3.2:compile
+- org.slf4j:slf4j-simple:jar:1.7.32:test
+- jakarta.validation:jakarta.validation-api:jar:3.0.0:compile
+- com.fasterxml.jackson.core:jackson-databind:jar:2.12.5:compile
| \- com.fasterxml.jackson.core:jackson-core:jar:2.12.5:compile
| +- org.reflections:reflections:jar:0.9.11:test
| | \- com.google.guava:guava:jar:30.1-jre:test
| | +- com.google.guava:failureaccess:jar:1.0.1:test
| | +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:test
| | +- com.google.code.findbugs:jsr305:jar:3.0.2:test
| | +- org.checkerframework:checker-qual:jar:3.5.0:test
| | +- com.google.errorprone:error_prone_annotations:jar:2.3.4:test
| | \- com.google.j2objc:j2objc-annotations:jar:1.3:test
| \- org.apache.commons:commons-lang3:jar:3.11:test
+- jakarta.el:jakarta.el-api:jar:4.0.0:compile
+- org.glassfish:jakarta.el:jar:4.0.2:compile
+- org.junit.jupiter:junit-jupiter-engine:jar:5.8.1:test
| +- org.junit.platform:junit-platform-engine:jar:1.8.1:test
| | +- org.opentest4j:opentest4j:jar:1.2.0:test
| | \- org.junit.platform:junit-platform-commons:jar:1.8.1:test
| +- org.junit.jupiter:junit-jupiter-api:jar:5.8.1:test
| \- org.apiguardian:apiguardian-api:jar:1.1.2:test
+- org.awaitility:awaitility:jar:3.1.6:test
| +- org.hamcrest:hamcrest-library:jar:1.3:test
| \- org.hamcrest:hamcrest-core:jar:1.3:test
+- org.hamcrest:hamcrest:jar:2.2:test
\- org.mockito:mockito-junit-jupiter:jar:3.12.4:test
|
|
Hmm, the referred example uses Also I see that in service files of cdi1x project is the CdiComponentProvider twice, once as an extension and once as a ComponentProvider impl - that can probably cause redundant initialization too. Or two standalone instances of the same class used as two standalone services ... If cpu caches would work like this, nothing would work ever ... prove it (the map is still protected by synchronized methods). Btw I was digging in my old examples and I found this: |
|
I'm also getting this error while the applications runs in production. It occurred the 11/01/2022 for a service started the 20/12/2021. |
|
@dmatej I think I found where the problem lie. It occurs when there is a concurrent initialization from Jersey server but also from each new client. Each new client triggers the initialization of injection which makes things worse. |
|
Hi @dmatej |
|
Hi, Any suggestions how to deal with this? |
With the same version of Jetty, HK2 and Jersey? Could you create a reproducer? |
|
Hi @dmatej, I'm afraid thats not possible: the error does not occur for every request, but maybe 4 or 5 times a day, with over a hundred requests per day. We are using Payara Server Enterprise 5.49.0 #badassfish (build 839). |
|
The issue is still happening on a regular basis in CI and production with Jersey 3.1.1 & Jetty 11.0.15. It happens more when the Jersey client is instantiated concurrently. |
|
@nhenneaux, issue happening in this project too? |
|
I don't remember so. It happens when doing concurrent client instantiations and I don't on this project. https://github.com/nhenneaux/jersey-httpclient-connector/blob/master/src/test/java/com/github/nhenneaux/jersey/connector/httpclient/JettyServerTest.java#L412-L413 |
For now several months, we are getting the following error for some of the builds we execute (I estimate it to 1-5% of all the builds).
It uses Jersey 3.0.2, Weld 4.0.2.Final with Java 16.
The error occurs when executing a JUnit test initializing CDI & JAX-RS server, similar to this test https://github.com/nhenneaux/jersey-httpclient-connector/blob/master/src/test/java/com/github/nhenneaux/jersey/connector/httpclient/JettyServerTest.java#L222
I suspect a race condition occurs since the class analyzer uses a HashMap and not a thread safe map (java.util.concurrent.ConcurrentMap). Indeed one CPU core cache could see the map full and another the map empty.
https://github.com/eclipse-ee4j/glassfish-hk2/blob/master/hk2-locator/src/main/java/org/jvnet/hk2/internal/ServiceLocatorImpl.java#L2464
Full stacktrace
The text was updated successfully, but these errors were encountered: