Quarkus 3.10 issue with OpenTelemetry?

[matzew]

Describe the bug

We get a unti test failure when trying to bump to Quarkus 3.10 (from 3.8).

java.lang.AssertionError: 
Expected size: 3 but was: 2 in:
[SpanData{spanContext=ImmutableSpanContext{traceId=b3af604008204ef273725923bf652a00, spanId=59f302b88a8e9bb0, traceFlags=01, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=true}, parentSpanContext=ImmutableSpanContext{traceId=b3af604008204ef273725923bf652a00, spanId=410e1d8a6485bd43, traceFlags=01, traceState=ArrayBasedTraceState{entries=[]}, remote=true, valid=true}, resource=Resource{schemaUrl=null, attributes={service.name="unknown_service:java", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.32.0"}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=io.vertx, version=null, schemaUrl=null, attributes={}}, name=POST, kind=SERVER, startEpochNanos=1721749113517586397, endEpochNanos=1721749113534142078, attributes=AttributesMap{data={http.request.method=POST, url.path=/broker-ns/broker-name, http.status_code=202, http.scheme=http, messaging.message_source=/hello, messaging.message_type=type, http.method=POST, http.response.status_code=202, http.url=http://localhost:8083/broker-ns/broker-name, messaging.message_id=1234}, capacity=128, totalAddedValues=10}, totalAttributeCount=10, events=[], totalRecordedEvents=0, links=[], totalRecordedLinks=0, status=ImmutableStatusData{statusCode=UNSET, description=}, hasEnded=true},
    SpanData{spanContext=ImmutableSpanContext{traceId=b3af604008204ef273725923bf652a00, spanId=410e1d8a6485bd43, traceFlags=01, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=true}, parentSpanContext=ImmutableSpanContext{traceId=00000000000000000000000000000000, spanId=0000000000000000, traceFlags=00, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=false}, resource=Resource{schemaUrl=null, attributes={service.name="unknown_service:java", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.32.0"}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=io.vertx, version=null, schemaUrl=null, attributes={}}, name=POST, kind=CLIENT, startEpochNanos=1721749113491976322, endEpochNanos=1721749113538305773, attributes=AttributesMap{data={http.request.method=POST, http.status_code=202, http.method=POST, url.full=http://localhost:8083/broker-ns/broker-name, http.response.status_code=202, http.url=http://localhost:8083/broker-ns/broker-name}, capacity=128, totalAddedValues=6}, totalAttributeCount=6, events=[], totalRecordedEvents=0, links=[], totalRecordedLinks=0, status=ImmutableStatusData{statusCode=UNSET, description=}, hasEnded=true}]

	at dev.knative.eventing.kafka.broker.receiver.impl.ReceiverVerticleTracingTest.traceIsPropagated(ReceiverVerticleTracingTest.java:190)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)

Expected behavior

Like in 3.8 and its telemetry libray no issue

Actual behavior

Test fails...

How to Reproduce?

• check out the branch: https://github.com/matzew/eventing-kafka-broker/tree/quarkus_3.10
• open project root in IDEA
• go to the data-plane project
• run the ReceiverVerticleTracingLoomImplTest test in the IDEA

Output of uname -a or ver

Fedora

Output of java -version

21 (prow) and 22 (my fedora)

Quarkus version or git rev

3.10.2

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

The error:

java.lang.AssertionError: 
Expected size: 3 but was: 2 in:
[SpanData{spanContext=ImmutableSpanContext{traceId=b3af604008204ef273725923bf652a00, spanId=59f302b88a8e9bb0, traceFlags=01, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=true}, parentSpanContext=ImmutableSpanContext{traceId=b3af604008204ef273725923bf652a00, spanId=410e1d8a6485bd43, traceFlags=01, traceState=ArrayBasedTraceState{entries=[]}, remote=true, valid=true}, resource=Resource{schemaUrl=null, attributes={service.name="unknown_service:java", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.32.0"}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=io.vertx, version=null, schemaUrl=null, attributes={}}, name=POST, kind=SERVER, startEpochNanos=1721749113517586397, endEpochNanos=1721749113534142078, attributes=AttributesMap{data={http.request.method=POST, url.path=/broker-ns/broker-name, http.status_code=202, http.scheme=http, messaging.message_source=/hello, messaging.message_type=type, http.method=POST, http.response.status_code=202, http.url=http://localhost:8083/broker-ns/broker-name, messaging.message_id=1234}, capacity=128, totalAddedValues=10}, totalAttributeCount=10, events=[], totalRecordedEvents=0, links=[], totalRecordedLinks=0, status=ImmutableStatusData{statusCode=UNSET, description=}, hasEnded=true},
    SpanData{spanContext=ImmutableSpanContext{traceId=b3af604008204ef273725923bf652a00, spanId=410e1d8a6485bd43, traceFlags=01, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=true}, parentSpanContext=ImmutableSpanContext{traceId=00000000000000000000000000000000, spanId=0000000000000000, traceFlags=00, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=false}, resource=Resource{schemaUrl=null, attributes={service.name="unknown_service:java", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.32.0"}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=io.vertx, version=null, schemaUrl=null, attributes={}}, name=POST, kind=CLIENT, startEpochNanos=1721749113491976322, endEpochNanos=1721749113538305773, attributes=AttributesMap{data={http.request.method=POST, http.status_code=202, http.method=POST, url.full=http://localhost:8083/broker-ns/broker-name, http.response.status_code=202, http.url=http://localhost:8083/broker-ns/broker-name}, capacity=128, totalAddedValues=6}, totalAttributeCount=6, events=[], totalRecordedEvents=0, links=[], totalRecordedLinks=0, status=ImmutableStatusData{statusCode=UNSET, description=}, hasEnded=true}]

	at dev.knative.eventing.kafka.broker.receiver.impl.ReceiverVerticleTracingTest.traceIsPropagated(ReceiverVerticleTracingTest.java:190)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)

[quarkus-bot]

/cc @geoand (knative), @iocanel (knative)

[geoand]

cc @brunobat

[matzew]

there are other tracing related errors, such as -> https://prow.knative.dev/view/gs/knative-prow/pr-logs/pull/knative-extensions_eventing-kafka-broker/3872/reconciler-tests_eventing-kafka-broker_main/1815775631011483648

all w/ the bump to 3.10

[brunobat]

looking into it

[brunobat]

@matzew what are the 3 traces that you are expecting on that test, here:
https://github.com/matzew/eventing-kafka-broker/blob/6e1d381b21f48d893aa925729ed50e903d378e83/data-plane/receiver/src/test/java/dev/knative/eventing/kafka/broker/receiver/impl/ReceiverVerticleTracingTest.java#L190

[matzew]

Hello @brunobat! Thanks for looking

the following is what I get w/ the main branch, which is based on 3.8.4.

[SpanData{spanContext=ImmutableSpanContext{traceId=6273d8419d395e889553065c78013690, spanId=ff7b1c1a3554ed79, traceFlags=01, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=true}, parentSpanContext=ImmutableSpanContext{traceId=6273d8419d395e889553065c78013690, spanId=6539dc73d2f4b5e7, traceFlags=01, traceState=ArrayBasedTraceState{entries=[]}, remote=true, valid=true}, resource=Resource{schemaUrl=null, attributes={service.name="unknown_service:java", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.32.0"}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=io.vertx, version=null, schemaUrl=null, attributes={}}, name=POST, kind=SERVER, startEpochNanos=1721837107852469627, endEpochNanos=1721837107870432134, attributes=AttributesMap{data={http.url=http://localhost:8083/broker-ns/broker-name, messaging.message_id=1234, http.path=/broker-ns/broker-name, messaging.message_source=/hello, http.scheme=http, http.method=POST, messaging.message_type=type, http.status_code=202}, capacity=128, totalAddedValues=8}, totalAttributeCount=8, events=[], totalRecordedEvents=0, links=[], totalRecordedLinks=0, status=ImmutableStatusData{statusCode=UNSET, description=}, hasEnded=true},
    SpanData{spanContext=ImmutableSpanContext{traceId=6273d8419d395e889553065c78013690, spanId=6539dc73d2f4b5e7, traceFlags=01, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=true}, parentSpanContext=ImmutableSpanContext{traceId=00000000000000000000000000000000, spanId=0000000000000000, traceFlags=00, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=false}, resource=Resource{schemaUrl=null, attributes={service.name="unknown_service:java", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.32.0"}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=io.vertx, version=null, schemaUrl=null, attributes={}}, name=POST, kind=CLIENT, startEpochNanos=1721837107819775285, endEpochNanos=1721837107874186712, attributes=AttributesMap{data={http.url=http://localhost:8083/broker-ns/broker-name, http.method=POST, http.status_code=202}, capacity=128, totalAddedValues=3}, totalAttributeCount=3, events=[], totalRecordedEvents=0, links=[], totalRecordedLinks=0, status=ImmutableStatusData{statusCode=UNSET, description=}, hasEnded=true},
    SpanData{spanContext=ImmutableSpanContext{traceId=6273d8419d395e889553065c78013690, spanId=f0fc29ae84665066, traceFlags=01, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=true}, parentSpanContext=ImmutableSpanContext{traceId=6273d8419d395e889553065c78013690, spanId=ff7b1c1a3554ed79, traceFlags=01, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=true}, resource=Resource{schemaUrl=null, attributes={service.name="unknown_service:java", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.32.0"}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=io.vertx, version=null, schemaUrl=null, attributes={}}, name=kafka_send, kind=PRODUCER, startEpochNanos=1721837107864910614, endEpochNanos=1721837107882621204, attributes=AttributesMap{data={peer.address=, message_bus.destination=topic-name-42, peer.service=kafka}, capacity=128, totalAddedValues=3}, totalAttributeCount=3, events=[], totalRecordedEvents=0, links=[], totalRecordedLinks=0, status=ImmutableStatusData{statusCode=UNSET, description=}, hasEnded=true}]

See: https://github.com/knative-extensions/eventing-kafka-broker/blob/main/data-plane/pom.xml#L64

and the version of telemetry -> https://github.com/knative-extensions/eventing-kafka-broker/blob/main/data-plane/pom.xml#L49

[matzew]

I see this here too, and I think it is just when I use the vertx-opentelemetry version 4.5.x:
knative-extensions/eventing-kafka-broker#3929 (review)

But not yet exactly sure, on what's the diff between 4.4.7 and 4.5.7 (which is what our core is based on, via quarkus)

[matzew]

@brunobat any idea on this?
Looks like is a diff between vertx (4.47 and 4.5.7).

With this regard, I also saw this in vertx: https://github.com/quarkusio/quarkus/blob/089a116ab626f320a8cb8854c406e0cacc534d38/extensions/opentelemetry/runtime/src/main/java/io/quarkus/opentelemetry/runtime/tracing/intrumentation/vertx/OpenTelemetryVertxTracer.java

but also not entirely sure if that could be used by us, and we could migrate off of the vertx-opentelemetry JAR.

[brunobat]

@matzew I'm not sure about the scope of this issue anymore.
From this issue it seems you cannot use quarkus-opentelemetry in version 3.10, however, in this issue: knative-extensions/eventing-kafka-broker#3929 it looks like you are instead instrumenting with vertx-opentelemetry which Quarkus doesn't support.
Can you please clarify what's your plan?
Also, above, I see you never answered to this question:

@matzew what are the 3 traces that you are expecting on that test, here:
https://github.com/matzew/eventing-kafka-broker/blob/6e1d381b21f48d893aa925729ed50e903d378e83/data-plane/receiver/src/test/java/dev/knative/eventing/kafka/broker/receiver/impl/ReceiverVerticleTracingTest.java#L190

In order to understand what's going on, we need to understand what are the contents of the span that's missing.