While running with splunk-otel-agent 1.2, can API 1.4.1 create custom spans via Tracer?

[woldie]

I am auto-instrumenting with the splunk-otel-agent 1.2, and I'd like to create custom spans using the same Tracer as the one the agent uses. I am currently attempting to acquire the Tracer with API 1.4.1 running on OpenJDK 11:

GlobalOpenTelemetry.getTracer(Objects.requireNonNull(applicationContext.getId()))

And I am trying to create custom spans with

tracer.spanBuilder("mySpan")
        .setSpanKind(SpanKind.SERVER)
        .startSpan();

(I have also tried SpanKind.INTERNAL and left setSpanKind() off of the expression, but it made no difference)

Unfortunately, my custom spans are not traced, and only the auto-instrumented spans the agent creates show up in my traces in SignalFX. I am calling Span#end on all my custom spans.

This link referenced in another similar answer is no longer relevant: https://github.com/open-telemetry/opentelemetry-java/blob/main/QUICKSTART.md#tracing

So, I would like to know the following given the latest and greatest splunk-otel-agent:

1. Is it possible to access/connect to the same Tracer that splunk-otel-agent must have internally?
2. If yes, is it possible (without using @WithSpan) to add custom spans to my service via that Tracer?

Thanks!

EDIT 1: I see two conflicting comments in the splunk-otel documentation. At this link, https://docs.splunk.com/Observability/gdi/get-data-in/application/java/instrumentation/instrument-java-application.html there is a note that reads:

If you need to add custom attributes to spans or want to manually generate spans, you need to manually
instrument your Java application or service. See Manually instrument Java applications.

but on the "Manually instrument Java applications" page, a note reads:

Manual OTel instrumentation is fully compatible with Splunk automatic JVM instrumentation and is fully
supported by Splunk.

So which is it?

[mateuszrzeszutek]

Hey @woldie ,
Can you share some more details about how you're using the OpenTelemetry API in your app? A small repo containing a reproduction scenario would be invaluable 🙏

This link referenced in another similar answer is no longer relevant: https://github.com/open-telemetry/opentelemetry-java/blob/main/QUICKSTART.md#tracing

It points to the official OpenTelemetry Java docs page that described how to use OpenTelemetry API in detail.

1. Is it possible to access/connect to the same Tracer that splunk-otel-agent must have internally?
2. If yes, is it possible (without using @WithSpan) to add custom spans to my service via that Tracer?

1. Internally the agent uses many different Tracer instances - but you definitely should be able to add your own telemetry to the agent's output.
2. Yes, most definitely - the scenario that you've described above should work.

EDIT 1: I see two conflicting comments in the splunk-otel documentation. At this link, https://docs.splunk.com/Observability/gdi/get-data-in/application/java/instrumentation/instrument-java-application.html there is a note that reads:

If you need to add custom attributes to spans or want to manually generate spans, you need to manually
instrument your Java application or service. See Manually instrument Java applications.

but on the "Manually instrument Java applications" page, a note reads:

Manual OTel instrumentation is fully compatible with Splunk automatic JVM instrumentation and is fully
supported by Splunk.

They're both correct: if you want to add custom attributes or spans, you need to manually instrument your application - AND manual instrumentation is fully supported by Splunk.

[mateuszrzeszutek]

Next thing to discover is whether we can add a custom BatchSpanProcessor to the same configured SDK as the agent's. (we want to do some custom logging at the end of every span, regardless of whether it is a custom span or agent-created span.)

If it's just some logging, please take a look at the SpanProcessor interface - it's definitely more lightweight than running another BatchSpanProcessor.

[jkwatson]

Also, I found I had some of my final custom spans associated with the traceId were not making it to SignalFX. But, when I did a span.setStatus(StatusCode.OK) just prior to scope.close() and span.end(), all of my custom spans made it to SignalFX. Not sure if I was witnessing a race condition in the agent/SDK or if there is more pilot error in my use of the system that the setStatus(OK) allowed me to work around.

This doesn't make any sense to me. Setting a span status should not change this at all. Can you enable debug logging in the agent and make sure that the spans are getting created, whether or not the the status has been explicitly set?

[woldie]

@mateuszrzeszutek thanks for the tip, that looks like exactly what I want

[woldie]

This page describes how to extend the agent with a custom SpanProcessor: https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/examples/extension/README.md

[mateuszrzeszutek]

This page describes how to extend the agent with a custom SpanProcessor: https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/examples/extension/README.md

Yes, that's correct.

[furkanali0325]

Hi Team,
I have integrated Splunk OTEL jar( open telemetry library) in my spring boot application for tracing and matrices.
So there is a SPLUNK_ACCESS_TOKEN that needs to be exposed to send data directly to the Splunk observability cloud. Currently, I have stored it in Kubernetes.yaml file. So I just want to know other approaches to expose this Splunk access token.

For example, there is a flag variable "OTEL_JAVAAGENT_ENABLED" used by the Splunk OTEL library and I have stored this value to "true" in Kubernetes.yaml file. So by this config OTEL jar is able to access it.

Is there any alternate way to store the splunk_access_token apart from Kubernetes.yaml or properties file?