feat: Introducing Tracing with OpenTelemetry API

[jimit-j-shah]

This change introduces Client-Side Tracing using OpenTelemetry API in Datastore Java SDK.

User Guide & Documentation:

• https://cloud.google.com/datastore/docs/client-side-traces
• https://cloud.google.com/datastore/docs/trace-span-references

Fixes #1428 ☕️

[cindy-peng]

Hi @jimit-j-shah, is this pull request combining all previous changes made on the feature branch tracing-main-3?

[jimit-j-shah]

Hi @jimit-j-shah, is this pull request combining all previous changes made on the feature branch tracing-main-3?

Yes. Just rebased on main's new changes into a new branch.

[jimit-j-shah]

Will wait to merge until we have launch approval.

[generated-files-bot]

Warning: This pull request is touching the following templated files:

• .kokoro/presubmit/graalvm-native.cfg
• samples/pom.xml
• samples/snapshot/pom.xml
• samples/snippets/pom.xml

[ehsannas]

The public APIs look good to me. I also briefly reviewed certain parts of the PR (skipped the tests). I've pointed out some nits below. I defer the approval to the team that maintains this repo.

[ehsannas]

adding opencensus code?

[jimit-j-shah]

Replaced TraceUtil usage in 45da20b

[ehsannas]

The main benefit of TraceUtil is that if tracing is disabled, no code from the OTEL library will ever be on the execution path of the SDK. This code violates that. I would highly recommend spending a bit more time to fix this before shipping.

[jimit-j-shah]

• Looks like ReadWriteTransactionCallable is scheduled to run on an Executor in the RetryHelper library provided by gax layer. So that complicates the Context propagation.
• Also tried "manually" propagating the parent SpanContext to the "callable" but that doesn't work either as we don't have control over the future lifetimes (which run in the RetryHelper) so the parent span may end() before the child spans are executed(?).
• This needs some significant overhauling or gax support for context propagation - both are non trivial and wouldn't want to block the launch on this.

[ehsannas]

Uh... not sure why using the TraceUtil directly doesn't work. Maybe it's because there's only 1 TraceUtil but several threads creating spans? If that's the case, can you try and see if this works:

Add an API TraceUtil.isTracingEnabled(), and then change:

private TraceUtil.Span startSpanWithParentContext(
        String spanName,
        TraceUtil.SpanContext parentSpanContext) {
  TraceUtil otelTraceUtil = datastore.getOptions().getTraceUtil();
  if (!otelTraceUtil.isTracingEnabled()) {
    return new TraceUtil.Span();
  }

  SpanBuilder spanBuilder =
      otelTraceUtil
          .getTracer()
          .spanBuilder(com.google.cloud.datastore.telemetry.TraceUtil.SPAN_NAME_TRANSACTION_RUN)
          .setSpanKind(SpanKind.PRODUCER)
          .setParent(
              Context.current()
                  .with(
                      io.opentelemetry.api.trace.Span.wrap(
                          parentSpanContext.getSpanContext())));
  return TraceUtil.Span(otelTraceUtil.addSettingsAttributesToCurrentSpan(spanBuilder).startSpan());
}

TraceUtil.Span span =  startSpanWithParentContext(
              com.google.cloud.datastore.telemetry.TraceUtil.SPAN_NAME_TRANSACTION_RUN,
              parentSpanContext);

try (TraceUtil.Scope ignored = span.makeCurrent()) {
...

This way the opentelemetry code will remain off the execution path if tracing is disabled, and OTEL types are also not used on the main code path.

[jimit-j-shah]

Hi @jimit-j-shah, is this pull request combining all previous changes made on the feature branch tracing-main-3?

Yes. Now I've done a fresh merge from main in #1576. Let's continue the conversation there.

[jimit-j-shah]

Will wait to merge until we have launch approval.

We now have launch approval.

[jimit-j-shah]

This PR is superseded by #1576 - will implement all the comments there.