Skip to content

MicroProfile: OpenTracing

Jump to bottom
kgibm edited this page Apr 2, 2018 · 32 revisions

The Eclipse MicroProfile project is an open collaboration between developers, the community, and vendors to create a programming model for microservices applications which is complementary to JEE.

The Eclipse MicroProfile OpenTracing project (MP OpenTracing) defines the specification and an API for tracing JAX-RS web service operations as they flow into, within, between, and out of servers. This trace information can be fed to OpenTracing-compliant distributed tracing systems such as Zipkin. Then, the distributed tracing systems may be used to display each JAX-RS call and how long it took, track performance, diagnose problems, feed load distribution algorithms, etc. This is particularly useful in a microservices architecture because such architectures often have many separate JVMs tied together with JAX-RS calls.

The MP OpenTracing project is largely a wrapper around the OpenTracing project which provides a vendor-neutral open standard for distributed tracing and MP OpenTracing uses parts of version 0.30.0 of OpenTracing's Java implementation.

Version 1.0 of MP OpenTracing was delivered as part of version 1.3 of the overall MicroProfile.

Important links:

  1. MP OpenTracing specification: https://github.com/eclipse/microprofile-opentracing/blob/master/spec/src/main/asciidoc/microprofile-opentracing.asciidoc

MP OpenTracing in OpenLiberty

OpenLiberty contains two features to support OpenTracing: opentracing-1.0 and mpOpenTracing-1.0. opentracing-1.0 was delivered before the MP OpenTracing 1.0 specification was finalized, and mpOpenTracing-1.0 delivered the full specification compliance with OpenTracing 1.0. Therefore, unless a user is on a version of Liberty before mpOpenTracing-1.0 was released, users should just use mpOpenTracing-1.0 which is a superset of opentracing-1.0 and implicitly enables it.

To use MP OpenTracing in OpenLiberty, the user must create a user feature with a factory that returns an io.opentracing.Tracer implementation. IBM has created sample code for a user feature that connects to Zipkin. After the Tracer implementation has been properly installed as a user feature, mpOpenTracing-1.0 and the user feature must be enabled as features in server.xml. For example:

<featureManager>
  <feature>mpOpenTracing-1.0</feature>
  <feature>usr:opentracingZipkin-0.30</feature>
</featureManager>

Additional configuration in server.xml may be needed for the particular user feature.

The way the user feature works is that it uses an OpenLiberty Service Provider Interface through OpentracingTracerFactory to provide the Tracer implementation.

In addition to automatically tracing JAX-RS endpoints and clients, an application may additionally trace arbitrary Java classes used within a JAX-RS call, and/or disable or change the operation name of particular JAX-RS endpoints using an injected Tracer and/or the @Traced annotation, respectively.

MP OpenTracing Development

Important links:

  1. Main development hub: https://github.com/eclipse/microprofile-opentracing/
  2. Real time development chat on Gitter: https://gitter.im/eclipse/microprofile-opentracing
  3. MicroProfile mailing list: https://groups.google.com/forum/#!forum/microprofile
  4. Google Calendar which may show periodic "Open Tracing hangout" meetings: https://calendar.google.com/calendar/embed?src=gbnbc373ga40n0tvbl88nkc3r4%40group.calendar.google.com

MP OpenTracing TCK

MP OpenTracing has a Technology Compatibility Toolkit (TCK) that's used to confirm that a Microprofile OpenTracing implementation (such as OpenLiberty) implements the MP OpenTracing specification.

The TCK tests are in OpentracingClientTests. The tests use @RunAsClient so that they're run in the TCK runner harness and they make JAX-RS calls to various web services deployed into OpenLiberty (example). Each test then makes a call to the getTracer web service which assumes that the Tracer implementation is a MockTracer. The getTracer web service reads all of the accumulated spans from the MockTracer, converts them to a tree of POJOs, and sends them back to the TCK runner which then compares the tree to an expected tree of spans and asserts the tree is correct.

The original FAT tests use a different approach of the OpentracingMockTracer which wraps the MockTracer and uses a toString method to send back the spans which the FAT checks. This is still the default for the FAT user feature's tracer factory. However, the TCK requires that the Tracer is a MockTracer, but to simplify development, the TCK FAT uses the same user feature, so if -DUSE_MOCK_TRACER=true is set on the JVM, then OpentracingMockTracerFactory will return a MockTracer instead. This is set in the TCK server in its jvm.options file.

The TCK FAT is defined in the tck-suite.xml, the TCK runner pom.xml, and the arquillian.xml. The original FAT tests are largely redundant with the TCK FAT tests but mainly they're kept around because they test just opentracing-1.0 without mpOpenTracing-1.0.

The logs of the TCK runner are in dev/com.ibm.ws.opentracing_fat/build/libs/autoFVT/results/mvnOutput_TCK and the logs of the server under test are in dev/com.ibm.ws.opentracing_fat/build/libs/autoFVT/output/servers/OpentracingTCKServer*/logs/trace.log.

To run the TCK FAT, run:

./gradlew :com.ibm.ws.opentracing_fat:buildandrun

After making a change to OpenLiberty, run:

./gradlew assemble && ./gradlew releaseNeeded && ./gradlew :com.ibm.ws.opentracing_fat:buildandrun

When doing open source development additions to a new version of the TCK, you'll need to update microprofile.opentracing.version in the pom.xml above to something like 1.1-SNAPSHOT and then build the TCK locally:

git clone https://github.com/eclipse/microprofile-opentracing
cd microprofile-opentracing
git checkout -b new_development
# make changes (including updating version to e.g. 1.1)
mvn install
# Then go run com.ibm.ws.opentracing_fat:buildandrun

OpenLiberty OpenTracing Development

Key Notes

  1. Incoming JAX-RS request starts: https://github.com/OpenLiberty/open-liberty/blob/18.0.0.1_RC/dev/com.ibm.ws.opentracing/src/com/ibm/ws/opentracing/OpentracingContainerFilter.java#L71
  2. Incoming JAX-RS request finishes: https://github.com/OpenLiberty/open-liberty/blob/18.0.0.1_RC/dev/com.ibm.ws.opentracing/src/com/ibm/ws/opentracing/OpentracingContainerFilter.java#L138
  3. Outgoing JAX-RS request starts: https://github.com/OpenLiberty/open-liberty/blob/18.0.0.1_RC/dev/com.ibm.ws.opentracing/src/com/ibm/ws/opentracing/OpentracingClientFilter.java#L81
  4. Outgoing JAX-RS request finishes: https://github.com/OpenLiberty/open-liberty/blob/18.0.0.1_RC/dev/com.ibm.ws.opentracing/src/com/ibm/ws/opentracing/OpentracingClientFilter.java#L152
  5. @Traced annotation processing: https://github.com/OpenLiberty/open-liberty/blob/18.0.0.1_RC/dev/com.ibm.ws.microprofile.opentracing/src/com/ibm/ws/microprofile/opentracing/cdi/TracedInterceptor.java#L34
  6. When mpOpenTracing-1.0 is enabled (instead of just opentracing-1.0), OpenTracingFilterHelper is injected into OpentracingContainerFilter and OpentracingClientFilter which performs the annotation processing that otherwise shouldn't happen for opentracing-1.0.
  7. Filtering (currently disabled; requires additional spec work): https://github.com/OpenLiberty/open-liberty/blob/18.0.0.1_RC/dev/com.ibm.ws.opentracing/src/com/ibm/ws/opentracing/filters/SpanFilter.java#L15

Key Concepts

  1. Review the API specification.
  2. A Span represents an execution of code such as a JAX-RS web service method, a method annotated with @Traced, or an explicitly traced block of code. The Span is the main conceptual unit, representing a traced piece of code, how long it took, and any tags or baggage that add metadata to the span (e.g. the URL of a request).
  3. The "main" way to create a span is to use Tracer.buildSpan with an operation name. When startActive is called, it returns an ActiveSpan object which must be deactivated (ActiveSpan implements Closeable so it can be wrapped in a try-with-resources statement to automatically close at the end of the block).

Debugging

The following trace specification should be used for the most common issues:

*=info:com.ibm.ws.opentracing*=all:com.ibm.ws.microprofile.opentracing*=all

If there may be CDI issues:

*=info:com.ibm.ws.opentracing*=all:com.ibm.ws.microprofile.opentracing*=all:CDI=all:com.ibm.ws.cdi*=all
Clone this wiki locally