Overview

OpenTelemetry (formed through a merger of the OpenTracing and OpenCensus projects) is a collection of APIs, SDKs, and tools to instrument, generate, collect, and export telemetry data like metrics, logs, and distributed traces. OpenTelemetry is vendor-neutral open-source observability framework and as an industry-standard, it is natively supported by a number of vendors. From the instrumentation perspective, there is support for quite a few programming languages, including dedicated Java APIs. Starting from 3.5.7 /3.6.2 / 4.0.3 releases, Apache CXF fully supports integration (through cxf-integration-tracing-opentelemetry module) with OpenTelemetry distributed tracing capabilities.

The section dedicated to Apache HTrace has pretty good introduction into distributed tracing basics however OpenTelemetry specification abstracts a lot of things, outlining just a general APIs to denote the Span lifecycle and injection points to propagate the context across many distributed components. As such, the intrinsic details about HTTP headers f.e. becomes an integral part of the distributed tracer of your choice, out of reach for Apache CXF.

Distributed Tracing in Apache CXF using OpenTelemetry

The current integration of the OpenTelemetry's distributed tracing in Apache CXF supports OpenTelemetry Java SDK 1.28.0+ (using semantic conventions) and provides full-fledged support of JAX-RS 2.x / JAX-WS applications. From high-level prospective, the JAX-RS integration consists of three main parts:

  • TracerContext (injectable through @Context annotation)
  • OpenTelemetryProvider (server-side JAX-RS provider) and OpenTelemetryClientProvider (client-side JAX-RS provider)
  • OpenTelemetryFeature (server-side JAX-RS feature) to simplify the configuration and integration

Similarly, from high-level perspective, JAX-WS integration includes:

  • OpenTelemetryStartInterceptor / OpenTelemetryStopInterceptor / OpenTelemetryFeature Apache CXF feature (server-side JAX-WS support)
  • OpenTelemetryClientStartInterceptor / OpenTelemetryClientStopInterceptor / OpenTelemetryClientFeature Apache CXF feature (client-side JAX-WS support)

Apache CXF uses HTTP headers to hand off tracing context from the client to the service and from the service to service. Those headers are specific to distributing tracing framework you have picked and are not configurable at the moment (unless the framework itself has a way to do that).

By default, OpenTelemetryClientProvider will use configured propagators to pass the currently active span through HTTP headers on each service invocation. If there is no active spans, the new span will be created and passed through HTTP headers on per-invocation basis. Essentially, for JAX-RS applications just registering OpenTelemetryClientProvider on the client and OpenTelemetryProvider on the server is enough to have tracing context to be properly passed everywhere. The only configuration part which is necessary are span exporters(s) and sampler(s) which are, not surprisingly, specific to distributing tracing vendor you have chosen.

It is also worth to mention the way Apache CXF attaches the description to spans. With regards to the client integration, the description becomes a full URL being invoked prefixed by HTTP method, for example: GET http://localhost:8282/books. On the server side integration, the description becomes a relative JAX-RS resource path prefixed by HTTP method, f.e.: GET books, POST book/123

Configuring Client

There are a couple of ways the JAX-RS client could be configured, depending on the client implementation. Apache CXF provides its own WebClient which could be configured just like that (in future versions, there would be a simpler ways to do that using client specific features):

final SdkTracerProvider sdkTracerProvider = SdkTracerProvider
     .builder()
     ...
     .build();

final OpenTelemetry openTelemetry = OpenTelemetrySdk
    .builder()
    .setTracerProvider(sdkTracerProvider)
    ...
    .buildAndRegisterGlobal();

final Tracer tracer =  openTelemetry.getTracer("web-client");
                  
Response response = WebClient
    .create("http://localhost:9000/catalog", Arrays.asList(new OpenTelemetryClientProvider(openTelemetry,  tracer))
    .accept(MediaType.APPLICATION_JSON)
    .get();

The configuration based on using the standard JAX-RS Client is very similar:

final SdkTracerProvider sdkTracerProvider = SdkTracerProvider
     .builder()
     ...
     .build();

final OpenTelemetry openTelemetry = OpenTelemetrySdk
    .builder()
    .setTracerProvider(sdkTracerProvider)
    ...
    .buildAndRegisterGlobal();

final Tracer tracer =  openTelemetry.getTracer("jaxrs-client");
final OpenTelemetryClientProvider provider = new OpenTelemetryClientProvider(openTelemetry, tracer);
final Client client = ClientBuilder.newClient().register(provider);
 
final Response response = client
    .target("http://localhost:9000/catalog")
    .request()
    .accept(MediaType.APPLICATION_JSON)
    .get();

Configuring Server

Server configuration is a bit simpler than the client one thanks to the feature class available, OpenTelemetryFeature. Depending on the way the Apache CXF is used to configure JAX-RS services, it could be part of JAX-RS application configuration, for example:

@ApplicationPath("/")
public class CatalogApplication extends Application {
    @Override
    public Set<Object> getSingletons() {
        final SdkTracerProvider sdkTracerProvider = SdkTracerProvider
           .builder()
           ...
           .build();

       final OpenTelemetry openTelemetry = OpenTelemetrySdk
           .builder()
           .setTracerProvider(sdkTracerProvider)
           ...
           .buildAndRegisterGlobal();

        final Tracer tracer =  openTelemetry.getTracer("tracer");
        return new HashSet<>(
                Arrays.asList(
                    new OpenTelemetryFeature(openTelemetry,  tracer)
                )
            );
    }
}

Or it could be configured using JAXRSServerFactoryBean as well, for example:

final SdkTracerProvider sdkTracerProvider = SdkTracerProvider
    .builder()
    ...
    .build();

final OpenTelemetry openTelemetry = OpenTelemetrySdk
    .builder()
    .setTracerProvider(sdkTracerProvider)
    ...
    .buildAndRegisterGlobal(); 
 
final Tracer tracer =  openTelemetry.getTracer("tracer"); 
final JAXRSServerFactoryBean factory = RuntimeDelegate.getInstance().createEndpoint(/* application instance */, JAXRSServerFactoryBean.class);
factory.setProvider(new OpenTelemetryFeature(openTelemetry,  tracer));
...
return factory.create();

Once the span processor(s) and sampler are properly configured, all generated spans are going to be collected and available for analysis and/or visualization.

Distributed Tracing In Action: Usage Scenarios

In the following subsections we are going to walk through many different scenarios to illustrate the distributed tracing in action, starting from the simplest ones and finishing with asynchronous JAX-RS services. All examples assume that configuration has been done (see please Configuring Client  and Configuring Server sections above).

Example #1: Client and Server with default distributed tracing configured

In the first example we are going to see the effect of using default configuration on the client and on the server, with only OpenTelemetryClientProvider and OpenTelemetryProvider registered. The JAX-RS resource endpoint is pretty basic stubbed method:

@Produces( { MediaType.APPLICATION_JSON } )
@GET
public Collection<Book> getBooks() {
    return Arrays.asList(
        new Book("Apache CXF Web Service Development", "Naveen Balani, Rajeev Hathi")
    );
}

The client is as simple as that:

final Response response = client
    .target("http://localhost:8282/books")
    .request()
    .accept(MediaType.APPLICATION_JSON)
    .get();

The actual invocation of the request by the client (with service name tracer-client) and consequent invocation of the service on the server side (service name tracer-server) is going to generate the following sample traces (taken from Jaeger UI):

Example #2: Client and Server with nested trace

In this example server-side implementation of the JAX-RS service is going to call an external system (simulated as a simple delay of 500ms) within its own span. The client-side code stays unchanged.

@Produces( { MediaType.APPLICATION_JSON } )
@GET
public Collection<Book> getBooks(@Context final TracerContext tracer) throws Exception {
    try(final Scope scope = tracer.startSpan("Calling External System")) {
        // Simulating a delay of 500ms required to call external system
        Thread.sleep(500);
             
        return Arrays.asList(
            new Book("Apache CXF Web Service Development", "Naveen Balani, Rajeev Hathi")
        );
    }
}

The actual invocation of the request by the client (with service name tracer-client) and consequent invocation of the service on the server side (service name tracer-server) is going to generate the following sample traces (taken from Jaeger UI):

Example #3: Client and Server trace with annotations

In this example server-side implementation of the JAX-RS service is going to add timeline to the active span. The client-side code stays unchanged.

@Produces( { MediaType.APPLICATION_JSON } )
@GET
public Collection<Book> getBooks(@Context final TracerContext tracer) throws Exception {
    tracer.timeline("Preparing Books");
    // Simulating some work using a delay of 100ms
    Thread.sleep(100);
          
    return Arrays.asList(
        new Book("Apache CXF Web Service Development", "Naveen Balani, Rajeev Hathi")
    );
}

The actual invocation of the request by the client (with service name tracer-client) and consequent invocation of the service on the server side (service name traceser-server) is going to generate the following sample traces (taken from Jaeger UI):

Please notice that timelines are treated as logs events in Jaeger.

Example #4: Client and Server with binary annotations (key/value)

In this example server-side implementation of the JAX-RS service is going to add key/value annotations to the active span. The client-side code stays unchanged.

@Produces( { MediaType.APPLICATION_JSON } )
@GET
public Collection<Book> getBooks(@Context final TracerContext tracer) throws Exception {
    final Collection<Book> books = Arrays.asList(
        new Book("Apache CXF Web Service Development", "Naveen Balani, Rajeev Hathi")
    );
          
    tracer.annotate("# of books", Integer.toString(books.size()));
    return books;
}

The actual invocation of the request by the client (with service name tracer-client) and consequent invocation of the service on the server side (service name tracer-server) is going to generate the following sample server trace properties (taken from Jaeger UI):

Example #5: Client and Server with parallel trace (involving thread pools)

In this example server-side implementation of the JAX-RS service is going to offload some work into thread pool and then return the response to the client, simulating parallel execution. The client-side code stays unchanged.

@Produces( { MediaType.APPLICATION_JSON } )
@GET
public Collection<Book> getBooks(@Context final TracerContext tracer) throws Exception {
    final Future<Book> book1 = executor.submit(
        tracer.wrap("Getting Book 1", new Traceable<Book>() {
            public Book call(final TracerContext context) throws Exception {
                // Simulating a delay of 100ms required to call external system
                Thread.sleep(100);
                      
                return new Book("Apache CXF Web Service Development",
                    "Naveen Balani, Rajeev Hathi");
            }
        })
    );
          
    final Future<Book> book2 = executor.submit(
        tracer.wrap("Getting Book 2", new Traceable<Book>() {
            public Book call(final TracerContext context) throws Exception {
                // Simulating a delay of 100ms required to call external system
                Thread.sleep(200);
                      
                return new Book("Developing Web Services with Apache CXF and Axis2",
                    "Kent Ka Iok Tong");
            }
        })
    );
         
    return Arrays.asList(book1.get(), book2.get());
}

The actual invocation of the request by the client (with service name tracer-client) and consequent invocation of the service on the server side (process name tracer-server) is going to generate the following sample traces (taken from Jaeger UI):

Example #6: Client and Server with asynchronous JAX-RS service (server-side)

In this example server-side implementation of the JAX-RS service is going to be executed asynchronously. It poses a challenge from the tracing prospective as request and response are processed in different threads (in general). At the moment, Apache CXF does not support the transparent tracing spans management (except for default use case) but provides the simple ways to do that (by letting to transfer spans from thread to thread). The client-side code stays unchanged.

@Produces( { MediaType.APPLICATION_JSON } )
@GET
public void getBooks(@Suspended final AsyncResponse response, @Context final TracerContext tracer) throws Exception {
    tracer.continueSpan(new Traceable<Future<Void>>() {
        public Future<Void> call(final TracerContext context) throws Exception {
            return executor.submit(
                tracer.wrap("Getting Book", new Traceable<Void>() {
                    public Void call(final TracerContext context) throws Exception {
                        // Simulating a processing delay of 50ms
                        Thread.sleep(50);
                              
                        response.resume(
                            Arrays.asList(
                                new Book("Apache CXF Web Service Development", "Naveen Balani, Rajeev Hathi")
                            )
                        );
                              
                        return null;
                    }
                })
            );
        }
    });
}

The actual invocation of the request by the client (with service name tracer-client) and consequent invocation of the service on the server side (service name tracer-server) is going to generate the following sample traces (taken from Jaeger UI):

Example #7: Client and Server with asynchronous invocation (client-side)

In this example server-side implementation of the JAX-RS service is going to be the default one:

@Produces( { MediaType.APPLICATION_JSON } )
@GET
public Collection<Book> getBooks() {
    return Arrays.asList(
        new Book("Apache CXF Web Service Development", "Naveen Balani, Rajeev Hathi")
    );
}

While the JAX-RS client implementation is going to perform the asynchronous invocation:

final Future<Response> future = client
    .target("http://localhost:8282/books")
    .request()
    .accept(MediaType.APPLICATION_JSON)
    .async()
    .get();

In this respect, there is no difference from the caller prospective however a bit more work is going under the hood to transfer the active tracing span from JAX-RS client request filter to client response filter as in general those are executed in different threads (similarly to server-side asynchronous JAX-RS resource invocation). The actual invocation of the request by the client (with service name tracer-client) and consequent invocation of the service on the server side (service name tracer-server) is going to generate the following sample traces (taken from Jaeger UI):

Distributed Tracing with OpenTelemetry and JAX-WS support

Distributed tracing in the Apache CXF is build primarily around JAX-RS 2.x implementation. However, JAX-WS is also supported but it requires to write some boiler-plate code and use OpenTelemetry Java SDK directly (the JAX-WS integration is going to be enhanced in the future). Essentially, from the server-side prospective the in/out interceptors, OpenTelemetryStartInterceptor and OpenTelemetryStopInterceptor respectively, should be configured as part of interceptor chains, either manually or using OpenTelemetryFeature. For example:

final SdkTracerProvider sdkTracerProvider = SdkTracerProvider
    .builder()
    ...
    .build();

final OpenTelemetry openTelemetry = OpenTelemetrySdk
    .builder()
    .setTracerProvider(sdkTracerProvider)
    ...
    .buildAndRegisterGlobal(); 
 
final Tracer tracer =  openTelemetry.getTracer("tracer");
final JaxWsServerFactoryBean sf = new JaxWsServerFactoryBean();
...
sf.getFeatures().add(new OpenTelemetryFeature(openTelemetry, tracer));
...
sf.create();

Similarly to the server-side, client-side needs own set of out/in interceptors, OpenTelemetryClientStartInterceptor and OpenTelemetryClientStopInterceptor (or OpenTelemetryClientFeature). Please notice the difference from server-side:  OpenTelemetryClientStartInterceptor becomes out-interceptor while OpenTelemetryClientStopInterceptor becomes in-interceptor. For example:

final SdkTracerProvider sdkTracerProvider = SdkTracerProvider
    .builder()
    ...
    .build();

final OpenTelemetry openTelemetry = OpenTelemetrySdk
    .builder()
    .setTracerProvider(sdkTracerProvider)
    ...
    .buildAndRegisterGlobal(); 
 
final Tracer tracer =  openTelemetry.getTracer("tracer");
final JaxWsProxyFactoryBean sf = new JaxWsProxyFactoryBean();
...
sf.getFeatures().add(new OpenTelemetryClientFeature(openTelemetry, tracer));
...
sf.create();

As it was mentioned before, you may use GlobalOpenTelemetry utility class to pass the tracer around so, for example, any JAX-WS service will be able to retrieve the current tracer by invoking GlobalOpenTelemetry.get() method.

Distributed Tracing with OpenTelemetry and OSGi

Most of the distributed tracers compatible with OpenTelemetry API could be deployed into OSGi container and as such, the integration is fully available for Apache CXF services running inside the container. For a complete example please take a look on jax_rs_tracing_opentelemetry_osgi sample project, but here is the typical OSGi  Blueprint snippet in case logging span exporter.

<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:cxf="http://cxf.apache.org/blueprint/core"
       xmlns:jaxrs="http://cxf.apache.org/blueprint/jaxrs"

       xsi:schemaLocation="http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd
                           http://cxf.apache.org/blueprint/core http://cxf.apache.org/schemas/blueprint/core.xsd
                           http://cxf.apache.org/jaxrs http://cxf.apache.org/schemas/blueprint/jaxrs.xsd">

    <bean id="tracingFeature" class="org.apache.cxf.tracing.opentelemetry.jaxrs.OpenTelemetryFeature">
        <argument index="0">
            <bean factory-ref="withPropagators" factory-method="build"/>
        </argument>
        <argument index="1" value="tracer" />
    </bean>

    <bean id="openTelemetrySdk" class="io.opentelemetry.sdk.OpenTelemetrySdk" factory-method="builder" />
    <bean id="sdkTracerProvider" class="io.opentelemetry.sdk.trace.SdkTracerProvider" factory-method="builder" />

    <bean id="withSampler" factory-ref="sdkTracerProvider" factory-method="setSampler">
        <argument index="0">
            <bean class="io.opentelemetry.sdk.trace.samplers.Sampler" factory-method="alwaysOn" />
        </argument>
    </bean>

    <bean id="processor" class="io.opentelemetry.sdk.trace.export.BatchSpanProcessor" factory-method="builder">
        <argument index="0">
            <bean class="io.opentelemetry.exporter.logging.LoggingSpanExporter" factory-method="create" />
        </argument>
    </bean>

    <bean id="withSpanProcessor" factory-ref="sdkTracerProvider" factory-method="addSpanProcessor">
        <argument index="0">
            <bean factory-ref="processor" factory-method="build"/>
        </argument>
    </bean>

    <bean id="withTracerProvider" factory-ref="openTelemetrySdk" factory-method="setTracerProvider">
        <argument index="0">
            <bean factory-ref="withSpanProcessor" factory-method="build"/>
        </argument>
    </bean>

    <bean id="withPropagators" factory-ref="withTracerProvider" factory-method="setPropagators">
        <argument index="0">
            <bean class="io.opentelemetry.context.propagation.ContextPropagators" factory-method="create">
                <argument index="0">
                    <bean class="io.opentelemetry.api.trace.propagation.W3CTraceContextPropagator" factory-method="getInstance" /> 
                </argument>
            </bean>
        </argument>
    </bean>

    <cxf:bus>
        <cxf:features>
            <cxf:logging />
        </cxf:features>
    </cxf:bus>

    <jaxrs:server id="catalogServer" address="/">
        <jaxrs:serviceBeans>
            ...
        </jaxrs:serviceBeans>
        <jaxrs:providers>
            <ref component-id="tracingFeature" />
        </jaxrs:providers>
    </jaxrs:server>

</blueprint>

As of now, OpenTelemetry does not provide dedicated OSGi bundles and the service loader mechanism is not working very well. It is very likely that you may need to declare own span exporters.

Accessing OpenTelemetry APIs

The Apache CXF  abstracts as much of the tracer-specific APIs behind TracerContext as possible. However, sometimes there is a need to get access to OpenTelemetry APIs in order to leverages the rich set of available instrumentations. To make it possible, TracerContext has a dedicated unwrap method which returns underlying Tracer instance. The snippet below shows off how to use this API and use OpenTelemetry instrumentation for OpenFeign client through OpenTracing shim (as of today, OpenFeign client does not provide native support for OpenTelemetry).

final SdkTracerProvider sdkTracerProvider = SdkTracerProvider
    .builder()
    ...
    .build();

final OpenTelemetry openTelemetry = OpenTelemetrySdk
    .builder()
    .setTracerProvider(sdkTracerProvider)
    ...
    .buildAndRegisterGlobal();

// Use OpenTelemetru OpenTracing shim for Feign OpenTracing integration
GlobalTracer.registerIfAbsent(OpenTracingShim.createTracerShim(openTelemetry));
@GET
@Path("/search")
@Produces(MediaType.APPLICATION_JSON)
public JsonObject search(@QueryParam("q") final String query, @Context final TracerContext tracing) throws Exception {
    final GoogleBooksApi api = Feign.builder()
        .client(new TracingClient(new ApacheHttpClient(), tracing.unwrap(Tracer.class)))
        .target(GoogleBooksApi.class, "https://www.googleapis.com");
 
    final Response response = api.search(query);
    try (final Reader reader = response.body().asReader()) {
        return Json.createReader(reader).readObject();
    }
}

The usage of tracer-specific APIs is not generally advisable (because of portability reasons) but in case there are no other options available, it is available.

Using non-JAX-RS clients

The  Apache CXF  uses native OpenTelemetry capabilities so the existing instrumentations for different HTTP clients work as expected. The usage of only JAX-RS client is not required. For example, the following snippet demonstrates the usage of traceble OkHttp client  to call JAX-RS resources, backed by Apache CXF and OpenTelemetry OkHttp3 instrumentation.

final SdkTracerProvider sdkTracerProvider = SdkTracerProvider
    .builder()
    ...
    .build();

final OpenTelemetry openTelemetry = OpenTelemetrySdk
    .builder()
    .setTracerProvider(sdkTracerProvider)
    ...
    .buildAndRegisterGlobal();

final OkHttpClient client = OkHttpClient.Builder().build();
final Call.Factory factory =  OkHttpTelemetry.builder(openTelemetry).build().newCallFactory(client));

final Request request = new Request.Builder()
    .url("http://localhost:9000/catalog")
    .header("Accept", "application/json")
    .build();
 
try (final Response response = factory.newCall(request).execute()) {
    // Do something with response.body()
}

Samples