OpenTelemetry Support in Mule Runtime

A trace is a sequence of spans representing a complete operation in a distributed system. Spans can be nested and represent a unit of work or operation. Each span includes:

When tracing a flow execution, each Mule component execution involved in processing the message is represented as a span. The component spans describe the relevant aspects of the execution. They can range from a single span for a simple component, such as the Set Payload, to more complex structures, such as a Batch Job.

The Mule runtime traces complex components, such as Batch Jobs.

The trace shows how the first step of the Batch Job processes each batch record, with each record span containing a child span that represents the unit of work the Logger component performs. Next, it displays the Batch Aggregator span, along with the For Each component and the processors within it, such as Transform Message and Logger. Then, the trace displays the other two batch steps and their respective components. Finally, it shows the Batch On Complete span, along with the Logger component that appears after the Batch Job in the flow.

In this example, one Mule app sends a request to another Mule app:

Distributed tracing captures both flows as part of the same trace because OpenTelemetry is instrumented in Mule and HTTP Connector.

If the execution of the unit of work represented by a span produces an error, the span’s status is set to Error. MuleSoft follows the semantic convention defined by OpenTelemetry that adds one or more exception events to the span:

In this example, when an error is produced, the span corresponding to the Raise Error component has an Error status:

Because this flow lacks error handling, the error propagates and fails the flow execution, causing the Error status to propagate to the flow span. A detailed view of the span also shows the associated exception events.

The HTTP Listener and HTTP Request operations contribute additional attributes to the spans, providing more context for tracing the HTTP interactions.

For flows with an HTTP Listener:

Additional attributes for an HTTP Listener associated flow span:

For flows with an HTTP Request:

Additional attributes for an HTTP Request associated flow span:

OpenTelemetry logging is a sequence of log records with a unified data model, where each record corresponds to a Java log call. Each log record contains:

Any entry added to the Log4j MDC is added to the OpenTelemetry logs as Log Record attributes, so all Log4j MDC data is exported as part of OpenTelemetry logs.

For adding custom attributes on both traces and logs, see Custom Attributes.

Mule runtime supports exporting all its telemetry data using the OpenTelemetry protocol (OTLP) and binary/protobuf format. You can configure an endpoint for each signal type (traces and logs), and data is sent to that endpoint. However, we strongly recommend using a collector because it provides the most flexible and feature-rich architecture for a wide range of use cases and deployment scenarios.

See OpenTelemetry Collector for more information.

Key reasons for using a collector include:

Mule runtime sends telemetry signals to one endpoint (the collector) in one (OTEL binary/protobuf) format, ensuring vendor neutrality. Then, the collector manages exporting signals to the selected backends. This is especially useful for monitoring solutions like Azure, which can’t ingest OTEL signals directly and require a specific exporter, or Grafana, which implements authentication and additional metrics via collector components.

Exporting the same signal to multiple third-party solutions is only possible through a collector because Mule runtime allows one endpoint per signal.

Most third-party monitoring solutions support or provide their own collector implementations.

Applications don’t need external Internet access to reach third-party monitoring solutions.

The collector handles authentication and authorization against third-party monitoring solutions.

You can use it as a single point for dedicated network policies.

You can optimize collectors for specific goals, scale independently, and distribute load without affecting Mule runtime instances.

Configuration changes don’t affect Mule runtime instances.

Collectors include multiple built in metrics and health checks.

A collector can centralize data enrichment and transformation for Mule runtime telemetry and additional non-Mule services.

Mule runtime includes predefined configuration templates for exporting traces and logs.

Tracing export setup properties:

Tracing export tuning properties:

Minimal configuration example that exports 50% of the traces to an HTTP OpenTelemetry collector endpoint:

Logging export setup properties:

Logging export tuning properties:

Minimal configuration example that sends logs to an HTTP OpenTelemetry collector endpoint:

Sampling helps reduce the costs of observability. Mule runtime supports Head sampling, which makes the sampling decision as early as possible. In case of a Mule application, that moment is when a message source is about to trigger a message processing, for example, when an HTTP listener has received a request.

Head sampling is a good fit for:

Regarding decision algorithms, Mule runtime currently supports these standardized OpenTelemetry algorithms: AlwaysOn, AlwaysOff, ParentBasedAlwaysOn, ParentBasedAlwaysOff, TraceIdRatio, and ParentBasedTraceIdRatio.

While Head sampling is simple and efficient, it has limitations. It can’t:

Tail sampling supports these advanced cases but is costly and can affect performance. Therefore, Mule runtime only supports Head sampling. If needed, implement Tail sampling in the OpenTelemetry collector.

Regarding sampling algorithms, Mule runtime currently supports these standardized OpenTelemetry algorithms: AlwaysOn, AlwaysOff, ParentBasedAlwaysOn, ParentBasedAlwaysOff, TraceIdRatio, and ParentBasedTraceIdRatio.

Mule runtime tracing is structured around three base tracing levels: OVERVIEW, MONITORING, and DEBUG. Each generated span has an associated tracing level. Flow spans use the OVERVIEW level, flow component spans use the MONITORING level, and internal flow component execution spans (parameter resolution, connection obtention, among others) use the DEBUG level.

In most cases, the default tracing level is the best choice. Use DEBUG for troubleshooting.

Service attributes identify the source of the telemetry data, Mule applications in this case. Mule runtime supports setting the service.name and service.namespace properties to custom values:

Additionally, a service.instance.id identifies the instance of the Mule application and is automatically generated as a unique identifier on each deployment or redeployment.

Adding custom data to the generated spans and log records can help with:

Mule runtime supports adding custom attributes to the spans and log records through the Tracing Module Any logging variable set by the module is also added as an attribute of the span that represents the set-logging-variable operation and the spans and log records that are generated during the flow’s execution, including subflows, flow references, and error handlers.

While useful, custom attributes must be handled with care. Include only relevant information. Too many attributes can affect performance and reach the feature limits:

Backpressure occurs when export queues for traces or logs are full. Backpressure strategies define how Mule runtime handles this event:

Dropping spans or log records results in missing data at the export destination. Blocking threads, on the other hand, can cause performance issues in Mule runtime. For both conditions, Mule runtime generates a WARNING log. Internal statistics track historical occurrences, and the log messages include cumulative totals since the export started. An alert is also triggered and is available at the Mule troubleshooting plugin DIAF files.

Export queue overflow: 2 spans have been dropped. Total spans dropped since the export started: 4

Export queue overflow: 2 spans have been delayed. Total spans delayed since the export started: 4

When errors occur during export requests, Mule runtime uses an exponential back-off with jitter to avoid overwhelming the destination until recovery. Only errors marked as retryable, as defined by OpenTelemetry standards, trigger retry logic. See HTTP transport protocol reference and gRPC transport protocol reference for more information.

Direct telemetry stream supports configuring the export for traces and logs to use mutual TLS authentication, which can be used to secure the communication with an external collector.

CERTIFICATES_PATH strategy is supported for Standalone and Hybrid Standalone deployments. Configure the certificates location when using this strategy:

CERTIFICATES_BASE64 strategy is supported for all the deployments. Encode the certificates in Base64 when using this strategy:

Mule runtime logs are the place to check for connection issues, export errors, and delayed or dropped traces and logs.

Mule diagnostics provides information about delayed or dropped traces and logs and are useful for quick checks. Note that diagnostics are temporal and cover only the last 60 minutes.

Use diagnostics as a starting point and complement your analysis by reviewing the logs.

Use an OpenTelemetry collector configured to report its internal metrics. See Collector internal metrics for more information. Having a collector with that setup can greatly improve the troubleshooting capabilities while moving the hard limits of your particular setup from the 3rd party solution to your own infrastructure where full observability is possible.

Keep standard Log4j logging enabled, or at least a variation that continues to log errors, to allow troubleshooting. Without this, you won’t be able to troubleshoot OpenTelemetry log export failures.

Do a performance test of your application before enabling this feature in a production environment. While our performance tests did not show huge performance costs, Direct telemetry stream is not a zero cost feature and therefore must be tested.

Always consider the limits of your third-party solution and configure your exports accordingly. Mule runtime can’t compensate for poor downstream performance scenarios without compromising its own stability.

The most common cause of export errors, delays, or drops is a bottleneck in the ingestion pipeline (the OpenTelemetry Collector and the third-party solution). Make sure these components operate within acceptable performance limits.

Be aware that tuning changes, such as modifying the queue size, can impact Mule application performance. Test these changes carefully before applying them to production.