Table of Contents

Configure OpenTelemetry

Note

OpenTelemetry export is opt-in. Every XMPro product ships with it disabled, and no telemetry is sent until you supply an OTLP endpoint.

Introduction

XMPro products export traces, metrics, and logs over OTLP to an endpoint you provide, either a collector you run or a vendor service that accepts OTLP directly. Configuration uses the same OpenTelemetry section in every product, so once you have configured one service the rest follow the same shape. The standard OpenTelemetry environment variables are also honoured, which means an existing collector deployment usually needs little XMPro-specific configuration.

This guide covers what must be set before export starts, the full list of configuration keys, how environment variables interact with appsettings.json, configuration for each deployment type, and troubleshooting when telemetry doesn't arrive. If you are new to XMPro's telemetry, read Observability and OpenTelemetry first.

Minimum Configuration

For most deployments the whole task is four values. Set them, then point your collector at whichever backend you run:

{
  "OpenTelemetry": {
    "Enabled": true,
    "Endpoint": "http://collector:4317",
    "Protocol": "Grpc",
    "Headers": "Authorization=Bearer your-token-here"
  }
}

Give the base URL only. For HttpProtobuf the exporter appends the signal path itself, as described in Choosing a Protocol. Drop Headers if your collector does not require authentication.

That is the complete setup. Exporting to a Backend covers routing the signals onward. The rest of this page is detail for tuning and for when something does not arrive.

Enabling OpenTelemetry

Three conditions must all be satisfied before a product exports anything. They are evaluated in order:

  1. The kill switch must not be set. If OTEL_SDK_DISABLED is true, OpenTelemetry is disabled regardless of every other setting.
  2. The feature must be enabled. OpenTelemetry:Enabled must be true.
  3. An endpoint must resolve. Either OTEL_EXPORTER_OTLP_ENDPOINT or OpenTelemetry:Endpoint must contain a value.
Important

If any of the three conditions is not met, the product starts normally and exports nothing. No error is logged and no warning is raised, because a disabled telemetry pipeline is a valid configuration. If telemetry is missing, check these three conditions first.

Configuration Keys

All keys live under the OpenTelemetry section:

Key Default Description
Enabled false Master switch for OTLP export. Must be true to export anything.
Endpoint (empty) OTLP endpoint, for example http://collector:4317. Required unless supplied by OTEL_EXPORTER_OTLP_ENDPOINT.
Protocol Grpc Either Grpc or HttpProtobuf.
Headers (empty) Export headers in key=value,key2=value2 format, used for authentication.
Tracing:ExcludedPaths /version, /health, /health/ping Request paths that do not produce trace spans.
Logging:MinimumLevel Information Minimum severity for log export. Accepts Verbose, Debug, Information, Warning, Error, or Fatal.

A complete appsettings.json section looks like this:

{
  "OpenTelemetry": {
    "Enabled": true,
    "Endpoint": "http://collector:4317",
    "Protocol": "Grpc",
    "Headers": "Authorization=Bearer your-token-here",
    "Tracing": {
      "ExcludedPaths": [ "/version", "/health", "/health/ping" ]
    },
    "Logging": {
      "MinimumLevel": "Information"
    }
  }
}

Stream Host has no HTTP server, so Tracing:ExcludedPaths has no effect there and is omitted from its configuration file.

Using Environment Variables

XMPro reads configuration from environment variables as well as appsettings.json, which is usually the better choice for Kubernetes and container deployments.

XMPro keys use the xm__ prefix with double underscores replacing the colons:

xm__OpenTelemetry__Enabled=true
xm__OpenTelemetry__Endpoint=http://collector:4317
xm__OpenTelemetry__Protocol=HttpProtobuf
xm__OpenTelemetry__Tracing__ExcludedPaths__0=/health
xm__OpenTelemetry__Tracing__ExcludedPaths__1=/version
xm__OpenTelemetry__Logging__MinimumLevel=Warning

The standard OpenTelemetry environment variables are also read, and they take priority over the equivalent appsettings.json values:

Variable Effect
OTEL_SDK_DISABLED When true, disables OpenTelemetry entirely.
OTEL_EXPORTER_OTLP_ENDPOINT Overrides OpenTelemetry:Endpoint.
OTEL_EXPORTER_OTLP_HEADERS Overrides OpenTelemetry:Headers.
OTEL_RESOURCE_ATTRIBUTES Adds resource attributes to every signal.
Warning

OTEL_EXPORTER_OTLP_HEADERS replaces OpenTelemetry:Headers completely. The two values are not merged. If you set an API key in appsettings.json and a tracing header in the environment variable, only the environment variable is used and the API key is lost.

Note

The standard variables in the table above do not apply to Stream Host, which reads only xm__-prefixed names. Use xm__OpenTelemetry__Endpoint and xm__OpenTelemetry__Headers there. This also has the advantage of covering the log sink, as described under Stream Host.

Choosing a Protocol

Both OTLP transports are supported, and the port your collector listens on differs between them:

Protocol Default collector port When to use
Grpc 4317 Direct connections to a collector, typically inside the same network or cluster.
HttpProtobuf 4318 Collectors reached through an HTTP ingress, load balancer, or proxy.

When HttpProtobuf is used, the exporter appends the signal path to your endpoint, so supply the base URL only. An endpoint of https://collector.example.com becomes https://collector.example.com/v1/traces for traces and https://collector.example.com/v1/metrics for metrics.

Note

The application default is Grpc, but the XMPro Helm charts set HttpProtobuf because Kubernetes deployments usually reach the collector through an ingress. If you deploy with Helm and expect gRPC, set the protocol explicitly.

Important

Identity Server is an exception: it always uses HttpProtobuf and cannot be changed. The protocol is fixed in the product rather than read from configuration. The is.otel.protocol Helm value exists, but it is ignored, so setting it has no effect. If your collector listens for gRPC on 4317, Identity Server still sends HTTP/protobuf and its telemetry will not arrive, with no error raised. Make sure your collector also accepts OTLP over HTTP on 4318, and point Identity Server at that port. Every other service honours the Protocol setting.

Excluding Paths From Traces

High-frequency utility endpoints generate spans that add cost without adding insight. Three paths are excluded by default: /version, /health, and /health/ping. These are the endpoints described in Health Checks.

Pattern matching works as follows:

  • Matching is case-insensitive.
  • A pattern matches that exact path only, unless it ends with /*.
  • A pattern ending in /* matches the prefix and everything beneath it. /health matches only /health, while /health/* also matches /health/ping.
  • Leading slashes are added if you omit them, blank entries are dropped, and duplicates are removed.
  • The catch-all entries / and /* are ignored, so you cannot accidentally disable all tracing this way.
Warning

The catch-all protection above does not currently apply to Identity Server. Identity Server accepts /* as a pattern, and because it matches every request this silently disables all of its trace spans. Avoid /* in the Identity Server excluded paths and list the paths you want dropped instead.

Example: to exclude a metrics scrape endpoint in addition to the defaults, list the full set rather than just the new value:

{
  "OpenTelemetry": {
    "Tracing": {
      "ExcludedPaths": [ "/version", "/health/*", "/internal/metrics" ]
    }
  }
}
Important

Excluded paths apply to trace spans only. Requests to excluded paths still appear in ASP.NET Core request metrics, which is deliberate because those metrics are low cardinality and remain useful for availability monitoring. If you exclude /health and still see health requests in http.server.* metrics, that is expected behaviour.

Configuring Log Export

Logs are exported by Serilog rather than by the OpenTelemetry SDK, and OpenTelemetry:Logging:MinimumLevel sets the floor for what reaches the OTLP endpoint. The default is Information.

This setting can only filter further, not add detail. Your root Serilog configuration decides which events are produced at all, and only those events are offered to the OTLP exporter. Raising Logging:MinimumLevel to Debug will not produce debug logs if your Serilog minimum level is Information. To export more detail, lower the Serilog level first, then set the export floor.

Values outside the accepted range fall back to Information rather than suppressing export, so a typo will not silently stop your logs.

Log events written inside an active trace carry the trace and span identifiers automatically, so your backend can link a log line to the request that produced it. No additional configuration is required for this correlation.

Deployment-Specific Configuration

The keys are the same everywhere, but how you supply them depends on how XMPro is deployed.

Kubernetes With Helm

This section covers the XMPro chart, which deploys App Designer, Data Stream Designer, Subscription Manager, Identity Server, XMPro AI, and Stream Host. Each has an otel block in the Helm values:

ad:
  otel:
    enabled: true
    protocol: "HttpProtobuf"
    clusterName: "my-cluster"
    tracing:
      excludedPaths: []
    logging:
      minimumLevel: ""

Not every service supports every key. Identity Server and Stream Host each accept a subset:

Service Values key Supported settings
App Designer ad.otel enabled, protocol, clusterName, tracing.excludedPaths, logging.minimumLevel
Data Stream Designer ds.otel As above
Subscription Manager sm.otel As above
XMPro AI ai.otel As above
Identity Server is.otel enabled, clusterName, tracing.excludedPaths. Does not accept protocol or logging.minimumLevel. Identity Server always uses HttpProtobuf, so it needs the HTTP port (4318) rather than the gRPC port. Its log export level is the application default, Information.
Stream Host sh.otel enabled, protocol, clusterName, logging.minimumLevel. Has no tracing block, because Stream Host serves no HTTP traffic.

The endpoint and authentication headers are never set in values. They are read from the secret named by global.secretsName, using the keys xmpro-otel-endpoint and xmpro-otel-headers.

Kubernetes resource attributes are wired automatically. OTEL_RESOURCE_ATTRIBUTES is populated with k8s.namespace.name and k8s.pod.name from the pod metadata, plus k8s.cluster.name from clusterName.

Important

Stream Host reads configuration only from xm__-prefixed environment variables, so the chart supplies its endpoint and headers as xm__OpenTelemetry__Endpoint and xm__OpenTelemetry__Headers. The standard OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS variables described in Using Environment Variables do not apply to Stream Host. If you deploy Stream Host with the standalone chart instead of the bundled one, supply those two xm__ names through sh.envSecretRefs.

Warning

excludedPaths in Helm values overrides the application defaults by index rather than appending to them. Because the chart writes each entry as xm__OpenTelemetry__Tracing__ExcludedPaths__{index}, supplying two entries replaces only the first two defaults and leaves the third in place. Always list the complete set of paths you want, or leave the list empty to keep the defaults.

On an AKS deployment, the Application Insights connection provisioned with the cluster continues to operate independently. Enabling OpenTelemetry does not disable it, and no Application Insights setting needs to change.

Terraform

If you deploy AKS or EKS with the XMPro Terraform modules, you set these values as module variables rather than editing Helm values directly. The site application layer exposes:

Variable Type Default Notes
enable_otel bool false Master switch for OTLP export
otel_endpoint string "" Written to the xmpro-otel-endpoint secret key
otel_headers string "" Marked sensitive; written to the xmpro-otel-headers secret key
otel_protocol string "HttpProtobuf"
otel_cluster_name string "" Becomes the k8s.cluster.name resource attribute
otel_tracing_excluded_paths list(string) [] Empty keeps the application defaults
otel_logging_minimum_level string "" Empty uses the application default, Information

This is where the secret keys in the Helm section come from: the module writes otel_endpoint and otel_headers into the secret store under those names.

A minimal set of values:

enable_otel       = true
otel_endpoint     = "https://collector.example.com"
otel_cluster_name = "my-aks-cluster"
# otel_headers supplied via TF_VAR_otel_headers, not committed

Because otel_headers is marked sensitive, supply it through a pipeline variable or TF_VAR_otel_headers rather than committing it to a .tfvars file.

Note

otel_logging_minimum_level is validated at plan time against Verbose, Debug, Information, Warning, Error, and Fatal. An invalid value fails the plan. This is stricter than the application itself, which falls back to Information when the value is not recognised.

Warning

enable_otel means something different in the Azure App Service deployment. In the terraform-xmpro-azure module it does not enable OTLP export. It creates Azure diagnostic settings on the App Services and container instances to ship logs to an Event Hub, and it requires observability_eventhub_rule_id to be set as well. The in-application telemetry path on Azure App Service is Application Insights, not the OpenTelemetry export described on this page. Everything here applies to the Kubernetes deployments.

Stream Host

Stream Host runs as a worker process on Docker, as a Windows service, or as a console application. Configure it through appsettings.json or environment variables:

{
  "OpenTelemetry": {
    "Enabled": true,
    "Endpoint": "http://collector:4317",
    "Protocol": "Grpc",
    "Logging": {
      "MinimumLevel": "Information"
    }
  }
}

Because Stream Host serves no HTTP traffic, it produces no ASP.NET Core spans or metrics and the Tracing section does not apply. It does emit its own stream and agent instruments in addition to the standard runtime and process metrics.

Important

On Stream Host, set authentication headers with OpenTelemetry:Headers (or xm__OpenTelemetry__Headers) rather than the standard OTEL_EXPORTER_OTLP_HEADERS variable. The standard variable reaches the trace and metric exporters, but not the log sink, so an authenticated collector accepts traces and metrics while rejecting logs. Setting the value in configuration covers all three signals.

Verifying the Configuration

Once you have enabled export and restarted the service, confirm telemetry is arriving:

  1. Check your collector is receiving data. Most collectors expose their own metrics for received spans and metric points. A collector that reports zero received items means the problem is on the XMPro side or in the network path.
  2. Look for the service name. Search your backend for the service.name values listed in Observability and OpenTelemetry. A service that has never exported will not appear at all.
  3. Generate traffic on a path that is not excluded. Requests to /health produce no spans by design, so use a normal application page or API call when testing traces.
  4. Confirm resource attributes. Each signal should carry service.version and xmpro.build.commit. Their presence confirms the resource configuration is being applied.

Troubleshooting

  1. No telemetry of any kind arrives

    Fix: Work through the three activation conditions in Enabling OpenTelemetry. Confirm OTEL_SDK_DISABLED is not set to true anywhere in the environment, confirm OpenTelemetry:Enabled is true, and confirm an endpoint is present in either OTEL_EXPORTER_OTLP_ENDPOINT or OpenTelemetry:Endpoint.

  2. Connection refused or timeouts in the service logs

    Fix: Check the protocol and port agree. Grpc normally uses 4317 and HttpProtobuf uses 4318. Sending gRPC to an HTTP receiver, or the reverse, fails even when the host and port are reachable.

  3. Traces and metrics arrive but logs do not

    Fix: Logs use a separate export path through Serilog. Check OpenTelemetry:Logging:MinimumLevel and confirm your root Serilog minimum level is producing events at that severity or lower.

    If your collector requires authentication and you set the credentials with OTEL_EXPORTER_OTLP_HEADERS, that variable does not reach the Stream Host log sink, so the collector rejects the logs while still accepting traces and metrics. Set OpenTelemetry:Headers in configuration instead, which covers all three signals.

  4. Authentication failures at the collector

    Fix: Confirm the header format is key=value,key2=value2 with no surrounding quotes. Remember that OTEL_EXPORTER_OTLP_HEADERS replaces the appsettings.json value rather than merging with it, so a header set in both places may not be what you expect.

  5. Requests to health endpoints still appear in metrics

    Fix: This is expected. Excluded paths suppress trace spans only. See Excluding Paths From Traces.

  6. A path added to excludedPaths in Helm is still traced

    Fix: Helm overrides the defaults by index rather than appending. List the complete set of paths you want excluded, including the defaults you wish to keep.

Security Considerations

Caution

Exported log events may contain values from your data and configuration. Treat the OTLP endpoint as a destination that receives potentially sensitive information, and secure it accordingly.

Before enabling export in production:

  • Use TLS for the OTLP endpoint, and prefer https over http for HttpProtobuf.
  • Authenticate the export path using Headers, and store the value in a secret store rather than in appsettings.json.
  • Review OpenTelemetry:Logging:MinimumLevel. Lower severities produce more detail and therefore more opportunity to export values you would rather keep inside the deployment.
  • Restrict network access to the endpoint so that only XMPro services can reach it.

Summary

Export starts only when the kill switch is clear, OpenTelemetry:Enabled is true, and an endpoint resolves, and a failure in any of the three is silent by design. Configuration is identical across products, with standard OpenTelemetry environment variables taking priority over appsettings.json and headers replacing rather than merging. Excluded paths reduce trace volume without affecting metrics, and log export runs through Serilog with its own severity floor that can filter but never add detail. Once telemetry is flowing, see Exporting to a Backend for routing it to your observability stack.

Last modified: September 28, 2026