Stream Host Installation

Overview

On XMPro 5.0, Stream Hosts run as pods in the same AKS or EKS cluster as the platform. There are two ways to deploy them, both driven by the layered Terraform deployment described in the AWS Kubernetes (EKS) and Azure Kubernetes (AKS) guides:

Default Stream Host - the site application layer (layer 3) can install one Stream Host alongside the platform. Its Collection is created for it during the DS database migration, so it registers with Data Stream Designer on first start.
Additional Stream Hosts - the Stream Hosts layer (layer 4, examples/layered/stream-hosts) deploys one standalone Stream Host per Collection. Each is its own Helm release of the streamhost chart with its own Collection ID and secret.

One Stream Host serves one Collection. To add capacity, add Collections and Stream Hosts rather than pointing several Stream Hosts at the same Collection.

Prerequisites

Layers 1 to 3 of the AKS or EKS deployment applied, with Data Stream Designer reachable at its public URL
kubectl access to the cluster and Terraform installed, as set up in the deployment guide
A Collection in Data Stream Designer for each additional Stream Host. See Manage Collections to create one and copy its ID and Secret.
The outputs of the site infrastructure and application layers (see the tables below)

Default Stream Host

The application layer's enable_default_stream_host variable controls the default Stream Host. It defaults to true on AKS and false on EKS, where the supported model is independent Stream Hosts through the Stream Hosts layer.

Application layer variable	Description
`enable_default_stream_host`	Deploy the default in-cluster Stream Host with the platform.
`stream_host_collection_id`	Collection ID for the default Stream Host. Leave empty to generate one. The DS database migration creates the Collection.
`stream_host_collection_secret`	Collection secret for the default Stream Host. Leave empty to generate one.
`stream_host_replica_count`	Replica count for the default Stream Host. Defaults to `1`.
`stream_host_image_tag`	Stream Host image tag. Empty falls back to the platform `image_tag`.

Set these in the application layer's terraform.tfvars and apply. The default Stream Host appears in Data Stream Designer under Stream Hosts once the pod is running.

Additional Stream Hosts

The Stream Hosts layer lives next to the other layers in the deployment repository and is applied after the application layer. From the application layer directory:

Mac/Linux:

cd ../stream-hosts
cp terraform.tfvars.example terraform.tfvars

Windows (PowerShell):

cd ..\stream-hosts
Copy-Item terraform.tfvars.example terraform.tfvars

Configure the layer

Fill terraform.tfvars with the values from your site. Most come from the outputs of the infrastructure and application layers.

AKS:

Variable	Value
`subscription_id`	Azure subscription
`cluster_name`	AKS cluster name, from layer 1
`name_suffix`	Site identifier, the same as the infrastructure and application layers
`aks_resource_group_name`	Resource group of the AKS cluster, from layer 1's `resource_group_name` output
`resource_group_name`	Site resource group, from the infrastructure layer output
`key_vault_name`	Site Key Vault, from the infrastructure layer output
`namespace`	Site namespace, from the application layer output
`cluster_secret_store_name`	`azurekv-<name_suffix>` unless you renamed it in the application layer
`image_registry`	`{{ACR_URL}}`
`image_tag`	`{{VERSION}}`
`image_pull_secrets`	Required for the XMPro registry. A list holding the application layer's `image_pull_secret_name` output, `["xmpro-registry"]` by default
`ds_server_url`	Data Stream Designer URL, from the application layer's `ds_url` output

EKS:

Variable	Value
`aws_region`	AWS region, the same as the other layers
`name_suffix`	Site identifier, from the infrastructure layer output
`cluster_name`	EKS cluster name, from layer 1
`namespace`	Site namespace, `name_suffix` by default
`cluster_secret_store_name`	`aws-secrets-<name_suffix>` unless you renamed it in the application layer
`secrets_kms_key_arn`	From the infrastructure layer's `secrets_kms_key_arn` output
`image_registry`	`{{ACR_URL}}`
`image_tag`	`{{VERSION}}`
`image_pull_secrets`	Required for the XMPro registry. A list holding the application layer's `image_pull_secret_name` output, `["xmpro-registry"]` by default
`ds_server_url`	Data Stream Designer URL, from the application layer's `ds_url` output

Define the Stream Hosts

Each entry in the stream_hosts map is one Stream Host. The key is a short lowercase identifier used in the Kubernetes and secret resource names.

stream_hosts = {
  ingest = {
    collection_id     = "<collection-id-from-DS>"
    collection_secret = "<collection-secret-from-DS>"
    gateway_name      = "SH-AKS-site1-ingest"
  }
  agg = {
    collection_id     = "<collection-id-from-DS>"
    collection_secret = "<collection-secret-from-DS>"
    resources = {
      limits   = { cpu = "2000m", memory = "8Gi" }
      requests = { cpu = "500m",  memory = "2Gi" }
    }
  }
}

Field	Description
`collection_id`	Required. The Collection ID from Data Stream Designer.
`gateway_name`	Display name in Data Stream Designer. Defaults to `SH-<name_suffix>-<key>` on AKS and `SH-EKS-<name_suffix>-<key>` on EKS.
`collection_secret`	Required. The Collection's Secret from Data Stream Designer. The layer stores it in Key Vault (AKS) or Secrets Manager (EKS) and External Secrets Operator delivers it to the pod. If it doesn't match the Collection, Data Stream Designer rejects the Stream Host and the pod restarts.
`image_tag`	Override the layer-wide `image_tag` for this instance.
`resources`	Container CPU and memory. The chart default is `500m` / `1Gi` limits and `100m` / `256Mi` requests. See Sizing.
`enable_app_insights`	Send telemetry to Application Insights. Requires `app_insights_secret_name` at the layer level.

The AKS layer also accepts, per entry:

Field	Description
`replica_count`	Number of pods for this Stream Host. Defaults to `1`.
`image_variant`	Image tag suffix for a runtime variant, for example `bookworm-slim-python3.12`. Appended to the effective tag.

On EKS each entry runs one pod, and the layer-wide enable_otel and otel_* variables push OTLP telemetry from the Stream Hosts to a collector. See Configure OpenTelemetry.

Apply

terraform init
terraform apply

Confirm each Stream Host shows online in Data Stream Designer under Stream Hosts. To add another, add an entry to stream_hosts and apply again. To remove one, delete its entry and apply.

Sizing

Start at a tier, monitor, then adjust. Actual needs depend on stream count, event rate and aggregation complexity.

Tier	`limits`	`requests`
Very Small (chart default)	`cpu = "500m"`, `memory = "1Gi"`	`cpu = "100m"`, `memory = "256Mi"`
Small	`cpu = "1000m"`, `memory = "4Gi"`	`cpu = "250m"`, `memory = "1Gi"`
Medium	`cpu = "2000m"`, `memory = "8Gi"`	`cpu = "500m"`, `memory = "2Gi"`
Large	`cpu = "4000m"`, `memory = "16Gi"`	`cpu = "1000m"`, `memory = "4Gi"`

Updating

To move Stream Hosts to a new release, change image_tag in terraform.tfvars and apply. The Helm release rolls the pods. Set image_tag on a single entry to move one Stream Host on its own.

Troubleshooting

Stream Host not online in Data Stream Designer

kubectl get pods -n <namespace> -l app=xmpro-sh-<key>-stream-host-sh
kubectl logs -n <namespace> deployment/xmpro-sh-<key>-stream-host-sh -c sh

Check ds_server_url matches the Data Stream Designer URL and is reachable from the cluster.
Check the Collection ID exists in Data Stream Designer, and that collection_secret matches the Collection's Secret. Unauthorized in the logs means it doesn't.
kubectl get externalsecret -n <namespace> xmpro-sh-<key> should show SecretSynced. If not, the ClusterSecretStore name is wrong or External Secrets Operator cannot read the secret.

Stream Host still rejected after you change collection_secret

External Secrets Operator refreshes the pod's secret once an hour, so the pod keeps the old value after an apply. On EKS the layer also doesn't update a secret it has already stored. To push the new value:

# EKS only: replace the stored secret so the new value is written
terraform apply -replace='module.stream_host["<key>"].aws_secretsmanager_secret_version.collection_secret'

# AKS and EKS: refresh the Kubernetes secret, then restart the pod
kubectl annotate externalsecret xmpro-sh-<key> -n <namespace> force-sync=$(date +%s) --overwrite
kubectl rollout restart -n <namespace> deployment/xmpro-sh-<key>-stream-host-sh

Pod stuck in ImagePullBackOff

The registry credential is missing or expired. Confirm image_pull_secrets names the application layer's pull secret and that the secret exists in the namespace.

Pod restarting

The container has a read-only root filesystem and runs as a non-root user. Agents can write only to /tmp, /home/app and /app/Cache.

Last modified: September 28, 2026

Table of Contents