Kiali – Documentation

News: KIALI-SECURITY-003 - Installation into ad-hoc namespaces

Tue, 11 May 2021 11:00:00 -0600

Description

Disclosure date: May 11, 2021
Affected Releases: prior to 1.33.0
Impact Score: 6.6 - AV:N/AC:L/PR:H/UI:N/S:C/C:L/I:L/A:L

A vulnerability was found in the Kiali Operator allowing installation of a specified image into any namespace.

Kiali users are exposed to this vulnerability if all the following conditions are met:

Kiali operator is used for installation.
Kiali CR was edited to install an image into an unapproved namespace.

This vulnerability is filed as CVE-2021-3495

Mitigation

If you can update:

Update to Kiali Operator v1.33.0 or later.

If you can not update:

Ensure only trusted individuals can create or edit a Kiali CRs (resources of kind “kiali”).

News: KIALI-SECURITY-002 - Authentication bypass when using the OpenID login strategy

Mon, 01 Jan 0001 00:00:00 +0000

Description

Disclosure date: March 5, 2021
Affected Releases: 1.26.0, 1.26.1, 1.26.2, 1.27.0, 1.28.0, 1.28.1, 1.29.0, 1.29.1, 1.30.0
Impact Score: 7.0 - AV:N/AC:L/PR:N/UI:N/S:C/C:L/I:L/A:N/E:F/RL:X/RC:C

A vulnerability was found in Kiali allowing an attacker to bypass the authentication mechanism. The vulnerability lets an attacker build forged credentials and use them to gain unauthorized access to Kiali.

Kiali users are exposed to this vulnerability if all the following conditions are met:

Kiali is setup with the openid authentication strategy.
As a result of configurations in both Kiali and your OpenID server, Kiali uses the implicit flow of the OpenID specification to negotiate authentication.
Kiali is setup with RBAC turned off.

This vulnerability is filed as CVE-2021-20278

Mitigation

If you can update:

Update to Kiali v1.31.0 or later.
If you need an earlier version, only Kiali 1.26.3 and 1.29.2 are fixed.

If you are locked with an older version of Kiali, you have three options:

Configure Kiali to use the authorization code flow of the OpenID specification; or
Configure Kiali to use the implicit flow of the OpenID specification and enable RBAC; or
Configure Kiali to use any of the other available authentication mechanisms.

News: KIALI-SECURITY-001 - Authentication bypass using forged credentials

Tue, 24 Mar 2020 11:00:00 -0600

Description

Disclosure date: March 25, 2020
Affected Releases: 0.4.0 to 1.15.0
Impact Score: 9.4 - AV:N/AC:L/PR:N/UI:N/S:U/C:L/I:H/A:H

A vulnerability was found in Kiali allowing an attacker to bypass the authentication mechanism. Currently, Kiali has four authentication mechanisms: login, token, openshift and ldap. All are vulnerable.

The vulnerability lets an attacker build forged credentials and use them to gain unauthorized access to Kiali.

Additionally, it was found that Kiali credentials were not being validated properly. Depending on the authentication mechanism configured in Kiali, this could facilitate unauthorized access into Kiali with forged and/or invalid credentials.

These vulnerabilities are filed as CVE-2020-1762 and CVE-2020-1764

Detection

Use the following bash script to check if you are vulnerable:

KIALI_VERSION=$(kubectl get pods -n istio-system -l app=kiali -o yaml | sed -n 's/^.*image: .*:v\(.*\)$/\1/p' | sort -u)
kubectl get deploy kiali -n istio-system -o yaml | grep -q LOGIN_TOKEN_SIGNING_KEY
TEST_KEY_ENV=$?
kubectl get cm kiali -n istio-system -o yaml | grep signing_key | grep -vq kiali
TEST_KEY_CFG=$?
VERSION_ENTRIES=(${KIALI_VERSION//./ })
echo "Your Kiali version found: ${KIALI_VERSION}"
[ ${VERSION_ENTRIES[0]} -lt "1" ] || ([ ${VERSION_ENTRIES[0]} -eq "1" ] && (\
  [ ${VERSION_ENTRIES[1]} -lt "15" ] || ([ ${VERSION_ENTRIES[1]} -eq "15" ] && ( \
  [ ${VERSION_ENTRIES[2]} -le "0" ])))) && echo "Your Kiali version is vulnerable"
[ $TEST_KEY_ENV -eq 1 ] && [ $TEST_KEY_CFG -eq 1 ] && echo "Your Kiali configuration looks vulnerable"

The script output will be similar to this:

Your Kiali version found: 1.14.0
Your Kiali version is vulnerable
Your Kiali configuration looks vulnerable

Mitigation

Update to Kiali 1.15.1 or later.

Alternatively, if you cannot update to version 1.15.1, mitigation is possible by setting a secure signing key when deploying Kiali. If you installed via Kiali operator, you could use the following bash script:

SIGN_KEY=$(chars=abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890; for i in {1..20}; do echo -n "${chars:RANDOM%${#chars}:1}"; done; echo)
kubectl get kiali -n $(kubectl get kiali --all-namespaces --no-headers -o custom-columns=NS:.metadata.namespace) -o yaml | sed "s/spec:/spec:\n    login_token:\n      signing_key: $SIGN_KEY/" | kubectl apply -f -

If you installed via Istio helm charts or istioctl command, you could use the following bash script:

KIALI_INSTALL_NAMESPACE=istio-system
SIGN_KEY=$(chars=abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890; for i in {1..20}; do echo -n "${chars:RANDOM%${#chars}:1}"; done; echo)
kubectl get cm kiali -n $KIALI_INSTALL_NAMESPACE -o yaml | sed "s/server:/login_token:\\n      signing_key: $SIGN_KEY\\n    server:/" | kubectl apply -f -
kubectl delete pod -l app=kiali -n $KIALI_INSTALL_NAMESPACE

Docs: Introduction

Mon, 01 Jan 0001 00:00:00 +0000

So far, we know how good Kiali can be to understand applications, their relationships with each other and with external applications.

In the previous tutorial, Kiali was setup to observe just a single cluster. Now, we will expand its capabilities to observe more than one cluster. The extra clusters are remotes, meaning that there is not a control plane on them, they only have user applications.

This topology is called primary-remote and it is very useful to spread applications into different clusters having just one primary cluster, which is where Istio and Kiali are installed.

This scenario is a good choice when as an application administrator or architect, you want to give a different set of clusters to different sets of developers and you also want all these applications to belong to the same mesh. This scenario is also very helpful to give applications high availability capabilities while keeping the observability together (we are referring to just applications in terms of high availability, for Istio, we might want to install a multi-primary deployment model, which is on the roadmap for the multicluster journey for Kiali).

In this tutorial we will be deploying Istio in a primary-remote deployment. At first, we will install the “east” cluster with Istio, then we will add the “west” remote cluster and join it to the mesh. Then we will see how Kiali allows us to observe and manage both clusters and their applications. Metrics will be aggregated into the “east” cluster using Prometheus federation and a single Kiali will be deployed on the “east” cluster.

If you already have a primary-remote deployment, you can skip to instaliing Kiali.

Docs: Introduction

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

Kiali uses Jaeger as a default distributed tracing backend. In this tutorial, we will replace it for Grafana Tempo.

We will setup a local environment in minikube, and install Kiali with Tempo as a distributed backend. This is a simplified architecture diagram:

We will install Tempo with the Tempo Operator and enable Jaeger query frontend to be compatible with Kiali in order to query traces.
We will setup Istio to send traces to the Tempo collector using the zipkin protocol. It is enabled by default from version 3.0 or higher of the Tempo Operator.
We will install MinIO and setup it up as object store, S3 compatible.

Environment

We use the following environment:

Istio 1.18.1
Kiali 1.72
Minikube 1.30
Tempo operator TempoStack v3.0

There are different installation methods for Grafana Tempo, but in this tutorial we will use the Tempo operator.

Docs: Prerequisites

Mon, 01 Jan 0001 00:00:00 +0000

Platform Setup

This tutorial assumes you have access to a Kubernetes cluster with Istio installed.

This tutorial is being updated and qualified using:

a Kind (Kubernetes in Docker) cluster with Helm-based Kiali install.
an OpenShift cluster with Kiali Operator install (qualification pending).

Tip

Platform-dependent tasks will be indicated with a special note like this.

Version pins will be recorded here as the tutorial is re-qualified. Use current stable releases of Kind, Istio, and Kiali unless noted otherwise.

Set up a Kind Cluster

Kind runs a local Kubernetes cluster using Docker. Istio and this tutorial also require a way to assign external IPs to LoadBalancer services (for the Istio ingress gateway).

Prerequisites

Install the following tools:

The Istio install script in the next section downloads Istio (including istioctl) if it is not already present.

Create the cluster

The Kiali project provides a script that creates a Kind cluster with MetalLB configured for LoadBalancer services. If you have the Kiali source repository, run:

./hack/start-kind.sh --name travels-tutorial

This creates a two-node Kind cluster named travels-tutorial with a MetalLB load balancer. Verify the cluster context:

kubectl config use-context kind-travels-tutorial
kubectl cluster-info

Kind

If you prefer not to use the Kiali script, follow the Kind load balancer guide to enable LoadBalancer services before installing Istio.

Install Istio

Kind

The Kiali project provides a script that installs Istio with the demo profile and telemetry addons (Prometheus, Grafana, Jaeger). From the Kiali source repository, run:

./hack/istio/install-istio-via-istioctl.sh -c kubectl -cp demo

This downloads Istio (if needed), installs the control plane and ingress gateway, and deploys the addons used later in this tutorial.

Verify the installation:

kubectl get pods -n istio-system
kubectl get svc istio-ingressgateway -n istio-system

The istio-ingressgateway service should show an EXTERNAL-IP (MetalLB assigns this on Kind). Join the Mesh uses that address for ingress.

Kind

If istio-ingressgateway stays <pending>, confirm MetalLB is running: kubectl get pods -n metallb-system

Prefer to install Istio yourself? Follow the Istio Getting Started guide using the demo profile, and install the Prometheus and Jaeger addons from ${ISTIO_HOME}/samples/addons/.

OpenShift

Follow the Istio OpenShift platform setup to install Istio on your cluster.

Alternatively, from the Kiali source repository, the install script defaults to the openshift profile when using oc:

./hack/istio/install-istio-via-istioctl.sh -c oc

Verify the control plane is running:

oc get pods -n istio-system

Join the Mesh uses an OpenShift route to expose the ingress gateway.

Install Kiali

Remove any Kiali installed from the Istio addons bundle before proceeding (set ISTIO_HOME to your Istio install directory, or the path under kiali/_output/ if you used the hack script):

kubectl delete -f ${ISTIO_HOME}/samples/addons/kiali.yaml --ignore-not-found

This tutorial uses different install methods depending on the platform. On Kind, a standalone Helm install keeps setup minimal. On OpenShift, install via the Kiali Operator — the recommended production method.

Kind

Install the Kiali server using the Quick Start Helm instructions. The Istio install above deploys Jaeger, but Kiali does not enable tracing integration by default — enable it explicitly:

helm install \
  --namespace istio-system \
  --set auth.strategy="anonymous" \
  --set external_services.tracing.enabled=true \
  --set external_services.tracing.internal_url="http://tracing.istio-system:16685/jaeger" \
  --repo https://kiali.org/helm-charts \
  kiali-server \
  kiali-server

Wait for the Kiali deployment to become ready:

kubectl rollout status deployment/kiali -n istio-system --timeout=300s
kubectl get pods,svc -n istio-system -l app.kubernetes.io/name=kiali

Confirm tracing is configured (the enabled field should be true):

kubectl get configmap kiali -n istio-system -o jsonpath='{.data.config\.yaml}' | grep -A2 'tracing:'

The kiali-server Helm chart is intended for demo and evaluation. For production clusters, use the Kiali Operator.

OpenShift

Install the Kiali Operator from OperatorHub in the OpenShift console, then create a Kiali CR in the istio-system namespace.

For a minimal tutorial setup with anonymous login, you can install the operator and CR in one step using Helm:

helm repo add kiali https://kiali.org/helm-charts
helm install \
  --set cr.create=true \
  --set cr.namespace=istio-system \
  --set cr.spec.auth.strategy="anonymous" \
  --set cr.spec.external_services.tracing.enabled=true \
  --set cr.spec.external_services.tracing.internal_url="http://tracing.istio-system:16685/jaeger" \
  --namespace kiali-operator \
  --create-namespace \
  kiali-operator \
  kiali/kiali-operator

Wait for the operator to reconcile the Kiali CR:

kubectl wait --for=condition=Successful kiali kiali -n istio-system --timeout=300s
kubectl get pods,svc -n istio-system -l app.kubernetes.io/name=kiali

See Creating and updating the Kiali CR for customization options. Production OpenShift deployments typically use the openshift auth strategy instead of anonymous.

Access the Kiali UI

Kind

Port-forward the Kiali service to your local machine:

kubectl port-forward svc/kiali 20001:20001 -n istio-system

Open http://localhost:20001/ in your browser.

The Kiali repo also provides a convenience script: ./hack/kiali-port-forward.sh

OpenShift

The Kiali operator creates an OpenShift route by default. Get the URL:

oc get route kiali -n istio-system

Open the route host in your browser (for example, https://<route-host>/).

See Accessing Kiali for more options.

After the Prerequisites you should be able to access Kiali. Verify its version by clicking the “?” icon and selecting “About”:

Docs: Quick Start

Mon, 01 Jan 0001 00:00:00 +0000

Run Kiali locally

Kiali can be run directly on your machine without being installed into a Kubernetes cluster. It uses your kubeconfig to connect to your cluster(s). If needed, it can port-forward into the cluster to connect to your external services (prometheus, tracing, istio, grafana).

Running Kiali locally is currently experimental. Functionality may change between releases.

Download the Kiali binary from the Kiali GitHub releases page for your OS and Arch.

Start Kiali which runs the backend server on localhost and opens your default browser to the Kiali UI.

kiali run

To see the full list of options

kiali run --help

If you want to run Kiali locally without a Prometheus instance, use the --disable-prometheus flag:

kiali run --disable-prometheus

This is useful for quickly exploring Kiali’s non-metrics features (workloads, services, Istio configuration, mesh topology) without needing a Prometheus deployment. See Disabling Prometheus for more details.

If the cluster name in your kubeconfig does not match the cluster name in Istio you can override this with --cluster-name-overrides kubeconfig-name=istio-cluster-name. The flag is a comma separated list so you can override as many names as you need.

Install Kiali

You can quickly install Kiali into your cluster via one of the following two methods.

These instructions are not recommended for production environments. Find more detailed information on installing Kiali, see the installation guide.

Before you install Kiali you must have already installed Istio. For full functionality, you should also install a telemetry storage addon (i.e. Prometheus), though Kiali can run without it if you disable Prometheus in the configuration. You might also consider installing Istio’s optional tracing addon (i.e. Jaeger) and optional Grafana addon but those are not required by Kiali. Refer to the Istio documentation for details.

Install via Istio Addons

If you downloaded Istio, the easiest way to install and try Kiali is by running:

kubectl apply -f ${ISTIO_HOME}/samples/addons/kiali.yaml

To uninstall:

kubectl delete -f ${ISTIO_HOME}/samples/addons/kiali.yaml --ignore-not-found

Install via Helm

Only Helm v3 has been tested. Previous Helm versions may or may not work.

To install the latest version of Kiali Server using Helm, run the following command:

helm install \
  --namespace istio-system \
  --set auth.strategy="anonymous" \
  --repo https://kiali.org/helm-charts \
  kiali-server \
  kiali-server

If you get a validation error, you may have to pass the option --disable-openapi-validation (this is needed on some versions of OpenShift, for example).

To uninstall:

helm uninstall --namespace istio-system kiali-server

Access to the UI

Run the following command:

kubectl port-forward svc/kiali 20001:20001 -n istio-system

Then, access Kiali by visiting https://localhost:20001/ in your preferred web browser.

News: Release Notes

Mon, 01 Jan 0001 00:00:00 +0000

For additional information check our sprint demo videos and blogs.

2.33.0

Release: October 05, 2026

Features:

Fixes:

The following field is no longer used by the Kiali CR and will be ignored if currently set.

spec.kiali_feature_flags.ui_defaults.i18n.show_selector

2.32.0

Release: September 14, 2026

Features:

Fixes:

Fix Mesh page duplicate istiod→ztunnel edges when multiple control planes are deployed

2.31.0

Release: August 21, 2026

Features:

Fixes:

2.30.0

Release: August 03, 2026

Features:

Fixes:

Upgrade Change Notes:

The fix for custom health status configuration could affect existing users. Degraded health status could now be reported as Failure. The behavior is correct, but could be unexepected. If affected, ensure proper setting of the failure threshold in your custom health configuration.

2.29.0

Release: July 13, 2026

Features:

Fixes:

2.28.0

Release: June 22, 2026

Features:

Fixes:

2.27.0

Release: June 01, 2026

Features:

Fixes:

The following fields are no longer used by the Kiali CR and will be ignored if currently set. The standard constant values are now used.

spec.istio_labels.egress_gateway_label
spec.istio_labels.ingress_gateway_label
spec.istio_labels.injection_label_name
spec.istio_labels.injection_label_rev

2.26.0

Release: May 29, 2026

Features:

Fixes:

2.25.0

Release: April 20, 2026

Features:

Fixes:

2.24.0

Release: March 30, 2026

Features:

Fixes:

Upgrade Change Notes:

kiali_health_status metric

Starting in v2.22, when the health cache and kiali metrics are both enabled (true by default) Kiali would also write a new metric, kiali_health_status. The initial implementation proved to be too heavy from a cardinality perspective. This metric has been redefined in v2.24 and will now generate a much lower cardinality of time series. Also, it is now opt-in, controlled by spec.server.observability.metrics.health_status.enabled. So, by default in v2.24, this metric will be disabled. The metric name remains the same, although attributes have been altered. Unless manually manipulated, existing series will remain in Prometheus until they naturally expire.

2.23.0

Release: March 09, 2026

Features:

Fixes:

2.22.0

Release: February 16, 2026

Features:

Fixes:

Upgrade Change Notes:

Health Status Pre-Compute and Caching

Kiali v2.22 introduces health status pre-compute and caching which is enabled by default. Production mesh sizes are growing and Kiali render times have been increasing, particularly for the Overview and List pages. In response, Kiali v2.22 changes its approach to health status calculation. In prior versions Kiali calculated health “on-demand”, based on the user’s selected duration, configuration settings, and other information, such as pod status. Starting in v2.22 Kiali will pre-calculate health status using a single, configurable duration, 5 minutes by default. The cached values increase the responsiveness of the Overview and List pages. Other pages, such as the Traffic graph and Detail pages will continue to calculate health status on-demand, and is based on the user’s selected duration. Users may notice that the Duration Dropdown selector has been removed from the Overview and List pages.

Users may notice an increase in backend resource utilization, as the Kiali server will now be calculating and refreshing health status, independent of user sessions. The Kiali CR introduces the following new configuration:

spec.health_config.compute.duration: 5m spec.health_config.compute.refresh_interval: 3m spec.health_config.compute.timeout: 10m

spec.kiali_internal.health_cache.enabled: false

It is recommended to keep the health cache enabled, as not all features will fall back to on-demand calculation.

Any questions, comments or feedback appreciated. Visit #kiali on Istio Slack or start a Discussion in Github at https://github.com/kiali/kiali.

2.21.0

Release: January 26, 2026

Features:

Fixes:

Upgrade Change Notes:

Traffic Graph Caching

Kiali v2.21 introduces traffic graph caching. Enabled by default. When a user navigates to the Traffic Graph and renders the initial graph, Kiali will start a background job to regenerate the graph periodically, based on the refresh interval set in the UI. The background job will cache the resulting graph and return it on subsequent UI requests. This can greatly improve re-render times, especially for larger graphs. Note that the initial graph render time will be unchanged. It is still recommended to use “Manual” refresh when working with large meshes, in order to fully define the desired graph before performing the initial request. Any fundamental change to the graph definition will invalidate the cache and restart a new refresh job. Users can navigate away and then back to the traffic graph, and resume with the latest cached graph, within the timeout period (10m by default).

Backend resource utilization may be affected, although is not anticipated to change significantly. The caching can be disabled in the Kiali config via:

spec.kiali_internal.graph_cache.enabled: false

Any questions, comments or feedback appreciated. Visit #kiali on Istio Slack or start a Discussion in Github at https://github.com/kiali/kiali.

2.20.0

Release: December 22, 2025

Features:

Fixes:

2.19.0

Release: November 24, 2025

Features:

Fixes:

2.18.0

Release: November 03, 2025

Features:

Fixes:

2.17.0

Release: October 13, 2025

Features:

Fixes:

Upgrade Change Notes:

The following fields are no longer used by the Kiali CR and MUST be removed, if currently set.

spec.external_services.istio.registry

2.16.0

Release: September 22, 2025

Features:

Fixes:

Upgrade Change Notes:

Discovery Selectors

Kiali now properly supports Istio control planes deployed into different namespaces. As part of this support both the spec.istio_namespace and spec.external_services.istio.root_namespace configuration fields have been removed. As such, Kiali Discovery Selectors, when defined, must include Istio’s control plane namespace(s). If you are using Kiali Discovery Selectors, please ensure that this new requirement is met. Note that Kiali’s deployment namespace is always included, and so co-located Istio control planes will be discovered.

The following fields are no longer used by the Kiali CR and MUST be removed, if currently set.

spec.external_services.istio.root_namespace

2.15.0

Release: September 02, 2025

Features:

Fixes:

Upgrade Change Notes:

Version 2.15.0 introduces a CRD schema for Kiali. The CRD version has not changed. But, validation will now occur on the cluster when the Kiali CRs are created or modified.

2.14.0

Release: August 08, 2025

Features:

Fixes:

Upgrade Change Notes:

The following fields are no longer used by the Kiali CR and MUST be removed, if currently set.

no longer used
- spec.istio_namespace
- spec.in_cluster
- spec.deployment.remote_secret_path
now auto-discovered
- spec.external_services.istio.config_map_name
- spec.external_services.istio.istiod_pod_monitoring_port
- spec.external_services.istio.envoy_admin_local_port
- spec.external_services.istio.istio_canary_version
- spec.external_services.istio.istio_injection_annotation
- spec.external_services.istio.istio_sidecar_annotation
- spec.external_services.istio.istiod_deployment_name
- spec.external_services.istio.istiod_pod_monitoring_port
- spec.external_services.istio.url_service_version

2.13.0

Release: July 21, 2025

Features:

Fixes:

MeshPage: Fix dataplane namespace count

2.12.0

Release: Jun 30, 2025

Features:

Fixes:

2.11.0

Release: Jun 09, 2025

Features:

Fixes:

2.10.0

Release: May 18, 2025

Features:

Fixes:

2.9.0

Release: Apr 25, 2025

Features:

Fixes:

2.8.0

Release: Apr 07, 2025

Features:

Fixes:

Deprecations:

Remove Cytoscape graph implementation

After a 4 month deprecation period the support for Kiali’s original, Cytoscape-based, graph implementation has ended. The ‘spec.kiali_feature_flags.ui_defaults.graph.impl’ configuration setting is no longer supported, and the sole implementation going forward uses PatternFly Topology. This has allowed for a significant cleanup of the Kiali code base, and removal of several dated dependencies. We’d like to thank the Cytoscape project, without which Kiali would not have existed. It is an excellent library, and our migration to PatternFly was motivated by a need to settle on a uniform component library.

2.7.0

Release: Mar 17, 2025

Features:

Fixes:

2.6.0

Release: Feb 21, 2025

Features:

Fixes:

Upgrade Notes:

The default values for the following Kiali CR fields have changed:

spec.istio_labels.app_label_name
- previous default: “app”
- new default: unset
spec.istio_labels.version_label_name
- previous default: “version”
- new default: unset

The change is related to the work done for Kiali issue 7603, included with this release. By default Kiali now allows for a mixing app labeling schemes, using the same set of app and version label pairings recognized by Istio:

service.istio.io/canonical-name, service.istio.io/canonical-revision
app.kubernetes.io/name, app.kubernetes.io/version
app, version

Users can configure a single labeling scheme by setting the existing CR fields, or leaving them set when upgrading.

2.5.0

Release: Feb 03, 2025

Features:

Fixes:

2.4.0

Release: Jan 13, 2025

Features:

Fixes:

2.3.0

Release: Dec 23, 2024

Features:

Fixes:

2.2.0

Release: Dec 02, 2024

Features:

Fixes:

2.1.0

Release: Nov 11, 2024

Features:

Fixes:

Deprecations:

RedHat Community Operator
- The community operator created confusion as to which operator to use on OpenShift. It will no longer be updated and will eventually be removed. OpenShift users are encouraged to use the productized operator, which is included with licensed copies of OpenShift.

2.0.0

Release: Oct 21, 2024

The first major Kiali release in over 5 years! There are two main reasons for the major version update:

There is a breaking change in Kiali’s namespace management configuration. To limit the namespaces accessible to Kiali, or made visible to users, Kiali v2.0 users will configure Discovery Selectors.

There is no longer support for the following deprecated configuration settings:

spec.deployment.accessible_namespaces
api.namespaces.exclude
api.namespaces.include
api.namespaces.label_selector_exclude
api.namespaces.label_selector_include

Kiali has a new traffic graph implementation.

The Cytoscape implementation has been deprecated and is no longer the default. Kiali has moved to PatternFly Topology to align with the rest of the Kiali interface, which is already implemented using PatternFly components. The old graph implementation will be removed as soon as the Kiali maintainers believe the new implementation has proven itself in the field. Until that time, it can still be accessed by setting:

spec:
  kiali_feature_flags:
    ui_defaults:
      graph:
        impl: "cy"

Features:

Fixes:

1.89.4

Release: Sep 30, 2024

Features:

The next feature release will be Kiali v 2.0.0

Fixes:

1.89.3

Release: Sep 09, 2024

Features:

The next feature release will be Kiali v 2.0.0

Fixes:

1.89.0

Release: Aug 19, 2024

Features:

Fixes:

1.88.0

Release: Jul 29, 2024

Features:

Fixes:

1.87.0

Release: Jul 08, 2024

Features:

Fixes:

Deprecations:

Kiali is deprecating its current namespace selection approach. For a description of the new mechanism see https://github.com/kiali/kiali/blob/master/design/KEPS/namespace-discovery/proposal.md. The following configuration is deprecated:
- spec.deployment.accessible_namespaces
Note that the following settings have already been deprecated and will soon be removed:
- api.namespaces.exclude
- api.namespaces.include
- api.namespaces.label_selector_exclude
- api.namespaces.label_selector_include

1.86.0

Release: Jun 17, 2024

Features:

Fixes:

1.85.0

Release: May 27

Features:

Fixes:

1.84.0

Release: May 06, 2024

Features:

Fixes:

1.83.0

Release: Apr 12, 2024

Features:

Fixes:

1.82.0

Release: Mar 22, 2024

Features:

Fixes:

1.81.0

Release: Mar 01, 2024

Features:

Kiali Server Helm Chart Support Custom NodePort

Fixes:

1.80.0

Release: Feb 09, 2024

Features:

Fixes:

1.79.0

Release: Jan 19, 2024

Features:

Fixes:

1.78.0

Release: Dec 08, 2023

Features:

Fixes:

Deprecations:

Kiali is deprecating use of the Jaeger exporter for Kiali’s own traces. Kiali will move to supporting only the OTel exporter.

1.77.0

Release: Nov 17, 2023

Features:

Fixes:

1.76.0

Release: Oct 27, 2023

Features:

Fixes:

1.75.0

Release: Oct 06, 2023

Features:

Fixes:

1.74.0

Release: Sep 15, 2023

Features:

Fixes:

Docs: Install Travel Demo

Mon, 01 Jan 0001 00:00:00 +0000

Deploy the Travel Demo

This demo application deploys several services grouped into three namespaces:

travel-control — business dashboard to configure traffic and view statistics
travel-portal — shop simulators that generate traffic
travel-agency — quote and pricing services

At this step, deploy the application without joining it to the service mesh. Sidecar injection is added in Join the Mesh.

Do not label the namespaces with istio-injection=enabled yet.

Kind

From the Kiali source repository, the Travel Demo install script creates the namespaces and deploys the manifests without enabling auto-injection:

./hack/istio/install-travel-agency-demo.sh -c kubectl -ai false

Alternatively, run the commands manually:

kubectl create namespace travel-agency
kubectl create namespace travel-portal
kubectl create namespace travel-control

kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travel_agency.yaml) -n travel-agency
kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travel_portal.yaml) -n travel-portal
kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travel_control.yaml) -n travel-control

If you have a local clone of the demos repository, you can apply the YAML files from travels/ instead of downloading them.

OpenShift

The install script also handles OpenShift-specific setup (NetworkAttachmentDefinitions and SecurityContextConstraints):

./hack/istio/install-travel-agency-demo.sh -c oc -ai false

Alternatively, substitute oc for kubectl in the manual commands above and add the necessary NetworkAttachmentDefinition to each namespace, along with SecurityContextConstraints for the service accounts in those namespaces (minimally, default).

Verify the deployment

Confirm workloads are running

Check that all deployments rolled out. Pods should show 1/1 ready — there are no sidecars yet:

kubectl get deployments -n travel-control
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
control   1/1     1            1           85s

kubectl get deployments -n travel-portal
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
travels   1/1     1            1           91s
viaggi    1/1     1            1           91s
voyages   1/1     1            1           91s

kubectl get deployments -n travel-agency
NAME            READY   UP-TO-DATE   AVAILABLE   AGE
cars-v1         1/1     1            1           96s
discounts-v1    1/1     1            1           96s
flights-v1      1/1     1            1           96s
hotels-v1       1/1     1            1           96s
insurances-v1   1/1     1            1           96s
mysqldb-v1      1/1     1            1           96s
travels-v1      1/1     1            1           96s

The portal workloads generate traffic to the travel agency services automatically.

Confirm in Kiali

Open Kiali and select Overview. This page summarizes what is in the service mesh — control planes, data planes, applications, and services that participate in the mesh.

Because the Travel Demo was deployed without sidecars, it is not part of the mesh yet. The Overview page should look much the same as before the demo was installed. You should still see the Istio control plane, and mesh counts such as data plane namespaces should remain at 0.

To see the demo namespaces, open the Namespaces page. The three Travel Demo namespaces should be listed:

travel-control
travel-portal
travel-agency

In the Type column, each namespace shows a - badge. Hover over the badge to see the tooltip Not part of the mesh:

Sidecar injection is covered in Join the Mesh, when these namespaces begin to appear as part of the mesh.

Understanding the demo application

The Travel Demo simulates a travel booking scenario across three namespaces. Traffic flows in one direction:

travel-control → travel-portal → travel-agency

The control dashboard configures how each portal shop behaves. The portal shops generate requests. The agency services respond with travel quotes.

How traffic flows

A typical request path looks like this:

Settings on the control dashboard determine how each portal shop sends traffic (device, user type, travel type, and volume).
A portal shop in travel-portal queries the travels service in travel-agency for available destinations.
The travels service queries hotels and returns destination options to the portal.
When a destination and travel type are selected, travels aggregates quotes from flights, cars, hotels, insurances, and discounts.
Cars, hotels, and flights may apply discounts depending on user type.

Travel Control namespace

The travel-control namespace hosts a business dashboard with two roles:

Configure every travel shop simulator — traffic ratio, device, user, and type of travel.
View a business summary of requests from travel-portal to travel-agency, grouped by shop, traffic type, and city.

Preview the Travel Dashboard (optional)

The control service is not exposed outside the cluster yet — that happens in Join the Mesh. To preview the dashboard now:

kubectl port-forward svc/control 8080:8080 -n travel-control

Open http://localhost:8080/ in your browser.

Travel Portal namespace

The travel-portal namespace runs several travel shop simulators (for example travels, viaggi, and voyages). Each shop represents a different portal with its own traffic characteristics.

Shops differ by channel (web or mobile), user type (new or registered), and travel type. Together they produce varied traffic patterns so you can explore realistic mesh scenarios in Kiali.

All portal shops call the travels service in the travel-agency namespace.

Travel Agency namespace

The travel-agency namespace provides backend quote services. The travels service is the main entry point: it receives a destination city and user, then aggregates a full travel budget from the supporting services:

hotels — lodging quotes
flights — airfare quotes
cars — car rental quotes
insurances — travel insurance quotes
discounts — special pricing for registered users
mysqldb — persistent storage for the demo

Each service calculates its portion independently; travels combines them into a single response.

Docs: Kiali and Tempo setup

Mon, 01 Jan 0001 00:00:00 +0000

Steps to install Kiali and Grafana Tempo

We will start minikube:

minikube start

It is a requirement to have cert-manager installed:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.12.0/cert-manager.yaml

Install the operator. It is important to download a version 3.0 or higher. In previous versions, the zipkin collector was not exposed, there was no way to change it as it was not defined in the CRD.

kubectl apply -f https://github.com/grafana/tempo-operator/releases/download/v0.3.0/tempo-operator.yaml

We will create the tempo namespace:

kubectl create namespace tempo

We will deploy minio, this is a sample minio.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  # This name uniquely identifies the PVC. Will be used in deployment below.
  name: minio-pv-claim
  labels:
    app: minio-storage-claim
spec:
  # Read more about access modes here: http://kubernetes.io/docs/user-guide/persistent-volumes/#access-modes
  accessModes:
    - ReadWriteOnce
  resources:
    # This is the request for storage. Should be available in the cluster.
    requests:
      storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: minio
spec:
  selector:
    matchLabels:
      app: minio
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        # Label is used as selector in the service.
        app: minio
    spec:
      # Refer to the PVC created earlier
      volumes:
        - name: storage
          persistentVolumeClaim:
            # Name of the PVC created earlier
            claimName: minio-pv-claim
      initContainers:
        - name: create-buckets
          image: busybox:1.28
          command:
            - "sh"
            - "-c"
            - "mkdir -p /storage/tempo-data"
          volumeMounts:
            - name: storage # must match the volume name, above
              mountPath: "/storage"
      containers:
        - name: minio
          # Pulls the default Minio image from Docker Hub
          image: minio/minio:latest
          args:
            - server
            - /storage
            - --console-address
            - ":9001"
          env:
            # Minio access key and secret key
            - name: MINIO_ACCESS_KEY
              value: "minio"
            - name: MINIO_SECRET_KEY
              value: "minio123"
          ports:
            - containerPort: 9000
            - containerPort: 9001
          volumeMounts:
            - name: storage # must match the volume name, above
              mountPath: "/storage"
---
apiVersion: v1
kind: Service
metadata:
  name: minio
spec:
  type: ClusterIP
  ports:
    - port: 9000
      targetPort: 9000
      protocol: TCP
      name: api
    - port: 9001
      targetPort: 9001
      protocol: TCP
      name: console
  selector:
    app: minio

And apply the yaml:

kubectl apply -n tempo -f minio.yaml

We will create a secret to access minio:

kubectl create secret generic -n tempo tempostack-dev-minio \
--from-literal=bucket="tempo-data" \
--from-literal=endpoint="http://minio:9000" \
--from-literal=access_key_id="minio" \
--from-literal=access_key_secret="minio123"

Install Grafana tempo with the operator. We will use the secret created in the previous step:

kubectl apply -n tempo -f - <<EOF
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: smm
spec:
  storageSize: 1Gi
  storage:
    secret:
      type: s3
      name: tempostack-dev-minio
  resources:
    total:
      limits:
        memory: 2Gi
        cpu: 2000m
  template:
    queryFrontend:
      jaegerQuery:
        enabled: true
        ingress:
          type: ingress
EOF

As an optional step, we can check if all the deployments have started correctly, and the services distributor has the port 9411 and the query frontend 16686:

kubectl get all -n tempo

(Optional) We can test if minio is working with a batch job to send some traces, in this case, to the open telemetry collector:

kubectl apply -f - <<EOF
apiVersion: batch/v1
kind: Job
metadata:
  name: tracegen
spec:
  template:
    spec:
      containers:
        - name: tracegen
          image: ghcr.io/open-telemetry/opentelemetry-collector-contrib/tracegen:latest
          command:
            - "./tracegen"
          args:
            - -otlp-endpoint=tempo-smm-distributor.tempo.svc.cluster.local:4317
            - -otlp-insecure
            - -duration=30s
            - -workers=1
      restartPolicy: Never
  backoffLimit: 4
EOF

And access the minio console:

kubectl port-forward --namespace istio-system service/minio 9001:9001

Install Istio with helm (Option I)

Istio can be installed with Helm following the instructions. The zipkin address needs to be set:

--set values.meshConfig.defaultConfig.tracing.zipkin.address="tempo-smm-distributor.tempo:9411"

And then, install Jaeger as Istio addon.

Install Istio using Kiali source code (Option II)

For development purposes, if we have Kiali source code, we can use the kiali hack scripts:

hack/istio/install-istio-via-istioctl.sh -c kubectl -a "prometheus grafana" -s values.meshConfig.defaultConfig.tracing.zipkin.address="tempo-smm-distributor.tempo:9411"

Install Kiali and bookinfo demo with some traffic generation

Install kiali:

helm install \
    --namespace istio-system \
    --set external_services.tracing.internal_url=http://tempo-smm-query-frontend.tempo:16685 \
    --set external_services.tracing.external_url=http://localhost:16686 \
    --set auth.strategy=anonymous \
    kiali-server \
    kiali/kiali-server

Install bookinfo with traffic generator

curl -L -o install-bookinfo.sh https://raw.githubusercontent.com/kiali/kiali/master/hack/istio/install-bookinfo-demo.sh
chmod +x install-bookinfo.sh
./install-bookinfo.sh -c kubectl -tg -id ${ISTIO_DIR}

And access Kiali:

kubectl port-forward svc/kiali 20001:20001 -n istio-system

Docs: Prerequisites

Mon, 01 Jan 0001 00:00:00 +0000

This tutorial is a walkthrough guide to install everything. For this reason, we will need:

minikube
istioctl
helm

This tutorial was tested on:

Minikube v1.30.1
Istio v1.18.1
Kiali v1.70

Clusters are provided by minikube instances, but this tutorial should work on on any Kubernetes environment.

We will set up some environment variables for the following commands:

CLUSTER_EAST="east"
CLUSTER_WEST="west"
ISTIO_DIR="absolute-path-to-istio-folder"

As Istio will be installed on more than one cluster and needs to communicate between clusters, we need to create certificates for the Istio installation. We will follow the Istio documentation related to certificates to achieve this:

mkdir -p certs
pushd certs

make -f $ISTIO_DIR/tools/certs/Makefile.selfsigned.mk root-ca

make -f $ISTIO_DIR/tools/certs/Makefile.selfsigned.mk $CLUSTER_EAST-cacerts
make -f $ISTIO_DIR/tools/certs/Makefile.selfsigned.mk $CLUSTER_WEST-cacerts

popd

The result is two certificates for then use when installing Istio in the future.

Docs: Deploy East cluster

Mon, 01 Jan 0001 00:00:00 +0000

Run the following commands to deploy the first cluster:

minikube start -p $CLUSTER_EAST --network istio --memory 8g --cpus 4

For both clusters, we need to configure MetalLB, which is a load balancer. This is because we need to assign an external IP to the required ingress gateways to enable cross cluster communication between Istio and the applications installed.

minikube addons enable metallb -p $CLUSTER_EAST

We set up some environment variables with IP ranges that MetalLB will then assign to the services:

MINIKUBE_IP=$(minikube ip -p $CLUSTER_EAST)
MINIKUBE_IP_NETWORK=$(echo $MINIKUBE_IP | sed -E 's/([0-9]+\.[0-9]+\.[0-9]+)\.[0-9]+/\1/')
MINIKUBE_LB_RANGE="${MINIKUBE_IP_NETWORK}.20-${MINIKUBE_IP_NETWORK}.29"

cat <<EOF | kubectl --context $CLUSTER_EAST apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses: [${MINIKUBE_LB_RANGE}]
EOF

We should have the first cluster deployed and ready to use.

Docs: Join the Mesh

Mon, 01 Jan 0001 00:00:00 +0000

Sidecar Proxies

The Travel Demo was deployed in the previous chapter without Istio sidecar proxies. These are Envoy proxies that can be injected into the application pods.

Without a sidecar, workloads do not connect to the Istio control plane and do not report mesh telemetry (metrics, access logs, or distributed traces).

Confirm in Kiali

As shown in Install Travel Demo, the demo namespaces appear on the Namespaces page with type - (Not part of the mesh):

The Overview page still summarizes only what is in the mesh — Istio control plane counts should remain unchanged and data plane namespaces should stay at 0.

On the Graph page, select the Travel Demo namespaces (travel-control, travel-portal, and travel-agency) in the namespace dropdown. Without sidecar proxies there is no request telemetry, so Kiali reports an empty graph:

The Workloads and Applications pages make the missing sidecars explicit. Open Workloads, select the travel-control namespace, and look for the missing-sidecar badge on the control workload:

Enable Sidecars

In this tutorial we add namespaces and workloads to the service mesh one step at a time. That makes it easier to see how Istio sidecar injection works before the rest of the demo joins the mesh.

We start with the control workload in the travel-control namespace.

Step 1

Enable Auto Injection on the travel-control namespace

Open Namespaces.
Click the travel-control namespace name to open its detail page.
Select Actions → Enable Auto Injection.
Confirm in the dialog.

This adds the istio-injection=enabled label to the namespace. Existing pods are not restarted yet — only new pods created after injection is enabled receive a sidecar automatically.

Step 2

Enable Auto Injection for the control workload

Open Workloads.
Select the travel-control namespace.
Click the control workload.
Select Actions → Enable Auto Injection.

Kiali updates the workload so the next pod receives an Istio sidecar. Kubernetes rolls out a new control pod; when it is ready you should see 2/2 containers (application + istio-proxy):

kubectl get pods -n travel-control
NAME                       READY   STATUS    RESTARTS   AGE
control-xxxxxxxxxx-xxxxx   2/2     Running   0          42s

Understanding what happened:

(i) Sidecar Injection

(ii) Automatic Sidecar Injection

Open Travel Demo to Outside Traffic

The control workload now has an Istio sidecar, but the application is still not reachable from outside the cluster.

In this section you expose the control service through an Istio Gateway and route external HTTP traffic to it with a VirtualService.

Step 1

Create a DNS entry for the control service using the external address of the Istio ingress gateway

There are several ways to provide a hostname for ingress depending on your platform. See Istio’s Determining the Ingress IP and Ports for general guidance.

Kind

If you created the cluster with ./hack/start-kind.sh, MetalLB is already configured. LoadBalancer services such as istio-ingressgateway receive an external IP address.

For Kind, check the external IP of the ingress gateway:

kubectl get svc istio-ingressgateway -n istio-system
NAME                   TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)                                                                      AGE
istio-ingressgateway   LoadBalancer   10.101.6.144   10.101.6.144   15021:30757/TCP,80:32647/TCP,443:30900/TCP,31400:30427/TCP,15443:31072/TCP   19h

Add an entry to /etc/hosts on the machine where you run the browser (use your cluster’s EXTERNAL-IP):

...
10.101.6.144 control.travel-control.istio-cluster.org
...

From that machine, control.travel-control.istio-cluster.org resolves to the Istio ingress gateway.

Kind

If EXTERNAL-IP stays <pending>, confirm MetalLB is running: kubectl get pods -n metallb-system

OpenShift

OpenShift does not populate Kubernetes EXTERNAL-IP for LoadBalancer services the same way. Expose the ingress gateway as a route instead.

For OpenShift, expose the ingress gateway as a route:

oc expose service istio-ingressgateway -n istio-system
oc get routes -n istio-system
NAME                   HOST/PORT                                  PATH   SERVICES               PORT    TERMINATION          WILDCARD
istio-ingressgateway   <YOUR_ROUTE_HOST>                                 istio-ingressgateway   http2                        None

Use <YOUR_ROUTE_HOST> wherever this chapter shows control.travel-control.istio-cluster.org (no /etc/hosts entry is required on OpenShift).

Step 2

Use the Request Routing wizard on the control service

Open Services.
Select the travel-control namespace.
Click the control service.
Select Actions → Request Routing.

Use Add Route Rule to add a default rule that sends all requests to the control workload.

Open Show advanced options, select the Gateways tab, enable Add Gateway, choose Create Gateway, and set the gateway host to control.travel-control.istio-cluster.org (port 80).

Before clicking Create, review the DestinationRule, Gateway, and VirtualService generated by the wizard:

Click Create to apply the configuration.

On the Istio Config page, confirm the new objects were created and validated in the travel-control namespace:

Step 3

Test the control service in your browser

Open http://control.travel-control.istio-cluster.org/ (on OpenShift, use your route host instead).

You should see the Travel Demo business dashboard — the same UI you previewed with kubectl port-forward in the previous chapter, now reachable through the mesh ingress.

Step 4

Review the travel-control namespace in Kiali

Open the Graph page, select the travel-control namespace, and refresh if needed. After browsing the dashboard, Kiali should show request telemetry from the ingress gateway through the control workload.

The graph may also show travel-portal services as destinations of outbound traffic from control. Only the control workload has a sidecar at this point — workloads in travel-portal and travel-agency still show missing sidecars on the Workloads page.

Understanding what happened:

External traffic enters the cluster through an Istio Gateway bound to the ingress gateway.
A VirtualService routes that traffic to the control service.
The control sidecar reports telemetry that Kiali displays on the graph.
Only the control workload participates in the mesh so far; the remaining demo workloads are added in Observe the Mesh.

(i) Istio Gateway

(ii) Istio Virtual Service

Docs: Install Istio on East cluster

Mon, 01 Jan 0001 00:00:00 +0000

The east cluster is the primary one, consequently is where the istiod process will be installed alongside other applications like Kiali.

Run the following commands to install Istio:

kubectl create namespace istio-system --context $CLUSTER_EAST

kubectl create secret generic cacerts -n istio-system --context $CLUSTER_EAST \
      --from-file=certs/$CLUSTER_EAST/ca-cert.pem \
      --from-file=certs/$CLUSTER_EAST/ca-key.pem \
      --from-file=certs/$CLUSTER_EAST/root-cert.pem \
      --from-file=certs/$CLUSTER_EAST/cert-chain.pem

kubectl --context=$CLUSTER_EAST label namespace istio-system topology.istio.io/network=network1

cat <<EOF > $CLUSTER_EAST.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: $CLUSTER_EAST
      network: network1
EOF

istioctl install -y --set values.pilot.env.EXTERNAL_ISTIOD=true --context=$CLUSTER_EAST -f $CLUSTER_EAST.yaml

After the installation, we need to create what we called an “east-west” gateway. It’s an ingress gateway just for the cross cluster configuration as we are opting to use the installation for different networks (this will be the case in the majority of the production scenarios).

$ISTIO_DIR/samples/multicluster/gen-eastwest-gateway.sh \
    --mesh mesh1 --cluster $CLUSTER_EAST --network network1 | \
    istioctl --context=$CLUSTER_EAST install -y -f -

Then, we need to expose the istiod service as well as the applications for the cross cluster communication:

kubectl apply --context=$CLUSTER_EAST -n istio-system -f \
    $ISTIO_DIR/samples/multicluster/expose-istiod.yaml

kubectl --context=$CLUSTER_EAST apply -n istio-system -f \
    $ISTIO_DIR/samples/multicluster/expose-services.yaml

export DISCOVERY_ADDRESS=$(kubectl \
    --context=$CLUSTER_EAST \
    -n istio-system get svc istio-eastwestgateway \
    -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

Finally, we need to install Prometheus, which is important and required for Kiali to operate:

kubectl --context $CLUSTER_EAST -n istio-system apply -f $ISTIO_DIR/samples/addons/prometheus.yaml

Docs: Observe the Mesh

Mon, 01 Jan 0001 00:00:00 +0000

Enable Sidecars in All Workloads

An Istio sidecar proxy connects a workload to the control plane and enables service mesh functionality.

Sidecars automatically collect metrics, access logs, and distributed traces — a major observability benefit of the mesh.

In Join the Mesh we injected a sidecar only into the control workload. The rest of the Travel Demo still lacks full mesh visibility.

Step 1

Use the Traffic Graph to identify missing sidecars in the Travel Demo

Open Traffic Graph.
Select all three Travel Demo namespaces (travel-control, travel-portal, and travel-agency).
Note the Graph Type menu → if necessary, set to the default: Versioned App graph.
Note the Display menu → if necessary, enable Missing Sidecars (under Show Badges).

The control workload reports telemetry for its traffic, but visibility is incomplete because travel-portal and travel-agency workloads still lack sidecar proxies.

Step 2

Enable proxy injection in travel-portal and travel-agency

In Join the Mesh we enabled injection for a single workload on purpose. For the remaining namespaces, a common pattern is to label namespaces before rollout so Istio injects sidecars automatically.

Label the namespaces and restart the deployments:

kubectl label namespace travel-agency istio-injection=enabled
kubectl label namespace travel-portal istio-injection=enabled

kubectl rollout restart deploy -n travel-portal
kubectl rollout restart deploy -n travel-agency

Kind

You can also enable injection from Kiali on each namespace (Namespaces → namespace detail → Actions → Enable Auto Injection), then restart workloads the same way.

Verify that workloads in all three demo namespaces have sidecars. Pods should show 2/2 ready (application + istio-proxy):

kubectl get pods -n travel-portal
kubectl get pods -n travel-agency

On the Workloads page, missing-sidecar badges should be gone for the restarted pods.

Step 3

Verify updated telemetry for travel-portal and travel-agency

Return to Traffic Graph, keep all three Travel Demo namespaces selected, and refresh if needed. You should now see traffic across the full demo topology.

Graph Walkthrough

The graph provides a powerful set of graph features to visualize service mesh traffic.

In this section we use the graph to explore the Travel Demo and identify the busiest services.

New to the Kiali graph? On the Traffic Graph page, click the Help icon in the toolbar to start the built-in graph tour. The tour introduces namespaces, display options, the summary panel, and other graph features.

Step 1

Select all travel- namespaces and enable Traffic Distribution edge labels

Open Traffic Graph.
Select travel-control, travel-portal, and travel-agency.
Open Display → under Show Edge Labels, enable Traffic Distribution.

The mesh looks healthy, but note that the hotels service carries more load than other services in travel-agency.

Step 2

Select the hotels service and inspect a trace in the summary panel

Click the hotels service node.
In the summary panel on the right, open the Traces tab.
Select a trace to inspect.

Combining telemetry and tracing shows traces that start from a portal and involve multiple services, and others that call only the hotels service.

Step 3

Drill into the main travels application graph

Right-click the travels application node. Right-click provides shortcuts to other Kiali pages.
Select Node Graph.

The graph focuses on the selected element so you can study one part of the topology in detail.

Application Details

Kiali provides detail views for applications, workloads, and services.

These views show structure, health, metrics, logs, traces, and Istio configuration for each component.

In this section we examine the main travels application in travel-agency.

Step 1

Navigate to the travels application

Open Applications.
Select the travel-agency namespace.
Click the travels application.

An application groups workloads and services that share the same application label. Telemetry and tracing signals are grouped by application even when multiple workloads are involved.

At this point the travels application consists of a travels-v1 workload exposed through the travels service. Click the travels-v1 workload link in the application overview.

Step 2

Examine Outbound Metrics for travels-v1

Open the Outbound Metrics tab on the travels-v1 workload.

The metrics tab shows charts built from Istio proxy telemetry. Expand the Request volume chart for a closer look:

Use Metrics Settings to change grouping and aggregation. Enable the spans checkbox to correlate metrics with tracing spans in the same chart.

In the context of the travels application, hotels request volume is higher than the other travel-agency services. Request duration looks normal, so the asymmetry is likely part of the application business logic rather than a performance problem.

Step 3

Review Logs for travels-v1

The Logs tab combines application container logs with Istio sidecar proxy logs. Enable spans for a correlated view of logs and traces.

Two main business methods appear in the application logs: GetDestinations and GetTravelQuote.

GetDestinations issues a GET /hotels request without parameters (visible in the sidecar proxy log):

GetTravelQuote calls multiple downstream services with a specific city parameter:

As described in the Travel Demo design, an initial query returns available hotels before the user selects a destination and requests quotes from the other services — which explains the higher hotels utilization.

Step 4

Review Traces for travels-v1

The hotels service handles more traffic than other travel-agency services. Next, use traces to see whether any step is slower than expected.

Open the Traces tab on travels-v1. Compare individual traces with the metrics histogram to judge whether a spike is unusual relative to average latency.

Click an interesting trace to see the trace details. Individual spans can be compared in more detail to pinpoint a slow step in a larger request flow.

Explore Further

This chapter focused on the graph and on the travels-v1 workload detail. The same observability tools are available throughout Kiali — browse the list pages and open any component to explore its detail view.

Namespaces — open travel-portal or travel-agency for namespace health and shortcuts to filtered list views.
Applications — compare travels in travel-agency with portal apps such as viaggi or voyages.
Services — inspect hotels, which carried more load in the graph walkthrough.
Workloads — open hotels-v1 or a portal workload and review the same tabs used here.
Istio Config — browse VirtualServices, DestinationRules, and other Istio objects with validation and YAML.

Each detail page starts with an Overview tab (mini-graph, health, and links to related objects). Applications, services, and workloads also provide Traffic, Metrics, and Traces tabs. Workloads add Logs and Envoy, as you saw on travels-v1.

See Detail Views for a full description of each tab.

When you are ready to change mesh behavior — not only observe it — continue to Control the Mesh, where Kiali wizards help configure request routing and other traffic management scenarios.

Docs: Control the Mesh

Mon, 01 Jan 0001 00:00:00 +0000

This chapter uses Kiali wizards to configure Istio traffic management on the Travel Demo. Each section introduces a scenario, walks through the wizard, verifies the result in Kiali, and ends with guidance to update or delete the generated configuration.

In Observe the Mesh you explored telemetry and tracing. Here you change how traffic flows through the mesh.

Request Routing

The Travel Demo application has several portals deployed on the travel-portal namespace consuming the travels service deployed on the travel-agency namespace.

The travels service is backed by a single workload called travels-v1 that receives requests from all portal workloads.

At some point in the lifecycle the business needs of the portals may differ and new versions of the travels service may be necessary.

This step will show how to route requests dynamically to multiple versions of the travels service.

Step 1

Deploy travels-v2 and travels-v3 workloads

Deploy the new versions of the travels service:

kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travels-v2.yaml) -n travel-agency
kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travels-v3.yaml) -n travel-agency

As there is no specific routing defined, when there are multiple workloads for the travels service the request distribution is fairly uniform.

Step 2

Investigate the HTTP headers used by the Travel Demo application

The Traffic Management features of Istio allow you to define matching conditions for dynamic request routing.

In our scenario we would like to perform the following routing logic:

All traffic from travels.uk routed to travels-v1
All traffic from viaggi.it routed to travels-v2
All traffic from voyages.fr routed to travels-v3

Portal workloads use HTTP/1.1 to call the travels service, so one strategy is to match on HTTP headers.

Where do those headers come from? That information belongs to the application domain — examine the code, documentation, or a live trace to see which headers are in use.

The Travel Demo uses an Istio annotation on the Deployment pod template to configure the proxy:

In our example the HTTP headers are added as part of the trace context. Tracing populates custom tags with the portal, device, user, and travel values.

Step 3

Use the Request Routing wizard on the travels service

Open Services.
Select the travel-agency namespace.
Click the travels service.
Select Actions → Request Routing.

Define three request-matching rules before clicking Create.

For the first rule, match when the portal header equals travels.uk:

Open the Request Matching tab.
Add an exact match for the portal header with value travels.uk.
Click Add Match to update Matching selected for this rule.

Open the Route To tab and set the destination workload for this rule.
Click Add Route Rule to save the first rule.

Return to the Request Matching tab and repeat the process, adding rules to route traffic from viaggi.it to travels-v2 and from voyages.fr to travels-v3.

When adding the additional rules make sure to clear the previous header match condition. If you make a mistake you can click Remove Rule from that rule’s kebab menu. The rule order does not matter.

When all three rules are defined, it should look like this:

Click Create to generate the Istio configuration. Review the generated config, then confirm to apply it.

The generated Istio config appears on the Istio Config card on the service detail page.

Step 4

Verify request routing from the travel-portal graph

Once routing is in place, verify that each portal sends traffic to its assigned travels workload:

Open Traffic Graph.
Select only the travel-portal namespace.
Set Graph Type to Workload graph.
Open Display → under Show Edge Labels, enable Traffic Distribution.
Open Display → disable Service Nodes.

An edge without a distribution label implies 100% of traffic on that path.

Navigate to any travels workload, open the Inbound Metrics tab, and expand the Request Volume chart for a similar view in the telemetry.

Increase the Duration to Last 30m (or use a custom time range) to show the time when our new routing was defined. We can see how the workload initially received traffic from all portals, then only from a single portal after request routing was applied.

Step 5

Update or delete Istio Configuration

Kiali wizards define high-level mesh scenarios and generate the Istio configuration needed to implement them (VirtualServices, DestinationRules, and related objects). Update or delete a scenario from the Actions menu on the service.

To experiment further, open the travels service and select Actions → Request Routing. When finished, use Actions to delete the generated Istio config.

Fault Injection

The Observe the Mesh chapter identified that the hotels service has additional traffic compared with other services deployed in the travel-agency namespace.

Also, this service is critical in the main business logic. It is responsible for querying all available destinations, presenting them to the user, and getting a quote for the selected destination.

This also means that the hotels service may be one of the weakest points of the Travel Demo application.

This step will show how to test the resilience of the Travel Demo application by injecting faults into the hotels service and then observing how the application reacts to this scenario.

Step 1

Use the Fault Injection wizard on the hotels service to inject a delay

Open Services.
Select the travel-agency namespace.
Click the hotels service.
Select Actions → Fault Injection.

Select HTTP Delay and set Delay percentage and Fixed Delay to the defaults, which introduce a 5 second delay on 100% of requests.

Create the HTTP Delay.

Step 2

Understanding source and destination metrics

Telemetry is collected from proxies and it is labeled with information about the source and destination workloads.

In our example, let’s say that the travels service (“Service A” in the Istio diagram below) invokes the hotels service (“Service B” in the diagram). The travels workload is the “source” and hotels is the “destination” workload. The travels proxy will report telemetry from the source perspective and hotels proxy will report telemetry from the destination perspective. Let’s look at the latency reporting from both perspectives.

The travels workload proxy has the Fault Injection configuration so it will perform the call to the hotels service and will apply the delay on the travels workload side (this is reported as source telemetry).

We can see in the hotels telemetry reported by the source (the travels proxy) that there is a visible gap showing 5 second delay in the request duration.

Note

The avg line reflects the injected delay most accurately. p50 and p95 are estimated from Istio’s Prometheus histogram buckets, not from individual request timings. When latency clusters near a bucket boundary (as it does with a fixed delay), quantiles can read lower than the configured delay — for example, with a 5s delay you may see avg just above 5s while p50 reads closer to 3.7s. The same pattern appears at other delay values (try updating the scenario to 4s to compare).

But as the Fault Injection delay is applied on the source proxy (travels), the destination proxy (hotels) is unaffected and its destination telemetry shows no delay.

Step 3

Study the impact of the travels service delay

The injected delay is propagated from the travels service to the downstream services deployed on travel-portal namespace, degrading the overall response time. But the downstream services are unaware, operate normally, and show a green status.

Step 4

Update or delete Istio Configuration

As part of this step you can update the Fault Injection scenario to test different delays. When finished, you can delete the generated Istio config for the hotels service.

Traffic Shifting

In the previous Request Routing step we deployed two new versions of the travels service using the travels-v2 and travels-v3 workloads.

That scenario showed how Istio can route specific requests to specific workloads. It was configured such that each portal deployed in the travel-portal namespace (travels.uk, viaggi.it and voyages.fr) was routed to a specific travels workload (travels-v1, travels-v2 and travels-v3).

This Traffic Shifting step will simulate a new scenario: the new travels-v2 and travels-v3 workloads will represent new improvements for the travels service that will be used by all requests.

These new improvements implemented in travels-v2 and travels-v3 represent two alternative ways to address a specific problem. Our goal is to test them before deciding which one to use as a next version.

At the beginning we will send 80% of the traffic into the original travels-v1 workload, and split 10% of the traffic each to travels-v2 and travels-v3.

Step 1

Use the Traffic Shifting wizard on the travels service

Open Services → travel-agency → travels.
Select Actions → Traffic Shifting.

Create a scenario with 80% of traffic to travels-v1 and 10% each to travels-v2 and travels-v3.

Step 2

Examine Traffic Shifting distribution from the travel-agency travels service Node Graph

Step 3

Compare travels workload and assess new changes proposed in travels-v2 and travels-v3

Istio Telemetry is grouped per logical application. That has the advantage of easily comparing different but related workloads, for one or more services.

In our example, we can use the “Inbound Metrics” and “Outbound Metrics” tabs in the travels application details, group by “Local version” and compare how travels-v2 and travels-v3 are working.

The charts show that the traffic distribution is working as expected and 80% is being distributed to the travels-v1 workload. They also show no big differences between travels-v2 and travels-v3 in terms of request duration.

Step 4

Update or delete Istio Configuration

As part of this step you can update the Traffic Shifting scenario to test different distributions. When finished, you can delete the generated Istio config for the travels service.

TCP Traffic Shifting

The Travel Demo application has a database service used by several services deployed in the travel-agency namespace.

At some point in the lifecycle of the application the telemetry shows that the database service degrades and starts to increase the average response time.

This is a common situation. In this case, a database specialist suggests an update of the original indexes due to data growth.

Our database specialist is suggesting two approaches and proposes to prepare two versions of the database service to test which may work better.

This step will show how the “Traffic Shifting” strategy can be applied to TCP services to test which new database indexing strategy works better.

Step 1

Deploy mysqldb-v2 and mysqldb-v3 workloads

To deploy the new versions of the mysqldb service execute the commands:

kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/mysql-v2.yaml) -n travel-agency
kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/mysql-v3.yaml) -n travel-agency

Step 2

Use the TCP Traffic Shifting Wizard on mysqldb service

Create a scenario with 80% of the traffic distributed to mysqldb-v1 workload and 10% of the traffic distributed each to mysqldb-v2 and mysqldb-v3.

Step 3

Examine Traffic Shifting distribution from the travel-agency Graph

Note that TCP telemetry has different types of metrics. As “Traffic Distribution” is only available for HTTP/gRPC services, for this service we need to use “Traffic Rate” to evaluate the distribution of data (bytes-per-second) between mysqldb workloads.

Step 4

Compare mysqldb workloads and study new indexes proposed in mysqldb-v2 and mysqldb-v3

TCP services have different telemetry but it’s still grouped by versions, allowing the user to compare and study pattern differences for mysqldb-v2 and mysqldb-v3.

The charts show more peaks in mysqldb-v2 compared to mysqldb-v3 but overall a similar behavior, so it’s probably safe to choose either strategy to shift all traffic.

Step 5

Update or delete Istio Configuration

As part of this step you can update the TCP Traffic Shifting scenario to test a different distribution. When finished, you can delete the generated Istio config for the mysqldb service.

Request Timeouts

In the Fault Injection step we showed how we could introduce a delay in the critical hotels service and test the resilience of the application.

The delay was propagated across services and Kiali showed how services accepted the delay without creating errors on the system.

But in real scenarios delays may have important consequences. Services may prefer to fail sooner, and recover, rather than propagating a delay across services.

This step will show how to add a request timeout for one of the portals deployed in travel-portal namespace. The travels.uk and viaggi.it portals will accept delays but voyages.fr will timeout and fail.

Step 1

Use the Fault Injection Wizard on hotels service to inject a delay

Repeat the Fault Injection step to add delay on hotels service.

Step 2

Use the Request Routing Wizard on travels service to add a route rule with delay for voyages.fr

Add a request timeout only on requests coming from the voyages.fr portal. This requires two route rules:

Use the Request Matching tab to add a matching condition for the portal header with voyages.fr value.

Use the Request Timeouts tab to add an HTTP Timeout with default values.

Add the first route rule to the scenario.

Add a second rule to match any request.

Create the request routing scenario. With this configuration, requests coming from voyages.fr will match the first rule and all others will match the second rule.

Step 3

Review the impact of the request timeout in the travels service

Create the rule. The Graph will show how requests coming from voyages.fr start to fail, due to the request timeout introduced.

Requests coming from other portals work without failures but are degraded by the hotels delay.

This scenario can be visualized in detail if we examine the “Inbound Metrics” and we group by “Remote app” and “Response code”.

As expected, the requests coming from voyages.fr don’t propagate the delay and they fail in the 2 seconds range, while requests from other portals don’t fail but they propagate the delay introduced in the hotels service.

Step 4

Update or delete Istio Configuration

As part of this step you can update the scenarios defined around hotels and travels services to experiment with more conditions, or you can delete the generated Istio config in both services.

Circuit Breaking

Distributed systems will benefit from failing quickly and applying back pressure, as opposed to propagating delays and errors through the system.

Circuit breaking is an important technique used to limit the impact of failures, latency spikes, and other types of network problems.

This step will show how to apply a Circuit Breaker into the travels service in order to limit the number of concurrent requests and connections.

Step 1

Deploy a new loadtester portal in the travel-portal namespace

In this example we are going to deploy a new workload that will simulate an important increase in the load of the system.

OpenShift

OpenShift users may also need to add the associated loadtester ServiceAccount to the necessary SecurityContextConstraints.

kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travel_loadtester.yaml) -n travel-portal

The loadtester workload will try to create 50 concurrent connections to the travels service, adding considerable pressure to the travel-agency namespace.

The Travel Demo application is capable of handling this load and at first glance it doesn’t show unhealthy status.

But in a real scenario an unexpected increase in the load of a service like this may have a significant impact on the overall system status.

Step 2

Use the Traffic Shifting Wizard on travels service to generate a traffic rule

Open travels service detail in the travel-agency namespace.
Select Actions → Request Routing.
Click Add Rule to distribute traffic (evenly) to the travels workloads.
Click Advanced Options → Circuit Breaker tab.
Enable Add Connection Pool and Add Outlier Detection with the default settings.
Create the Circuit Breaker.

The “Connection Pool” settings will indicate that the proxy sidecar will reject requests when the number of concurrent connections and requests exceeds more than one.

The “Outlier Detection” will eject a host from the connection pool if there is more than one consecutive error.

Step 3

Study the behavior of the Circuit Breaker in the travels service

In the loadtester versioned-app Graph we can see that the travels service’s Circuit Breaker accepts some, but fails most, connections.

Remember that these connections are stopped by the proxy on the loadtester side. That “fail sooner” pattern prevents overloading the network.

Using the Graph we can select the failed edge, check the Flags tab, and see that those requests are closed by the Circuit breaker.

If we examine the “Request volume” metric from the “Outbound Metrics” tab we can see the evolution of the requests, and how the introduction of the Circuit Breaker made the proxy reduce the request volume.

Step 4

Update or delete Istio Configuration

As part of this step you can update the scenarios defined around the travels service to experiment with more Circuit Breaker settings, or you can delete the generated Istio config in the service.

Understanding what happened:

(i) Circuit Breaking

(ii) Outlier Detection

(iii) Connection Pool Settings

(iv) Envoy’s Circuit Breaking Architecture

Mirroring

This tutorial has shown several scenarios where Istio can route traffic to different versions in order to compare versions and evaluate which one works best.

The Traffic Shifting step was focused on the travels service adding new travels-v2 and travels-v3 workloads and the TCP Traffic Shifting showed how this scenario can be used on TCP services like the mysqldb service.

Mirroring (or shadowing) is a particular case of the Traffic Shifting scenario where the proxy sends a copy of live traffic to a mirrored service.

The mirrored traffic happens out of band of the primary request path. It allows for testing of alternate services, in production environments, with minimal risk.

Istio mirrored traffic is only supported for HTTP/gRPC protocols.

This step will show how to apply mirrored traffic into the travels service.

Step 1

Use the Traffic Shifting Wizard on travels service

We will simulate the following:

travels-v1 is the original traffic and it will keep 80% of the traffic
travels-v2 is the new version to deploy, it’s being evaluated and it will get 20% of the traffic to compare against travels-v1
But travels-v3 will be considered as a new, experimental version for testing outside of the regular request path. It will be defined as a mirrored workload on 50% of the original requests.

Open travels service detail in the travel-agency namespace.
Select Actions → Traffic Shifting.
Set the 80%/20% loads for travels-v1 and travels-v2.
Click the Mirroring icon for travels-v3.
Set the 50% mirroring load.
Create the Traffic Shifting with Mirroring.

Step 2

Examine Traffic Shifting distribution from the travel-agency Graph

Note that Istio does not report mirrored traffic telemetry from the source proxy. It is reported from the destination proxy, although it is not flagged as mirrored, and therefore an edge from travels to the travels-v3 workload will appear in the graph. Note the traffic rates reflect the expected ratio of 80/20 between travels-v1 and travels-v2, with travels-v3 at about half of that total.

This can be examined better using the “Source” and “Destination” metrics from the “Inbound Metrics” tab.

The “Source” proxy, in this case the proxies injected into the workloads of travel-portal namespace, won’t report telemetry for travels-v3 mirrored workload.

But the “Destination” proxy, in this case the proxy injected in the travels-v3 workload, will collect the telemetry from the mirrored traffic.

Step 3

Update or delete Istio Configuration

As part of this step you can update the Mirroring scenario to test different mirrored distributions.

When finished you can delete the generated Istio config for the travels service.

Docs: Install Kiali

Mon, 01 Jan 0001 00:00:00 +0000

Run the following command to install Kiali using the Kiali operator:

kubectl config use-context $CLUSTER_EAST

helm upgrade --install --namespace istio-system --set auth.strategy=anonymous --set deployment.logger.log_level=debug --set deployment.ingress.enabled=true --repo https://kiali.org/helm-charts kiali-server kiali-server

Verify that Kiali is running with the following command:

istioctl dashboard kiali

There are other alternatives to expose Kiali or other Addons in Istio. Check Remotely Accessing Telemetry Addons for more information.

Docs: Navigating Multiple Meshes with OSSMC

Mon, 01 Jan 0001 00:00:00 +0000

OpenShift Service Mesh Console (OSSMC) includes Istios and Kialis pages in the OpenShift Console Service Mesh menu. These pages let you browse and inspect every Istio and Kiali CR on the cluster directly from the console — without requiring a Kiali server to be connected to the plugin.

This is especially useful when a cluster hosts many mesh instances. Platform teams that operate dozens or hundreds of Istio control planes (and the Kiali instances that observe them) on a cluster can navigate that inventory in one place instead of jumping between CLI commands, individual operator UIs, or separate Kiali routes.

Why this matters

One inventory for the whole cluster — List every Istio CR and every Kiali CR from the OpenShift Console sidebar.
No Kiali server required — The pages talk to the Kubernetes API. You can install the OSSMC plugin independently of any Kiali Server and still get this multi-mesh navigation.
Works alongside Kiali-powered observability — When you do connect a Kiali server to the plugin, Overview, Traffic Graph, and the other observability pages appear for that connected instance. Istios and Kialis stay in the menu so you can still manage the full fleet.

Istio control planes

The Istios page lists every cluster-scoped Istio CR managed by the OSSM/Sail Operator. Open a row to see details for that Istio installation.

Use this page when you need a console view of mesh control-plane instances on the cluster — for example, confirming which Istio CRs exist, opening one for more information, or orienting yourself before connecting a related Kiali instance.

Kiali instances

The Kialis page lists every Kiali CR on the cluster. Open a row to see configuration from that Kiali CR.

Both pages only list CRs your OpenShift user has permission to read. If your account has namespace-scoped access instead of cluster-wide access to istios.sailoperator.io or kialis.kiali.io, the list may show fewer instances than actually exist on the cluster — or appear empty. If you expect to see an instance that is missing, ask your cluster administrator to grant list/get permission on the relevant CR in that namespace (or cluster-wide).

The list includes:

Connected — Shows Active when that Kiali instance is the OSSMC backend, or Inactive when it is not. If OSSMConsole resources cannot be read, the status shows Unknown.
Observe — When route hosts are available, links to the Kiali UI and the OpenShift Console (Kiali | Console).
Actions — Connect or Disconnect to change which Kiali server OSSMC uses.

From the list or detail page you can Connect or Disconnect a Kiali instance:

Connect — Connects that Kiali server to the OSSMC plugin so Overview, Traffic Graph, Mesh, and the other Kiali-powered pages become available for that instance.
Disconnect — Disconnects that Kiali instance from the plugin. Istios and Kialis remain available so you can keep navigating the fleet and connect a different instance when needed.

Connect and disconnect require permission to patch the OSSMConsole CR (ossmconsoles.kiali.io).

OSSMC supports a single tenant at a time for Kiali-powered observability pages. Connect the Kiali instance you want to use for deep observability; use Istios and Kialis to navigate the rest of the fleet and switch when needed.

When a Kiali server is connected — and when it is not

What you see in the Service Mesh menu depends on whether OSSMC has a connected, reachable Kiali server:

When no Kiali server is connected — Expect Istios and Kialis only.
When a Kiali instance is connected and reachable — Expect observability pages (Overview, Traffic Graph, Mesh, and others) in the menu, with Istios and Kialis at the bottom, below a separator.

If OSSMC is configured to use a Kiali server but cannot reach it, observability pages show a Service Mesh is not configured message. Istios and Kialis continue to work — the same as when no server is connected.

Getting started

Install the Kiali Operator (a Kiali server is optional for these pages).
Create an OSSMConsole CR to install the plugin — see The OSSMConsole CR.
Open Service Mesh → Istios or Kialis in the OpenShift Console.

To connect a Kiali server for deep observability on one instance, use Connect on the Kialis page or set spec.kiali on the OSSMConsole CR. See Connecting a Kiali instance to the console.

For a tour of the Kiali-powered pages (Overview, Traffic Graph, and the rest), see the OSSMC User Guide.

Docs: Install Travels on East cluster

Mon, 01 Jan 0001 00:00:00 +0000

Run the following commands to install Travels application on east cluster:

kubectl create namespace travel-agency --context $CLUSTER_EAST
kubectl create namespace travel-portal --context $CLUSTER_EAST
kubectl create namespace travel-control --context $CLUSTER_EAST

kubectl label namespace travel-agency istio-injection=enabled --context $CLUSTER_EAST
kubectl label namespace travel-portal istio-injection=enabled --context $CLUSTER_EAST
kubectl label namespace travel-control istio-injection=enabled --context $CLUSTER_EAST

kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travel_agency.yaml) -n travel-agency --context $CLUSTER_EAST
kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travel_portal.yaml) -n travel-portal --context $CLUSTER_EAST
kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travel_control.yaml) -n travel-control --context $CLUSTER_EAST

After the installation, we can see that the Travels application is running on the east cluster:

It is important to note that Kiali only observes one istio-system namespace as we did not configure it for multicluster yet.

Go to the Graph page and select the three namespaces related to the Travels demo in the namespace dropdown menu. This shows you the in-cluster traffic:

So far, we installed everything on one cluster, similarly to the Travels tutorial for a single cluster.

Now we will expand this topology to include a remote cluster. As we commented this situation can be very common in a production scenario, either because we might want to split some applications into different clusters, generally because they are maintained by different developers or for high availability or just making applications available in other zones to reduce latencies.

Docs: Secure the Mesh

Mon, 01 Jan 0001 00:00:00 +0000

Authorization Policies and Sidecars

Security is one of Istio’s main pillars.

The Istio Security High Level Architecture provides a comprehensive solution to design and implement multiple security scenarios.

In this tutorial we will show how Kiali can use telemetry information to create security policies for the workloads deployed in a given namespace.

Istio telemetry aggregates the ServiceAccount information used in workload communication. This information can be used to define authorization policies that deny and allow actions based on observed live traffic.

Additionally, Istio sidecars can be created to limit the hosts with which a given workload can communicate. This improves traffic control, and also reduces the memory footprint of the proxies.

This step will show how we can define authorization policies for the travel-agency namespace, in the Travel Demo application, for all existing traffic in a given time period.

Once authorization policies are defined, a new workload will be rejected if it doesn’t match the security rules defined.

Step 1

Undeploy the loadtester workload from travel-portal namespace

In this example we will use the loadtester workload as the “intruder” in our security rules.

If we have followed the previous tutorial steps, we need to undeploy it from the system.

kubectl delete -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travel_loadtester.yaml) -n travel-portal

We should validate that telemetry has updated the travel-portal namespace and “Security” can be enabled in the Graph Display options.

Step 2

Create Authorization Policies, and Istio Sidecars, for current traffic for travel-agency namespace

Every workload in the cluster uses a Service Account.

travels.uk, viaggi.it and voyages.fr workloads use the default cluster.local/ns/travel-portal/sa/default ServiceAccount defined automatically per namespace.

This information is propagated into the Istio Telemetry and Kiali can use it to define a set of AuthorizationPolicy rules, and Istio Sidecars.

The Sidecars restrict the list of hosts with which each workload can communicate, based on the current traffic.

Open the travel-agency Namespace details page and use Actions → Create Traffic Policies.

This will generate a main DENY ALL rule to protect the whole namespace, and an individual ALLOW rule per workload identified in the telemetry.

It will also create an individual Sidecar per workload, each of them containing the set of hosts.

As an example, we can see that for the travels-v1 workload, the following hosts are added to the sidecar.

Step 3

Deploy the loadtester portal in the travel-portal namespace

If the loadtester workload uses a different ServiceAccount, then when it’s deployed it won’t comply with the AuthorizationPolicy rules defined in the previous step.

OpenShift

OpenShift users may also need to add the associated loadtester ServiceAccount to the necessary SecurityContextConstraints.

kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travel_loadtester.yaml) -n travel-portal

Now, the travels workload will reject requests made by the loadtester workload and that situation will be reflected in Graph:

This can also be verified in the details page using the Outbound Metrics tab grouped by response code (only the 403 line is present).

Inspecting the Logs tab confirms that loadtester workload is getting a HTTP 403 Forbidden response from travels workloads, as expected.

Step 4

Update travels-v1 AuthorizationPolicy to allow loadtester ServiceAccount

AuthorizationPolicy resources are defined per workload using matching selectors.

As part of the example, we can show how a ServiceAccount can be added into an existing rule to allow traffic from loadtester workload into the travels-v1 workload only.

As expected, now we can see that the travels-v1 workload accepts requests from all travel-portal namespace workloads, but travels-v2 and travels-v3 continue to reject requests from the loadtester source.

Using “Outbound Metrics” tab from the loadtester workload we can group per “Remote version” and “Response code” to get a detailed view of this AuthorizationPolicy change.

Step 5

Verify the proxies clusters list is limited by the Sidecars

According to Istio Sidecar documentation, Istio configures all mesh sidecar proxies to reach every mesh workload. After the sidecars are created, the list of hosts is reduced according to the current traffic. To verify this, we can look for the clusters configured in each proxy.

As an example, looking into the cars-v1 workload, we can see that there is a reduced number of clusters with which the proxy can communicate.

Step 6

Update or delete Istio Configuration

As part of this step, you can update the AuthorizationPolicies and Istio Sidecars generated for the travel-agency namespace, and experiment with more security rules. Or, you can delete the generated Istio config for the namespace.

Docs: Deploy West cluster

Mon, 01 Jan 0001 00:00:00 +0000

Run the following commands to deploy the second cluster:

minikube start -p $CLUSTER_WEST --network istio --memory 8g --cpus 4

Similar to the east cluster, we configure MetalLB:

minikube addons enable metallb -p $CLUSTER_WEST

MINIKUBE_IP=$(minikube ip -p $CLUSTER_WEST)
MINIKUBE_IP_NETWORK=$(echo $MINIKUBE_IP | sed -E 's/([0-9]+\.[0-9]+\.[0-9]+)\.[0-9]+/\1/')
MINIKUBE_LB_RANGE="${MINIKUBE_IP_NETWORK}.30-${MINIKUBE_IP_NETWORK}.39"

cat <<EOF | kubectl --context $CLUSTER_WEST apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses: [${MINIKUBE_LB_RANGE}]
EOF

Docs: Install Istio on West cluster

Mon, 01 Jan 0001 00:00:00 +0000

This installation will be different as this cluster will be a remote. In a remote cluster, it won’t be an Istio control plane. Istio will install some resources that allows the primary control plane to configure the workloads in the remote cluster like injecting the sidecars and configuring the low level routing.

kubectl create namespace istio-system --context $CLUSTER_WEST

kubectl create secret generic cacerts -n istio-system --context $CLUSTER_WEST \
      --from-file=certs/$CLUSTER_WEST/ca-cert.pem \
      --from-file=certs/$CLUSTER_WEST/ca-key.pem \
      --from-file=certs/$CLUSTER_WEST/root-cert.pem \
      --from-file=certs/$CLUSTER_WEST/cert-chain.pem

kubectl --context=$CLUSTER_WEST annotate namespace istio-system topology.istio.io/controlPlaneClusters=$CLUSTER_EAST
kubectl --context=$CLUSTER_WEST label namespace istio-system topology.istio.io/network=network2

cat <<EOF > $CLUSTER_WEST.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: remote
  values:
    istiodRemote:
      injectionPath: /inject/cluster/$CLUSTER_WEST/net/network2
    global:
      remotePilotAddress: ${DISCOVERY_ADDRESS}
EOF

istioctl install -y --context=$CLUSTER_WEST -f $CLUSTER_WEST.yaml

We will also install a Prometheus instance on the remote. We will federate both Prometheus, with the east’s one being the place where all metrics will be gathered together:

kubectl apply -f $ISTIO_DIR/samples/addons/prometheus.yaml --context $CLUSTER_WEST

An important step is to create a secret on east cluster allowing it to fetch information of the remote cluster:

istioctl x create-remote-secret \
    --context=$CLUSTER_WEST \
    --name=$CLUSTER_WEST | \
    kubectl apply -f - --context=$CLUSTER_EAST

Finally, we create the east-west gateway

$ISTIO_DIR/samples/multicluster/gen-eastwest-gateway.sh \
    --mesh mesh1 --cluster $CLUSTER_WEST --network network2 | \
    istioctl --context=$CLUSTER_WEST install -y -f -

Prometheus federation

Kiali requires unified metrics from a single Prometheus endpoint for all clusters, even in a multi-cluster environment. In this tutorial, we will federate the two Prometheus instances, meaning that all the remote’s metrics should be fetched by the main Prometheus.

We will configure east’s Prometheus to fetch west’s metrics:

kubectl patch svc prometheus -n istio-system --context $CLUSTER_WEST -p "{\"spec\": {\"type\": \"LoadBalancer\"}}"

WEST_PROMETHEUS_ADDRESS=$(kubectl --context=$CLUSTER_WEST -n istio-system get svc prometheus -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

curl -L -o prometheus.yaml https://raw.githubusercontent.com/kiali/kiali/master/hack/istio/multicluster/prometheus.yaml

sed -i "s/WEST_PROMETHEUS_ADDRESS/$WEST_PROMETHEUS_ADDRESS/g" prometheus.yaml

kubectl --context=$CLUSTER_EAST apply -f prometheus.yaml -n istio-system

Docs: Configure Kiali for multicluster

Mon, 01 Jan 0001 00:00:00 +0000

We will configure Kiali to access the remote cluster. This will require a secret (similar to the Istio secret) containing the credentials for Kiali to fetch information from the remote cluster:

curl -L -o kiali-prepare-remote-cluster.sh https://raw.githubusercontent.com/kiali/kiali/master/hack/istio/multicluster/kiali-prepare-remote-cluster.sh

chmod +x kiali-prepare-remote-cluster.sh

./kiali-prepare-remote-cluster.sh --kiali-cluster-context $CLUSTER_EAST --remote-cluster-context $CLUSTER_WEST

Finally, upgrade the installation for Kiali to pick up the secret:

kubectl config use-context $CLUSTER_EAST

helm upgrade --install --namespace istio-system --set auth.strategy=anonymous --set deployment.logger.log_level=debug --set deployment.ingress.enabled=true --repo https://kiali.org/helm-charts kiali-server kiali-server

As result, we can quickly see that a new namespace appear in the Overview, the istio-system namespace from west cluster:

Docs: Anonymous strategy

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

The anonymous strategy removes any authentication requirement. Users will have access to Kiali without providing any credentials.

Although the anonymous strategy doesn’t provide any access protection, it’s valid for some use-cases. Some examples known from the community:

Exposing Kiali through a reverse proxy, where the reverse proxy is providing a custom authentication mechanism.
Exposing Kiali on an already limited network of trusted users.
When Kiali is accessed through kubectl port-forward or alike commands that allow usage of the cluster’s RBAC capabilities to limit access.
When developing Kiali, where a developer has a private instance on his own machine.

It’s worth to emphasize that the anonymous strategy will leave Kiali unsecured. If you are using this option, make sure that Kiali is available only to trusted users, or access is protected by other means.

Set-up

To use the anonymous strategy, use the following configuration in the Kiali CR:

spec:
  auth:
    strategy: anonymous

The anonymous strategy doesn’t have any additional configuration.

Access control

When using the anonymous strategy, the content displayed in Kiali is based on the permissions of the Kiali service account. By default, the Kiali service account has cluster wide access and will be able to display everything in the cluster.

OpenShift

If you are running Kiali in OpenShift, access can be customized by changing privileges to the Kiali ServiceAccount. For example, to reduce permissions to individual namespaces, first, remove the cluster-wide permissions granted by default:

  oc delete clusterrolebindings kiali

Then grant the kiali role only in needed namespaces. For example:

  oc adm policy add-role-to-user kiali system:serviceaccount:istio-system:kiali-service-account -n ${NAMESPACE}

View only

You can tell the Kiali Operator to install Kiali in “view only” mode (this does work for either OpenShift or Kubernetes). You do this by setting the view_only_mode to true in the Kiali CR, which allows Kiali to read service mesh resources found in the cluster, but it does not allow any change:

spec:
  deployment:
    view_only_mode: true

Docs: Grafana

Mon, 01 Jan 0001 00:00:00 +0000

Grafana configuration

Istio provides preconfigured Grafana dashboards for the most relevant metrics of the mesh. Although Kiali offers similar views in its metrics dashboards, it is not in Kiali’s goals to provide the advanced querying options, nor the highly customizable settings, that are available in Grafana. Thus, it is recommended that you use Grafana if you need those advanced options.

Kiali can provide a direct link from its metric dashboards to the equivalent or most similar Grafana dashboard, which is convenient if you need the powerful Grafana options.

The Grafana links will appear in the Kiali metrics pages. For example:

For these links to appear in Kiali you need to manually configure the Grafana URL and the dashboards that come preconfigured with Istio, like in the following example:

Kiali will query Grafana and try to fetch the configured dashboards. For this reason Kiali must be able to reach Grafana, authenticate, and find the Istio dashboards. The Istio dashboards must be installed in Grafana for the links to appear in Kiali.

spec:
  external_services:
    grafana:
      enabled: true
      # Grafana service name is "grafana" and is in the "telemetry" namespace.
      internal_url: 'http://grafana.telemetry:3000/'
      # Public facing URL of Grafana
      external_url: 'http://my-ingress-host/grafana'
      # Grafana datasource UID when there are multiple
      datasource_uid: ""
      dashboards:
      - name: "Istio Service Dashboard"
        variables:
          datasource: "var-datasource"
          namespace: "var-namespace"
          service: "var-service"
      - name: "Istio Workload Dashboard"
        variables:
          datasource: "var-datasource"          
          namespace: "var-namespace"
          workload: "var-workload"
          datasource: "var-datasource"
      - name: "Istio Mesh Dashboard"
      - name: "Istio Control Plane Dashboard"
      - name: "Istio Performance Dashboard"
      - name: "Istio Wasm Extension Dashboard"

The described configuration is done in the Kiali CR when Kiali is installed using the Kiali Operator. If Kiali is installed with the Helm chart then the correct way to configure this is via regular –set flags.

Grafana authentication configuration

The Kiali CR provides authentication configuration that will be used to connect to your Grafana instance and for detecting your Grafana version in the Mesh graph.

spec:
  external_services:
    grafana:
      enabled: true
      auth:
        insecure_skip_verify: false
        password: "pwd"
        token: ""
        type: "basic"
        use_kiali_token: false
        username: "user"
      health_check_url: ""

To configure a secret to be used as a password, see this FAQ entry.

To authenticate using OAuth2 client_credentials flow, set type: "oauth2" and provide the oauth2 block:

spec:
  external_services:
    grafana:
      auth:
        type: "oauth2"
        oauth2:
          client_id: "my-client-id"
          client_secret: "secret:my-oauth2-secret:client_secret"
          token_url: "https://idp.example.com/token"
          scopes: []            # optional: list of OAuth2 scopes to request
          audience: ""          # optional: some providers require this
          auth_style: "header"  # "header" (default) or "params"

The client_secret field supports the secret:<secretName>:<secretKey> pattern for automatic secret mounting and rotation without pod restart. See the FAQ entry for details.

insecure_skip_verify applies only to the Grafana connection, not to the OAuth2 token endpoint. The token endpoint always validates TLS certificates. To trust a private CA for the token endpoint, add the CA to the kiali-cabundle ConfigMap as described in the TLS Configuration page.

TLS Certificate Configuration

If your Grafana server uses HTTPS with a certificate issued by a private CA, see the TLS Configuration page to learn how to configure Kiali to trust your CA.

Docs: How to Contribute

Mon, 01 Jan 0001 00:00:00 +0000

Contributing to the Docs

To contribute to the kiali.io docs see kiali.io on github.

In short you will:

Fork the kiali.io repo on GitHub.
Make your changes and send a pull request (PR).
If you’re not yet ready for a review, add “WIP” to the PR name to indicate it’s a work in progress.
1. Don’t add the Hugo property “draft = true” to the page front matter, it prevents auto-deployment of the content preview.
Wait for the automated PR workflow to do some checks. When it’s ready, you should see a comment like this: deploy/netlify — Deploy preview ready!
Click Details to the right of “Deploy preview ready” to see a preview of your updates.
Continue updating your doc and pushing your changes until you’re happy with the content.

Updating a single page

If you’ve just spotted something you’d like to change while using the docs, there is a shortcut for you:

Click Edit this page in the top right hand corner of the page.
If you don’t already have an up to date fork of the project repo, you are prompted to get one:
1. Click Fork this repository and propose changes or Update your Fork to get an up to date version of the project to edit.
2. The appropriate page in your fork is displayed in edit mode.
Follow the steps above to make, preview, and propose your changes.

Creating an issue

If you’ve found a problem in the docs, but you’re not sure how to fix it yourself, please create an issue in the kiali.io repo. You can also create an issue about a specific page by clicking the Create Documentation Issue button in the top right hand corner of the page.

Contributing to the Code

For code contribution see the kiali project’s CONTRIBUTING page.

Docs: Install Travels on West cluster

Mon, 01 Jan 0001 00:00:00 +0000

We are going to deploy two new services just to distribute traffic on the new cluster. These services are travels v2 and v3:

kubectl create ns travel-agency --context $CLUSTER_WEST

kubectl label namespace travel-agency istio-injection=enabled --context $CLUSTER_WEST

kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travels-v2.yaml) -n travel-agency --context $CLUSTER_WEST
kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travels-v3.yaml) -n travel-agency --context $CLUSTER_WEST

cat <<EOF | kubectl -n travel-agency --context $CLUSTER_WEST apply -f -
apiVersion: v1
kind: Service
metadata:
  name: travels
  labels:
    app: travels
spec:
  ports:
    - name: http
      port: 8000
  selector:
    app: travels
---
apiVersion: v1
kind: Service
metadata:
  name: insurances
  labels:
    app: insurances
spec:
  ports:
    - name: http
      port: 8000
  selector:
    app: insurances
---
apiVersion: v1
kind: Service
metadata:
  name: hotels
  labels:
    app: hotels
spec:
  ports:
    - name: http
      port: 8000
  selector:
    app: hotels
---
apiVersion: v1
kind: Service
metadata:
  name: flights
  labels:
    app: flights
spec:
  ports:
    - name: http
      port: 8000
  selector:
    app: flights
---
apiVersion: v1
kind: Service
metadata:
  name: discounts
  labels:
    app: discounts
spec:
  ports:
    - name: http
      port: 8000
  selector:
    app: discounts
---
apiVersion: v1
kind: Service
metadata:
  name: cars
  labels:
    app: cars
spec:
  ports:
    - name: http
      port: 8000
  selector:
    app: cars
EOF

After the installation, we can see that traffic is flowing to the remote cluster too:

This is happening automatically, Istio balances the traffic to both services. The key thing to notice here is that there is a concept called namespace sameness in Istio that is very important when planning our multicluster setup.

In both clusters, we can see that we have the same namespaces. They are called the same in both. Also, we can see that the services in both clusters need to exist and be called the same.

When we created the west’s namespaces, they are called the same, and also notice that even if we do not have instances of insurances or cars, we created the services. This is because travel services from the cluster will try to communicate with these services, not caring at all if the applications are in the west or east cluster. Istio will handle the routing in the back.

From this moment, we can start playing with Kiali to introduce some scenarios previously seen in the Travels tutorial.

Docs: Jaeger

Mon, 01 Jan 0001 00:00:00 +0000

Jaeger configuration

Jaeger is a highly recommended service because Kiali uses distributed tracing data for several features, providing an enhanced experience.

By default, Kiali will try to reach Jaeger at the GRPC-enabled URL of the form http://tracing.<istio_namespace_name>:16685/jaeger, which is the usual case if you are using the Jaeger Istio add-on. If this endpoint is unreachable, Kiali will disable features that use distributed tracing data.

If your Jaeger instance has a different service name or is installed to a different namespace, you must manually provide the endpoint where it is available, like in the following example:

spec:
  external_services:
    tracing:
      # Enabled by default. Kiali will anyway fallback to disabled if
      # Jaeger is unreachable.
      enabled: true
      # Jaeger service name is "tracing" and is in the "telemetry" namespace.
      # Make sure the URL you provide corresponds to the non-GRPC enabled endpoint
      # if you set "use_grpc" to false.
      internal_url: "http://tracing.telemetry:16685/jaeger"
      use_grpc: true
      # Public facing URL of Jaeger
      external_url: "http://my-jaeger-host/jaeger"

Minimally, you must provide spec.external_services.tracing.internal_url to enable Kiali features that use distributed tracing data. However, Kiali can provide contextual links that users can use to jump to the Jaeger console to inspect tracing data more in depth. For these links to be available you need to set the spec.external_services.tracing.external_url to the URL where you expose Jaeger outside the cluster.

Default values for connecting to Jaeger are based on the Istio’s provided sample add-on manifests. If your Jaeger setup differs significantly from the sample add-ons, make sure that Istio is also properly configured to push traces to the right URL.

Jaeger authentication configuration

The Kiali CR provides authentication configuration that will be used also for querying the version check to provide information in the Mesh graph.

spec:
  external_services:
    tracing:
      enabled: true
      auth:
        insecure_skip_verify: false
        password: "pwd"
        token: ""
        type: "basic"
        use_kiali_token: false
        username: "user"
      health_check_url: ""

To configure a secret to be used as a password, see this FAQ entry.

To authenticate using OAuth2 client_credentials flow, set type: "oauth2" and provide the oauth2 block:

spec:
  external_services:
    tracing:
      use_grpc: false           # must be false; oauth2 token injection is not implemented for gRPC transport
      auth:
        type: "oauth2"
        oauth2:
          client_id: "my-client-id"
          client_secret: "secret:my-oauth2-secret:client_secret"
          token_url: "https://idp.example.com/token"
          scopes: []            # optional: list of OAuth2 scopes to request
          audience: ""          # optional: some providers require this
          auth_style: "header"  # "header" (default) or "params"

OAuth2 authentication requires use_grpc: false. OAuth2 token injection is not implemented for gRPC transport.

insecure_skip_verify applies only to the Jaeger connection, not to the OAuth2 token endpoint. The token endpoint always validates TLS certificates. To trust a private CA for the token endpoint, add the CA to the kiali-cabundle ConfigMap as described in the TLS Configuration page.

The client_secret field supports the secret:<secretName>:<secretKey> pattern for automatic secret mounting and rotation without pod restart. See the FAQ entry for details.

TLS Certificate Configuration

If your Jaeger server uses HTTPS with a certificate issued by a private CA, see the TLS Configuration page to learn how to configure Kiali to trust your CA.

Docs: Kiali Backstage Plugin

Mon, 01 Jan 0001 00:00:00 +0000

The Kiali Backstage Plugin provides information about each Service Mesh object related with an entity in Backstage, a framework for building developer portals.

The plugin has different views to be included in Backstage:

Added as cards to see resource lists
Added as a tab which includes all the predefined Kiali cards
Added as a page, which views are not going to be filtered by entities and offer a full Kiali view

The Kiali Backstage plugin is released as a technology preview in Red Hat Developer Hub.

Documentation

Get Involved

Development guide

Docs: Kiali Chatbot

Mon, 01 Jan 0001 00:00:00 +0000

Kiali Chatbot is Kiali’s built-in AI assistant in the Kiali UI. It lets you ask questions about your service mesh and get answers backed by live data from Kiali and its configured backends (Prometheus, tracing, Kubernetes, etc.).

It does not require an external MCP server. Kiali includes its own set of MCP-style tools internally, so the AI can call them without depending on a separate MCP deployment.

Status

The Kiali chatbot was first released in Kiali version 2.22 and it is in Dev preview.

How does it work

At a high level:

The Kiali UI sends your chat request (prompt + context + selected model) to the Kiali backend.
Kiali selects the configured provider/model from ai.chat.
The provider calls the LLM with a set of internal MCP tools (defined in Kiali under kiali/ai/mcp).
The LLM may request tool calls (e.g. mesh graph, traces, resource details, workload logs, Istio config operations).
Kiali executes those tool calls against Kiali/Kubernetes/Prometheus/tracing backends and returns the final answer, including optional UI navigation actions and documentation citations.
The response is delivered as streaming events (for example: start, token, tool_call, tool_result, end, error) so the UI can progressively render tokens and tool activity in real time.

For configuration keys (enable/disable, tool filters, providers/models, store), see the ai.chat section in the Kiali CR spec.

Tool schemas (inputs/outputs)

Kiali Chatbot uses internal tools with defined input schemas and structured outputs.

Configuring the Kiali Chatbot

The chat_ai configuration is deprecated since v2.31 and has been moved under ai.chat.

The Kiali Chatbot is disabled by default. To enable it, set ai.enabled: true and ai.chat.enabled: true. When enabled, you will see the chatbot icon in the Kiali UI:

You must also configure at least one provider and model (including an API key), and pick a default provider/model.

Restricting access

By default, when the Chatbot is enabled, it is available to all users. You can restrict access to specific users by configuring the allowed_users list:

ai:
  enabled: true
  chat:
    enabled: true
    allowed_users:
      - "user1"
      - "user2"

If allowed_users is empty (the default), all users have access. If it is not empty, only the users listed will see and be able to use the Chatbot.

Switching model providers

Kiali Chatbot providers and models are configured in ai.chat:

Providers: OpenAI (type: openai), Google (type: google), Anthropic (type: anthropic), and LightSpeed (type: lightspeed).
Models are selected by name (per-provider) and can be enabled/disabled.
API keys can be set inline (not recommended) or via secret:<secret-name>:<key-in-secret>.
Tool exposure can be filtered globally with ai.chat.tools and further restricted per provider with ai.chat.providers[].tools.

Example configuration (showing three providers: OpenAI, Google, and Anthropic):

ai:
  enabled: true
  chat:
    default_provider: "openai"
    tools:
      disabled_tools:
        - "manage_istio_config"
    providers:
      - name: "openai"
        enabled: true
        description: "OpenAI provider"
        type: "openai"
        config: "default"
        default_model: "gpt"
        tools:
          enabled_tools:
            - "get_logs"
            - "get_mesh_status"
            - "list_traces"
        models:
          - name: "gpt"
            enabled: true
            model: "<openai-model-name>"
            key: "secret:my-key-secret:openai-api-key"
      - name: "google"
        enabled: true
        description: "Google provider"
        type: "google"
        config: "gemini"
        default_model: "gemini"
        models:
          - name: "gemini"
            enabled: true
            model: "gemini-2.5-pro"
            description: "Model provided by Google with OpenAI API Support"
            endpoint: "https://generativelanguage.googleapis.com/v1beta/openai"
            key: "secret:my-key-secret:google-api-key"
      - name: "anthropic"
        enabled: true
        description: "Anthropic provider"
        type: "anthropic"
        config: "default"
        default_model: "claude-haiku"
        key: "secret:my-key-secret:claude-api-key"
        models:
          - name: claude-sonnet
            model: "claude-sonnet-4-5"
            enabled: true
            endpoint: "https://api.anthropic.com/"
          - name: claude-haiku
            model: "claude-haiku-4-5"
            enabled: true

enabled_tools acts as an allowlist: when set, only the listed tool names are exposed. disabled_tools acts as a denylist and is applied afterwards. You can define these filters globally under ai.chat.tools and/or per provider under ai.chat.providers[].tools. Provider-level filters can only further restrict the already-allowed global toolset.

To see the available built-in tool names you can use in these lists, see Kiali Chatbot tools.

LightSpeed provider example:

ai:
  enabled: true
  chat:
    providers:
      - name: "LightSpeed"
        description: "Openshift LightSpeed"
        type: "lightspeed"
        endpoint: "<LightSpeed endpoint>"
        enabled: true

TLS verification

By default, Kiali verifies the TLS certificate of AI provider endpoints. Kiali uses its CA bundle (kiali-cabundle ConfigMap) and the platform TLS policy when connecting to providers.

If the provider uses a self-signed certificate that is not in Kiali’s CA bundle, you can disable TLS verification per provider:

ai:
  enabled: true
  chat:
    providers:
      - name: "LightSpeed"
        type: "lightspeed"
        endpoint: "https://lightspeed-app-server.openshift-lightspeed.svc:8443"
        enabled: true
        insecure_skip_verify: true

Setting insecure_skip_verify: true disables certificate validation for that provider. Use this only for development or testing. In production, add the provider’s CA certificate to the kiali-cabundle ConfigMap instead.

You can also select the configured models and providers in the chatbot window:

When the assistant uses a tool, Kiali shows a tool-result card directly in the chat so you can see which tool was executed:

You can click the square tool-result card to open the full tool output in a modal window:

In this modal view you can inspect the complete tool response in detail (for example returned resources, metrics, or logs) before continuing the conversation.

Streaming events in the chat

Kiali Chatbot UI updates are powered by server-sent streaming events from the backend. This is why responses appear incrementally (token by token), and why tool usage is shown as it happens.

In practice:

token events render incremental assistant text.
tool_call and tool_result events render the tool card and its status/output.
end finalizes the answer, including optional UI actions and documentation references.
error reports failures without waiting for a full response timeout.

What you can ask

Examples of tasks that work well:

Mesh/namespace topology and summaries (graph, status)
Basic observability questions (metrics, traces)
Troubleshooting workflows (get logs for a workload, identify failing namespaces)

Predefined prompts

Kiali Chatbot displays a set of suggested prompts tailored to the page you are currently viewing. The suggestions change as you navigate between pages (Overview, Graph, Mesh, Namespaces, Applications, Services, Workloads, Istio Config), so the most relevant questions for your current context are always one click away.

The prompts are served by a built-in server-side catalog through the GET /api/chat/prompts endpoint. Use the optional ?category=<page> query parameter to fetch prompts for a specific page. If the endpoint is unavailable, the chatbot falls back to a set of static prompts.

Clicking a suggested prompt populates the message input so you can review or edit the text before sending it.

Example prompts

“Show me the mesh graph for namespace bookinfo.”
“Which workloads in istio-system look unhealthy and why?”
“Get traces for service productpage in bookinfo for the last 30m.”

Ask vs. Troubleshoot mode

The chatbot input area includes a mode selector (the dropdown next to the send button) that lets you switch between two interaction modes: Ask and Troubleshoot.

Ask mode is a conversational Q&A mode. Use it to ask general questions about your mesh, namespaces, services, or Istio configuration. The assistant answers using its knowledge and the available internal tools, but the interaction is free-form and exploratory.

Troubleshoot mode is a structured diagnostic mode. Use it when you have a specific issue to investigate. In this mode, the chatbot:

Accepts a description of the problem as the starting point.
Automatically selects a relevant set of tools (logs, traces, metrics, Istio config checks) to gather evidence.
Follows a systematic diagnostic workflow and summarizes its findings with a recommended course of action.

The difference between the two modes depends a lot on the model used. In the case of LightSpeed, the modes are handled natively by the provider. For other models, the system prompt is adapted for each mode.

Next step

If you want to use an AI assistant outside the Kiali UI (for example, in an IDE), see Kiali MCP.

Docs: OpenShift

Mon, 01 Jan 0001 00:00:00 +0000

These are specific notes for the External Kiali deployment model on OpenShift.

Installation

It is highly recommended that the Kiali Operator be deployed on all clusters, even if the Kiali Server itself is not deployed on some clusters. This will ensure that the proper namespace and remote cluster resources can be created. Clusters without a Kiali Server will require only the remote cluster resources necessary for remote Kiali Server authentication. To install these resources, configure the Kiali CR with:

spec.deployment.remote_cluster_resources_only: true

This Kiali CR will result in an installation requiring very limited resources.

Authorization Strategy

When using the openshift authentication strategy on OpenShift, make sure to read and apply any guidance found in the notes for multi-cluster.

Docs: OSSM Console

Mon, 01 Jan 0001 00:00:00 +0000

OpenShift Service Mesh Console (OSSMC) is a dynamic plugin for OpenShift Console based on OpenShift dynamic plugins technology.

OSSMC lets you navigate multiple meshes from the OpenShift Console through Istios and Kialis pages — browse every Istio and Kiali instance on the cluster without requiring a Kiali server connected to the plugin. When a Kiali instance is connected and reachable, OSSMC also provides Kiali-powered observability — a dedicated Service Mesh navigation category with pages for overview, traffic graph, mesh infrastructure, namespaces, applications, services, workloads, and Istio configuration, plus Service Mesh tabs on OpenShift resource detail pages.

OSSMC was first released in September 2022 as a developer preview. It has since been released GA in October 2023.

Documentation

Navigating Multiple Meshes with OSSMC — Istios and Kialis pages for multi-mesh inventory in the console
User guide — Kiali-powered observability pages in the OpenShift Console
Install via OSSMConsole CR

Get Involved

Releases

Release list

Docs: OSSMC User Guide

Mon, 01 Jan 0001 00:00:00 +0000

The OpenShift Service Mesh Console (OSSMC) is an extension to the OpenShift Console which provides visibility into your Service Mesh. With the OSSMC plugin installed, a new Service Mesh menu category is available in the navigation menu on the left side of the web console.

Which pages appear depends on whether a Kiali server is connected to the plugin:

With a connected Kiali server — When a Kiali instance is connected and reachable, OSSMC provides the full Kiali-powered experience: dedicated list and detail pages for mesh components, plus Service Mesh tabs on OpenShift Workloads, Services, Projects, and Istio configuration detail pages. The features match those of the standalone Kiali Console, organized to integrate with the OpenShift Console.
Without a connected Kiali server — When no Kiali instance is connected (or Kiali is not reachable), OSSMC still provides Istios and Kialis pages that list every Istio and Kiali CR on the cluster. These pages do not require a Kiali server and are especially useful for navigating many mesh instances from one console. See Navigating Multiple Meshes.

The OSSMC plugin does not replace the standalone Kiali Console. After installing OSSMC, you can still access Kiali through its own route when a Kiali server is deployed. This User Guide discusses the extensions you see from within the OpenShift Console itself — primarily the Kiali-powered pages. For Istios and Kialis, see Navigating Multiple Meshes.

The OSSMC only supports a single tenant today. Whether that tenant is configured to access only a subset of OpenShift projects or has access cluster-wide to all projects does not matter, however, only a single tenant can be accessed.

The OSSMC plugin adds a Service Mesh category to the OpenShift Console sidebar. The pages available depend on whether a Kiali server is connected to the plugin.

Pages that require a connected Kiali server

When a Kiali instance is connected and reachable, the following pages appear in the Service Mesh menu. Each is documented in its own section later in this guide.

Overview — Namespace summary with health and metric cards
Traffic Graph — Full mesh topology view
Mesh — Istio infrastructure status
Namespaces — Namespace list with health, labels, and detail links
Applications — Application list with a dedicated detail page
Services — Service list with detail pages
Workloads — Workload list with health, type, and Istio configuration
Istio Config — Istio configuration list with validation status

All of these pages are standalone routes within the OSSMC plugin, providing a consistent Kiali-powered experience without leaving the OpenShift Console context.

OSSMC connects to the connected Kiali instance through the plugin proxy configured on the OSSMConsole CR. Service Mesh tabs on workload, service, project, and Istio resource detail pages are also available when a Kiali server is connected.

If OSSMC is configured to use a Kiali server but cannot connect to it, pages that require Kiali show a Service Mesh is not configured message with guidance to install and configure Kiali through the Kiali Operator. Istios and Kialis remain available — see Navigating Multiple Meshes.

If a page that requires Kiali fails unexpectedly, OSSMC displays a Service Mesh Console Unavailable message instead of affecting the rest of the OpenShift Console. If you use the Fleet Service Mesh perspective and are not using those Kiali-backed pages on that cluster, you can ignore this message.

Navigating Multiple Meshes with OSSMC

Istios and Kialis are always available in the Service Mesh menu. When a Kiali server is also connected, they appear at the bottom of the menu below a separator. Use them to browse every Istio and Kiali instance on the cluster and to connect or disconnect which Kiali server the plugin uses for observability.

See Navigating Multiple Meshes for details.

Overview

The Overview page provides a summary of your mesh by showing cards representing the namespaces participating in the mesh. Each namespace card has summary metric graphs and additional health details. There are links in the cards that take you to other pages within OSSMC.

Traffic Graph

The Traffic Graph page provides the full topology view of your mesh. The mesh is represented by nodes and edges — each node representing a component of the mesh and each edge representing traffic flowing through the mesh between components.

Mesh

The Mesh page provides detailed information about the Istio infrastructure status. It shows an infrastructure topology view with core and add-on components, their health, and how they are connected to each other.

Namespaces

The Namespaces page provides a list of namespaces participating in the mesh along with their health status and labels. Clicking a namespace opens a detail page with a split-panel layout: the left panel contains stacked cards showing namespace attributes, resource links, and health information, while the right panel displays a namespace-scoped traffic minigraph.

Applications

The Applications page provides a list of applications detected in the mesh. Clicking an application opens a dedicated detail page with sub-tabs for Overview, Traffic, Inbound Metrics, and Traces. The application detail page displays an application badge and provides the same level of detail as the standalone Kiali Console.

Services

The Services page provides a list of services in the mesh. Clicking a service opens a detail page with sub-tabs for Overview, Traffic, Inbound Metrics, and Traces.

Workloads

The Workloads page provides a list of workloads in the mesh along with their health status, type, and associated Istio configuration. Clicking a workload navigates to the corresponding Kubernetes resource detail page (Deployment, ReplicaSet, DaemonSet, StatefulSet, etc.) with the Service Mesh tab selected.

Istio Config

The Istio Config page provides a list of all Istio configuration files in your mesh with a column that provides a quick way to know if the configuration for each resource is valid. You can also create new Istio configuration resources from this page. The list page uses Kiali’s full filtering capabilities, including type, name, and validation status filters.

Workload Details

The Workloads detail view (accessible from the OpenShift Workloads pages such as Deployments, Pods, ReplicaSets, StatefulSets, and DaemonSets) has a tab Service Mesh that provides mesh-related detail for the selected workload. The details are grouped into several sub-tabs: Overview, Traffic, Logs, Inbound Metrics, Outbound Metrics, Traces, and Envoy.

Workload: Overview

The Workload: Overview sub-tab provides a summary of the selected workload including a localized topology graph showing the workload with all inbound and outbound edges and nodes.

Workload: Traffic

The Workload: Traffic sub-tab provides information about all inbound and outbound traffic to the workload.

Workload: Logs

The Workload: Logs sub-tab provides the logs for the workload’s containers. You can view container logs individually or in a unified fashion, ordered by log time. This is especially helpful to see how the Envoy sidecar proxy logs relate to your workload’s application logs. You can enable the tracing span integration which then allows you to see which logs correspond to trace spans.

Workload: Metrics

You can see both inbound and outbound metric graphs in the corresponding sub-tabs. All the workload metrics can be displayed here, providing you with a detail view of the performance of your workload. You can enable the tracing span integration which allows you to see which spans occurred at the same time as the metrics. You can then click on a span marker in the graph to view the specific spans associated with that timeframe.

Workload: Traces

The Traces sub-tab provides a chart showing the trace spans collected over the given timeframe. Click on a bubble to drill down into those trace spans; the trace spans can provide you the most low-level detail within your workload application, down to the individual request level.

The trace details view will give further details, including heatmaps that provide you with a comparison of one span in relation to other requests and spans in the same timeframe.

When the OpenShift tracing UI plugin is enabled, Kiali will try to auto discover the plugin settings and the View in Tracing Kiali link will redirect to the plugin (for Kiali 2.8.0+). If the plugin config needs to be adjusted, the following settings should be included in the plugin-conf ConfigMap:

{
  ...
  "observability": {
    "instance": "sample",
    "namespace": "tempo",
    "tenant": "default"
  }
}

Workload: Envoy

The Envoy sub-tab provides information about the Envoy sidecar configuration. This is useful when you need to dig down deep into the sidecar configuration when debugging things such as connectivity issues.

Application Details

The Applications detail page provides mesh-related detail for the selected application, including a localized topology graph, traffic information, inbound metrics, and traces. The detail page is accessible both from the OSSMC Applications list page and from graph node navigation.

Service Details

The Services detail view has a tab Service Mesh that provides mesh-related detail for the selected service. The details are grouped into several sub-tabs: Overview, Traffic, Inbound Metrics, Traces. These sub-tabs are similar in nature as the Workload sub-tabs with the same names and serve the same functions.

Project Details

The Projects detail view has a tab Service Mesh that provides mesh-related detail for that project with a split-panel layout showing project attributes, resource links, health information, and a namespace-scoped traffic minigraph.

Istio Config Details

The detail pages for Istio configuration resources (such as VirtualService, DestinationRule, Gateway, AuthorizationPolicy, and others) have a Service Mesh tab that shows an overview and validation status for the resource.

Docs: Prometheus

Mon, 01 Jan 0001 00:00:00 +0000

Prometheus configuration

Kiali uses Prometheus to generate the topology graph, show metrics, calculate health and for several other features. Prometheus is enabled by default and is required for full Kiali functionality.

Disabling Prometheus

If you want to run Kiali without a Prometheus instance, you can disable it:

spec:
  external_services:
    prometheus:
      enabled: false

When Prometheus is disabled, Kiali will still start and serve non-metrics features such as workload/service/app listing, Istio configuration, and mesh topology. However, the graph, metrics tabs, traffic tabs, and request-rate health will be unavailable. Health badges for workloads and apps will degrade to show only Kubernetes-level status (replica counts).

The UI will display a subtle informational message reminding you that metrics features are unavailable due to your configuration choice.

When Prometheus is Unreachable

When Prometheus is enabled (the default) but Kiali cannot reach it at startup, Kiali will still start successfully with metrics features temporarily unavailable.

The UI will display a warning notification explaining why metrics are unavailable. The Prometheus component will still appear in the masthead status and the mesh topology page, reported as unhealthy, so you have clear visibility into the misconfiguration.

To restore full metrics functionality after a startup failure, fix the Prometheus connectivity issue (correct the URL, ensure the Prometheus server is running, etc.) and restart Kiali.

Configuring the Prometheus URL

By default, Kiali assumes that Prometheus is available at the URL of the form http://prometheus.<istio_namespace_name>:9090, which is the usual case if you are using the Prometheus Istio add-on. If your Prometheus instance has a different service name or is installed in a different namespace, you must manually provide the endpoint where it is available, like in the following example:

spec:
  external_services:
    prometheus:
      # Prometheus service name is "metrics" and is in the "telemetry" namespace
      url: "http://metrics.telemetry:9090/"

Notice that you don’t need to expose Prometheus outside the cluster. It is enough to provide the Kubernetes internal service URL.

Kiali maintains an internal cache of some Prometheus queries to improve performance (mainly, the queries to calculate Health indicators). It would be very rare to see data delays, but should you notice any delays you may tune caching parameters to values that work better for your environment.

See the Kiali CR reference page for the current default values.

Compatibility with Prometheus-like servers

Although Kiali assumes a Prometheus server and is tested against it, there are TSDBs that can be used as a Prometheus replacement despite not implementing the full Prometheus API.

Community users have faced two issues when using Prometheus-like TSDBs:

Kiali may report that the TSDB is unreachable, and/or
Kiali may show empty metrics if the TSBD does not implement the /api/v1/status/config.

To fix these issues, you may need to provide a custom health check endpoint for the TSDB and/or manually provide the configurations that Kiali reads from the /api/v1/status/config API endpoint:

spec:
  external_services:
    prometheus:
      # Fix the "Unreachable" metrics server warning.
      health_check_url: "http://custom-tsdb-health-check-url"
      # Fix for the empty metrics dashboards
      thanos_proxy:
        enabled: true
        retention_period: "7d"
        scrape_interval: "30s"

Prometheus tuning

For production optimization — recording rules, federation, metric thinning, scrape intervals, and retention — see Tuning.

Prometheus authentication configuration

The Kiali CR provides authentication configuration that will be used also for querying the version check to provide information in the Mesh graph.

spec:
  external_services:
    prometheus:
      auth:
        insecure_skip_verify: false
        password: "pwd"
        token: ""
        type: "basic"
        use_kiali_token: false
        username: "user"
      health_check_url: ""

To configure a secret to be used as a password, see this FAQ entry.

To authenticate using OAuth2 client_credentials flow (for example, Azure Monitor Managed Prometheus or any OAuth2-protected endpoint), set type: "oauth2" and provide the oauth2 block:

spec:
  external_services:
    prometheus:
      auth:
        type: "oauth2"
        oauth2:
          client_id: "my-client-id"
          client_secret: "secret:my-oauth2-secret:client_secret"
          token_url: "https://login.microsoftonline.com/<tenant>/oauth2/v2.0/token"
          scopes:
          - "https://prometheus.monitor.azure.com/.default"
          audience: ""          # optional: some providers require this
          auth_style: "header"  # "header" (default) or "params"

The client_secret field supports the secret:<secretName>:<secretKey> pattern for automatic secret mounting and rotation without pod restart. See the FAQ entry for details.

insecure_skip_verify applies only to the Prometheus connection, not to the OAuth2 token endpoint. The token endpoint always validates TLS certificates. To trust a private CA for the token endpoint, add the CA to the kiali-cabundle ConfigMap as described in the TLS Configuration page.

TLS Certificate Configuration

If your Prometheus server uses HTTPS with a certificate issued by a private CA, see the TLS Configuration page to learn how to configure Kiali to trust your CA.

Docs: Kiali Chatbot tools (schemas)

Mon, 01 Jan 0001 00:00:00 +0000

Kiali Chatbot uses internal MCP-style tools (implemented inside Kiali) to fetch live data and perform safe actions. These are not external MCP server tools.

Administrators can control which of these tools are exposed to the AI by using ai.chat.tools for global filtering and ai.chat.providers[].tools for provider-specific filtering. Use the exact tool names below in enabled_tools and disabled_tools.

The tool input schemas are defined in Kiali under kiali/ai/mcp/tools/*.yaml. The tool outputs are JSON structures returned by the Kiali backend and consumed by the model and/or UI.

Tool list

get_action_ui: returns UI navigation actions (buttons/links).
get_logs: returns workload or pod logs with optional filtering.
get_mesh_status: returns high-level mesh health, control plane, observability stack, and connectivity status.
get_mesh_traffic_graph: returns a compact service-to-service traffic topology with metrics such as throughput, response time, and mTLS.
get_metrics: returns Istio or Envoy metrics for services, workloads, or apps.
get_pod_performance: returns current pod CPU and memory usage versus requests and limits.
get_referenced_docs: returns relevant Istio and Kiali documentation links.
get_trace_details: returns the hierarchy and span details for a specific trace.
list_or_get_resources: lists resources or returns details for services, workloads, apps, and namespaces.
list_traces: returns a compact list of distributed traces for a service.
manage_istio_config_read: lists or gets Istio config in read-only mode.
manage_istio_config: creates, patches, or deletes Istio config with a confirmation flow for sensitive actions.

Docs: Prerequisites

Mon, 01 Jan 0001 00:00:00 +0000

Istio

Before you install Kiali you must have already installed Istio. For full functionality, you should also install a telemetry storage addon (e.g. Prometheus), though Kiali can run without it if you disable Prometheus in the configuration. You might also consider installing Istio’s optional tracing addon (e.g. Tempo) and optional Grafana addon but those are not required by Kiali. Refer to the Istio documentation for details.

Optionally Enable the Debug Interface

Like istioctl, Kiali can make use of Istio’s port 8080 “Debug Interface” API. Despite the naming, this is required for accessing the status of the proxies.

The ENABLE_DEBUG_ON_HTTP setting controls the relevant API access. Istio suggests to disable this for security, but Kiali requires ENABLE_DEBUG_ON_HTTP=true, which is the default.

If you prefer not to enable the Istio API then certain Kiali features will be unavailable. If disabled, set spec.external_services.istio.istio_api_enabled: false in the Kiali CR.

For more information, see the Istio documentation.

Version Compatibility

It is always recommended that users run a supported version of Istio. The Istio news page posts end-of-support (EOL) dates. Supported Kiali versions include only the Kiali versions associated with supported Istio versions.

Starting with Kiali v2.4, each Kiali release is tested against the currently supported Istio releases. Unless otherwise noted, a Kiali release will be compatible with those releases. Older, untested Istio versions may also be compatible. Known incompatibilities will be noted in the table below. Prior to Kiali v2.4, compatibility is guaranteed only against the latest Istio release at the time. Although compatibility may be fine with other versions.

Istio	Tested Kiali Versions	Notes
1.30	2.26 and higher
1.29	2.21 and higher
1.28	2.17-2.30	Istio 1.28 is out of support.
1.27	2.12-2.25	Istio 1.27 is out of support.
1.26	2.9-2.20	Istio 1.26 is out of support.
1.25	2.5-2.16	Istio 1.25 is out of support.
1.24	2.0-2.13	Istio 1.24 is out of support.
1.23	1.87, 2.4-2.8	Istio 1.23 is out of support. Kiali v2 requires migration from Kiali v1 non-default namespace management (i.e. accessible_namespaces) to Discovery Selectors.
1.22	1.87, 2.4-2.5	Istio 1.22 is out of support. Kiali v1.86 is the recommended minimum for Istio Ambient users. Starting with Kiali v1.86,.1 Istio v1.22 is required.
1.21	1.81	Istio 1.21 is out of support.
1.20	1.78	Istio 1.20 is out of support.
1.19	1.75	Istio 1.19 is out of support.
1.18	1.73	Istio 1.18 is out of support.
1.17	1.66	Istio 1.17 is out of support. Avoid 1.63.0,1.63.1 due to a regression.
1.16	1.63	Istio 1.16 is out of support. Avoid 1.62.0,1.63.0,1.63.1 due to a regression.
1.15	1.59	Istio 1.15 is out of support.
1.14	1.54	Istio 1.14 is out of support.
1.13	1.49	Istio 1.13 is out of support.
1.12	1.44	Istio 1.12 is out of support.
1.11	1.41	Istio 1.11 is out of support.
1.10	1.37	Istio 1.10 is out of support.
1.9	1.33	Istio 1.9 is out of support.
1.8	1.28	Istio 1.8 is out of support. It removes all support for mixer/telemetry V1, as does Kiali 1.26.0. Use earlier versions of Kiali for mixer.
1.7	1.25	Istio 1.7 is out of support. Istioctl no longer installs Kiali. Use the Istio samples/addons for quick demo installs.
1.6	1.21	Istio 1.6 is out of support. Kiali 1.17 is recommended for Istio < 1.6.

OpenShift Service Mesh Version Compatibility

If you are running Red Hat OpenShift Service Mesh (OSSM), use only the bundled, supported version of Kiali.

OSSM	Kiali	Notes
3.4	2.27
3.3	2.22
3.2	2.17
3.1	2.11
3.0	2.4
2.6	1.73	OSSM 2.6 is out of support
2.5	1.73	OSSM 2.5 is out of support
2.4	1.65	OSSM 2.4 is out of support
2.3	1.57	OSSM 2.3 is out of support
2.2	1.48	OSSM 2.2 is out of support

OpenShift Console Plugin (OSSMC) Version Compatibility

The OSSMC plugin can be installed independently of any Kiali Server — a Kiali Server is not required for the Istios and Kialis pages used to navigate multiple meshes in the OpenShift Console.

When a Kiali instance is connected to OSSMC, the Kiali server version must match the OSSMC plugin version. See the compatibility table below.

OpenShift	OSSMC Min	OSSMC Max	Notes
4.19+	2.20
4.15+	1.84	2.19
4.12-4.18	1.73	1.83	OSSMC is out of support in this version range.

Maistra Version Compatibility

Maistra (OSSM 2.x) has reached end of life and is out of support.

Browser Compatibility

Kiali requires a modern web browser and supports the last two versions of Chrome, Firefox, Safari or Edge.

Hardware Requirements

Any machine capable of running a Kubernetes based cluster should also be able to run Kiali.

However, Kiali tends to grow in resource usage as your cluster grows. Usually the more namespaces and workloads you have in your cluster, the more memory you will need to allocate to Kiali.

Platform-specific requirements

OpenShift

If you are installing on OpenShift, you must grant the cluster-admin role to the user that is installing Kiali. If OpenShift is installed locally on the machine you are using, the following command should log you in as user system:admin which has this cluster-admin role:

$ oc login -u system:admin

For most commands listed on this documentation, the Kubernetes CLI command kubectl is used to interact with the cluster environment. On OpenShift you can simply replace kubectl with oc, unless otherwise noted.

Google Cloud Private Cluster

Private clusters on Google Cloud have network restrictions. Kiali needs your cluster’s firewall to allow access from the Kubernetes API to the Istio Control Plane namespace, for both the 8080 and 15000 ports.

To review the master access firewall rule:

gcloud compute firewall-rules list --filter="name~gke-${CLUSTER_NAME}-[0-9a-z]*-master"

To replace the existing rule and allow master access:

gcloud compute firewall-rules update <firewall-rule-name> --allow <previous-ports>,tcp:8080,tcp:15000

Istio deployments on private clusters also need extra ports to be opened. Check the Istio installation page for GKE to see all the extra installation steps for this platform.

Docs: ACM Observability

Mon, 01 Jan 0001 00:00:00 +0000

OpenShift Only: This guide is specifically for Red Hat OpenShift environments using Red Hat Advanced Cluster Management (ACM) for Kubernetes. ACM is an OpenShift-specific product.

Overview

Red Hat Advanced Cluster Management (ACM) provides centralized observability for multi-cluster OpenShift environments through its Observability Service. Metrics from all managed clusters (including the hub cluster itself) are collected and aggregated into a central Thanos-based storage system

This guide uses the multicluster observability add-on (MCOA) to aggregate mesh metrics on managed clusters (introduced in ACM 4.17), federate selected series, and remote-write them to the central Thanos-based storage system.

Kiali can query these aggregated metrics either through ACM’s external Observatorium API (using mTLS authentication) or directly through internal Thanos services. This guide explains both options, with detailed steps for the Observatorium API approach.

Architecture

Components

On the Hub Cluster:

ACM Observability Service: Centralized observability platform
- Observatorium API: External HTTPS endpoint with mTLS authentication
- Thanos: Metrics storage and query engine (Query, Query Frontend, Receive, Store)

On Managed Clusters (Hub + Spokes):

User Workload Monitoring (UWM): OpenShift’s Prometheus for user workloads
PodMonitor/ServiceMonitor: Scrape Istio metrics from:
- Sidecar proxies (in application namespaces)
- Control plane (istiod in istio-system)
- Ztunnel (in ztunnel namespace, for L4 metrics in Ambient mode)
- Waypoint proxies (in application namespaces, for L7 metrics in Ambient mode)
One MCOA PrometheusRule in a dedicated aggregation namespace: Aggregates raw per-proxy traffic series from all scraped namespaces into workload:istio_* series on UWM
MCOA user-workload Prometheus Agent: Federates selected UWM series, restores standard istio_* metric names, and remote-writes them to hub Thanos

Kiali Deployment Location:

Kiali can be deployed on any cluster with network access to:

The hub cluster’s metrics backend (Observatorium API or internal Thanos services)
Each managed cluster’s Kubernetes API (for workload and configuration data)

Common deployment locations:

Hub cluster (recommended): Co-located with ACM for lower latency metric queries and simplified networking. Can use internal Thanos services (HTTP) or external Observatorium API (HTTPS). Typically requires external deployment mode (ignore_home_cluster: true) since the hub usually doesn’t run mesh workloads or an Istio control plane.
Spoke/managed cluster: Kiali deployed alongside the mesh workloads or the Istio control plane. Must use external Observatorium API route.
Separate management cluster: Kiali deployed externally in dedicated “external deployment” mode (see External Kiali). Must use external Observatorium API route.

This guide assumes Kiali is deployed on the hub cluster in external deployment mode, but the configuration applies to any deployment location.

Metrics Flow

There are two independent flows:

Ingestion (managed cluster → hub):

Istio data plane components (sidecars, ztunnel, or waypoint proxies) expose metrics at :15020/stats/prometheus.
User Workload Monitoring Prometheus scrapes those metrics (typically every 30s).
UWM evaluates the cross-namespace recording rule in the dedicated mesh-observability namespace and produces workload:istio_* series.
The MCOA user-workload Prometheus Agent federates selected series through /federate, relabels workload:istio_* back to istio_*, and remote-writes them to the hub (every 5 minutes by default).
The hub stores them in Thanos Receive/Store and serves them through Thanos Query Frontend.

Query (Kiali → hub):

Kiali can query metrics through either of these paths:

Via Observatorium API Route (HTTPS with mTLS):

Kiali queries the external Observatorium API route.
Observatorium forwards the request to Thanos Query Frontend.
Thanos Query Frontend reads from Thanos Store/Receive and returns the result back through Observatorium to Kiali.

Via Internal Thanos Service (HTTP):

Kiali queries the internal Thanos Query Frontend service directly within the cluster, bypassing Observatorium.

Expected Latency: 5-6 minutes from traffic generation to visibility in Kiali due to the 5-minute (default) push interval.

Prerequisites

1. ACM Observability Service

ACM MultiClusterObservability must be installed on the hub cluster:

# Verify ACM Observability is running
oc get mco observability

# Check Observatorium API route
oc get route observatorium-api -n open-cluster-management-observability

2. User Workload Monitoring

User Workload Monitoring must be enabled on all clusters (hub and spokes):

# Enable UWM by editing cluster-monitoring-config
oc -n openshift-monitoring edit configmap cluster-monitoring-config

# Add:
# data:
#   config.yaml: |
#     enableUserWorkload: true

# Verify UWM pods are running
oc get pods -n openshift-user-workload-monitoring

See: Enabling monitoring for user-defined projects

3. Istio Metrics Collection

Create ServiceMonitor and PodMonitor resources to collect Istio metrics. The PodMonitor for sidecars must be created in each namespace with Istio sidecars because OpenShift monitoring ignores namespaceSelector in these resources. The ServiceMonitor for istiod is created once in istio-system.

ServiceMonitor for istiod (in istio-system):

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: istiod-monitor
  namespace: istio-system
spec:
  targetLabels:
  - app
  selector:
    matchLabels:
      istio: pilot
  endpoints:
  - port: http-monitoring
    interval: 30s

PodMonitor for Istio proxies (must be applied in every mesh namespace):

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: istio-proxies-monitor-<your-mesh-namespace>
  namespace: <your-mesh-namespace>
spec:
  selector:
    matchExpressions:
    - key: istio-prometheus-ignore
      operator: DoesNotExist
  podMetricsEndpoints:
  - path: /stats/prometheus
    interval: 30s
    relabelings:
    - action: keep
      sourceLabels: ["__meta_kubernetes_pod_container_name"]
      regex: "istio-proxy"
    - action: keep
      sourceLabels: ["__meta_kubernetes_pod_annotationpresent_prometheus_io_scrape"]
    - action: replace
      regex: (\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})
      replacement: '[$2]:$1'
      sourceLabels: ["__meta_kubernetes_pod_annotation_prometheus_io_port","__meta_kubernetes_pod_ip"]
      targetLabel: "__address__"
    - action: replace
      regex: (\d+);((([0-9]+?)(\.|$)){4})
      replacement: '$2:$1'
      sourceLabels: ["__meta_kubernetes_pod_annotation_prometheus_io_port","__meta_kubernetes_pod_ip"]
      targetLabel: "__address__"
    - sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_name","__meta_kubernetes_pod_label_app"]
      separator: ";"
      targetLabel: "app"
      action: replace
      regex: "(.+);.*|.*;(.+)"
      replacement: "${1}${2}"
    - sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_version","__meta_kubernetes_pod_label_version"]
      separator: ";"
      targetLabel: "version"
      action: replace
      regex: "(.+);.*|.*;(.+)"
      replacement: "${1}${2}"
    - action: replace
      regex: "(.+)"
      replacement: "${1}"
      sourceLabels:
      - __meta_kubernetes_pod_label_app_kubernetes_io_name
      targetLabel: app_kubernetes_io_name
    - action: replace
      regex: "(.+)"
      replacement: "${1}"
      sourceLabels:
      - __meta_kubernetes_pod_label_app_kubernetes_io_version
      targetLabel: app_kubernetes_io_version
    - sourceLabels: ["__meta_kubernetes_namespace"]
      action: replace
      targetLabel: namespace
    - action: replace
      replacement: "<your-mesh-identification-string>"
      targetLabel: mesh_id

See: Configuring OpenShift Monitoring with Service Mesh

Ambient Mode Metrics

If you are using Istio’s Ambient mode instead of (or in addition to) sidecar mode, you need additional PodMonitors to collect metrics from the Ambient data plane components.

Understanding Ambient Mode Metrics

Ambient mode uses a layered architecture with two metric sources:

Ztunnel (L4 metrics only)

Runs as a DaemonSet (namespace varies by installation)
Handles all L4 traffic for pods enrolled in ambient mode
Produces TCP-level metrics:
- istio_tcp_sent_bytes_total
- istio_tcp_received_bytes_total
- istio_tcp_connections_opened_total
- istio_tcp_connections_closed_total
Does not produce HTTP metrics

Waypoint proxies (L7 metrics)

Run as Deployments in application namespaces
Optional L7 proxies deployed per-namespace or per-service
Produce full HTTP metrics (same as sidecars):
- istio_requests_total
- istio_request_duration_milliseconds_*
- istio_request_bytes_*
- istio_response_bytes_*
- Plus all TCP metrics listed above

If you only use ztunnel (no waypoints), Kiali will show TCP traffic but not HTTP-level details like response codes or latency histograms.

PodMonitor for Ztunnel

Create a PodMonitor in the namespace where ztunnel runs. Ztunnel pods expose metrics using the same interface as sidecars:

Container name: istio-proxy
Annotation: prometheus.io/scrape: "true"
Metrics path: /stats/prometheus on port 15020

Because ztunnel uses the same metrics interface, you can use the same PodMonitor configuration shown in the Istio Metrics Collection section above, changing only the namespace field to match your ztunnel namespace.

Note: The ztunnel namespace location depends on your Istio installation method. Verify your ztunnel namespace with: oc get pods -l app=ztunnel -A

PodMonitor for Waypoint Proxies

Create a PodMonitor in each namespace with a waypoint. Waypoint pods also expose metrics using the same interface as sidecars:

Container name: istio-proxy
Annotation: prometheus.io/scrape: "true"
Metrics path: /stats/prometheus on port 15020

Because waypoints use the same metrics interface, you can use the same PodMonitor configuration shown in the Istio Metrics Collection section above.

4. Enable the ACM multicluster observability add-on

MCOA is disabled by default. It uses the OpenShift Cluster Observability Operator’s Prometheus Operator and Prometheus Agent to federate and remote-write metrics from managed clusters. Install the Cluster Observability Operator as required by ACM, then enable both platform and user-workload metrics on the hub:

apiVersion: observability.open-cluster-management.io/v1beta2
kind: MultiClusterObservability
metadata:
  name: observability
spec:
  capabilities:
    platform:
      metrics:
        default:
          enabled: true
    userWorkloads:
      metrics:
        default:
          enabled: true

Enabling these capabilities replaces the legacy metrics collectors with MCOA collectors. See the ACM 2.17 MCOA documentation.

The hub-level custom allowlist used by the legacy ACM collector is still a valid legacy configuration: the observability operator merges it into the configuration distributed to managed clusters. It is not sufficient after switching to MCOA because MCOA replaces those collectors. ACM 2.17 documents migrating custom allowlists to ScrapeConfig and PrometheusRule resources; this guide uses that MCOA resource model directly.

5. Aggregate and federate Istio metrics

Use UWM as the Edge Prometheus described in Recording Rules and Federation. MCOA’s Prometheus Agent is the federation and remote-write layer, and ACM Observatorium/Thanos is the long-retention federated backend that Kiali queries.

Create the following resources on the hub in namespace open-cluster-management-observability: one PrometheusRule targeting the dedicated mesh-observability namespace, one shared user-workload ScrapeConfig, and one cluster-wide platform ScrapeConfig for container CPU and memory. Application namespaces still need their ServiceMonitor/PodMonitor resources, but they are not MCOA rule targets.

Before propagating the rule, exempt mesh-observability from UWM label enforcement on every managed cluster. This is required because UWM otherwise injects the rule namespace into selectors and recorded series, preventing one rule from aggregating metrics across application namespaces:

apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    namespacesWithoutLabelEnforcement:
    - mesh-observability

Create one PrometheusRule named kiali-istio-aggregation. Its annotation tells MCOA to propagate it into mesh-observability; the label selects UWM Prometheus, rather than Thanos Ruler, to evaluate the rule on each managed cluster.

In the below PrometheusRule rules, only the “core tier” metrics are listed. If you want the “dashboard tier” or the “kiali tier” metrics, too, simply add them to the list (see Recording Rules and Federation for the full list of metrics for all tiers).

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: kiali-istio-aggregation
  namespace: open-cluster-management-observability
  annotations:
    observability.open-cluster-management.io/target-namespace: mesh-observability
  labels:
    app.kubernetes.io/component: user-workload-metrics-collector
    openshift.io/prometheus-rule-evaluation-scope: leaf-prometheus
spec:
  groups:
  - name: istio.workload-aggregation
    interval: 30s
    rules:
    - record: workload:istio_requests_total
      expr: sum without (pod, pod_template_hash, instance, job, node) (istio_requests_total)
    - record: workload:istio_request_messages_total
      expr: sum without (pod, pod_template_hash, instance, job, node) (istio_request_messages_total)
    - record: workload:istio_response_messages_total
      expr: sum without (pod, pod_template_hash, instance, job, node) (istio_response_messages_total)
    - record: workload:istio_tcp_sent_bytes_total
      expr: sum without (pod, pod_template_hash, instance, job, node) (istio_tcp_sent_bytes_total)
    - record: workload:istio_tcp_received_bytes_total
      expr: sum without (pod, pod_template_hash, instance, job, node) (istio_tcp_received_bytes_total)
    - record: workload:istio_tcp_connections_opened_total
      expr: sum without (pod, pod_template_hash, instance, job, node) (istio_tcp_connections_opened_total)
    - record: workload:istio_tcp_connections_closed_total
      expr: sum without (pod, pod_template_hash, instance, job, node) (istio_tcp_connections_closed_total)
    - record: workload:istio_request_duration_milliseconds_bucket
      expr: sum without (pod, pod_template_hash, instance, job, node) (istio_request_duration_milliseconds_bucket)
    - record: workload:istio_request_duration_milliseconds_sum
      expr: sum without (pod, pod_template_hash, instance, job, node) (istio_request_duration_milliseconds_sum)
    - record: workload:istio_request_duration_milliseconds_count
      expr: sum without (pod, pod_template_hash, instance, job, node) (istio_request_duration_milliseconds_count)
    - record: workload:istio_request_bytes_bucket
      expr: sum without (pod, pod_template_hash, instance, job, node) (istio_request_bytes_bucket)
    - record: workload:istio_request_bytes_sum
      expr: sum without (pod, pod_template_hash, instance, job, node) (istio_request_bytes_sum)
    - record: workload:istio_request_bytes_count
      expr: sum without (pod, pod_template_hash, instance, job, node) (istio_request_bytes_count)
    - record: workload:istio_response_bytes_bucket
      expr: sum without (pod, pod_template_hash, instance, job, node) (istio_response_bytes_bucket)
    - record: workload:istio_response_bytes_sum
      expr: sum without (pod, pod_template_hash, instance, job, node) (istio_response_bytes_sum)
    - record: workload:istio_response_bytes_count
      expr: sum without (pod, pod_template_hash, instance, job, node) (istio_response_bytes_count)

Next, create this UWM ScrapeConfig. It federates the aggregated traffic metrics and the non-aggregated Istio, control-plane, process, and Envoy metrics that Kiali needs. The relabeling restores the original istio_* names before remote write. In effect, these are the metrics that will be stored in the federated Prometheus (i.e. the hub Thanos) that Kiali will see.

In the below ScrapeConfig match list, only the “core tier” metrics are listed. If you want the “dashboard tier” or the “kiali tier” metrics, too, simply add them to the list (see Recording Rules and Federation for the full list of metrics for all tiers).

apiVersion: monitoring.rhobs/v1alpha1
kind: ScrapeConfig
metadata:
  name: kiali-istio-federation
  namespace: open-cluster-management-observability
  labels:
    app.kubernetes.io/component: user-workload-metrics-collector
spec:
  honorLabels: true
  jobName: kiali-istio-federation
  metricRelabelings:
  - action: replace
    regex: 'workload:(.*)'
    replacement: '${1}'
    sourceLabels: [__name__]
    targetLabel: __name__
  metricsPath: /federate
  params:
    match[]:
    - '{__name__=~"workload:istio_requests_total"}'
    - '{__name__=~"workload:istio_request_bytes_(bucket|count|sum)"}'
    - '{__name__=~"workload:istio_request_duration_milliseconds_(bucket|count|sum)"}'
    - '{__name__=~"workload:istio_request_messages_total"}'
    - '{__name__=~"workload:istio_response_bytes_(bucket|count|sum)"}'
    - '{__name__=~"workload:istio_response_messages_total"}'
    - '{__name__=~"workload:istio_tcp_connections_(opened|closed)_total"}'
    - '{__name__=~"workload:istio_tcp_(received|sent)_bytes_total"}'
    - '{__name__=~"istio_build|process_cpu_seconds_total|process_resident_memory_bytes"}'
    - '{__name__=~"pilot_info|pilot_proxy_convergence_time_(sum|count)|pilot_services|pilot_xds$|pilot_xds_pushes"}'
    - '{__name__=~"workload_manager_active_proxy_count"}'
    - '{__name__=~"envoy_cluster_upstream_cx_active|envoy_cluster_upstream_rq_total|envoy_listener_downstream_cx_active|envoy_listener_http_downstream_rq|envoy_server_memory_allocated|envoy_server_memory_heap_size|envoy_server_uptime"}'
  scrapeInterval: 5m

Now create this platform ScrapeConfig once per cluster. It collects CPU and memory metrics for pods in all namespaces; no namespace matcher is needed.

apiVersion: monitoring.rhobs/v1alpha1
kind: ScrapeConfig
metadata:
  name: kiali-istio-platform-federation
  namespace: open-cluster-management-observability
  labels:
    app.kubernetes.io/component: platform-metrics-collector
spec:
  jobName: kiali-istio-platform-federation
  metricsPath: /federate
  params:
    match[]:
    - '{__name__=~"container_cpu_usage_seconds_total|container_memory_working_set_bytes"}'
  scrapeInterval: 5m

Finally, add references for the single PrometheusRule and both ScrapeConfig objects to the selected placement in the existing ClusterManagementAddOn object named multicluster-observability-addon. The following is a fragment of that placement’s configs list; merge these entries with its existing entries rather than applying this as a replacement for the whole add-on object.

A placement is ACM’s hub-side selection of managed clusters that receive an add-on configuration. Use the name and namespace from the placement entry that MCOA already uses:

oc get clustermanagementaddon multicluster-observability-addon -o yaml

Look under spec.installStrategy.placements. If multiple entries exist, choose the one that selects the managed clusters hosting this mesh. For example, if you see name: global and namespace: open-cluster-management-global-set, use those values for <placement-name> and <placement-namespace>.

spec:
  installStrategy:
    placements:
    - name: <placement-name>
      namespace: <placement-namespace>
      configs:
      - group: monitoring.coreos.com
        resource: prometheusrules
        name: kiali-istio-aggregation
        namespace: open-cluster-management-observability
      - group: monitoring.rhobs
        resource: scrapeconfigs
        name: kiali-istio-federation
        namespace: open-cluster-management-observability
      - group: monitoring.rhobs
        resource: scrapeconfigs
        name: kiali-istio-platform-federation
        namespace: open-cluster-management-observability

For a repeatable development installation, the Kiali repository’s hack/configure-acm-mcoa.sh install helper creates these same objects and placement references without changing the current kubeconfig context. It is a convenience, not a prerequisite for this configuration.

The mesh-observability namespace must exist on every managed cluster selected by the placement before MCOA propagates its recording rule. If clusters use different application namespace layouts, create the appropriate ServiceMonitor and PodMonitor resources locally on each cluster. The same MCOA aggregation rule can still be propagated to mesh-observability on every cluster.

The kiali-istio-federation user-workload ScrapeConfig collects the aggregated traffic metrics rather than raw per-proxy series. The cluster-wide platform ScrapeConfig collects CPU and memory separately because those metrics belong to OpenShift platform monitoring rather than UWM.

Configuring Kiali for ACM Observability

Choosing Between Observatorium API and Internal Thanos Services

You have two options for connecting Kiali to ACM metrics:

Option 1: Observatorium API Route (HTTPS with mTLS)

external_services:
  prometheus:
    url: "https://observatorium-api-<namespace>.<apps-domain>/api/metrics/v1/default"
    auth:
      type: none
      cert_file: "secret:acm-observability-certs:tls.crt"
      key_file: "secret:acm-observability-certs:tls.key"

Provides:

HTTPS with mTLS authentication and encryption
External access (can be accessed from outside the cluster if needed)
RBAC enforcement via Observatorium
Multi-tenant isolation
Requires certificate setup

Option 2: Internal Thanos Service (HTTP)

external_services:
  prometheus:
    url: "http://observability-thanos-query-frontend.open-cluster-management-observability.svc:9090"
    auth:
      type: none

Provides:

Simpler setup (no certificates required)
Direct access to Thanos (potentially lower latency)
Internal cluster networking only
HTTP only (no encryption between Kiali and Thanos)

Recommendation: Use the Observatorium API for production environments where you want encrypted connections and proper authentication. Use internal services for development/testing environments where simplicity is preferred or where network security is already provided by the cluster infrastructure.

The rest of this guide focuses on the Observatorium API approach with mTLS authentication.

Step 1: Obtain mTLS Certificates from ACM

ACM automatically creates long-lived client certificates (1 year validity) for accessing the Observatorium API. Extract these from the hub cluster:

# Extract client certificate (for authentication)
oc get secret observability-grafana-certs \
  -n open-cluster-management-observability \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > tls.crt

# Extract client key (for authentication)
oc get secret observability-grafana-certs \
  -n open-cluster-management-observability \
  -o jsonpath='{.data.tls\.key}' | base64 -d > tls.key

Note: These certificates are created automatically when ACM MultiClusterObservability is deployed and are already trusted by the Observatorium API.

ACM Version Note: Secret names may vary depending on your ACM version. Before proceeding, verify the secret exists:

oc get secrets -n open-cluster-management-observability | grep -i cert

If observability-grafana-certs doesn’t exist, look for similar secrets containing client certificates.

Step 2: Extract Server CA Certificate

Extract the CA certificate that signed the Observatorium API server certificate. This is used by Kiali to validate the server’s TLS certificate.

First, identify which CA issued the server certificate:

# Get the Observatorium API route hostname
HOST=$(oc get route observatorium-api -n open-cluster-management-observability -o jsonpath='{.spec.host}')

# Check who issued the server certificate
echo | openssl s_client -connect "${HOST}:443" -servername "${HOST}" -showcerts 2>/dev/null | openssl x509 -noout -issuer

Example output:

issuer=C=US, O=Red Hat, Inc., CN=observability-server-ca-certificate

Then, extract the matching CA certificate based on the issuer CN:

If the issuer CN is observability-server-ca-certificate:

oc get secret observability-server-ca-certs \
  -n open-cluster-management-observability \
  -o jsonpath='{.data.ca\.crt}' | base64 -d > server-ca.crt

If the issuer CN is observability-client-ca-certificate:

oc get secret observability-client-ca-certs \
  -n open-cluster-management-observability \
  -o jsonpath='{.data.ca\.crt}' | base64 -d > server-ca.crt

Note: Both secrets are in the open-cluster-management-observability namespace. The exact CA used may vary depending on your ACM version and configuration.

Step 3: Create Kubernetes Resources

Note: <kiali-namespace> and ${KIALI_NAMESPACE} are used as a placeholder for the namespace where Kiali is deployed. This is commonly istio-system but is not required to be - replace with your actual Kiali namespace.

Create the mTLS certificate secret in Kiali’s namespace:

KIALI_NAMESPACE="istio-system"  # Replace with your Kiali namespace

oc create secret generic acm-observability-certs \
  -n ${KIALI_NAMESPACE} \
  --from-file=tls.crt=tls.crt \
  --from-file=tls.key=tls.key

Create the CA bundle ConfigMap in Kiali’s namespace:

oc create configmap kiali-cabundle \
  -n ${KIALI_NAMESPACE} \
  --from-file=additional-ca-bundle.pem=server-ca.crt

On OpenShift: The Kiali Operator (or Helm chart) automatically creates a separate ConfigMap named kiali-cabundle-openshift for the OpenShift service CA, then uses a projected volume to combine it with your custom kiali-cabundle ConfigMap. You only need to create/manage kiali-cabundle with your ACM CA - the system handles merging.

For more details about CA bundle configuration, see TLS Configuration.

Step 4: Get Observatorium API URL

Find the external Observatorium API route URL:

oc get route observatorium-api \
  -n open-cluster-management-observability \
  -o jsonpath='https://{.spec.host}/api/metrics/v1/default'

The URL format is: https://observatorium-api-<namespace>.<apps-domain>/api/metrics/v1/default

Step 5: Configure Kiali

Using Kiali Operator (Kiali CR):

spec:
  external_services:
    prometheus:
      # Use Observatorium API route
      url: "<observatorium-api-url>"

      auth:
        type: none  # mTLS authentication at TLS layer, no Authorization header
        cert_file: "secret:acm-observability-certs:tls.crt"
        key_file: "secret:acm-observability-certs:tls.key"

      # Enable Thanos proxy mode
      thanos_proxy:
        enabled: true
        retention_period: "365d"
        scrape_interval: "5m"

Using Server Helm Chart:

OBSERVATORIUM_API_URL="$(oc get route observatorium-api -n open-cluster-management-observability -o jsonpath='https://{.spec.host}/api/metrics/v1/default')"

helm install kiali kiali-server \
  --namespace ${KIALI_NAMESPACE} \
  --set external_services.prometheus.url="${OBSERVATORIUM_API_URL}" \
  --set external_services.prometheus.auth.type="none" \
  --set external_services.prometheus.auth.cert_file="secret:acm-observability-certs:tls.crt" \
  --set external_services.prometheus.auth.key_file="secret:acm-observability-certs:tls.key" \
  --set external_services.prometheus.thanos_proxy.enabled="true" \
  --set external_services.prometheus.thanos_proxy.retention_period="365d" \
  --set external_services.prometheus.thanos_proxy.scrape_interval="5m"

Important Configuration Notes

Scrape Intervals

This architecture has several independent intervals. They control different stages of the metrics pipeline and should not be treated as one global scrape period:

Edge target scraping — ServiceMonitor.spec.endpoints[].interval and PodMonitor.spec.podMetricsEndpoints[].interval control how often the spoke’s UWM Prometheus scrapes Istiod, sidecars, ztunnel, and waypoint metrics. If omitted, UWM uses the prometheus.scrapeInterval default of 30s.
Recording-rule evaluation — PrometheusRule.spec.groups[].interval is how often UWM evaluates the aggregation rules. This is an evaluation interval, not a target scrape interval. If omitted, UWM uses the prometheus.evaluationInterval default of 30s.
UWM-to-hub federation — ScrapeConfig.spec.scrapeInterval controls how often the MCOA PrometheusAgent fetches metrics from the spoke UWM Prometheus /federate endpoint and remote-writes them to hub Thanos. If omitted, the MCOA PrometheusAgent default is 300s (5m).
Kiali’s Thanos hint — external_services.prometheus.thanos_proxy.scrape_interval tells Kiali what interval to use when calculating rate windows and query steps. It does not control federation. If omitted, Kiali defaults it to 30s; for ACM, set it to the same effective interval as the MCOA federation job, whose default is 5m.

If a ServiceMonitor or PodMonitor omits its endpoint interval, OpenShift UWM uses the default from the user-workload-monitoring-config ConfigMap. If a PrometheusRule group omits its interval, UWM uses the ConfigMap’s rule evaluation default. These defaults can be configured without changing every monitor or rule:

apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
      evaluationInterval: 30s
      scrapeInterval: 30s

The ConfigMap settings are defaults for omitted ServiceMonitor, PodMonitor, and PrometheusRule interval fields. They do not configure UWM-to-hub federation: ScrapeConfig.spec.scrapeInterval defaults to 5m, and Kiali’s thanos_proxy.scrape_interval defaults to 30s. For ACM, configure Kiali’s value to match the effective federation interval.

These intervals must be selected as a pipeline. The 30s edge scrape and rule evaluation intervals determine how quickly UWM discovers new samples and refreshes the aggregated recording-rule series; they do not make those samples available in hub Thanos every 30 seconds. The 5m federation interval determines how frequently new data points arrive in hub Thanos, so it determines both the normal metrics delay and the minimum useful range for PromQL rate() queries. A rate window should contain at least two federated samples, which means it should normally be at least twice the effective federation interval (10m for a 5m federation interval).

If the federation interval changes, update Kiali’s thanos_proxy.scrape_interval, dashboard and alert rate() windows, and any health-check duration settings together. If Kiali’s thanos_proxy.scrape_interval remains at its 30s default while federation remains at 5m, Kiali can calculate rate windows and query steps that are too short for the samples actually arriving in Thanos. This can produce empty or misleading rate results even though the underlying metrics exist. Explicit PromQL range windows, such as [5m], are separate settings and must also be long enough to contain at least two federated samples. Conversely, changing only the edge scrape interval affects freshness and UWM resource usage, but does not reduce the hub’s federation delay.

Metrics Latency

The MCOA PrometheusAgent scrapes its federation endpoint every 300 seconds by default and remote-writes the result to Thanos. This means there is normally a 5-6 minute delay before new metrics appear in Kiali.

To use a different interval for the Kiali federation jobs, set spec.scrapeInterval on the kiali-istio-federation and kiali-istio-platform-federation ScrapeConfig resources. For example, the following makes the shared user-workload job run every minute:

spec:
  jobName: kiali-istio-federation
  scrapeInterval: 1m
  # Other fields from the ScrapeConfig in step 5 remain unchanged.

Set Kiali’s external_services.prometheus.thanos_proxy.scrape_interval to the same interval. Do not change the generated MCOA PrometheusAgent merely to alter these Kiali jobs: that changes the default interval for the agent’s other jobs as well.

Initial warm-up period: After deploying a new application, it takes approximately twice the collection interval before data appears in Kiali’s graph and metrics tab. This is because Kiali uses PromQL rate() functions which require at least two data points to compute a result, and with ACM’s collection interval, two data points take at least two collection cycles to accumulate. For example, with the default 5-minute interval, expect a ~10-minute warm-up period. After this initial warm-up, all time ranges in Kiali should display data normally. However, keep in mind that the most recent data visible in Kiali will always be at least one collection interval old, since metrics must complete a full collection cycle before they appear in Thanos.

Thanos Proxy Mode

Enable thanos_proxy when using ACM/Thanos:

external_services:
  prometheus:
    thanos_proxy:
      enabled: true
      retention_period: "365d" # Match your actual ACM Thanos retention
      scrape_interval: "5m"   # Must match the MCOA federation interval

When enabled: true, Kiali uses the configured scrape_interval and retention_period values directly, rather than querying Prometheus’s /api/v1/status/config and /api/v1/status/runtimeinfo endpoints to discover them. This is necessary because Thanos does not expose these Prometheus configuration endpoints.

Why these values matter:

scrape_interval: Kiali’s UI uses this value to compute PromQL rate() intervals and query step sizes. The rate interval must be large enough to contain at least two data points for rate() to produce results. With ACM, data points arrive in Thanos at the ACM collection interval (default 5 minutes), not at the local Prometheus scrape interval (typically 15-30 seconds). If scrape_interval is set too low (e.g., “30s”), the computed rate windows will be too narrow to capture two ACM data points, causing Kiali’s metrics tab to show empty charts even though data exists in Thanos.

Critical: Set scrape_interval to match the effective MCOA federation interval (default "5m"), not the local UWM scrape interval. MCOA’s default PrometheusAgent.spec.scrapeInterval is 300s; a per-job ScrapeConfig interval can override it. If you customize either value, configure Kiali with the effective interval.

retention_period: Used to limit time range queries to available data. ACM defaults to 365d retention when spec.advanced.retentionConfig is not explicitly configured in the MultiClusterObservability CR. If using the default, set retention_period to “365d”. If configuring custom retention, use at least 10d minimum (a Thanos requirement for downsampling to function). Always match retention_period to your actual ACM retention configuration. Use 14d only when your MultiClusterObservability retention policy is configured for 14 days.

Multi-Cluster Setup

For multi-cluster service mesh deployments with ACM:

1. Metrics Aggregation (Handled by ACM)

ACM automatically aggregates metrics from all managed clusters. Each cluster’s metrics include a cluster label with the cluster name (the metadata.name of the ManagedCluster resource). To get a list of all the clusters managed by ACM, run oc get managedcluster on the hub cluster.

Kiali can filter metrics by cluster using query_scope. The query_scope configuration adds label filters to every Prometheus query:

external_services:
  prometheus:
    # Example 1: Filter to a single cluster
    query_scope:
      cluster: "east-cluster"

    # Example 2: Filter by mesh_id and cluster
    query_scope:
      mesh_id: "mesh-1"
      cluster: "east-cluster"

Each key-value pair in query_scope is added as key="value" to every query. For example, cluster: "east-cluster" adds cluster="east-cluster" to all PromQL queries.

2. Remote Cluster Access (For Workload/Config Data)

While metrics come from ACM’s central Thanos, Kiali still needs direct API access to each cluster for:

Workload and service discovery
Istio configuration validation
Kubernetes resource details

Create remote cluster secrets as described in the multi-cluster setup guide. When the external Kiali deployment uses auth.strategy: openshift, also follow the OpenShift multi-cluster authentication guidance. In particular, a remote cluster that provides only remote_cluster_resources_only: true needs its Kiali OAuthClient callback URI to point to the external Kiali route and end in /api/auth/callback/<remote-cluster-name>. This remote API authentication is independent of ACM/MCOA metrics federation.

3. External Deployment Model

For multi-cluster with ACM, if you deploy Kiali on the hub cluster (or on a separate management cluster), you will typically want to run Kiali in external deployment mode:

clustering:
  ignore_home_cluster: true  # Kiali is external to mesh

kubernetes_config:
  cluster_name: "<management-cluster-name>"  # Unique name for the cluster where Kiali runs

This is Kiali’s own home-cluster identity. It is not required to match the ACM ManagedCluster name or an Istio clusterName; choose a value unique among the clusters configured in Kiali. In this external deployment model, set clustering.ignore_home_cluster: true so Kiali does not attempt to treat the management cluster as a mesh cluster.

See the External Kiali guide for complete external deployment instructions.

Certificate Management

Automatic Rotation

ACM-issued certificates (stored in the observability-grafana-certs secret in the ACM observability namespace) have 1-year validity and are automatically rotated by ACM before expiration. When certificates are rotated:

ACM updates the observability-grafana-certs secret in open-cluster-management-observability namespace
You must update the acm-observability-certs secret in Kiali’s namespace with the new certificate data. Options include:
- Re-run the extraction commands from Step 1: Obtain mTLS Certificates from ACM manually
- Use an ACM ConfigurationPolicy with hub cluster templating to automatically distribute and update the secret to the cluster where Kiali runs (see ACM Governance documentation for details)
Kubernetes updates the mounted files in Kiali pod (within 60 seconds after the secret update)
Kiali automatically uses new certificates on next connection (no pod restart needed)

Using Custom Certificates

If you prefer to use your own certificate infrastructure instead of ACM’s certificates:

Generate/obtain certificates signed by a CA trusted by ACM Observatorium API
Configure ACM to trust your CA (consult ACM documentation)
Create the acm-observability-certs secret with your certificates

Verification

Check Certificate Configuration

# Verify secret exists
oc get secret acm-observability-certs -n ${KIALI_NAMESPACE}

# Check certificate expiration
oc get secret acm-observability-certs -n ${KIALI_NAMESPACE} \
  -o jsonpath='{.data.tls\.crt}' | base64 -d | \
  openssl x509 -noout -enddate

# Verify CA bundle
oc get configmap kiali-cabundle -n ${KIALI_NAMESPACE} \
  -o jsonpath='{.data.additional-ca-bundle\.pem}' | \
  openssl x509 -noout -subject

Check Kiali Logs

Verify certificates are loaded successfully:

oc logs -n ${KIALI_NAMESPACE} deployment/kiali | grep -i "credential\|certificate"

# Expected output (at "info" log level):
# INF Loaded [1] valid CA certificate(s) from [/kiali-cabundle/additional-ca-bundle.pem]
#
# Additional output (at "debug" log level):
# DBG Credential file path configured: [/kiali-override-secrets/prometheus-cert/tls.crt]
# DBG Credential file path configured: [/kiali-override-secrets/prometheus-key/tls.key]

Test Metrics

Generate mesh traffic in one of your managed clusters
Wait for the initial warm-up period (approximately twice the ACM collection interval; default ~10 minutes) for metrics to propagate to Thanos and for enough data points to accumulate for rate calculations. The graph may appear sooner (after ~5 minutes).
Access Kiali UI and navigate to a workload
Verify metrics appear in the Metrics tab and traffic graph

Ambient Mode: If you are using Ambient mode:

Ztunnel-only traffic (no waypoint): You’ll see TCP metrics and traffic edges in the graph, but HTTP details (response codes, latency) will not be available.
Traffic through waypoints: You’ll see full L7 metrics, same as sidecar mode.

Verify Metrics in Thanos Directly

Validate each stage of the federation pipeline: raw Istio metrics in the managed cluster’s edge UWM Prometheus, workload:* recording-rule output in that same Prometheus, and the relabeled metrics in hub Thanos. Set the three values first:

HUB_CONTEXT="<hub-kubecontext>"
MANAGED_CONTEXT="<managed-cluster-kubecontext>"
MANAGED_CLUSTER_NAME="<acm-managed-cluster-name>" # List names with: oc --context="${HUB_CONTEXT}" get managedcluster -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'

PROM_POD=$(oc --context="${MANAGED_CONTEXT}" \
  -n openshift-user-workload-monitoring \
  get pods -l app.kubernetes.io/name=prometheus \
  -o jsonpath='{.items[0].metadata.name}')

# Edge UWM: raw counters scraped from Istio proxies
oc --context="${MANAGED_CONTEXT}" \
  -n openshift-user-workload-monitoring \
  exec -c prometheus "${PROM_POD}" -- \
  wget -qO- \
  'http://localhost:9090/api/v1/query?query=sum%28istio_requests_total%29' \
  | jq '.data.result'

# Edge UWM: aggregates produced by the recording rules
oc --context="${MANAGED_CONTEXT}" \
  -n openshift-user-workload-monitoring \
  exec -c prometheus "${PROM_POD}" -- \
  wget -qO- \
  'http://localhost:9090/api/v1/query?query=sum%28workload%3Aistio_requests_total%29' \
  | jq '.data.result'

# Hub Thanos: federated aggregates, relabeled back to istio_requests_total
oc --context="${HUB_CONTEXT}" get --raw \
  "/api/v1/namespaces/open-cluster-management-observability/services/http:observability-thanos-query-frontend:9090/proxy/api/v1/query?query=sum%28istio_requests_total%7Bcluster%3D%22${MANAGED_CLUSTER_NAME}%22%7D%29" \
  | jq '.data.result'

Generate traffic before querying and allow at least one MCOA collection interval. Each query should return a non-empty result. The workload: prefix exists only on the edge recording-rule series; MCOA removes it before writing the metric to hub Thanos. Because the hub sample can lag the edge by the federation interval, compare presence, labels, and approximately corresponding counter values rather than expecting exact point-in-time equality.

List the Istio and Envoy metric names in edge UWM. This output includes the workload:* series produced by the recording rules:

oc --context="${MANAGED_CONTEXT}" \
  -n openshift-user-workload-monitoring \
  exec -c prometheus "${PROM_POD}" -- \
  wget -qO- \
  'http://localhost:9090/api/v1/label/__name__/values' \
  | jq -r '
  [.data[] | select(test("istio|envoy"; "i"))] | unique
  | .[] ,
  "===\nTOTAL COUNT OF istio_ AND envoy_ METRICS: \(length)"'

List the metrics stored in hub Thanos. The result does not contain workload:* names because federation relabels those series, and it should have far fewer metric names than the edge:

oc --context="${HUB_CONTEXT}" get --raw \
  "/api/v1/namespaces/open-cluster-management-observability/services/http:observability-thanos-query-frontend:9090/proxy/api/v1/label/__name__/values" \
  | jq -r '
  [.data[] | select(test("istio|envoy|pilot"; "i"))] | unique
  | .[] ,
  "===\nTOTAL COUNT OF istio_, envoy_, AND pilot_ METRICS: \(length)"'

The following additional hub queries inspect the backend metric datastore used by ACM.

Note: These commands use jq to format JSON output. If you don’t have jq installed, simply omit | jq . to see the full, unfiltered and raw JSON.

# List available metric names (Kiali uses istio_*, pilot_*, and envoy_* metrics)
oc --context="${HUB_CONTEXT}" get --raw "/api/v1/namespaces/open-cluster-management-observability/services/http:observability-thanos-query-frontend:9090/proxy/api/v1/label/__name__/values" | jq -r '.data[] | select(startswith("istio_") or startswith("pilot_") or startswith("envoy_"))'

# Count timeseries for key Istio metrics (shows which metrics have data and how many unique timeseries)
oc --context="${HUB_CONTEXT}" get --raw "/api/v1/namespaces/open-cluster-management-observability/services/http:observability-thanos-query-frontend:9090/proxy/api/v1/query?query=count%20by%20(__name__)%20({__name__=~%22istio_requests_total|istio_tcp.*total%22})" | jq -r '.data.result[] | "\(.metric.__name__): \(.value[1])"'

# Query Istio request metrics with full details (limited to first result to show structure)
oc --context="${HUB_CONTEXT}" get --raw "/api/v1/namespaces/open-cluster-management-observability/services/http:observability-thanos-query-frontend:9090/proxy/api/v1/query?query=istio_requests_total" | jq '.data.result |= .[0:1]'

Troubleshooting

Empty Graph or No Metrics

Symptom: Kiali shows an empty graph, “No metrics” in the metrics tab, or both.

Causes and Solutions:

scrape_interval too low: If thanos_proxy.scrape_interval is set lower than the ACM collection interval (e.g., “30s” instead of “5m”), Kiali’s rate calculations will use windows too narrow to capture enough data points from Thanos
- Solution: Set thanos_proxy.scrape_interval to match the ACM collection interval (default “5m”). See Thanos Proxy Mode for details
Still in warm-up period: After deploying a new application, it takes approximately twice the ACM collection interval (~10 minutes by default) before enough data points exist for rate calculations
- Solution: Wait for the warm-up period to elapse
MCOA federation resources missing: The recording rules or federation job were not propagated to the managed cluster
- Solution: On the hub, verify the correctness of the kiali-istio-federation and kiali-istio-platform-federation ScrapeConfig resources and the kiali-istio-aggregation PrometheusRule. Then verify their references under the selected ClusterManagementAddOn placement. On a managed cluster, run oc get prometheusagent,scrapeconfig -A to find MCOA’s configured agent namespace (the default is open-cluster-management-agent-addon) and run oc get prometheusrule -n mesh-observability to verify the propagated rule.
PodMonitor missing: Prometheus not scraping Istio data plane components
- Solution: Create an istio-proxies-monitor-<namespace> PodMonitor in each mesh namespace (including the ztunnel namespace and namespaces with waypoint proxies if using Ambient mode)
UWM not enabled: User Workload Monitoring not configured
- Solution: Enable enableUserWorkload: true in cluster-monitoring-config ConfigMap in openshift-monitoring namespace
Missing source/destination labels: The graph builds its topology from workload and namespace labels in the metrics. Verify Istio metrics have proper labels
Namespace not selected: Ensure the namespace is selected in the graph’s namespace dropdown
Query scope mismatch: Check query_scope cluster names match actual cluster label values

See also the Why is my graph empty? FAQ for additional troubleshooting information.

TLS/Certificate Errors

Symptom: Kiali logs show “x509: certificate signed by unknown authority” or “tls: bad certificate”

Solutions:

Verify CA bundle: Ensure kiali-cabundle ConfigMap has the correct CA
```
oc get configmap kiali-cabundle -n ${KIALI_NAMESPACE} -o yaml
```

Check certificate chain: Verify client cert is signed by expected CA

oc get secret acm-observability-certs -n ${KIALI_NAMESPACE} \
  -o jsonpath='{.data.tls\.crt}' | base64 -d | \
  openssl x509 -noout -issuer

Verify projected volume: Check both ConfigMaps are mounted

oc exec -n ${KIALI_NAMESPACE} deploy/kiali -- ls -la /kiali-cabundle/
# Should show: additional-ca-bundle.pem, service-ca.crt

Connection Refused / Timeout

Symptom: Kiali cannot reach Observatorium API

Solutions:

Verify route exists:

oc get route observatorium-api -n open-cluster-management-observability

Check ACM is ready (should return “True”):

oc get mco observability -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}{"\n"}'

Test connectivity (should return “OK”):

oc get --raw "/api/v1/namespaces/open-cluster-management-observability/services/http:observability-thanos-query-frontend:9090/proxy/-/ready"

Check NetworkPolicies: Ensure no policies block egress from Kiali’s namespace

Ambient Mode: No HTTP Metrics

Symptom: Ambient mode workloads show TCP traffic in Kiali but no HTTP metrics (response codes, latency)

Possible causes:

No waypoint deployed: Ztunnel only provides L4 (TCP) metrics. Deploy a waypoint proxy for L7 (HTTP) visibility.
Missing waypoint PodMonitor: Even with a waypoint, metrics won’t be collected without a PodMonitor:
- Verify waypoint pod exists: oc get pods -n <namespace> -l gateway.networking.k8s.io/gateway-class-name=istio-waypoint
- Create PodMonitor in the waypoint’s namespace (same config as sidecar PodMonitor)
Missing recording rule for waypoint metrics: Verify that the shared kiali-istio-aggregation rule is present in mesh-observability and that the waypoint namespace has a PodMonitor. The platform ScrapeConfig is cluster-wide and does not need a namespace-specific copy.

Ambient Mode: No Ztunnel Metrics

Symptom: Ambient mode workloads show no traffic at all in Kiali

Possible causes:

Missing ztunnel PodMonitor: Create an istio-proxies-monitor-<ztunnel-namespace> PodMonitor in the ztunnel namespace
Wrong ztunnel namespace: Verify ztunnel location: oc get pods -l app=ztunnel -A
Missing recording rule: Verify that the shared kiali-istio-aggregation rule is present in mesh-observability and that the ztunnel namespace has a PodMonitor. The platform ScrapeConfig is cluster-wide and does not need a namespace-specific copy.

Reference

This example represents a fully configured Kiali installation using ACM Observability via the Observatorium API with mTLS:

apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
  name: kiali
  namespace: <kiali-namespace>
spec:
  clustering:
    ignore_home_cluster: true  # External deployment

  kubernetes_config:
    cluster_name: "<management-cluster-name>"

  external_services:
    prometheus:
      url: "<observatorium-api-url>"

      auth:
        type: none
        cert_file: "secret:acm-observability-certs:tls.crt"
        key_file: "secret:acm-observability-certs:tls.key"

      thanos_proxy:
        enabled: true
        retention_period: "365d"
        scrape_interval: "5m"

Required Kubernetes resources:

---
# mTLS client certificates (from ACM)
# Data extracted from Secret observability-grafana-certs in namespace open-cluster-management-observability
apiVersion: v1
kind: Secret
metadata:
  name: acm-observability-certs
  namespace: <kiali-namespace>
type: Opaque
data:
  tls.crt: <base64-encoded-certificate>  # From observability-grafana-certs secret, tls.crt key
  tls.key: <base64-encoded-key>          # From observability-grafana-certs secret, tls.key key

---
# Server CA trust (from ACM)
# Data extracted from Secret observability-client-ca-certs (or observability-server-ca-certs) in namespace open-cluster-management-observability
apiVersion: v1
kind: ConfigMap
metadata:
  name: kiali-cabundle
  namespace: <kiali-namespace>
data:
  additional-ca-bundle.pem: |
    -----BEGIN CERTIFICATE-----
    <ACM Observability CA certificate>  # From ca.crt or tls.crt key (see Step 2 for extraction commands)
    -----END CERTIFICATE-----

Additional Resources

Docs: Header strategy

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

The header strategy assumes a reverse proxy is in front of Kiali, such as OpenUnison or OAuth2 Proxy, injecting the user’s identity into each request to Kiali as an Authorization header. This token can be an OpenID Connect token or any other token the cluster recognizes.

In addition to a user token, the header strategy supports impersonation headers. If the impersonation headers are present in the request, then Kiali will act on behalf of the user specified by the impersonation (assuming the token supplied in the Authorization header is authorized to do so).

The header strategy allows for namespace access control.

The header strategy is only supported for single cluster.

Set-up

The header strategy will work with any Kubernetes cluster. The token provided must be supported by that cluster. For instance, most “on-prem” clusters support OpenID Connect, but cloud hosted clusters do not. For clusters that don’t support a token, the impersonation headers can be injected by the reverse proxy.

spec:
  auth:
    strategy: header

The header strategy doesn’t have any additional configuration.

HTTP Header

The header strategy looks for a token in the Authorization HTTP header with the Bearer prefix. The HTTP header should look like:

Authorization: Bearer TOKEN

Where TOKEN is the appropriate token for your cluster. This TOKEN will be submitted to the API server via a TokenReview to validate the token ONLY on the first access to Kiali. On subsequent calls the TOKEN is passed through directly to the API server.

Security Considerations

Network Policies

A policy should be put in place to make sure that the only “client” for Kiali is the authenticating reverse proxy. This helps limit potential abuse and ensures that the authenticating reverse proxy is the source of truth for who accessed Kiali.

Short Lived Tokens

The authenticating reverse proxy should inject a short lived token in the Authorization header. A shorter lived token is less likely to be abused if leaked. Kiali will take whatever token is passed into the reqeuest, so as tokens are regenerated Kiali will use the new token.

Impersonation

TokenRequest API

The authenticating reverse proxy should use the TokenRequest API instead of static ServiceAccount tokens when possible while using impersonation. The ServiceAccount that can impersonate users and groups is privileged and having it be short lived cuts down on the possibility of a token being leaked while it’s being passed between different parts of the infrastructure.

Drop Incoming Impersonation Headers

The authenticating proxy MUST drop any headers it receives from a remote client that match the impersonation headers. Not only do you want to make sure that the authenticating proxy can’t be overriden on which user to authenticate, but also what groups they’re a member of.

Docs: Install via Helm

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

Helm is a popular tool that lets you manage Kubernetes applications. Applications are defined in a package named Helm chart, which contains all of the resources needed to run an application.

Kiali has a Helm Charts Repository at https://kiali.org/helm-charts. Two Helm Charts are provided:

The kiali-operator Helm Chart installs the Kiali operator which in turn installs Kiali when you create a Kiali CR.
The kiali-server Helm Chart installs a standalone Kiali without the need of the Operator nor a Kiali CR.

The kiali-server Helm Chart does not provide all the functionality that the Kiali Operator provides. Some features you read about in the documentation may only be available if you install the Kiali Server using the Kiali Operator (see this FAQ for details). Therefore, although the kiali-server Helm Chart is actively maintained, it is not recommended and is only provided for convenience. If using Helm, the recommended method is to install the kiali-operator Helm Chart and then create a Kiali CR to let the Operator deploy Kiali.

Make sure you have the helm command available by following the Helm installation docs.

Helm version 3.10 is the minimum required Helm version. Older versions will not work. Newer versions have not been tested.

Adding the Kiali Helm Charts repository

Add the Kiali Helm Charts repository with the following command:

$ helm repo add kiali https://kiali.org/helm-charts

All helm commands in this page assume that you added the Kiali Helm Charts repository as shown.

If you already added the repository, you may want to update your local cache to fetch latest definitions by running:

$ helm repo update

Installing Kiali using the Kiali operator

This installation method gives Kiali access to existing namespaces as well as namespaces created later. See Namespace Management for more information.

Once you’ve added the Kiali Helm Charts repository, you can install the latest Kiali Operator along with the latest Kiali server by running the following command:

$ helm install \
    --set cr.create=true \
    --set cr.namespace=istio-system \
    --set cr.spec.auth.strategy="anonymous" \
    --namespace kiali-operator \
    --create-namespace \
    kiali-operator \
    kiali/kiali-operator

The --namespace kiali-operator and --create-namespace flags instructs to create the kiali-operator namespace (if needed), and deploy the Kiali operator on it. The --set cr.create=true and --set cr.namespace=istio-system flags instructs to create a Kiali CR in the istio-system namespace. Since the Kiali CR is created in advance, as soon as the Kiali operator starts, it will process it to deploy Kiali. After Kiali has started, you can access Kiali UI through ‘http://localhost:20001’ by executing kubectl port-forward service/kiali -n istio-system 20001:20001 because of --set cr.spec.auth.strategy="anonymous". But realize that anonymous mode will allow anyone to be able to see and use Kiali. If you wish to require users to authenticate themselves by logging into Kiali, use one of the other auth strategies.

The Kiali Operator Helm Chart is configurable. Check available options and default values by running:

$ helm show values kiali/kiali-operator

You can pass the --version X.Y.Z flag to the helm install and helm show values commands to work with a specific version of Kiali.

The kiali-operator Helm Chart mirrors all settings of the Kiali CR as chart values that you can configure using regular --set flags. For example, the Kiali CR has a spec.server.web_root setting which you can configure in the kiali-operator Helm Chart by passing --set cr.spec.server.web_root=/your-path to the helm install command.

For more information about the Kiali CR, see the Creating and updating the Kiali CR page.

Operator-Only Install

To install only the Kiali Operator, omit the --set cr.create and --set cr.namespace flags of the helm command previously shown. For example:

$ helm install \
    --namespace kiali-operator \
    --create-namespace \
    kiali-operator \
    kiali/kiali-operator

This will omit creation of the Kiali CR, which you will need to create later to install Kiali Server. This option is good if you plan to do large customizations to the installation.

Installing Multiple Instances of Kiali

By installing a single Kiali operator in your cluster, you can install multiple instances of Kiali by simply creating multiple Kiali CRs. For example, if you have two Istio control planes in namespaces istio-system and istio-system2, you can create a Kiali CR in each of those namespaces to install a Kiali instance in each control plane.

If you wish to install multiple Kiali instances in the same namespace, or if you need the Kiali instance to have different resource names than the default of kiali, you can specify spec.deployment.instance_name in your Kiali CR. The value for that setting will be used to create a unique instance of Kiali using that instance name rather than the default kiali. One use-case for this is to be able to have unique Kiali service names across multiple Kiali instances in order to be able to use certain routers/load balancers that require unique service names.

Since the spec.deployment.instance_name field is used for the Kiali resource names, including the Service name, you must ensure the value you assign this setting follows the Kubernetes DNS Label Name rules. If it does not, the operator will abort the installation. And note that because Kiali uses this as a prefix (it may append additional characters for some resource names) its length is limited to 40 characters.

Standalone Kiali installation

To install the Kiali Server without the operator, use the kiali-server Helm Chart:

$ helm install \
    --namespace istio-system \
    kiali-server \
    kiali/kiali-server

The kiali-server Helm Chart mirrors all settings of the Kiali CR as chart values that you can configure using regular --set flags. For example, the Kiali CR has a spec.server.web_fqdn setting which you can configure in the kiali-server Helm Chart by passing the --set server.web_fqdn flag as follows:

$ helm install \
    --namespace istio-system \
    --set server.web_fqdn=example.com \
    kiali-server \
    kiali/kiali-server

Upgrading Helm installations

If you want to upgrade to a newer Kiali version (or downgrade to older versions), you can use the regular helm upgrade commands. For example, the following command should upgrade the Kiali Operator to the latest version:

$ helm upgrade \
    --namespace kiali-operator \
    --reuse-values \
    kiali-operator \
    kiali/kiali-operator

WARNING: No migration paths are provided. However, Kiali is a stateless application and if the helm upgrade command fails, please uninstall the previous version and then install the new desired version.

By upgrading the Kiali Operator, existent Kiali Server installations managed with a Kiali CR will also be upgraded once the updated operator starts.

Managing configuration of Helm installations

After installing either the kiali-operator or the kiali-server Helm Charts, you may be tempted to manually modify the created resources to modify the installation. However, we recommend using helm upgrade to update your installation.

For example, assuming you have the following installation:

$ helm list -n kiali-operator
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
kiali-operator  kiali-operator  1               2021-09-14 18:00:45.320351026 -0500 CDT deployed        kiali-operator-1.40.0   v1.40.0

Notice that the current installation is version 1.40.0 of the kiali-operator. Let’s assume you want to use your own mirrors of the Kiali Operator container images. You can update your installation with the following command:

$ helm upgrade \
    --namespace kiali-operator \
    --reuse-values \
    --set image.repo=your_mirror_registry_url/owner/kiali-operator-repo \
    --set image.tag=your_mirror_tag \
    --version 1.40.0 \
    kiali-operator \
    kiali/kiali-operator

Make sure that you specify the --reuse-values flag to take the configuration of your current installation. Then, you only need to specify the new settings you want to change using --set flags.

Make sure that you specify the --version X.Y.Z flag with the version of your current installation. Otherwise, you may end up upgrading to a new version.

Uninstalling

Removing the Kiali operator and managed Kialis

If you used the kiali-operator Helm chart, first you must ensure that all Kiali CRs are deleted. For example, the following command will agressively delete all Kiali CRs in your cluster:

$ kubectl delete kiali --all --all-namespaces

The previous command may take some time to finish while the Kiali operator removes all Kiali installations.

Then, remove the Kiali operator using a standard helm uninstall command. For example:

$ helm uninstall --namespace kiali-operator kiali-operator
$ kubectl delete crd kialis.kiali.io

You have to manually delete the kialis.kiali.io CRD because Helm won’t delete it.

If you fail to delete the Kiali CRs before uninstalling the operator, a proper cleanup may not be done.

Known problem: uninstall hangs (unable to delete the Kiali CR)

Typically this happens if not all Kiali CRs are deleted prior to uninstalling the operator. To force deletion of a Kiali CR, you need to clear its finalizer. For example:

$ kubectl patch kiali kiali -n istio-system -p '{"metadata":{"finalizers": []}}' --type=merge

This forces deletion of the Kiali CR and will skip uninstallation of the Kiali Server. Remnants of the Kiali Server may still exist in your cluster which you will need to manually remove.

Removing standalone Kiali

If you installed a standalone Kiali by using the kiali-server Helm chart, use the standard helm uninstall commands. For example:

$ helm uninstall --namespace istio-system kiali-server

Docs: Kiali MCP

Mon, 01 Jan 0001 00:00:00 +0000

Kiali MCP is an integration that allows MCP-capable AI assistants to query (and optionally manage) Kiali-related data by calling tools exposed by an MCP server.

The implementation is provided as part of the Kubernetes MCP Server upstream and also for Openshift MCP server. It exposes a kiali toolset (see upstream guide: docs/KIALI.md).

Prerequisites

A reachable Kiali endpoint (Route/Ingress/Service URL).
Kubernetes credentials available to the MCP server (kubeconfig or in-cluster config).

Enable the `kiali` toolset

Create a TOML config file and enable kiali in toolsets.

toolsets = ["core", "kiali"]

[toolset_configs.kiali]
url = "https://kiali.example" # Endpoint/route to reach the Kiali console
# insecure = true  # optional: allow insecure TLS (not recommended in production)
# certificate_authority = "/path/to/ca.crt"  # CA bundle for Kiali's TLS cert

Notes:

If url is https:// and insecure = false, you must provide certificate_authority.
Authentication to Kiali is performed using the server’s Kubernetes credentials (it obtains/uses a bearer token for Kiali calls).

Connect from an MCP client

How you wire this into a specific client depends on the client, but the core idea is the same: start the MCP server with your kubeconfig and your TOML config.

Example (conceptual) command:

kubernetes-mcp-server --config /path/to/config.toml --read-only

Once connected, your assistant can use the Kiali tools (for example: mesh graph, metrics, traces, workload logs) to power a chatbot-like experience outside the Kiali UI (for example, in an IDE).

Docs: Perses

Mon, 01 Jan 0001 00:00:00 +0000

Perses configuration

The Perses community dashboards provide preconfigured Perses dashboards for the most relevant mesh metrics. Although Kiali offers similar views in its metrics dashboards, it is not in Kiali’s goals to provide the advanced querying options, nor the highly customizable settings, that are available in Perses. They are the same as those provided by Istio’s Grafana add-on. Thus, it is recommended that you use Perses if you need those advanced options.

Kiali, from version v2.15, can provide a direct link from its metric dashboards to the equivalent or most similar Perses dashboard, which is convenient if you need the powerful Perses options.

The Perses links will appear in the Kiali metrics pages. For example:

For these links to appear in Kiali you need to manually configure the Perses URL and the dashboards that come preconfigured with Istio, like in the following example:

Kiali will query Perses and try to fetch the configured dashboards. For this reason Kiali must be able to reach Perses, authenticate, and find the Istio dashboards. The Istio dashboards must be installed in Perses for the links to appear in Kiali.

spec:
  external_services:
    perses:
      enabled: true
      # Perses service name is "perses" and is in the "telemetry" namespace.
      internal_url: 'http://perses.telemetry:4000/'
      # Public facing URL of Perses
      external_url: 'http://my-ingress-host/perses'
      dashboards:
        - name: "Istio Service Dashboard"
          variables:
            namespace: "var-namespace"
            service: "var-service"
            datasource: "var-datasource"
        - name: "Istio Workload Dashboard"
          variables:
            namespace: "var-namespace"
            workload: "var-workload"
        - name: "Istio Mesh Dashboard"

        - name: "Istio Ztunnel Dashboard"
          variables:
            namespace: "var-namespace"
            workload: "var-workload"
      # Perses project
      project: "istio"

When running Perses with the cluster observability operator in OpenShift, it requires an additional configuration item (Available from Kiali >2.17), so the url format can be compatible with the plugin UI URL:

spec:
  external_services:
    perses:
      ...
      url_format: "openshift"

The internal URL shouldn’t be set to avoid an internal validation of the Dashboards. The external URL should be set to the OpenShift cluster, without the additional path.

Perses authentication configuration

The Kiali CR provides authentication configuration that will be used to connect to your Perses instance and for detecting your Perses version in the Mesh graph.

Basic and OAuth2 client_credentials authentication are supported.

spec:
  external_services:
    perses:
      enabled: true
      auth:
        insecure_skip_verify: false
        password: "pwd"
        type: "basic"
        username: "user"
      health_check_url: ""

To configure a secret to be used as a user or password, see this FAQ entry.

To authenticate using OAuth2 client_credentials flow, set type: "oauth2" and provide the oauth2 block:

spec:
  external_services:
    perses:
      auth:
        type: "oauth2"
        oauth2:
          client_id: "my-client-id"
          client_secret: "secret:my-oauth2-secret:client_secret"
          token_url: "https://idp.example.com/token"
          scopes: []            # optional: list of OAuth2 scopes to request
          audience: ""          # optional: some providers require this
          auth_style: "header"  # "header" (default) or "params"

The client_secret field supports the secret:<secretName>:<secretKey> pattern for automatic secret mounting and rotation without pod restart. See the FAQ entry for details.

insecure_skip_verify applies only to the Perses connection, not to the OAuth2 token endpoint. The token endpoint always validates TLS certificates. To trust a private CA for the token endpoint, add the CA to the kiali-cabundle ConfigMap as described in the TLS Configuration page.

TLS Certificate Configuration

If your Perses server uses HTTPS with a certificate issued by a private CA, see the TLS Configuration page to learn how to configure Kiali to trust your CA.

Docs: Tempo

Mon, 01 Jan 0001 00:00:00 +0000

Grafana Tempo Configuration

There are two possibilities to integrate Kiali with Grafana Tempo:

Using the Grafana Tempo API: This option returns the traces from the Tempo API in OpenTelemetry format.
Using the Jaeger frontend with the Grafana Tempo backend.
Appendix: Configuration table

Using the Grafana Tempo API

There are two steps to set up Kiali and Grafana Tempo:

Set up the Kiali CR updating the Tracing and Grafana sections.
Set up a Tempo data source in Grafana.

Set up the Kiali CR

This is a configuration example to set up Kiali tracing with Grafana Tempo:

spec:
  external_services:
    tracing:
      # Enabled by default. Kiali will anyway fallback to disabled if
      # Tempo is unreachable.
      enabled: true
      health_check_url: "https://tempo-instance.grafana.net"
      # Tempo service name is "query-frontend" and is in the "tempo" namespace.
      # Make sure the URL you provide corresponds to the non-GRPC enabled endpoint
      # It does not support grpc yet, so make sure "use_grpc" is set to false.
      internal_url: "http://tempo-tempo-query-frontend.tempo.svc.cluster.local:3200/"
      provider: "tempo"
      tempo_config:
        org_id: "1"
        datasource_uid: "a8d2ef1c-d31c-4de5-a90b-e7bc5252cd00"
        url_format: "grafana"
      use_grpc: false
      # Public facing URL of Tempo 
      external_url: "https://grafana-istio-system.apps-crc.testing/"

Kiali uses the external_url to construct “View in tracing” links in the UI. For the Tempo provider the default url_format is grafana. So, by default the URL will have the Grafana UI format when linking to specific services and traces.

It is also possible to set url_format to openshift. In this case the URL will redirect to the UI Plugin in the OpenShift console. When it is set to openshift, there are other settings as well:

spec:
  external_services:
    tracing:
      tempo_config:
        name: "sample"
        namespace: "tempo"
        tenant: "default"
        url_format: "openshift"

When the tenant is specified, if internal_url doesn’t have a path, it will be autocompleted with the Tempo path. For this example:

internal_url: https://tempo-sample-gateway.tempo.svc.cluster.local:8080/

Will be autocompleted to: https://tempo-sample-gateway.tempo.svc.cluster.local:8080/api/traces/v1/{tenant}/tempo

The other valid option for url_format is jaeger, used when the Jaeger UI is available in Tempo.

Set up a Tempo Datasource in Grafana

We can optionally set up a default Tempo datasource in Grafana so that you can view the Tempo tracing data within the Grafana UI, as you see here:

To set up the Tempo datasource, go to the Home menu in the Grafana UI, click Data sources, then click the Add new data source button and select the Tempo data source. You will then be asked to enter some data to configure the new Tempo data source:

The most important values to set up are the following:

Mark the data source as default, so the URL that Kiali uses will redirect properly to the Tempo data source.
Update the HTTP URL. This is the internal URL of the HTTP tempo frontend service. e.g. http://tempo-tempo-query-frontend.tempo.svc.cluster.local:3200/

Additional configuration

The Traces tab in the Kiali UI will show your traces in a bubble chart:

Increasing performance is achievable by enabling gRPC access, specifically for query searches. However, accessing the HTTP API will still be necessary to gather information about individual traces. This is an example to configure the gRPC access:

spec:
  external_services:
    tracing:
      enabled: true
      # grpc port defaults to 9095
      grpc_port: 9095 
      internal_url: "http://query-frontend.tempo:3200"
      provider: "tempo"
      use_grpc: true
      external_url: "http://my-tempo-host:3200"

Service check URL

By default, Kiali will check the service health in the endpoint /status/services, but sometimes, this is exposed in a different url, which can lead to a component unreachable message:

This can be changed with the health_check_url configuration option.

spec:
  external_services:
    tracing:
      health_check_url: "http://query-frontend.tempo:3200"

Configuration for the Grafana Tempo Datasource

In order to correctly redirect Kiali to the right Grafana Tempo Datasource, there are a couple of configuration options to update:

spec:
  external_services:
    tracing:
      tempo_config:
        org_id: "1"
        datasource_uid: "a8d2ef1c-d31c-4de5-a90b-e7bc5252cd00"

org_id is usually not needed since “1” is the default value which is also Tempo’s default org id. The datasource_uid needs to be updated in order to redirect to the right datasource in Grafana versions 10 or higher.

Using the Jaeger frontend with Grafana Tempo tracing backend

It is possible to use the Grafana Tempo tracing backend exposing the Jaeger API. tempo-query is a Jaeger storage plugin. It accepts the full Jaeger query API and translates these requests into Tempo queries.

Since Tempo is not yet part of the built-in addons that are part of Istio, you need to manage your Tempo instance.

Tanka

The official Grafana Tempo documentation explains how to deploy a Tempo instance using Tanka. You will need to tweak the settings from the default Tanka configuration to:

Expose the Zipkin collector
Expose the GRPC Jaeger Query port

When the Tempo instance is deployed with the needed configurations, you have to set meshConfig.defaultConfig.tracing.zipkin.address from Istio to the Tempo Distributor service and the Zipkin port. Tanka will deploy the service in distributor.tempo.svc.cluster.local:9411.

The external_services.tracing.internal_url Kiali option needs to be set to: http://query-frontend.tempo.svc.cluster.local:16685.

Tempo Operator

The Tempo Operator for Kubernetes provides a native Kubernetes solution to deploy Tempo easily in your system.

After installing the Tempo Operator in your cluster, you can create a new Tempo instance with the following CR:

kubectl create namespace tempo
kubectl apply -n tempo -f - <<EOF
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: smm
spec:
  storageSize: 1Gi
  storage:
    secret:
      type: s3
      name: object-storage
  template:
    queryFrontend:
      component:
        resources:
          limits:
            cpu: "2"
            memory: 2Gi
      jaegerQuery:
        enabled: true
        ingress:
          type: ingress
EOF

Note the name of the bucket where the traces will be stored in our example is called object-storage. Check the Tempo Operator documentation to know more about what storages are supported and how to create the secret properly to provide it to your Tempo instance.

Now, you are ready to configure the meshConfig.defaultConfig.tracing.zipkin.address field in your Istio installation. It needs to be set to the 9411 port of the Tempo Distributor service. For the previous example, this value will be tempo-smm-distributor.tempo.svc.cluster.local:9411.

Now, you need to configure the internal_url setting from Kiali to access the Jaeger API. You can point to the 16685 port to use GRPC or 16686 if not. For the given example, the value would be http://tempo-ssm-query-frontend.tempo.svc.cluster.local:16685.

There is a related tutorial with detailed instructions to setup Kiali and Grafana Tempo with the Operator.

Configuration table

Supported versions

Kiali Version	Jaeger	Tempo	Tempo with JaegerQuery
<= 1.79 (OSSM 2.5)	✅	❌	✅
> 1.79	✅	✅	✅

Minimal configuration for Kiali <= 1.79

In external_services.tracing

	http	grpc
Jaeger	`.internal_url = 'http://jaeger_service_url:16686/jaeger'` `.use_grpc = false`	`.internal_url = 'http://jaeger_service_url:16685/jaeger'` `.use_grpc = true (Not required: by default)`
Tempo	`.internal_url = 'http://query_frontend_url:16686'` `.use_grpc = false`	`.internal_url = 'http://query_frontend_url:16685'` `.use_grpc = true (Not required: by default)`

Minimal configuration for Kiali > 1.79

	http	grpc
Jaeger	`.internal_url = 'http://jaeger_service_url:16686/jaeger'` `.use_grpc = false`	`.internal_url = 'http://jaeger_service_url:16685/jaeger'` `.use_grpc = true (Not required: by default)`
Tempo	`internal_url = 'http://query_frontend_url:3200'` `.use_grpc = false` `.provider = 'tempo'`	`.internal_url = 'http://query_frontend_url:3200'` `.grpc_port: 9095` `.provider: 'tempo'` `.use_grpc = true (Not required: by default)`

Tempo tuning

Resources consumption

Grafana Tempo is a powerful tool, but it can lead to performance issues when not configured correctly. For example, the following configuration is not recommended and may lead to OOM issues for simple queries in the query-frontend component:

spec:
  resources:
    total:
      limits:
        memory: 2Gi
        cpu: 2000m

These resources are shared between all the Tempo components. When needed, apply resources to each specific component, instead of applying the resources globally:

spec:
  template:
    queryFrontend:
      component:
        resources:
          limits:
            cpu: "2"
            memory: 2Gi

This Grafana Dashboard is available to measure the resources used in the tempo namespace.

Caching

Tempo offers multi-level caching that is used by default with Tanka and Helm deployment examples. It uses external cache, supporting Memcached and Redis. The lower level cache has a higher hit rate, and caches bloom filters and parquet data. The higher level caches frontend-search data.

Optimizing the cache depends on the application usage, and can be done modifying different parameters:

Connection limit for MemCached: Should be increased in large deployments, as MemCached is set to 1024 by default.
Cache size control: Should be increased when the working set is larger than the size of cache.

Tune search pipeline

There are many parameters to tune the search pipeline, some of these:

max_concurrent_queries: If it is too high it can cause OOM.
concurrent_jobs: How many jobs are done concurrently.
max_retries: When it is too high it can result in a lot of load.

Dedicated attribute columns

When using the vParquet3 storage format , defining dedicated attribute columns can improve the query performance. In order to best choose those columns (Up to 10), a good criteria is to choose attributes that contribute growing the block size (And not those commonly used).

Tempo authentication configuration

The Kiali CR provides authentication configuration that will be used also for querying the version check to provide information in the Mesh graph.

spec:
  external_services:
    tracing:
      enabled: true
      auth:
        insecure_skip_verify: false
        password: "pwd"
        token: ""
        type: "basic"
        use_kiali_token: false
        username: "user"
      health_check_url: ""

To configure a secret to be used as a password, see this FAQ entry.

To authenticate using OAuth2 client_credentials flow, set type: "oauth2" and provide the oauth2 block:

spec:
  external_services:
    tracing:
      use_grpc: false           # must be false; oauth2 token injection is not implemented for gRPC transport
      auth:
        type: "oauth2"
        oauth2:
          client_id: "my-client-id"
          client_secret: "secret:my-oauth2-secret:client_secret"
          token_url: "https://idp.example.com/token"
          scopes: []            # optional: list of OAuth2 scopes to request
          audience: ""          # optional: some providers require this
          auth_style: "header"  # "header" (default) or "params"

OAuth2 authentication requires use_grpc: false. OAuth2 token injection is not implemented for gRPC transport.

insecure_skip_verify applies only to the Tempo connection, not to the OAuth2 token endpoint. The token endpoint always validates TLS certificates. To trust a private CA for the token endpoint, add the CA to the kiali-cabundle ConfigMap as described in the TLS Configuration page.

The client_secret field supports the secret:<secretName>:<secretKey> pattern for automatic secret mounting and rotation without pod restart. See the FAQ entry for details.

TLS Certificate Configuration

If your Tempo server uses HTTPS with a certificate issued by a private CA, see the TLS Configuration page to learn how to configure Kiali to trust your CA.

Tempo cache

Kiali 2.2 includes a simple tracing cache for Tempo that stores the last N traces. By default, it is enabled and it keeps the last 200 traces. It can be modified in the Kiali CR with:

spec:
  external_services:
    tracing:
      enabled: true
      tempo_config:
        cache_enabled: true
        cache_capacity: 200

Kiali emits some cache metrics. The following query obtains the cache hit rate:

(sum(kiali_cache_hits_total{name="tempo"})/sum(kiali_cache_requests_total{name="tempo"})) * 100

Docs: Tuning

Mon, 01 Jan 0001 00:00:00 +0000

Production environments should not be using the Istio Prometheus add-on, or carrying over its configuration settings. That is useful only for small, or demo installations. Instead, Prometheus should have been installed in a production-oriented way, following the Prometheus documentation.

This page is primarily for users where Prometheus is being used specifically for Kiali, and possible optimizations that can be made knowing that Kiali does not utilize all of the default Istio and Envoy telemetry.

Istio and Envoy generate a large amount of telemetry for analysis and troubleshooting. This can result in significant resources being required to ingest and store the telemetry, and to support queries into the data. If you use the telemetry specifically to support Kiali, it is possible to drop unnecessary metrics and unnecessary labels on required metrics. This FAQ Entry displays the metrics and attributes required for Kiali to operate.

Option 1: Recording Rules and Federation (Recommended)

For production meshes at scale, metric thinning on a single Prometheus TSDB reduces storage somewhat but still retains per-proxy Istio series. A more efficient approach—aligned with Istio Observability Best Practices—is to:

Aggregate using a short-lived “edge” Prometheus. Scrape your raw metrics and then use recording rules that “sum away” per-proxy labels (pod, instance, etc.) into workload:* series.
- “Sum away” means that several time-series will be aggregated into one, by combining those with like values for specified fields. The resulting value is the sum of the individual time-series values.
Federate using a long-lived Federated Prometheus. Use Prometheus federation to pull the edge aggregates (plus required non-aggregated metrics) into Federated Prometheus.
- Federating will relabel the workload:* aggregates back to standard metric names (e.g. workload:istio_requests_total back to istio_requests_total) on Federated Prometheus so Kiali queries standard metric names.

Kiali already aggregates traffic at workload/service granularity in its PromQL; it does not use per-pod Istio labels. Pre-aggregated counters and histograms are therefore compatible with the traffic graph, health monitoring, and metrics tabs.

Architecture

  Edge Prometheus                         Federated Prometheus
  (scrapes Istio/Envoy)                   (long retention; Kiali queries here)
        │                                           ▲
        │  recording rules                          │  /federate
        │  istio_*  →  workload:*                   │  + relabel workload: → istio_
        │  short retention                          │
        └───────────────────────────────────────────┘

Edge Prometheus is whichever instance scrapes Istio mesh telemetry (often not the same as your platform monitoring stack). It evaluates recording rules and keeps a short retention (for example 6 hours) on raw and aggregated series.

Federated Prometheus is the long-retention TSDB Kiali should query. It federates selected series from the edge, relabels workload:* back to istio_*, and holds long retention. Both instances are production components; only Federated Prometheus is Kiali’s query target.

Configure Kiali accordingly:

The two Prometheus instances are often in different namespaces or clusters in real deployments. The Kiali repository includes a demo lab harness that co-locates both in istio-system for learning and CI—it is not a production installer.

spec:
  external_services:
    prometheus:
      # Federated Prometheus URL — not the edge Istio scraper
      url: "http://prometheus-federated.monitoring:9090/"

If you use Istio Perses (or Grafana) dashboards with Kiali, configure them to point at the same Federated Prometheus URL.

Query target: In this pattern, external_services.prometheus.url always targets Federated Prometheus—the long-retention TSDB that holds federated mesh metrics. Edge Prometheus exists only to collect raw telemetry, evaluate recording rules, and federate upstream; it is not Kiali’s database.

kiali_* self-monitoring metrics must also end up in that Federated Prometheus TSDB, but may reach it via edge aggregation and federation or via direct scrape—see Kiali self-monitoring metrics.

Metric tiers

Federation configuration is split into tiers (metric groupings) so that Federated Prometheus pulls only needed metrics:

Tier	Purpose	Required for
Core	Kiali required metrics	Traffic graph, health, lists, mesh overview
Istio Dashboards	Optional control-plane, perf, ztunnel, and WASM metrics	Perses Istio dashboards
Kiali self-monitoring	Kiali operational metrics (`kiali_*`)	Kiali Internal Metrics dashboard, optional health-status alerting

Some Perses dashboards work with the Core tier alone (Mesh, service, and workload dashboards). To ensure all of the Istio dashboards are supported, enable the Istio Dashboards tier. Enable Kiali self-monitoring when the built-in Kiali metrics dashboard or kiali_health_status alerting is needed. Note that if you define your own Kiali Custom Dashboards, you will need to ensure any required metrics are also configured for the federated Prometheus.

Production reference files

For production deployments, merge the YAML below from hack/prometheus/federation/ into your existing Prometheus configuration. These files are reference snippets—not a Kiali installer and not full deployments. Integrate them the same way you manage other Prometheus rules and scrape jobs (rule_files, ConfigMap volume mounts, PrometheusRule CRs, Helm values, etc.).

File	Integrate into	Purpose
`core-recording-rules.yml`	Edge Prometheus (scrapes Istio/Envoy)	Recording rules producing `workload:*` series
`core-federation-match.yml`	Federated Prometheus federation job	Core-tier `match[]` selectors
`core-metrics.yml`	(reference)	Canonical core Istio metric list
`istio-dashboard-federation-match.yml`	Production federation job (optional)	Perses dashboard `match[]` selectors
`istio-dashboard-metrics.yml`	(reference)	Optional Istio dashboard metric list
`kiali-metrics-recording-rules.yml`	Kiali edge Prometheus (optional)	Recording rules for `kiali:*` series
`kiali-metrics-federation-match.yml`	Production federation job (optional)	Federation selectors for `kiali:*` series
`kiali-metrics.yml`	(reference)	Canonical Kiali self-monitoring metric list

The recording rules, federation, and production checklist sections below describe how to apply these files. Optional tiers (Istio dashboards, Kiali self-monitoring) are added only when those features are enabled.

Demo lab files (not for production)

The demo/ subdirectory under the same path contains scripts and sample Kubernetes deployments for learning and CI. They patch the Istio add-on Prometheus in istio-system and deploy sample prometheus-federated / prometheus-kiali-edge instances. Do not use demo/ in production clusters—use the reference files above with your own Prometheus instead.

File	Purpose
`demo/install.sh` / `demo/uninstall.sh`	Demo lab deploy and teardown
`demo/prometheus-federated.yaml`	Sample Federated Prometheus deployment
`demo/prometheus-kiali-edge.yaml`	Sample dedicated Kiali edge Prometheus
`demo/render-*.py`, `demo/merge-recording-rules.py`	Render demo manifests from the production reference YAML

See Demo walkthrough below to try the pattern locally.

Recording rules (edge Prometheus)

Apply core-recording-rules.yml on the Prometheus instance that scrapes Istio traffic (or merge the groups into your existing rule set). The rules drop scrape/infrastructure labels (pod, pod_template_hash, instance, job, node) while preserving Istio semantic labels and the Prometheus scrape namespace label. The same without (...) shape is used in kiali-metrics-recording-rules.yml for consistency.

groups:
- name: istio.workload-aggregation
  interval: 30s
  rules:
  - record: workload:istio_requests_total
    expr: sum without (pod, pod_template_hash, instance, job, node) (istio_requests_total)
  - record: workload:istio_request_duration_milliseconds_bucket
    expr: sum without (pod, pod_template_hash, instance, job, node) (istio_request_duration_milliseconds_bucket)
  # ... remaining traffic counters and histogram components — see core-recording-rules.yml

The recording rules do not apply rate(). Kiali applies rate() at query time with user-selected durations. Use sum without (...) rather than sum by (...) so required labels are not dropped accidentally.

How you install the rules depends on your platform—for example a rule_files entry in prometheus.yml, a ConfigMap volume mount, or a Prometheus Operator PrometheusRule CR in the namespace where edge Prometheus runs.

On OpenShift User Workload Monitoring, a cross-namespace aggregation rule needs a dedicated namespace that is exempt from UWM label enforcement. For the ACM/MCOA pattern used by the multicluster tutorials, use mesh-observability and add it to namespacesWithoutLabelEnforcement in user-workload-monitoring-config on each managed cluster. This is what allows one propagated rule to aggregate metrics scraped in the Istio and application namespaces; it does not remove the need for ServiceMonitor or PodMonitor resources in those namespaces.

Federation (federated Prometheus)

Add a federation scrape job to your existing long-retention federated Prometheus. Use core-federation-match.yml for the complete core-tier match[] list. Federate workload:* traffic metrics from the edge and relabel names before storage:

- job_name: istio-mesh-federate
  honor_labels: true
  metrics_path: /federate
  scrape_interval: 30s
  params:
    match[]:
      - '{__name__=~"workload:istio_requests_total"}'
      - '{__name__=~"workload:istio_request_bytes_(bucket|count|sum)"}'
      - '{__name__=~"workload:istio_request_duration_milliseconds_(bucket|count|sum)"}'
      # ... see core-federation-match.yml for the full core-tier list
  metric_relabel_configs:
  - source_labels: [__name__]
    regex: 'workload:(.*)'
    target_label: __name__
    action: replace
  static_configs:
  - targets:
    - '<edge-prometheus-host>:9090'

Set honor_labels: true on every federation scrape job. Without it, Prometheus renames conflicting labels from the federated edge with an exported_ prefix (for example destination_workload becomes exported_destination_workload). Kiali queries assume standard Istio label names (source_*, destination_*, reporter, and so on); omitting honor_labels can break the traffic graph, health, and metrics tabs even when match[] and relabel rules are correct.

Also federate non-traffic metrics that Kiali needs but does not aggregate (for example istio_build, pilot_xds, container_cpu_usage_seconds_total) directly by name—see core-metrics.yml and core-federation-match.yml.

To include Istio dashboard metrics, append the selectors from istio-dashboard-federation-match.yml to match[].

Configure network access, TLS, and authentication between Federated and Edge Prometheus according to your environment. Kiali authentication for the Federated Prometheus URL is configured separately (see Prometheus authentication configuration).

Demo walkthrough (lab only)

The demo/install.sh script is a learning and CI harness only. It does not replace the production integration above—it automates the same reference YAML against the Istio add-on Prometheus in istio-system so you can validate the pattern locally:

# From a clone of github.com/kiali/kiali, with Istio add-on Prometheus running:
./hack/prometheus/federation/demo/install.sh

# Optional: also federate Perses dashboard metrics
./hack/prometheus/federation/demo/install.sh --with-dashboards

# Optional: federate Kiali self-monitoring (shared Istio edge Prometheus)
./hack/prometheus/federation/demo/install.sh --with-kiali-metrics

# Optional: federate Kiali self-monitoring (dedicated Kiali edge Prometheus)
./hack/prometheus/federation/demo/install.sh --with-kiali-metrics --kiali-edge dedicated

# Optional: point Kiali at the demo federated Prometheus
./hack/prometheus/federation/demo/install.sh --switch-kiali

# Combine flags as needed, for example:
./hack/prometheus/federation/demo/install.sh --with-dashboards --with-kiali-metrics --switch-kiali

After install, port-forward and verify:

# Istio edge Prometheus (recording rules)
kubectl port-forward -n istio-system svc/prometheus 9091:9090

# Demo federated Prometheus (federated data; point Kiali here)
kubectl port-forward -n istio-system svc/prometheus-federated 9092:9090

# Dedicated Kiali edge (only with --kiali-edge dedicated)
kubectl port-forward -n istio-system svc/prometheus-kiali-edge 9093:9090

# Edge: workload:* aggregates
curl -s 'http://localhost:9091/api/v1/query?query=count(workload:istio_requests_total)'

# Production: istio_* and kiali_* (no workload: or kiali: prefix)
curl -s 'http://localhost:9092/api/v1/query?query=count(istio_requests_total)'
curl -s 'http://localhost:9092/api/v1/query?query=count({__name__=~"kiali_.*"})'

The Kiali Internal Metrics dashboard only works when Kiali queries federated Prometheus (--switch-kiali or external_services.prometheus.url → prometheus-federated).

Run demo/uninstall.sh for teardown.

Integration checklist

Use this checklist when integrating the production reference files into your own Prometheus stack (not the demo/ installer):

Identify edge Prometheus—the TSDB that scrapes Istio/Envoy (may be in a monitoring namespace, a remote cluster, or a managed service—not necessarily istio-system).
Merge core-recording-rules.yml onto the edge; set short retention on raw mesh telemetry.
Add a federation scrape job to federated Prometheus using core-federation-match.yml.
Optionally extend match[] with Istio dashboard-tier selectors if Istio dashboards are enabled.
If Kiali self-monitoring is enabled, choose an option from Kiali self-monitoring metrics: apply kiali-metrics-recording-rules.yml on the Kiali edge and federate kiali-metrics-federation-match.yml to Federated Prometheus (Options 1–2), or scrape Kiali directly into Federated Prometheus (Option 3).
Point external_services.prometheus.url at federated Prometheus (and the same URL for Perses/Grafana, if using).
Validate equivalence between edge aggregates and federated data (see below).
Tune intervals using Interval tuning below.

For multi-cluster deployments, apply the same edge → federated pattern per mesh cluster: each cluster’s Edge Prometheus scrapes local Istio/Envoy telemetry, evaluates recording rules, and federates into that cluster’s Federated Prometheus (or into a shared central Federated Prometheus, if your organization consolidates metrics that way). Kiali already supports per-cluster Prometheus URLs in multicluster configuration—set each cluster’s external_services.prometheus.url to the Federated Prometheus instance that holds that cluster’s federated mesh metrics, not the local Edge scraper.

Interval tuning

Several independent intervals affect freshness, CPU use, and the minimum time windows Kiali can use for rate() queries. Set them together—not in isolation.

What is configured	Where configured	Setting	Role
Metric Scrape interval	Edge Prometheus	`global.scrape_interval` (or per-job override on Istio/Envoy targets)	How often raw `istio_*` metrics are scraped
Recording rule interval	Edge Prometheus	`interval` on the rule group in `core-recording-rules.yml`	How often `workload:*` aggregates are recomputed
Federation interval	Federated Prometheus	`scrape_interval` on the federated Prometheus federation job	How often Prometheus pulls `workload:*` (and other federated series) from the edge
Metric retention	Edge Prometheus	`storage.tsdb.retention.time`	How long raw and `workload:*` series are kept before expiry (short, e.g. 6h)
Metric retention	Federated Prometheus	`storage.tsdb.retention.time`	Long-term history Kiali and dashboards query (as desired)

Rules of thumb:

Set all three intervals equal: edge scrape = recording rule = federation scrape. – The recommended default is 30s/30s/30s. This is straightforward, efficient, and correct for Kiali. Evaluating rules faster than the scrape interval wastes CPU (re-sums unchanged data). Evaluating slower means workload:* updates lag behind available raw data. Federation at the same rate ensures each federation scrape captures a freshly evaluated aggregate.
Set scrape_timeout below scrape_interval on the federation job. – (for example 25s timeout with 30s interval) so slow federation scrapes do not overlap.
Kiali minimum duration depends on the federation scrape interval alone. – Each federation scrape produces one data point in the federated TSDB. Prometheus rate() requires at least two data points in the range window, so the minimum useful Kiali duration is 2 × federation_scrape_interval. With 30s federation, that is 60s (1m) — matching Kiali’s smallest offered duration.

Recommended settings — 30s/30s/30s:

This is a common production default (for example kube-prometheus-stack). The Kiali reference bundle uses these values:

Setting	Recommended value	Notes
Edge `scrape_interval`	`30s`	Common production baseline
Recording rule `interval`	`30s`	Matches scrape; one eval per scrape cycle
Federation `scrape_interval`	`30s`	Determines Kiali’s sampling rate
Federation `scrape_timeout`	`25s`	Slightly less than scrape interval
Edge retention	`6h`	Enough for troubleshooting; raw series expire after federation
Minimum Kiali duration	`1m` (`60s`)	`2 × 30s` federation; Kiali’s smallest dropdown value

Example edge rule group header and federated Prometheus federation job:

# Edge Prometheus — recording rules
groups:
- name: istio.workload-aggregation
  interval: 30s          # match 30s scrape_interval
  rules:
  - record: workload:istio_requests_total
    expr: sum without (pod, pod_template_hash, instance, job, node) (istio_requests_total)

# Federated Prometheus — federation job
- job_name: istio-mesh-federate
  scrape_interval: 30s
  scrape_timeout: 25s
  metrics_path: /federate
  # ... match[] and relabel configs

Expected freshness: with 30s/30s/30s, the three intervals run on independent, unsynchronized clocks. Worst-case staleness of the most recent data point is ~90s (three consecutive 30s waits); average staleness is ~45s. This staleness applies to each counter snapshot. Rate accuracy between consecutive federation data points is unaffected — rate() computes the slope between correctly ordered samples regardless of their absolute delay from live traffic.

When to use different intervals:

Edge scrape	Recording rules	Federation	Min Kiali duration	When to use
`15s`	`15s`	`30s`	`60s` (1m)	Edge also serves alerting or direct queries needing fine granularity. Doubles scrape load vs 30s; no Kiali benefit via federation.
`30s` (recommended)	`30s`	`30s`	`60s` (1m)	Default. Efficient, straightforward.
`1m`	`1m`	`1m`	`2m`	Very large clusters where reducing scrape load outweighs freshness.

Istio’s own examples use interval: 5s for recording rules with a 15s scrape. That configuration was designed for their quick-start addon where the same Prometheus serves direct queries; faster rule eval keeps workload:* fresh for local consumers. In a federation architecture where no one queries the edge directly, this benefit disappears and the extra CPU is wasted.

Kiali duration dropdown:

Kiali reads globalScrapeInterval from the Prometheus at external_services.prometheus.url and filters durations to >= 2 x globalScrapeInterval. With federation, Kiali queries the federated Prometheus. If the federated Prometheus global.scrape_interval is set to 30s (matching the federation job interval), Kiali auto-detects the correct minimum duration of 60s (1m) and no additional configuration is needed.

If the federated Prometheus global.scrape_interval differs from the federation job interval (for example global.scrape_interval: 15s but the federation job runs every 30s), Kiali may offer durations shorter than the federation sampling supports. Ensure the federated Prometheus global.scrape_interval matches or exceeds the federation job’s scrape_interval.

Kiali self-monitoring metrics

Kiali can export its own Prometheus metrics (kiali_*) for performance and optional health-status monitoring. These are not Istio mesh metrics—they are not produced on the edge by Envoy, not listed in core-metrics.yml, and not part of the Istio federation tiers.

Because Kiali queries federated Prometheus, kiali_* series must ultimately be available in that same TSDB. The metric rules KEP describes three deployment options:

Option	Kiali metrics scraped by	Before production
1. Shared Istio edge	Same edge Prom as Istio/Envoy	Recording rules + federation (parallel to `workload:*`)
2. Dedicated Kiali edge	Separate edge Prom for Kiali only	Recording rules + federation to prod
3. Direct to Federated	Federated Prom directly	Raw scrape; dedup required in queries for HA

Config	Default	Purpose
`server.observability.metrics.enabled`	`true`	Operational metrics (API, graph, cache, validation, etc.)
`server.observability.metrics.health_status.enabled`	`false`	`kiali_health_status` gauge per entity (opt-in; higher cardinality)

The metrics HTTP listener (port server.observability.metrics.port, default 9090) starts when either flag is true.

kiali_health_status and HA: Multiple Kiali replicas export the same gauge values for the same entities (duplicate series, not partition-of-work). With Options 1 or 2, use max without (pod, instance, …) in recording rules to deduplicate—not sum. With Option 3, apply the same dedup in alert/dashboard queries (for example max by (cluster, namespace, health_type, name) (kiali_health_status)), because raw scrape retains per-replica copies.

The built-in Kiali Internal Metrics custom dashboard queries external_services.prometheus.url and filters metrics with the Prometheus scrape namespace label (for example {namespace="istio-system", app="kiali", ...}). It only works when federated Prometheus holds the kiali_* series (via federation or direct scrape) with the namespace label retained after edge recording rules—see kiali-metrics-recording-rules.yml.

Reference files for Options 1–2: kiali-metrics-recording-rules.yml, kiali-metrics-federation-match.yml, and (for Option 2) demo/prometheus-kiali-edge.yaml. Try them in the demo walkthrough with --with-kiali-metrics and optionally --kiali-edge dedicated.

For kiali_health_status alerting on OpenShift, see the OSSM health status alerts tutorial.

Validation

Confirm that federated series match edge aggregates (on federated Prometheus, after relabel):

# Edge Prometheus
sum(rate(workload:istio_requests_total{destination_workload_namespace="bookinfo"}[5m]))

# Federated Prometheus
sum(rate(istio_requests_total{destination_workload_namespace="bookinfo"}[5m]))

Compare series counts on the edge before federation:

count({__name__="istio_requests_total"})
count({__name__="workload:istio_requests_total"})

The workload:* count should be substantially lower when workloads have been associated with multiple pods (replicas or restarts).

Option 2: Metric Thinning

If the Federation option is not possible and you are limited to a single TSDB instance, this may be helpful.

To reduce the default telemetry to only what is needed by Kiali¹ users can add the following snippet to their Prometheus configuration. Because things can change with different versions, it is recommended to ensure you use the correct version of this documentation based on your Kiali/Istio version.

The metric_relabel_configs: attribute should be added under each job name defined to scrape Istio or Envoy metrics. Below we show it under the kubernetes-pods job, but you should adapt as needed. Be careful of indentation.

    - job_name: kubernetes-pods
      metric_relabel_configs:
      - action: drop
        source_labels: [__name__]
        regex: istio_agent_.*|istiod_.*|istio_build|citadel_.*|galley_.*|pilot_[^psx].*|envoy_cluster_[^u].*|envoy_cluster_update.*|envoy_listener_[^dh].*|envoy_server_[^mu].*|envoy_wasm_.*
      - action: labeldrop
        regex: chart|destination_app|destination_version|heritage|.*operator.*|istio.*|release|security_istio_io_.*|service_istio_io_.*|sidecar_istio_io_inject|source_app|source_version

Applying this configuration should reduce the number of stored metrics by about 20%, as well as reducing the number of attributes stored on many remaining metrics.

Metric Thinning with Disabled Features

The section above drops metrics unused by Kiali. As such, making those configuration changes should not negatively impact Kiali behavior in any way. But some very heavy metrics remain. These metrics can also be dropped, but their removal will impact the behavior of Kiali. This may be OK if you don’t use the affected features of Kiali, or if you are willing to sacrifice the feature for the associated metric savings. In particular, these are “Histogram” metrics. Istio is planning to make some improvements to help users better configure these metrics, but as of this writing they are still defined with fairly inefficient default “buckets”, making the number of associated time-series quite large, and the overhead of maintaining and querying the metrics, intensive. Each histogram actually is comprised of 3 stored metrics. For example, a histogram named xxx would result in the following metrics stored into Prometheus:

xxx_bucket
- The most intensive metric, and is required to calculate percentile values.
xxx_count
- Required to calculate ‘avg’ values.
xxx_sum
- Required to calculate rates over time, and for ‘avg’ values.

When considering whether to thin the Histogram metrics, one of the following three approaches is recommended:

If the relevant Kiali reporting is needed, keep the histogram as-is.
If the relevant Kiali reporting is not needed, or not worth the additional metric overhead, drop the entire histogram.
If the metric chart percentiles are not required, drop only the xxx_bucket metric. This removes the majority of the histogram overhead while keeping rate and average (non-percentile) values in Kiali.

These are the relevant Histogram metrics:

istio_request_bytes

This metric is used to produce the Request Size chart on the metric tabs. It also supports Request Throughput edge labels on the graph.

Appending |istio_request_bytes_.* to the drop regex above would drop all associated metrics and would prevent any request size/throughput reporting in Kiali.
Appending |istio_request_bytes_bucket to the drop regex above, would prevent any request size percentile reporting in the Kiali metric charts.

istio_response_bytes

This metric is used to produce the Response Size chart on the metric tabs. And also supports Response Throughput edge labels on the graph

Appending |istio_response_bytes_.* to the drop regex above would drop all associated metrics and would prevent any response size/throughput reporting in Kiali.
Appending |istio_response_bytes_bucket to the drop regex above would prevent any response size percentile reporting in the Kiali metric charts.

istio_request_duration_milliseconds

This metric is used to produce the Request Duration chart on the metric tabs. It also supports Response Time edge labels on the graph.

Appending |istio_request_duration_milliseconds_.* to the drop regex above would drop all associated metrics and would prevent any request duration/response time reporting in Kiali.
Appending |istio_request_duration_milliseconds_bucket to the drop regex above would prevent any request duration/response time percentile reporting in the Kiali metric charts or graph edge labels.

Scrape Interval

The Prometheus globalScrapeInterval is an important configuration option². The scrape interval can have a significant effect on metrics collection overhead as it takes effort to pull all of those configured metrics and update the relevant time-series. And although it doesn’t affect time-series cardinality, it does affect storage for the data-points, as well as having impact when computing query results (the more data-points, the more processing and aggregation).

Users should think carefully about their configured scrape interval. Note that the Istio addon for prometheus configures it to 15s. This is great for demos but may be too frequent for production scenarios. The prometheus helm charts set a default of 1m, which is more reasonable for most installations, but may not be the desired frequency for any particular setup.

The recommendation for Kiali is to set the longest interval possible, while still providing a useful granularity. The longer the interval the less data points scraped, thus reducing processing, storage, and computational overhead. But the impact on Kiali should be understood. It is important to realize that request rates (or byte rates, message rates, etc) require a minumum of two data points:

rate = (dp2 - dp1) / timePeriod

That means for Kiali to show anything useful in the graph, or anywhere rates are used (many places), the minimum duration must be >= 2 x globalScrapeInterval. Kiali will eliminate invalid Duration options given the globalScrapeInterval.

Kiali does a lot of aggregation and querying over time periods. As such, the number of data points will affect query performance, especially for larger time periods.

For more information, see the Prometheus documentation.

TSDB retention time

The Prometheus tsdbRetentionTime is an important configuration option. It has a significant effect on metrics storage, as Prometheus will keep each reported data-point for that period of time, performing compaction as needed. The larger the retention time, the larger the required storage. Note also that Kiali queries against large time periods, and very large data-sets, may result in poor performance or timeouts.

The recommendation for Kiali is to set the shortest retention time that meets your needs and/or operational limits. In some cases users may want to offload older data to a secondary store. Kiali will eliminate invalid Duration options given the tsdbRetentionTime.

For more information, see the Prometheus documentation.

Some non-essential telemetry remains in order to not over-complicate the configuration change. The remaining telemetry is typically negligible. ↩︎
Note that Prometheus can be configured such that individual scrape points can override the global setting, but Kiali is not currently concerned with this corner case. ↩︎

Docs: MultiCluster on OpenShift

Mon, 01 Jan 0001 00:00:00 +0000

This guide sets up a two-cluster OpenShift environment from scratch where:

The hub cluster runs Red Hat Advanced Cluster Management (ACM) for fleet management and centralized metrics collection (ACM Observability / Thanos)
The spoke cluster is imported into ACM and runs OpenShift Service Mesh 3 (OSSM 3) with Kiali
The spoke mesh has two demo application namespaces: one using Istio ambient mode (via the ZTunnel CR), one using Istio sidecar injection
Kiali queries metrics via the ACM Observatorium API on the hub cluster (mTLS), giving it access to Istio metrics collected and forwarded by ACM from the spoke’s User Workload Monitoring Prometheus

The result is a working Kiali installation that shows traffic graphs, metrics, and mesh topology across both the ambient-mode and sidecar-mode workloads running on the spoke.

The diagram below shows the environment after this guide completes. Badges such as G1:P3.2-3.8 mark which guide section(s) install each component (Guide 1, §§3.2–3.8). Click the diagram to open a full-size SVG in a new tab.

Prerequisites

Before starting, you need:

Hub cluster kubeconfig context named ossm-kiali-hub — every oc command in this guide that targets the hub passes --context=ossm-kiali-hub
Spoke cluster kubeconfig context named ossm-kiali-spoke — every oc command that targets the spoke passes --context=ossm-kiali-spoke

oc CLI installed and both contexts present in ~/.kube/config (or KUBECONFIG). Verify with:

oc --context=ossm-kiali-hub   whoami --show-server
oc --context=ossm-kiali-spoke whoami --show-server

openssl installed locally (for generating Istio CA certificates)
Both clusters must be OpenShift 4.19 or later (required for OSSM 3 and Istio 1.30 as used in this guide)
Both clusters must have access to Red Hat OperatorHub (i.e., connected to the Red Hat operator catalog)
jq available locally (used in verification commands)

Environment Setup

Set these variables in your shell before running any commands. They are referenced throughout this guide.

# Name for the spoke cluster in ACM (must be a valid Kubernetes resource name)
export SPOKE_CLUSTER_NAME="spoke"

# Istio version to install. Must be a version supported by the installed OSSM operator.
# After installing the operator (Phase 3.2), you can list supported versions with:
#   oc --context=ossm-kiali-spoke get crd istios.sailoperator.io \
#     -o jsonpath='{.spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.version.enum}'
# Must be >= 1.30 for ambient cross-cluster traffic routing.
export ISTIO_VERSION="1.30.1"

# meshID - arbitrary identifier for this mesh
export MESH_ID="mesh1"

# SeaweedFS S3 credentials for in-cluster Thanos object storage (ACM Observability)
# These are only used inside the cluster — no external storage account is required
export SEAWEEDFS_ACCESS_KEY="seaweedfs"
export SEAWEEDFS_SECRET_KEY="seaweedfs123"

Verify both kubeconfig contexts are reachable:

oc --context=ossm-kiali-hub   whoami --show-server
oc --context=ossm-kiali-spoke whoami --show-server

Phase 1: ACM on the Hub Cluster

1.1 Install ACM Operator

Detect the latest available ACM channel, then create the open-cluster-management namespace and install the operator via OLM. ACM channels follow the naming pattern release-X.Y (e.g. release-2.17):

ACM_CHANNEL=$(oc --context=ossm-kiali-hub get packagemanifest advanced-cluster-management \
  -n openshift-marketplace \
  -o jsonpath='{.status.channels[*].name}' | \
  tr ' ' '\n' | sort -V | tail -1)
echo "Using ACM channel: ${ACM_CHANNEL}"

oc --context=ossm-kiali-hub create namespace open-cluster-management 2>/dev/null || true

oc --context=ossm-kiali-hub apply -f - <<'EOF'
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: open-cluster-management
  namespace: open-cluster-management
spec:
  targetNamespaces:
  - open-cluster-management
EOF

oc --context=ossm-kiali-hub apply -f - <<EOF
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: acm-operator-subscription
  namespace: open-cluster-management
spec:
  sourceNamespace: openshift-marketplace
  source: redhat-operators
  channel: ${ACM_CHANNEL}
  installPlanApproval: Automatic
  name: advanced-cluster-management
EOF

Wait for the ACM operator to install its CRDs. The multiclusterhubs CRD being Established confirms the operator is running and ready:

until oc --context=ossm-kiali-hub get crd \
  multiclusterhubs.operator.open-cluster-management.io &>/dev/null; do
  echo "Waiting for MCH CRD to appear..."
  sleep 5
done

oc --context=ossm-kiali-hub wait crd/multiclusterhubs.operator.open-cluster-management.io \
  --for=condition=Established \
  --timeout=300s

1.2 Create the MultiClusterHub

Wait for the operator pod to be fully ready before creating the MultiClusterHub. The MCH CR is validated by an admission webhook served by the operator — applying the CR before the webhook endpoint is ready causes an immediate rejection:

until oc --context=ossm-kiali-hub get pods \
  -l name=multiclusterhub-operator \
  -n open-cluster-management \
  --no-headers 2>/dev/null | grep -q .; do
  sleep 5
done

oc --context=ossm-kiali-hub wait pod \
  -l name=multiclusterhub-operator \
  -n open-cluster-management \
  --for=condition=Ready \
  --timeout=300s

oc --context=ossm-kiali-hub apply -f - <<'EOF'
apiVersion: operator.open-cluster-management.io/v1
kind: MultiClusterHub
metadata:
  name: multiclusterhub
  namespace: open-cluster-management
spec: {}
EOF

Wait for ACM to be fully ready. This typically takes 5–10 minutes on a fresh cluster:

echo "Waiting for MultiClusterHub to reach Running status..."
while true; do
  PHASE=$(oc --context=ossm-kiali-hub get mch multiclusterhub \
    -n open-cluster-management \
    -o jsonpath='{.status.phase}' 2>/dev/null)
  if [ "${PHASE}" = "Running" ]; then
    echo "MultiClusterHub is Running"
    break
  fi
  echo "  Current phase: ${PHASE} — waiting..."
  sleep 15
done

1.3 Verify ACM is Ready

oc --context=ossm-kiali-hub get multiclusterhub multiclusterhub -n open-cluster-management \
  -o jsonpath='{.status.phase}{"\n"}'
# Expected: Running

1.4 Enable ACM Observability (MultiClusterObservability)

ACM Observability collects metrics from all managed clusters and stores them in Thanos on the hub. Kiali will query these aggregated metrics via the Observatorium API.

ACM needs an S3-compatible object store as its Thanos backend. This guide deploys SeaweedFS in-cluster so that no external storage account is required.

Create the observability namespace:

oc --context=ossm-kiali-hub create namespace open-cluster-management-observability 2>/dev/null || true

Deploy SeaweedFS as a single-pod in-cluster object store. Its mini mode runs the S3 gateway and storage services in one process and creates the thanos bucket at startup:

oc --context=ossm-kiali-hub apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: seaweedfs
  namespace: open-cluster-management-observability
spec:
  replicas: 1
  selector:
    matchLabels:
      app: seaweedfs
  template:
    metadata:
      labels:
        app: seaweedfs
    spec:
      containers:
      - name: seaweedfs
        image: ghcr.io/chrislusf/seaweedfs:4.47
        args:
        - mini
        - -dir=/data
        - -admin.port=12646
        - -master.telemetry=false
        env:
        - name: AWS_ACCESS_KEY_ID
          value: "${SEAWEEDFS_ACCESS_KEY}"
        - name: AWS_SECRET_ACCESS_KEY
          value: "${SEAWEEDFS_SECRET_KEY}"
        - name: S3_BUCKET
          value: thanos
        ports:
        - containerPort: 8333
          name: s3
        volumeMounts:
        - name: data
          mountPath: /data
        readinessProbe:
          tcpSocket:
            port: 8333
          initialDelaySeconds: 10
          periodSeconds: 5
        livenessProbe:
          tcpSocket:
            port: 8333
          initialDelaySeconds: 10
          periodSeconds: 5
      volumes:
      - name: data
        emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: seaweedfs
  namespace: open-cluster-management-observability
spec:
  ports:
  - port: 8333
    name: s3
    targetPort: 8333
  selector:
    app: seaweedfs
EOF

Wait for SeaweedFS and its pre-created Thanos bucket to be ready:

oc --context=ossm-kiali-hub rollout status deployment/seaweedfs \
  -n open-cluster-management-observability \
  --timeout=120s

Create the Thanos object storage secret pointing at the in-cluster SeaweedFS S3 endpoint:

oc --context=ossm-kiali-hub apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: thanos-object-storage
  namespace: open-cluster-management-observability
type: Opaque
stringData:
  thanos.yaml: |
    type: s3
    config:
      bucket: thanos
      endpoint: seaweedfs.open-cluster-management-observability.svc:8333
      insecure: true
      access_key: ${SEAWEEDFS_ACCESS_KEY}
      secret_key: ${SEAWEEDFS_SECRET_KEY}
EOF

Create the MultiClusterObservability CR on the hub. This deploys the hub-side Thanos and Observatorium services and enables the MCOA control plane. For managed clusters selected by the MCOA add-on placement, MCOA deploys PrometheusAgent collectors that federate platform and—after UWM is enabled separately—user-workload metrics to the hub.

The capabilities block enables both the platform and user-workload metric collection paths that MCOA uses to federate Istio and container metrics. The retention configuration is explicit: retentionInLocal is set to 24h for short-lived local hub storage, while raw, 5-minute, and 1-hour Thanos blocks are retained for 365d in the long-lived aggregated store. Kiali queries that aggregated Thanos data, so its thanos_proxy.retention_period is also set to 365d:

oc --context=ossm-kiali-hub apply -f - <<'EOF'
apiVersion: observability.open-cluster-management.io/v1beta2
kind: MultiClusterObservability
metadata:
  name: observability
spec:
  capabilities:
    platform:
      metrics:
        default:
          enabled: true
    userWorkloads:
      metrics:
        default:
          enabled: true
  observabilityAddonSpec: {}
  storageConfig:
    metricObjectStorage:
      name: thanos-object-storage
      key: thanos.yaml
    alertmanagerStorageSize: 1Gi
    compactStorageSize: 10Gi
    receiveStorageSize: 10Gi
    ruleStorageSize: 1Gi
    storeStorageSize: 10Gi
  advanced:
    retentionConfig:
      retentionInLocal: 24h
      retentionResolution1h: 365d
      retentionResolution5m: 365d
      retentionResolutionRaw: 365d
    alertmanager:
      replicas: 1
      resources:
        requests:
          cpu: 20m
          memory: 64Mi
    compact:
      resources:
        requests:
          cpu: 50m
          memory: 128Mi
    grafana:
      replicas: 1
      resources:
        requests:
          cpu: 20m
          memory: 64Mi
    observatoriumAPI:
      replicas: 1
      resources:
        requests:
          cpu: 20m
          memory: 64Mi
    query:
      replicas: 1
      resources:
        requests:
          cpu: 50m
          memory: 128Mi
    queryFrontend:
      replicas: 1
      resources:
        requests:
          cpu: 50m
          memory: 64Mi
    queryFrontendMemcached:
      replicas: 1
      resources:
        requests:
          cpu: 20m
          memory: 64Mi
    rbacQueryProxy:
      replicas: 1
      resources:
        requests:
          cpu: 20m
          memory: 64Mi
    receive:
      resources:
        requests:
          cpu: 50m
          memory: 128Mi
    rule:
      replicas: 1
      resources:
        requests:
          cpu: 50m
          memory: 128Mi
    store:
      replicas: 1
      resources:
        requests:
          cpu: 50m
          memory: 128Mi
    storeMemcached:
      replicas: 1
      resources:
        requests:
          cpu: 20m
          memory: 64Mi
EOF

Wait for ACM Observability to be ready. This can take 5–10 minutes as Thanos components start up:

echo "Waiting for MultiClusterObservability to be ready..."
while true; do
  READY=$(oc --context=ossm-kiali-hub get mco observability \
    -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true)
  READY=${READY:-Unknown}
  if [ "${READY}" = "True" ]; then
    echo "MultiClusterObservability is Ready"
    break
  fi
  echo "  Ready=${READY} — waiting..."
  sleep 15
done

Verify the Observatorium API route exists:

oc --context=ossm-kiali-hub get route observatorium-api \
  -n open-cluster-management-observability \
  -o jsonpath='{.spec.host}{"\n"}'

1.5 Configure MCOA Federation

How the metrics pipeline works: User Workload Monitoring (UWM) on each spoke is the short-lived edge metrics store. It scrapes raw istio_* series and other user-workload metrics. Recording rules aggregate the selected high-cardinality Istio series into workload:istio_* series. MCOA federates those aggregated series and other core metrics from UWM to hub Thanos. A separate cluster-wide platform federation job collects container CPU and memory metrics from all namespaces. Hub Thanos is the data source Kiali ultimately queries. For the interval relationships, timing expectations, and the consequences of changing an interval, see Scrape Intervals.

Hub Thanos retains its own local data for 24 hours and retains raw, 5-minute, and 1-hour aggregate data for 365 days. Kiali queries hub Thanos through the Observatorium API; its thanos_proxy.retention_period is set to match the hub’s 365-day aggregate retention.

First, identify which MCOA placement to use. A “placement” selects the managed clusters that receive the MCOA add-on configuration. The tutorial uses the placement already referenced by ACM’s multicluster-observability-addon, so the recording rules and federation configuration are propagated only to the clusters selected by that placement.

Read the ClusterManagementAddOn and capture the placement name and namespace:

ADDON_JSON=$(oc --context=ossm-kiali-hub get clustermanagementaddon \
  multicluster-observability-addon -o json)

PLACEMENT_COUNT=$(echo "${ADDON_JSON}" | \
  jq '(.spec.installStrategy.placements // []) | length')

if [ "${PLACEMENT_COUNT}" -eq 1 ]; then
  MCOA_PLACEMENT_NAME=$(echo "${ADDON_JSON}" | \
    jq -r '.spec.installStrategy.placements[0].name')
  MCOA_PLACEMENT_NS=$(echo "${ADDON_JSON}" | \
    jq -r '.spec.installStrategy.placements[0].namespace')
  echo "Using placement: ${MCOA_PLACEMENT_NS}/${MCOA_PLACEMENT_NAME}"
elif [ "${PLACEMENT_COUNT}" -gt 1 ]; then
  echo "Multiple placements found — set MCOA_PLACEMENT_NAME and MCOA_PLACEMENT_NS manually:"
  echo "${ADDON_JSON}" | jq -r \
    '(.spec.installStrategy.placements // [])[] | "  name=\(.name) namespace=\(.namespace)"'
else
  echo "ERROR: no MCOA placements found; ensure MCO is Ready"
fi

Create the shared user-workload ScrapeConfig. This federates aggregated Istio traffic metrics and other core Kiali metrics from UWM. Container CPU and memory are excluded here — they come from platform monitoring and require separate jobs below. The metric relabeling rule renames workload:istio_* series back to istio_* so Kiali queries work unchanged:

oc --context=ossm-kiali-hub apply -f - <<'EOF'
apiVersion: monitoring.rhobs/v1alpha1
kind: ScrapeConfig
metadata:
  labels:
    app.kubernetes.io/component: user-workload-metrics-collector
    app.kubernetes.io/managed-by: kiali-mcoa-federation
  name: kiali-istio-federation
  namespace: open-cluster-management-observability
spec:
  honorLabels: true
  jobName: kiali-istio-federation
  metricRelabelings:
  - action: replace
    regex: 'workload:(.*)'
    replacement: '${1}'
    sourceLabels: [__name__]
    targetLabel: __name__
  - action: replace
    regex: 'kiali:(.*)'
    replacement: '${1}'
    sourceLabels: [__name__]
    targetLabel: __name__
  metricsPath: /federate
  params:
    match[]:
    # CORE TIER
    # Istio traffic (pod-aggregated on edge via recording rules)
    - '{__name__=~"workload:istio_requests_total"}'
    - '{__name__=~"workload:istio_request_bytes_(bucket|count|sum)"}'
    - '{__name__=~"workload:istio_request_duration_milliseconds_(bucket|count|sum)"}'
    - '{__name__=~"workload:istio_request_messages_total"}'
    - '{__name__=~"workload:istio_response_bytes_(bucket|count|sum)"}'
    - '{__name__=~"workload:istio_response_messages_total"}'
    - '{__name__=~"workload:istio_tcp_connections_(opened|closed)_total"}'
    - '{__name__=~"workload:istio_tcp_(received|sent)_bytes_total"}'
    # Istio info (not aggregated)
    - '{__name__=~"istio_build"}'
    # Control plane overview (Kiali mesh page)
    - '{__name__=~"process_cpu_seconds_total"}'
    - '{__name__=~"process_resident_memory_bytes"}'
    - '{__name__=~"pilot_info"}'
    - '{__name__=~"pilot_proxy_convergence_time_(sum|count)"}'
    - '{__name__=~"pilot_services"}'
    - '{__name__=~"pilot_xds$"}'
    - '{__name__=~"pilot_xds_pushes"}'
    - '{__name__=~"workload_manager_active_proxy_count"}'
    # Envoy workload details (all metrics used by Kiali Envoy tab)
    - '{__name__=~"envoy_cluster_upstream_cx_active"}'
    - '{__name__=~"envoy_cluster_upstream_rq_total"}'
    - '{__name__=~"envoy_listener_downstream_cx_active"}'
    - '{__name__=~"envoy_listener_http_downstream_rq"}'
    - '{__name__=~"envoy_server_memory_allocated"}'
    - '{__name__=~"envoy_server_memory_heap_size"}'
    - '{__name__=~"envoy_server_uptime"}'
    # KIALI TIER
    # Kiali self-monitoring metrics
    - '{__name__=~"kiali:kiali_.*"}'
    # ISTIO DASHBOARD TIER
    # istio-control-plane + istio-performance (Istiod runtime)
    - '{__name__=~"pilot_k8s_(reg|cfg)_events"}'
    - '{__name__=~"pilot_push_triggers"}'
    - '{__name__=~"pilot_total_xds_(rejects|internal_errors)"}'
    - '{__name__=~"pilot_xds_push_time_bucket"}'
    - '{__name__=~"pilot_xds_config_size_bytes_bucket"}'
    - '{__name__=~"go_goroutines"}'
    - '{__name__=~"go_memstats_(alloc_bytes|alloc_bytes_total|heap_alloc_bytes|heap_inuse_bytes|heap_sys_bytes|mallocs_total|stack_inuse_bytes)"}'
    # istio-performance (container/process detail)
    - '{__name__=~"container_fs_usage_bytes"}'
    - '{__name__=~"process_open_fds"}'
    - '{__name__=~"process_virtual_memory_bytes"}'
    # istio-ztunnel-dashboard (ambient)
    - '{__name__=~"istio_dns_requests_total"}'
    - '{__name__=~"istio_tcp_sockets_open"}'
    - '{__name__=~"istio_xds_connection_terminations_total"}'
    - '{__name__=~"istio_xds_message_total"}'
    - '{__name__=~"workload_manager_pending_proxy_count"}'
    # istio-extension-dashboard (WASM)
    - '{__name__=~"envoy_wasm_.*"}'
  scrapeInterval: 5m
EOF

Create one cluster-wide platform ScrapeConfig for container CPU and memory. Platform monitoring (not UWM) owns these series, so they require a separate federation job. This job intentionally federates those metrics from all namespaces.

The tutorial creates one platform ScrapeConfig for the cluster. In a production deployment, create the same single cluster-wide resource in the observability namespace used by your Kiali instance.

oc --context=ossm-kiali-hub apply -f - <<'EOF'
apiVersion: monitoring.rhobs/v1alpha1
kind: ScrapeConfig
metadata:
  labels:
    app.kubernetes.io/component: platform-metrics-collector
    app.kubernetes.io/managed-by: kiali-mcoa-federation
  name: kiali-istio-platform-federation
  namespace: open-cluster-management-observability
spec:
  jobName: kiali-istio-platform-federation
  metricsPath: /federate
  params:
    match[]:
    - '{__name__=~"container_cpu_usage_seconds_total|container_memory_working_set_bytes"}'
  scrapeInterval: 5m
EOF

Create one aggregation PrometheusRule in a dedicated namespace. MCOA propagates it into mesh-observability on every managed cluster. Exempt that namespace from UWM label enforcement so the rule can select and record metrics from all scraped application namespaces. If user-workload-monitoring-config already exists, merge this key into its existing config.yaml rather than replacing unrelated UWM settings.

oc --context=ossm-kiali-spoke create namespace mesh-observability --dry-run=client -o yaml | \
  oc --context=ossm-kiali-spoke apply -f -
oc --context=ossm-kiali-spoke apply -f - <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    namespacesWithoutLabelEnforcement:
    - mesh-observability
EOF

The PrometheusRule and ScrapeConfig resources have distinct roles. The PrometheusRule is applied to the spoke’s UWM Prometheus and evaluates sum without (...) recording rules. Those rules reduce many per-pod/per-proxy istio_* series into lower-cardinality workload:istio_* series while preserving the labels Kiali needs. The ScrapeConfig configures MCOA’s PrometheusAgent to fetch those already-aggregated workload:istio_* series from UWM’s /federate endpoint, remove the workload: prefix so they again use standard istio_* names, and remote-write them to hub Thanos. Without the PrometheusRule, MCOA could federate raw per-proxy metrics, but the aggregated hub store would retain much higher-cardinality data. Without the ScrapeConfig, the aggregated recording-rule series would remain only on the spoke and would never reach hub Thanos.

oc --context=ossm-kiali-hub apply -f - <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  annotations:
    observability.open-cluster-management.io/target-namespace: mesh-observability
  labels:
    app.kubernetes.io/component: user-workload-metrics-collector
    app.kubernetes.io/managed-by: kiali-mcoa-federation
    openshift.io/prometheus-rule-evaluation-scope: leaf-prometheus
  name: kiali-istio-aggregation
  namespace: open-cluster-management-observability
spec:
  groups:
  - interval: 30s
    name: istio.workload-aggregation
    rules:
    # CORE TIER
    - expr: sum without (pod, pod_template_hash, instance, job, node) (istio_requests_total)
      record: workload:istio_requests_total
    - expr: sum without (pod, pod_template_hash, instance, job, node) (istio_request_messages_total)
      record: workload:istio_request_messages_total
    - expr: sum without (pod, pod_template_hash, instance, job, node) (istio_response_messages_total)
      record: workload:istio_response_messages_total
    - expr: sum without (pod, pod_template_hash, instance, job, node) (istio_tcp_sent_bytes_total)
      record: workload:istio_tcp_sent_bytes_total
    - expr: sum without (pod, pod_template_hash, instance, job, node) (istio_tcp_received_bytes_total)
      record: workload:istio_tcp_received_bytes_total
    - expr: sum without (pod, pod_template_hash, instance, job, node) (istio_tcp_connections_opened_total)
      record: workload:istio_tcp_connections_opened_total
    - expr: sum without (pod, pod_template_hash, instance, job, node) (istio_tcp_connections_closed_total)
      record: workload:istio_tcp_connections_closed_total
    - expr: sum without (pod, pod_template_hash, instance, job, node) (istio_request_duration_milliseconds_bucket)
      record: workload:istio_request_duration_milliseconds_bucket
    - expr: sum without (pod, pod_template_hash, instance, job, node) (istio_request_duration_milliseconds_sum)
      record: workload:istio_request_duration_milliseconds_sum
    - expr: sum without (pod, pod_template_hash, instance, job, node) (istio_request_duration_milliseconds_count)
      record: workload:istio_request_duration_milliseconds_count
    - expr: sum without (pod, pod_template_hash, instance, job, node) (istio_request_bytes_bucket)
      record: workload:istio_request_bytes_bucket
    - expr: sum without (pod, pod_template_hash, instance, job, node) (istio_request_bytes_sum)
      record: workload:istio_request_bytes_sum
    - expr: sum without (pod, pod_template_hash, instance, job, node) (istio_request_bytes_count)
      record: workload:istio_request_bytes_count
    - expr: sum without (pod, pod_template_hash, instance, job, node) (istio_response_bytes_bucket)
      record: workload:istio_response_bytes_bucket
    - expr: sum without (pod, pod_template_hash, instance, job, node) (istio_response_bytes_sum)
      record: workload:istio_response_bytes_sum
    - expr: sum without (pod, pod_template_hash, instance, job, node) (istio_response_bytes_count)
      record: workload:istio_response_bytes_count
    # KIALI TIER
    # Counters
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_ai_requests_total)
      record: kiali:kiali_ai_requests_total
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_ai_store_evictions_total)
      record: kiali:kiali_ai_store_evictions_total
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_api_failures_total)
      record: kiali:kiali_api_failures_total
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_cache_hits_total)
      record: kiali:kiali_cache_hits_total
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_cache_requests_total)
      record: kiali:kiali_cache_requests_total
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_graph_cache_evictions_total)
      record: kiali:kiali_graph_cache_evictions_total
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_graph_cache_hits_total)
      record: kiali:kiali_graph_cache_hits_total
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_graph_cache_misses_total)
      record: kiali:kiali_graph_cache_misses_total
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_health_cache_hits_total)
      record: kiali:kiali_health_cache_hits_total
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_health_cache_misses_total)
      record: kiali:kiali_health_cache_misses_total
    # Gauges
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_ai_store_conversations_total)
      record: kiali:kiali_ai_store_conversations_total
    - expr: max without (pod, pod_template_hash, instance, job, node) (kiali_graph_nodes)
      record: kiali:kiali_graph_nodes
    - expr: max without (pod, pod_template_hash, instance, job, node) (kiali_kubernetes_clients)
      record: kiali:kiali_kubernetes_clients
    - expr: max without (pod, pod_template_hash, instance, job, node) (kiali_health_status)
      record: kiali:kiali_health_status
    # Histograms
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_ai_request_duration_seconds_bucket)
      record: kiali:kiali_ai_request_duration_seconds_bucket
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_ai_request_duration_seconds_sum)
      record: kiali:kiali_ai_request_duration_seconds_sum
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_ai_request_duration_seconds_count)
      record: kiali:kiali_ai_request_duration_seconds_count
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_api_processing_duration_seconds_bucket)
      record: kiali:kiali_api_processing_duration_seconds_bucket
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_api_processing_duration_seconds_sum)
      record: kiali:kiali_api_processing_duration_seconds_sum
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_api_processing_duration_seconds_count)
      record: kiali:kiali_api_processing_duration_seconds_count
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_checker_processing_duration_seconds_bucket)
      record: kiali:kiali_checker_processing_duration_seconds_bucket
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_checker_processing_duration_seconds_sum)
      record: kiali:kiali_checker_processing_duration_seconds_sum
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_checker_processing_duration_seconds_count)
      record: kiali:kiali_checker_processing_duration_seconds_count
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_graph_appender_duration_seconds_bucket)
      record: kiali:kiali_graph_appender_duration_seconds_bucket
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_graph_appender_duration_seconds_sum)
      record: kiali:kiali_graph_appender_duration_seconds_sum
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_graph_appender_duration_seconds_count)
      record: kiali:kiali_graph_appender_duration_seconds_count
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_graph_generation_duration_seconds_bucket)
      record: kiali:kiali_graph_generation_duration_seconds_bucket
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_graph_generation_duration_seconds_sum)
      record: kiali:kiali_graph_generation_duration_seconds_sum
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_graph_generation_duration_seconds_count)
      record: kiali:kiali_graph_generation_duration_seconds_count
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_graph_marshal_duration_seconds_bucket)
      record: kiali:kiali_graph_marshal_duration_seconds_bucket
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_graph_marshal_duration_seconds_sum)
      record: kiali:kiali_graph_marshal_duration_seconds_sum
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_graph_marshal_duration_seconds_count)
      record: kiali:kiali_graph_marshal_duration_seconds_count
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_health_refresh_duration_seconds_bucket)
      record: kiali:kiali_health_refresh_duration_seconds_bucket
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_health_refresh_duration_seconds_sum)
      record: kiali:kiali_health_refresh_duration_seconds_sum
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_health_refresh_duration_seconds_count)
      record: kiali:kiali_health_refresh_duration_seconds_count
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_prometheus_processing_duration_seconds_bucket)
      record: kiali:kiali_prometheus_processing_duration_seconds_bucket
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_prometheus_processing_duration_seconds_sum)
      record: kiali:kiali_prometheus_processing_duration_seconds_sum
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_prometheus_processing_duration_seconds_count)
      record: kiali:kiali_prometheus_processing_duration_seconds_count
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_single_validation_processing_duration_seconds_bucket)
      record: kiali:kiali_single_validation_processing_duration_seconds_bucket
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_single_validation_processing_duration_seconds_sum)
      record: kiali:kiali_single_validation_processing_duration_seconds_sum
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_single_validation_processing_duration_seconds_count)
      record: kiali:kiali_single_validation_processing_duration_seconds_count
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_tracing_processing_duration_seconds_bucket)
      record: kiali:kiali_tracing_processing_duration_seconds_bucket
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_tracing_processing_duration_seconds_sum)
      record: kiali:kiali_tracing_processing_duration_seconds_sum
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_tracing_processing_duration_seconds_count)
      record: kiali:kiali_tracing_processing_duration_seconds_count
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_validation_processing_duration_seconds_bucket)
      record: kiali:kiali_validation_processing_duration_seconds_bucket
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_validation_processing_duration_seconds_sum)
      record: kiali:kiali_validation_processing_duration_seconds_sum
    - expr: sum without (pod, pod_template_hash, instance, job, node) (kiali_validation_processing_duration_seconds_count)
      record: kiali:kiali_validation_processing_duration_seconds_count
EOF

Register the ScrapeConfig and PrometheusRule resources you just created above with the selected MCOA placement in ACM’s cluster-scoped ClusterManagementAddon named multicluster-observability-addon. Specifically, add them to that placement’s configs list. This instructs MCOA to propagate those resources from the hub to every managed cluster selected by the placement.

The jq-based merge in the code below is idempotent — it skips any reference that is already present:

add_mcoa_ref() {
  local group=$1 resource=$2 name=$3
  local patch exists
  exists=$(oc --context=ossm-kiali-hub get clustermanagementaddon \
    multicluster-observability-addon -o json | jq -r \
    --arg g "${group}" --arg r "${resource}" --arg n "${name}" \
    --arg ns "open-cluster-management-observability" \
    --arg pn "${MCOA_PLACEMENT_NAME}" --arg pns "${MCOA_PLACEMENT_NS}" \
    '[.spec.installStrategy.placements[] |
      select(.name == $pn and .namespace == $pns) |
      .configs[]? |
      select(.group == $g and .resource == $r and .name == $n and .namespace == $ns)] | length')
  if [ "${exists}" -eq 0 ]; then
    local idx
    idx=$(oc --context=ossm-kiali-hub get clustermanagementaddon \
      multicluster-observability-addon -o json | jq -r \
      --arg pn "${MCOA_PLACEMENT_NAME}" --arg pns "${MCOA_PLACEMENT_NS}" \
      '.spec.installStrategy.placements | to_entries[] |
        select(.value.name == $pn and .value.namespace == $pns) | .key')
    local configs
    configs=$(oc --context=ossm-kiali-hub get clustermanagementaddon \
      multicluster-observability-addon -o json | jq -r \
      --argjson i "${idx}" \
      '.spec.installStrategy.placements[$i].configs | type == "array"')
    local op path
    if [ "${configs}" = true ]; then
      op="add"; path="/spec/installStrategy/placements/${idx}/configs/-"
    else
      op="add"; path="/spec/installStrategy/placements/${idx}/configs"
    fi
    oc --context=ossm-kiali-hub patch clustermanagementaddon \
      multicluster-observability-addon --type=json -p="[{
        \"op\": \"${op}\",
        \"path\": \"${path}\",
        \"value\": {\"group\": \"${group}\", \"resource\": \"${resource}\",
                    \"name\": \"${name}\",
                    \"namespace\": \"open-cluster-management-observability\"}
      }]"
    echo "Added ${resource}/${name} to placement ${MCOA_PLACEMENT_NS}/${MCOA_PLACEMENT_NAME}"
  else
    echo "Reference ${resource}/${name} already present — skipping"
  fi
}

# User-workload ScrapeConfig (shared)
add_mcoa_ref monitoring.rhobs scrapeconfigs kiali-istio-federation

# Cluster-wide platform ScrapeConfig
add_mcoa_ref monitoring.rhobs scrapeconfigs kiali-istio-platform-federation

# Shared aggregation PrometheusRule
add_mcoa_ref monitoring.coreos.com prometheusrules kiali-istio-aggregation

The hub-side MCOA configuration resources are now registered with the MCOA placement. To verify, in the ClusterManagementAddon named multicluster-observability-addon, look for the selected placement’s configs list:

oc --context=ossm-kiali-hub get clustermanagementaddon multicluster-observability-addon \
  -o jsonpath-as-json='{.spec.installStrategy.placements[*].configs}'

It should now contain references to the Kiali resources created in this section: the shared kiali-istio-federation ScrapeConfig, the cluster-wide kiali-istio-platform-federation ScrapeConfig, and the single kiali-istio-aggregation PrometheusRule.

Once the spoke is imported, MCOA propagates the PrometheusRule into mesh-observability and the ScrapeConfig objects to the managed-cluster agent. The agent federates metrics from the local UWM.

Phase 2: Import the Spoke Cluster into ACM

2.1 Create the ManagedCluster Resource and Namespace

oc --context=ossm-kiali-hub create namespace "${SPOKE_CLUSTER_NAME}" 2>/dev/null || true

oc --context=ossm-kiali-hub apply -f - <<EOF
apiVersion: cluster.open-cluster-management.io/v1
kind: ManagedCluster
metadata:
  name: ${SPOKE_CLUSTER_NAME}
  labels:
    cloud: auto-detect
    vendor: auto-detect
spec:
  hubAcceptsClient: true
EOF

2.2 Create the Auto-Import Secret

The simplest way to import a spoke is to give ACM the spoke’s kubeconfig directly. ACM’s import controller detects the auto-import-secret and installs the klusterlet agent on the spoke automatically.

Extract the spoke’s kubeconfig context into a standalone file:

oc config view --context=ossm-kiali-spoke --minify --flatten \
  > /tmp/spoke-kubeconfig.yaml

# Verify it connects to the spoke
oc --kubeconfig=/tmp/spoke-kubeconfig.yaml whoami --show-server

Create the auto-import secret on the hub:

oc --context=ossm-kiali-hub create secret generic auto-import-secret \
  -n "${SPOKE_CLUSTER_NAME}" \
  --from-file=kubeconfig=/tmp/spoke-kubeconfig.yaml

ACM consumes and deletes this secret automatically once the klusterlet is installed.

2.3 Create a KlusterletAddonConfig

This enables the standard ACM add-ons on the spoke:

oc --context=ossm-kiali-hub apply -f - <<EOF
apiVersion: agent.open-cluster-management.io/v1
kind: KlusterletAddonConfig
metadata:
  name: ${SPOKE_CLUSTER_NAME}
  namespace: ${SPOKE_CLUSTER_NAME}
spec:
  applicationManager:
    enabled: true
  certPolicyController:
    enabled: true
  policyController:
    enabled: true
  searchCollector:
    enabled: true
EOF

2.4 Wait for the Spoke to Join

echo "Waiting for ${SPOKE_CLUSTER_NAME} to join and become available..."
while true; do
  STATUS=$(oc --context=ossm-kiali-hub get managedcluster "${SPOKE_CLUSTER_NAME}" \
    -o jsonpath='{range .status.conditions[*]}{.type}={.status}{" "}{end}' 2>/dev/null)
  echo "  Status: ${STATUS}"
  if echo "${STATUS}" | grep -q "ManagedClusterJoined=True" && \
     echo "${STATUS}" | grep -q "ManagedClusterConditionAvailable=True"; then
    echo "${SPOKE_CLUSTER_NAME} is joined and available"
    break
  fi
  sleep 15
done

Clean up the temporary kubeconfig:

rm -f /tmp/spoke-kubeconfig.yaml

Verify both clusters are managed:

oc --context=ossm-kiali-hub get managedclusters
# Should show local-cluster and ${SPOKE_CLUSTER_NAME} both with JOINED=True, AVAILABLE=True

Phase 3: OpenShift Service Mesh 3 (Spoke Cluster)

3.1 Enable User Workload Monitoring

Check if UWM is already enabled on the spoke cluster:

oc --context=ossm-kiali-spoke get configmap cluster-monitoring-config \
  -n openshift-monitoring \
  -o jsonpath='{.data.config\.yaml}' 2>/dev/null | \
  grep -q "enableUserWorkload: true" && \
  echo "Already enabled" || echo "Not enabled"

If UWM is not yet enabled, enable it following the instructions below.

This tutorial intentionally does not configure a retention period for the spoke’s UWM Prometheus. OpenShift therefore uses its default 24-hour retention for user-workload metrics. This is appropriate because UWM is the short-lived edge metrics store. MCOA will federate the metrics to hub Thanos, where this tutorial retains the aggregated data for a longer period – 365 days.

If you need a different UWM retention period other than 24 hours, configure it independently on each spoke in the user-workload-monitoring-config ConfigMap (see example yaml below). If you already have this ConfigMap, you will want to merge this prometheus.retention setting with any existing data in your ConfigMap.

apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
      retention: 24h

The following enables UWM safely by merging enableUserWorkload: true without clobbering other monitoring settings.

if oc --context=ossm-kiali-spoke get configmap cluster-monitoring-config \
    -n openshift-monitoring &>/dev/null 2>&1; then
  # ConfigMap already exists — patch only the enableUserWorkload key
  EXISTING=$(oc --context=ossm-kiali-spoke get configmap cluster-monitoring-config \
    -n openshift-monitoring -o jsonpath='{.data.config\.yaml}' 2>/dev/null || true)
  if echo "${EXISTING}" | grep -q "enableUserWorkload:"; then
    oc --context=ossm-kiali-spoke get configmap cluster-monitoring-config \
      -n openshift-monitoring -o json \
      | jq '.data["config.yaml"] |= sub("enableUserWorkload:\\s*\\w+"; "enableUserWorkload: true")' \
      | oc --context=ossm-kiali-spoke apply -f -
  else
    oc --context=ossm-kiali-spoke get configmap cluster-monitoring-config \
      -n openshift-monitoring -o json \
      | jq '.data["config.yaml"] = ((.data["config.yaml"] // "") + "\nenableUserWorkload: true\n")' \
      | oc --context=ossm-kiali-spoke apply -f -
  fi
else
  # ConfigMap does not exist — create it and label it as tutorial-owned
  oc --context=ossm-kiali-spoke create configmap cluster-monitoring-config \
    -n openshift-monitoring \
    --from-literal=config.yaml="enableUserWorkload: true"
  oc --context=ossm-kiali-spoke label configmap cluster-monitoring-config \
    -n openshift-monitoring kiali.io/tutorial-owned=true
fi

Wait for UWM pods to appear and become ready. The pods take a moment to be created after the ConfigMap is applied:

until oc --context=ossm-kiali-spoke get pods \
  -l app.kubernetes.io/name=prometheus \
  -n openshift-user-workload-monitoring \
  --no-headers 2>/dev/null | grep -q .; do
  sleep 5
done

oc --context=ossm-kiali-spoke wait pod \
  --for=condition=Ready \
  -l app.kubernetes.io/name=prometheus \
  -n openshift-user-workload-monitoring \
  --timeout=300s

3.2 Install OpenShift Service Mesh 3 Operator

Install the OSSM 3 operator cluster-wide via OLM. The operator manages Istio control planes across all namespaces:

oc --context=ossm-kiali-spoke apply -f - <<'EOF'
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: openshift-service-mesh-operator
  namespace: openshift-operators
spec:
  channel: stable
  installPlanApproval: Automatic
  name: servicemeshoperator3
  source: redhat-operators
  sourceNamespace: openshift-marketplace
EOF

Wait for the operator pod to appear and become ready:

until oc --context=ossm-kiali-spoke get pods \
  -l app.kubernetes.io/created-by=servicemeshoperator3 \
  -n openshift-operators \
  --no-headers 2>/dev/null | grep -q .; do
  sleep 5
done

oc --context=ossm-kiali-spoke wait pod \
  --for=condition=Ready \
  -l app.kubernetes.io/created-by=servicemeshoperator3 \
  -n openshift-operators \
  --timeout=300s

3.3 Create Required Namespaces

The ztunnel namespace must have the istio-discovery: enabled label so that istiod discovers it and distributes the istio-ca-root-cert ConfigMap there — without which ztunnel pods fail to start:

oc --context=ossm-kiali-spoke create namespace istio-system 2>/dev/null || true
oc --context=ossm-kiali-spoke create namespace istio-cni 2>/dev/null || true
oc --context=ossm-kiali-spoke create namespace ztunnel 2>/dev/null || true

oc --context=ossm-kiali-spoke label namespace ztunnel istio-discovery=enabled

3.4 Generate and Apply Istio CA Certificates

A self-signed Istio root CA is required for mTLS within the mesh:

mkdir -p /tmp/istio-certs && cd /tmp/istio-certs

# Root CA key and certificate
openssl genrsa -out root-key.pem 4096

cat > root-ca.conf <<'CONF'
encrypt_key = no
prompt = no
utf8 = yes
default_md = sha256
default_bits = 4096
req_extensions = req_ext
x509_extensions = req_ext
distinguished_name = req_dn
[ req_ext ]
subjectKeyIdentifier = hash
basicConstraints = critical, CA:true
keyUsage = critical, digitalSignature, nonRepudiation, keyEncipherment, keyCertSign
[ req_dn ]
O = Istio
CN = Root CA
CONF

openssl req -sha256 -new \
  -key root-key.pem \
  -config root-ca.conf \
  -out root-cert.csr

openssl x509 -req -sha256 -days 3650 \
  -signkey root-key.pem \
  -extensions req_ext -extfile root-ca.conf \
  -in root-cert.csr \
  -out root-cert.pem

# Intermediate CA for the spoke
cat > intermediate.conf <<'CONF'
[ req ]
encrypt_key = no
prompt = no
utf8 = yes
default_md = sha256
default_bits = 4096
req_extensions = req_ext
x509_extensions = req_ext
distinguished_name = req_dn
[ req_ext ]
subjectKeyIdentifier = hash
basicConstraints = critical, CA:true, pathlen:0
keyUsage = critical, digitalSignature, nonRepudiation, keyEncipherment, keyCertSign
subjectAltName=@san
[ san ]
DNS.1 = istiod.istio-system.svc
[ req_dn ]
O = Istio
CN = Intermediate CA
L = spoke
CONF

openssl genrsa -out ca-key.pem 4096

openssl req -new \
  -config intermediate.conf \
  -key ca-key.pem \
  -out cluster-ca.csr

openssl x509 -req -sha256 -days 3650 \
  -CA root-cert.pem \
  -CAkey root-key.pem -CAcreateserial \
  -extensions req_ext -extfile intermediate.conf \
  -in cluster-ca.csr \
  -out ca-cert.pem

cat ca-cert.pem root-cert.pem > cert-chain.pem

cd -

Load the CA certificates into the istio-system namespace:

oc --context=ossm-kiali-spoke get secret cacerts -n istio-system &>/dev/null || \
oc --context=ossm-kiali-spoke create secret generic cacerts -n istio-system \
  --from-file=ca-cert.pem=/tmp/istio-certs/ca-cert.pem \
  --from-file=ca-key.pem=/tmp/istio-certs/ca-key.pem \
  --from-file=root-cert.pem=/tmp/istio-certs/root-cert.pem \
  --from-file=cert-chain.pem=/tmp/istio-certs/cert-chain.pem

3.5 Install IstioCNI

IstioCNI is required on OpenShift for both sidecar and ambient modes. It handles pod network setup:

oc --context=ossm-kiali-spoke apply -f - <<EOF
apiVersion: sailoperator.io/v1
kind: IstioCNI
metadata:
  name: default
spec:
  namespace: istio-cni
  profile: openshift-ambient
  version: v${ISTIO_VERSION}
EOF

Wait for IstioCNI to be ready:

oc --context=ossm-kiali-spoke wait istiocni default \
  --for=condition=Ready \
  --timeout=300s

3.6 Install the Istio Control Plane

The Istio CR uses the openshift-ambient profile and discoverySelectors to scope which namespaces Istio manages. The trustedZtunnelNamespace field tells istiod where the ZTunnel CR will deploy ztunnel:

oc --context=ossm-kiali-spoke apply -f - <<EOF
apiVersion: sailoperator.io/v1
kind: Istio
metadata:
  name: default
spec:
  namespace: istio-system
  profile: openshift-ambient
  updateStrategy:
    type: InPlace
  values:
    global:
      meshID: ${MESH_ID}
    meshConfig:
      discoverySelectors:
      - matchLabels:
          istio-discovery: enabled
    pilot:
      trustedZtunnelNamespace: ztunnel
  version: v${ISTIO_VERSION}
EOF

Wait for the Istio control plane to be ready:

oc --context=ossm-kiali-spoke wait istio default \
  --for=condition=Ready \
  --timeout=300s

3.7 Install ZTunnel

ZTunnel is the Ambient mode per-node L4 proxy. The dedicated ZTunnel CR is the recommended way to manage ztunnel in OSSM 3 — it gives you independent lifecycle control over the ztunnel DaemonSet.

oc --context=ossm-kiali-spoke apply -f - <<EOF
apiVersion: sailoperator.io/v1
kind: ZTunnel
metadata:
  name: default
spec:
  namespace: ztunnel
  version: v${ISTIO_VERSION}
EOF

Wait for the ZTunnel DaemonSet to be ready:

oc --context=ossm-kiali-spoke wait ztunnel default \
  --for=condition=Ready \
  --timeout=300s

# Confirm ztunnel pods are running on all nodes
oc --context=ossm-kiali-spoke get pods -n ztunnel -l app=ztunnel

3.8 Configure Istio Metrics Collection

MCOA deploys the managed-cluster Prometheus operator it needs on each selected spoke. A separate COO subscription is not required for metric federation; Guide 3 installs the full COO product only when its Perses dashboards are needed.

Ensure the aggregation namespace exists and is exempt from UWM label enforcement. The other namespaces below are created for their mesh monitors, not as MCOA rule targets:

for NS in mesh-observability istio-system ztunnel ambient-demo bookinfo; do
  oc --context=ossm-kiali-spoke create namespace "${NS}" --dry-run=client -o yaml | \
    oc --context=ossm-kiali-spoke apply -f -
done

Create the ServiceMonitors and PodMonitors that tell UWM Prometheus what to scrape.

ServiceMonitor for istiod (control plane metrics):

oc --context=ossm-kiali-spoke apply -f - <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: istiod-monitor
  namespace: istio-system
spec:
  targetLabels:
  - app
  selector:
    matchLabels:
      istio: pilot
  endpoints:
  - port: http-monitoring
    interval: 30s
EOF

PodMonitor for ztunnel (L4 TCP metrics for ambient-mode traffic):

oc --context=ossm-kiali-spoke apply -f - <<EOF
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: ztunnel-monitor
  namespace: ztunnel
spec:
  selector:
    matchExpressions:
    - key: istio-prometheus-ignore
      operator: DoesNotExist
  podMetricsEndpoints:
  - path: /stats/prometheus
    interval: 30s
    relabelings:
    - action: keep
      sourceLabels: ["__meta_kubernetes_pod_container_name"]
      regex: "istio-proxy"
    - action: keep
      sourceLabels: ["__meta_kubernetes_pod_annotationpresent_prometheus_io_scrape"]
    - action: replace
      regex: (\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})
      replacement: '[\$2]:\$1'
      sourceLabels: ["__meta_kubernetes_pod_annotation_prometheus_io_port","__meta_kubernetes_pod_ip"]
      targetLabel: "__address__"
    - action: replace
      regex: (\d+);((([0-9]+?)(\.|$)){4})
      replacement: '\$2:\$1'
      sourceLabels: ["__meta_kubernetes_pod_annotation_prometheus_io_port","__meta_kubernetes_pod_ip"]
      targetLabel: "__address__"
    - sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_name","__meta_kubernetes_pod_label_app"]
      separator: ";"
      targetLabel: "app"
      action: replace
      regex: "(.+);.*|.*;(.+)"
      replacement: "\${1}\${2}"
    - sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_version","__meta_kubernetes_pod_label_version"]
      separator: ";"
      targetLabel: "version"
      action: replace
      regex: "(.+);.*|.*;(.+)"
      replacement: "\${1}\${2}"
    - action: replace
      regex: "(.+)"
      replacement: "\${1}"
      sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
      targetLabel: "app_kubernetes_io_name"
    - action: replace
      regex: "(.+)"
      replacement: "\${1}"
      sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_version"]
      targetLabel: "app_kubernetes_io_version"
    - sourceLabels: ["__meta_kubernetes_namespace"]
      action: replace
      targetLabel: namespace
    - action: replace
      replacement: "${MESH_ID}"
      targetLabel: mesh_id
EOF

Phase 4: Kiali — Metrics Certs (Hub) then Install (Spoke)

Kiali on the spoke queries metrics from the hub’s ACM Observatorium API using mTLS. This phase has two parts: first extract the necessary certificates from the hub, then install and configure Kiali on the spoke.

4.1 Extract ACM Observatorium Certificates (Hub Cluster)

Get the Observatorium API URL — you will need this for the Kiali CR:

export OBSERVATORIUM_URL=$(oc --context=ossm-kiali-hub get route observatorium-api \
  -n open-cluster-management-observability \
  -o jsonpath='https://{.spec.host}/api/metrics/v1/default')
echo "Observatorium URL: ${OBSERVATORIUM_URL}"

Extract the client certificate and key. ACM automatically creates long-lived (1 year) client certificates in the observability-grafana-certs secret:

oc --context=ossm-kiali-hub get secret observability-grafana-certs \
  -n open-cluster-management-observability \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/obs-tls.crt

oc --context=ossm-kiali-hub get secret observability-grafana-certs \
  -n open-cluster-management-observability \
  -o jsonpath='{.data.tls\.key}' | base64 -d > /tmp/obs-tls.key

Identify which CA signed the Observatorium API’s server certificate, then extract it:

HOST=$(oc --context=ossm-kiali-hub get route observatorium-api \
  -n open-cluster-management-observability \
  -o jsonpath='{.spec.host}')

echo | openssl s_client \
  -connect "${HOST}:443" \
  -servername "${HOST}" \
  -showcerts 2>/dev/null | openssl x509 -noout -issuer

If the issuer CN is observability-server-ca-certificate:

oc --context=ossm-kiali-hub get secret observability-server-ca-certs \
  -n open-cluster-management-observability \
  -o jsonpath='{.data.ca\.crt}' | base64 -d > /tmp/obs-server-ca.crt

If the issuer CN is observability-client-ca-certificate:

oc --context=ossm-kiali-hub get secret observability-client-ca-certs \
  -n open-cluster-management-observability \
  -o jsonpath='{.data.ca\.crt}' | base64 -d > /tmp/obs-server-ca.crt

4.2 Create Cert Resources on the Spoke

Load the extracted certificates into the istio-system namespace where Kiali runs on the spoke:

Create the mTLS client certificate secret:

oc --context=ossm-kiali-spoke create secret generic acm-observability-certs \
  -n istio-system \
  --from-file=tls.crt=/tmp/obs-tls.crt \
  --from-file=tls.key=/tmp/obs-tls.key

Create the CA bundle ConfigMap so Kiali trusts the Observatorium API’s server certificate:

oc --context=ossm-kiali-spoke create configmap kiali-cabundle \
  -n istio-system \
  --from-file=additional-ca-bundle.pem=/tmp/obs-server-ca.crt

4.3 Install the Kiali Operator (Spoke)

oc --context=ossm-kiali-spoke apply -f - <<'EOF'
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: kiali-ossm
  namespace: openshift-operators
spec:
  channel: stable
  installPlanApproval: Automatic
  name: kiali-ossm
  source: redhat-operators
  sourceNamespace: openshift-marketplace
EOF

Wait for the Kiali operator pod to appear and become ready:

until oc --context=ossm-kiali-spoke get pods \
  -l app.kubernetes.io/name=kiali-operator \
  -n openshift-operators \
  --no-headers 2>/dev/null | grep -q .; do
  sleep 5
done

oc --context=ossm-kiali-spoke wait pod \
  --for=condition=Ready \
  -l app.kubernetes.io/name=kiali-operator \
  -n openshift-operators \
  --timeout=300s

4.4 Install Kiali

Kiali is deployed in istio-system and queries metrics from the hub’s Observatorium API using the mTLS certificates created above. The openshift auth strategy authenticates users with the home cluster’s OpenShift OAuth. This tutorial explicitly disables Kiali impersonation for compatibility with Kiali releases that do not support it. If your Kiali version supports impersonation and you want users to authenticate only once, enable it on both Kiali CRs.

The scrape_interval: "5m" matches the default ACM metrics collection interval. The retention_period: "365d" matches the long-lived aggregate retention configured in the MCO CR above:

oc --context=ossm-kiali-spoke apply -f - <<EOF
apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
  name: kiali
  namespace: istio-system
spec:
  auth:
    strategy: openshift
    openshift:
      impersonation:
        enabled: false
  deployment:
    cluster_wide_access: true
    instance_name: kiali
    logger:
      log_level: debug
    namespace: istio-system
    replicas: 1
  external_services:
    grafana:
      enabled: false
    prometheus:
      auth:
        cert_file: secret:acm-observability-certs:tls.crt
        key_file: secret:acm-observability-certs:tls.key
        type: none
        use_kiali_token: false
      thanos_proxy:
        enabled: true
        retention_period: "365d"
        scrape_interval: "5m"
      url: "${OBSERVATORIUM_URL}"
  version: default
EOF

Wait for the Kiali CR to reconcile successfully:

oc --context=ossm-kiali-spoke wait kiali kiali \
  -n istio-system \
  --for=condition=Successful \
  --timeout=300s

Wait for the Kiali deployment to roll out:

oc --context=ossm-kiali-spoke rollout status deployment/kiali -n istio-system

4.5 Install the OpenShift Service Mesh Console Plugin

The OSSMConsole CR instructs the Kiali Operator to register a console plugin that adds the Service Mesh menu to the OpenShift console, providing an integrated Kiali view within the OCP UI:

oc --context=ossm-kiali-spoke apply -f - <<'EOF'
apiVersion: kiali.io/v1alpha1
kind: OSSMConsole
metadata:
  name: ossmconsole
  namespace: istio-system
spec: {}
EOF

until oc --context=ossm-kiali-spoke get ossmconsole ossmconsole \
  -n istio-system \
  -o jsonpath='{.status.conditions[?(@.type=="Successful")].status}' 2>/dev/null | grep -q "True"; do
  echo "Waiting for OSSMConsole reconciliation..."
  sleep 10
done
echo "OSSMConsole ready"

Phase 5: Demo Applications (Spoke Cluster)

Two demo namespaces are created: one in ambient mode (ztunnel handles L4), one with sidecar injection (envoy proxy per pod). Both namespaces are labeled istio-discovery: enabled so that istiod’s discoverySelectors includes them.

5.1 Ambient Demo App — Helloworld

This namespace uses ambient mode. Ztunnel provides L4 TCP mTLS automatically — no sidecar containers are injected. The demo deploys the standard Istio helloworld application in two versions (v1 and v2), which allows Kiali to show version-differentiated traffic distribution in the topology graph.

Create the ambient-demo namespace with the ambient mode and discovery labels:

oc --context=ossm-kiali-spoke apply -f - <<'EOF'
apiVersion: v1
kind: Namespace
metadata:
  name: ambient-demo
  labels:
    istio.io/dataplane-mode: ambient
    istio-discovery: enabled
EOF

Deploy the helloworld service and both versions using the OSSM sample manifests:

ISTIO_MINOR=$(echo "${ISTIO_VERSION}" | cut -d. -f1-2)

# Download the helloworld manifest once to avoid GitHub rate limits on repeated requests
curl -sL "https://raw.githubusercontent.com/openshift-service-mesh/istio/release-${ISTIO_MINOR}/samples/helloworld/helloworld.yaml" \
  -o /tmp/helloworld.yaml

oc --context=ossm-kiali-spoke apply -n ambient-demo -l service=helloworld -f /tmp/helloworld.yaml
oc --context=ossm-kiali-spoke apply -n ambient-demo -l version=v1 -f /tmp/helloworld.yaml
oc --context=ossm-kiali-spoke apply -n ambient-demo -l version=v2 -f /tmp/helloworld.yaml
rm -f /tmp/helloworld.yaml

Wait for both versions to be ready:

oc --context=ossm-kiali-spoke wait deployment/helloworld-v1 \
  -n ambient-demo --for=condition=Available --timeout=120s
oc --context=ossm-kiali-spoke wait deployment/helloworld-v2 \
  -n ambient-demo --for=condition=Available --timeout=120s

Deploy a traffic generator that continuously calls the helloworld service. Requests are round-robined between v1 and v2 by kube-proxy, which Kiali will show as weighted traffic edges to each version:

oc --context=ossm-kiali-spoke apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: traffic-gen
  namespace: ambient-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: traffic-gen
  template:
    metadata:
      labels:
        app: traffic-gen
    spec:
      containers:
      - name: client
        image: registry.access.redhat.com/ubi9/ubi-minimal:latest
        command: ["/bin/sh", "-c"]
        args:
        - |
          while true; do
            curl -s --max-time 5 http://helloworld:5000/hello || echo "failed"
            sleep 2
          done
EOF

Verify all pods are running. No pod should have more than one container — ambient mode adds no sidecars:

oc --context=ossm-kiali-spoke get pods -n ambient-demo
# Each pod should show 1/1 READY (no istio-proxy sidecar)

5.1.1 Deploy a Waypoint for L7 Metrics

Without a waypoint, ztunnel only processes L4 traffic. Kiali will show traffic edges and TCP-level metrics (istio_tcp_*) but no HTTP details — no response codes, no latency, no request rates. A waypoint proxy is an Envoy-based L7 proxy that intercepts traffic inside the ambient mesh and produces the full set of HTTP metrics Kiali needs for its traffic graph and workload dashboards.

Deploy a waypoint for the ambient-demo namespace. The istio.io/waypoint-for: service label tells ztunnel that this waypoint handles traffic addressed to Kubernetes Services (not pod IPs). Valid values are service, workload, all, and none — service is the default and the right choice here since traffic-gen calls helloworld via its Service VIP:

oc --context=ossm-kiali-spoke apply -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: waypoint
  namespace: ambient-demo
  labels:
    istio.io/waypoint-for: service
spec:
  gatewayClassName: istio-waypoint
  listeners:
  - name: mesh
    port: 15008
    protocol: HBONE
EOF

Wait for the waypoint to be programmed, then enroll the namespace — this label tells ztunnel to redirect traffic to services in ambient-demo through the waypoint for L7 processing:

oc --context=ossm-kiali-spoke wait gateway/waypoint \
  -n ambient-demo \
  --for=condition=Programmed=True \
  --timeout=120s

oc --context=ossm-kiali-spoke label namespace ambient-demo \
  istio.io/use-waypoint=waypoint

Verify the Gateway and namespace are configured correctly:

# Confirm the Gateway has istio.io/waypoint-for: service
oc --context=ossm-kiali-spoke get gateway waypoint -n ambient-demo \
  -o jsonpath='waypoint-for={.metadata.labels.istio\.io/waypoint-for}{"\n"}'
# Expected: waypoint-for=service

# Confirm the namespace is enrolled
oc --context=ossm-kiali-spoke get namespace ambient-demo \
  -o jsonpath='use-waypoint={.metadata.labels.istio\.io/use-waypoint}{"\n"}'
# Expected: use-waypoint=waypoint

# Confirm the waypoint pod is running
oc --context=ossm-kiali-spoke get pods -n ambient-demo \
  -l gateway.networking.k8s.io/gateway-name=waypoint

Double edges in Kiali: Once the waypoint is active, Kiali will show two edges between workloads in ambient-demo — one from ztunnel (TCP/L4 metrics) and one from the waypoint (HTTP/L7 metrics). This is expected. Use the Traffic menu in the Kiali graph toolbar and select Waypoint to filter to L7-only edges, or select ZTunnel to see L4-only edges.

Create a PodMonitor so UWM Prometheus scrapes the waypoint’s Envoy metrics:

oc --context=ossm-kiali-spoke apply -f - <<EOF
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: istio-proxies-monitor
  namespace: ambient-demo
spec:
  selector:
    matchExpressions:
    - key: istio-prometheus-ignore
      operator: DoesNotExist
  podMetricsEndpoints:
  - path: /stats/prometheus
    interval: 30s
    relabelings:
    - action: keep
      sourceLabels: ["__meta_kubernetes_pod_container_name"]
      regex: "istio-proxy"
    - action: keep
      sourceLabels: ["__meta_kubernetes_pod_annotationpresent_prometheus_io_scrape"]
    - action: replace
      regex: (\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})
      replacement: '[\$2]:\$1'
      sourceLabels: ["__meta_kubernetes_pod_annotation_prometheus_io_port","__meta_kubernetes_pod_ip"]
      targetLabel: "__address__"
    - action: replace
      regex: (\d+);((([0-9]+?)(\.|$)){4})
      replacement: '\$2:\$1'
      sourceLabels: ["__meta_kubernetes_pod_annotation_prometheus_io_port","__meta_kubernetes_pod_ip"]
      targetLabel: "__address__"
    - sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_name","__meta_kubernetes_pod_label_app"]
      separator: ";"
      targetLabel: "app"
      action: replace
      regex: "(.+);.*|.*;(.+)"
      replacement: "\${1}\${2}"
    - sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_version","__meta_kubernetes_pod_label_version"]
      separator: ";"
      targetLabel: "version"
      action: replace
      regex: "(.+);.*|.*;(.+)"
      replacement: "\${1}\${2}"
    - action: replace
      regex: "(.+)"
      replacement: "\${1}"
      sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
      targetLabel: "app_kubernetes_io_name"
    - action: replace
      regex: "(.+)"
      replacement: "\${1}"
      sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_version"]
      targetLabel: "app_kubernetes_io_version"
    - sourceLabels: ["__meta_kubernetes_namespace"]
      action: replace
      targetLabel: namespace
    - action: replace
      replacement: "${MESH_ID}"
      targetLabel: mesh_id
EOF

Generate traffic, then follow the timing guidance in Scrape Intervals before querying hub Thanos for reporter=waypoint:

oc --context=ossm-kiali-hub get --raw \
  "/api/v1/namespaces/open-cluster-management-observability/services/http:observability-thanos-query-frontend:9090/proxy/api/v1/query?query=istio_requests_total%7Breporter%3D%22waypoint%22%7D" \
  | jq '.data.result | length'
# Returns the count of waypoint reporter timeseries — must be > 0

5.2 Sidecar Demo App — Bookinfo

The Bookinfo application is the standard Istio demo app. It consists of four microservices (productpage, details, ratings, reviews) connected via Envoy sidecar proxies, producing rich L7 HTTP metrics that Kiali uses for its traffic graph and workload views.

Create the bookinfo namespace and label it for sidecar injection and Istio discovery:

oc --context=ossm-kiali-spoke create namespace bookinfo 2>/dev/null || true

oc --context=ossm-kiali-spoke label namespace bookinfo \
  istio-injection=enabled \
  istio-discovery=enabled

Deploy the Bookinfo application using the OSSM-maintained sample manifests. The Istio version in the URL should match your installed Istio version:

ISTIO_MINOR=$(echo "${ISTIO_VERSION}" | cut -d. -f1-2)

oc --context=ossm-kiali-spoke apply -n bookinfo \
  -f "https://raw.githubusercontent.com/openshift-service-mesh/istio/release-${ISTIO_MINOR}/samples/bookinfo/platform/kube/bookinfo.yaml"

Wait for all Bookinfo pods to be ready. Each pod should show 2/2 containers (app + istio-proxy sidecar):

oc --context=ossm-kiali-spoke wait pods \
  --for=condition=Ready \
  --all \
  -n bookinfo \
  --timeout=180s

oc --context=ossm-kiali-spoke get pods -n bookinfo

Verify the app is reachable inside the cluster:

oc --context=ossm-kiali-spoke exec \
  "$(oc --context=ossm-kiali-spoke get pod -l app=ratings -n bookinfo \
     -o jsonpath='{.items[0].metadata.name}')" \
  -c ratings -n bookinfo -- \
  curl -sS productpage:9080/productpage | grep -o "<title>.*</title>"
# Expected: <title>Simple Bookstore App</title>

Expose the Bookinfo productpage externally using Gateway API:

oc --context=ossm-kiali-spoke apply -n bookinfo \
  -f "https://raw.githubusercontent.com/openshift-service-mesh/istio/release-${ISTIO_MINOR}/samples/bookinfo/gateway-api/bookinfo-gateway.yaml"

oc --context=ossm-kiali-spoke wait \
  --for=condition=Programmed \
  gateway/bookinfo-gateway \
  -n bookinfo \
  --timeout=120s

export BOOKINFO_URL="http://$(oc --context=ossm-kiali-spoke get gateway bookinfo-gateway \
  -n bookinfo \
  -o jsonpath='{.status.addresses[0].value}'):$(oc --context=ossm-kiali-spoke get gateway bookinfo-gateway \
  -n bookinfo \
  -o jsonpath='{.spec.listeners[?(@.name=="http")].port}')/productpage"
echo "Bookinfo URL: ${BOOKINFO_URL}"

Open ${BOOKINFO_URL} in a browser to verify the app. To generate continuous traffic for Kiali’s graph without manual browser interaction, deploy a traffic generator:

oc --context=ossm-kiali-spoke apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: traffic-gen
  namespace: bookinfo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: traffic-gen
  template:
    metadata:
      labels:
        app: traffic-gen
    spec:
      containers:
      - name: client
        image: registry.access.redhat.com/ubi9/ubi-minimal:latest
        command: ["/bin/sh", "-c"]
        args:
        - |
          while true; do
            curl -s -o /dev/null -w "%{http_code}\n" --max-time 5 http://productpage:9080/productpage || echo "failed"
            sleep 2
          done
EOF

5.3 PodMonitor for the Bookinfo Namespace

Sidecar (Envoy proxy) metrics must be scraped by UWM Prometheus. OpenShift UWM does not support namespaceSelector in PodMonitors, so a PodMonitor must be created in each namespace that has sidecar-injected pods:

oc --context=ossm-kiali-spoke apply -f - <<EOF
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: istio-proxies-monitor
  namespace: bookinfo
spec:
  selector:
    matchExpressions:
    - key: istio-prometheus-ignore
      operator: DoesNotExist
  podMetricsEndpoints:
  - path: /stats/prometheus
    interval: 30s
    relabelings:
    - action: keep
      sourceLabels: ["__meta_kubernetes_pod_container_name"]
      regex: "istio-proxy"
    - action: keep
      sourceLabels: ["__meta_kubernetes_pod_annotationpresent_prometheus_io_scrape"]
    - action: replace
      regex: (\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})
      replacement: '[\$2]:\$1'
      sourceLabels: ["__meta_kubernetes_pod_annotation_prometheus_io_port","__meta_kubernetes_pod_ip"]
      targetLabel: "__address__"
    - action: replace
      regex: (\d+);((([0-9]+?)(\.|$)){4})
      replacement: '\$2:\$1'
      sourceLabels: ["__meta_kubernetes_pod_annotation_prometheus_io_port","__meta_kubernetes_pod_ip"]
      targetLabel: "__address__"
    - sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_name","__meta_kubernetes_pod_label_app"]
      separator: ";"
      targetLabel: "app"
      action: replace
      regex: "(.+);.*|.*;(.+)"
      replacement: "\${1}\${2}"
    - sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_version","__meta_kubernetes_pod_label_version"]
      separator: ";"
      targetLabel: "version"
      action: replace
      regex: "(.+);.*|.*;(.+)"
      replacement: "\${1}\${2}"
    - action: replace
      regex: "(.+)"
      replacement: "\${1}"
      sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
      targetLabel: "app_kubernetes_io_name"
    - action: replace
      regex: "(.+)"
      replacement: "\${1}"
      sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_version"]
      targetLabel: "app_kubernetes_io_version"
    - sourceLabels: ["__meta_kubernetes_namespace"]
      action: replace
      targetLabel: namespace
    - action: replace
      replacement: "${MESH_ID}"
      targetLabel: mesh_id
EOF

Phase 6: Verification

Before running the metrics checks (6.3) and checking the Kiali traffic graph (6.4): Review Scrape Intervals for the expected federation delay, warm-up time, and query-window requirements. The mesh health checks (6.1) and traffic flow checks (6.2) can be run immediately.

6.1 Verify Mesh Components

Check that all Istio and ztunnel components are healthy on the spoke:

oc --context=ossm-kiali-spoke get istio default
# Expect READY=1, IN USE=1, and STATUS=Healthy.

oc --context=ossm-kiali-spoke get istiocni default
# Expect READY=True and STATUS=Healthy.

oc --context=ossm-kiali-spoke get ztunnel default
# Expect READY=True and STATUS=Healthy.

oc --context=ossm-kiali-spoke get pods -n istio-system
# The istiod pod should be 1/1 Ready and Running.

oc --context=ossm-kiali-spoke get pods -n istio-cni
# Each istio-cni-node DaemonSet pod should be 1/1 Ready and Running.

oc --context=ossm-kiali-spoke get pods -n ztunnel
# Each ztunnel DaemonSet pod should be 1/1 Ready and Running.

6.2 Verify Traffic is Flowing

Check that traffic-gen pods are successfully sending requests:

# Ambient demo (expect "Hello version: v1" or "Hello version: v2" responses)
oc --context=ossm-kiali-spoke logs -n ambient-demo deployment/traffic-gen --tail=5

# Sidecar demo (expect HTTP 200 status codes)
oc --context=ossm-kiali-spoke logs -n bookinfo deployment/traffic-gen --tail=5

6.3 Verify Istio Metrics from Edge UWM to Hub Thanos

The metrics pipeline has two hops (spoke UWM → MCOA federation → hub Thanos). Query all three stages to isolate scrape, recording-rule, and federation failures:

# Locate the spoke's user-workload Prometheus pod
PROM_POD=$(oc --context=ossm-kiali-spoke \
  -n openshift-user-workload-monitoring \
  get pods -l app.kubernetes.io/name=prometheus \
  -o jsonpath='{.items[0].metadata.name}')

# Edge UWM: raw counters scraped from Istio proxies
oc --context=ossm-kiali-spoke \
  -n openshift-user-workload-monitoring \
  exec -c prometheus "${PROM_POD}" -- \
  wget -qO- \
  'http://localhost:9090/api/v1/query?query=istio_requests_total' \
  | jq '.data.result'

# Edge UWM: aggregates produced by the recording rules
oc --context=ossm-kiali-spoke \
  -n openshift-user-workload-monitoring \
  exec -c prometheus "${PROM_POD}" -- \
  wget -qO- \
  'http://localhost:9090/api/v1/query?query=workload%3Aistio_requests_total' \
  | jq '.data.result'

# Hub Thanos: federated aggregates, relabeled back to istio_requests_total
oc --context=ossm-kiali-hub get --raw \
  "/api/v1/namespaces/open-cluster-management-observability/services/http:observability-thanos-query-frontend:9090/proxy/api/v1/query?query=istio_requests_total%7Bcluster%3D%22${SPOKE_CLUSTER_NAME}%22%7D" \
  | jq '.data.result'

# List all Istio metric names present in hub Thanos
oc --context=ossm-kiali-hub get --raw \
  "/api/v1/namespaces/open-cluster-management-observability/services/http:observability-thanos-query-frontend:9090/proxy/api/v1/label/__name__/values" \
  | jq -r '.data[]' | grep "^istio_"

Generate traffic before running the queries. The edge queries should become non-empty after UWM scrapes and evaluates its rules. Hub values follow the MCOA federation cadence, so review Scrape Intervals for the expected delay and rate-query timing. Compare presence and approximately corresponding counter values rather than expecting exact point-in-time equality.

List the Istio and Envoy metric names in edge UWM. This output includes the workload:* series produced by the recording rules:

oc --context=ossm-kiali-spoke \
  -n openshift-user-workload-monitoring \
  exec -c prometheus "${PROM_POD}" -- \
  wget -qO- \
  'http://localhost:9090/api/v1/label/__name__/values' \
  | jq -r '
  [.data[] | select(test("istio|envoy"; "i"))] | unique
  | .[] ,
  "===\nTOTAL COUNT OF istio_ AND envoy_ METRICS: \(length)"'

Now list the metrics stored in hub Thanos. The result does not contain workload:* names because federation relabels those series, and it should have far fewer metric names than the edge:

oc --context=ossm-kiali-hub get --raw \
  "/api/v1/namespaces/open-cluster-management-observability/services/http:observability-thanos-query-frontend:9090/proxy/api/v1/label/__name__/values" \
  | jq -r '
  [.data[] | select(test("istio|envoy"; "i"))] | unique
  | .[] ,
  "===\nTOTAL COUNT OF istio_ AND envoy_ METRICS: \(length)"'

You should see istio_tcp_sent_bytes_total, istio_tcp_connections_opened_total (from ztunnel for the ambient namespace) and istio_requests_total (from sidecar proxies for the sidecar namespace).

If no Istio metrics appear after 15 minutes, verify the MCOA federation pipeline:

# Confirm MCOA capabilities are enabled on the MCO
oc --context=ossm-kiali-hub get mco observability \
  -o jsonpath='{.spec.capabilities}' | jq .

# Confirm hub-side ScrapeConfigs exist
oc --context=ossm-kiali-hub get scrapeconfig \
  -n open-cluster-management-observability | grep kiali

# Confirm hub-side PrometheusRules exist
oc --context=ossm-kiali-hub get prometheusrule \
  -n open-cluster-management-observability | grep kiali

# Confirm the MCOA add-on is Available on the spoke
oc --context=ossm-kiali-hub get managedclusteraddon \
  multicluster-observability-addon -n "${SPOKE_CLUSTER_NAME}"

# Confirm the MCOA PrometheusAgent exists on the spoke
oc --context=ossm-kiali-spoke get prometheusagent \
  -n open-cluster-management-agent-addon

# Confirm the shared aggregation PrometheusRule propagated into the exempt namespace
oc --context=ossm-kiali-spoke get prometheusrule \
  kiali-istio-aggregation -n mesh-observability

# Confirm PodMonitors and ServiceMonitors are in place
oc --context=ossm-kiali-spoke get podmonitor,servicemonitor -A | \
  grep -E "ztunnel|istiod|bookinfo|ambient"

# Confirm UWM Prometheus pods are running
oc --context=ossm-kiali-spoke get pods -n openshift-user-workload-monitoring

6.4 Verify ACM Can See the Spoke

On the hub cluster, confirm the spoke cluster is healthy and visible to ACM:

oc --context=ossm-kiali-hub get managedcluster "${SPOKE_CLUSTER_NAME}"
oc --context=ossm-kiali-hub get managedclusteraddons -n "${SPOKE_CLUSTER_NAME}"

6.5 Access the Kiali UI

Kiali is accessible in two ways:

Standalone UI — the Kiali route URL:

oc --context=ossm-kiali-spoke get route kiali -n istio-system -o jsonpath='https://{.spec.host}{"\n"}'

OpenShift console — the Service Mesh item in the left-hand menu found at the spoke OpenShift console URL:

oc --context=ossm-kiali-spoke get route console -n openshift-console \
  -o jsonpath='https://{.spec.host}{"\n"}'

Open either URL and log in with your OpenShift credentials. You should see:

Overview page: shows 2 data planes - 1 ambient and 1 sidecar. Viewing data planes shows ambient-demo and bookinfo
Traffic Graph: navigate to the Traffic Graph page and select ambient-demo and bookinfo from the namespace dropdown at the top. Traffic edges should appear for each namespace:
- ambient-demo: traffic-gen → helloworld split to helloworld-v1 and helloworld-v2. To control what is shown, open the Display menu in the graph toolbar and toggle “Waypoint Proxies”. In the Traffic menu in the graph toolbar under the Ambient section you will find Waypoint, Ztunnel, and Total toggles. Enable Waypoint to see L7 HTTP edges; enable Ztunnel to see L4 TCP edges. If only TCP edges appear, the waypoint’s L7 metrics may need another ACM collection cycle (~5 minutes) before appearing.
- bookinfo: full L7 graph across productpage → details, reviews → ratings with HTTP response codes and latency
Mesh page: navigate to the Mesh page to see the overall mesh topology — the control plane, ztunnel, and the istio-system namespace can all be represented in the mesh graph.

6.5.1 Verify Waypoint Metrics in Kiali

The waypoint emits the HTTP/L7 metrics, but Kiali displays those metrics on the application workloads and services involved in the request. The waypoint workload page is primarily for checking the waypoint status, enrolled services and workloads, and Envoy details; its normal inbound and outbound workload metric views are not the right place to validate application traffic.

After traffic-gen has generated requests to the helloworld Service, use the following Kiali views:

Workload details → helloworld-v1 or helloworld-v2 → Inbound Metrics: shows HTTP/L7 traffic received by the workload through the waypoint.
Workload details → traffic-gen → Outbound Metrics: shows the HTTP/L7 traffic sent from the traffic generator through the waypoint.
Service details → helloworld → Inbound Metrics: shows inbound traffic for the helloworld Service.
Traffic Graph → Traffic → Waypoint: filters the graph to the waypoint’s L7 HTTP edges. Select Ztunnel to see the separate L4/TCP edges, or Total to see both.

Seeing both a waypoint edge and a ztunnel edge is expected in ambient mode. The waypoint provides HTTP details such as response codes and latency, while ztunnel provides L4/TCP telemetry. If the waypoint views are empty, first confirm that the hub Thanos query for reporter="waypoint" in section 5.1.1 returns a result, then allow another ACM collection cycle for the metrics to reach hub Thanos.

For Kiali to show the Ambient badge and ztunnel details it needs access to the ztunnel namespace. The cluster_wide_access: true setting in the Kiali CR (configured in Phase 4) covers this automatically.

Because Kiali queries ACM’s hub Thanos rather than the spoke’s local Prometheus, the graph follows the MCOA federation cadence. See Scrape Intervals for the expected latency and warm-up behavior.

Cleanup

The cleanup order below mirrors the install order in reverse and is dependency-safe: MCOA federation resources are removed while ACM is still running, then ACM is removed.

Step 1 — Remove workloads from the spoke:

oc --context=ossm-kiali-spoke delete gateway waypoint -n ambient-demo --ignore-not-found
oc --context=ossm-kiali-spoke delete namespace ambient-demo bookinfo --ignore-not-found
oc --context=ossm-kiali-spoke delete ossmconsole ossmconsole -n istio-system --ignore-not-found
oc --context=ossm-kiali-spoke delete kiali kiali -n istio-system --ignore-not-found
oc --context=ossm-kiali-spoke delete secret acm-observability-certs cacerts -n istio-system --ignore-not-found
oc --context=ossm-kiali-spoke delete configmap kiali-cabundle -n istio-system --ignore-not-found
oc --context=ossm-kiali-spoke delete ztunnel default --ignore-not-found
oc --context=ossm-kiali-spoke delete istio default --ignore-not-found
oc --context=ossm-kiali-spoke delete istiocni default --ignore-not-found
oc --context=ossm-kiali-spoke delete namespace ztunnel istio-system istio-cni --ignore-not-found

Step 2 — Remove MCOA federation resources from the hub (must happen while ACM is still running):

# Read the placement so we know which index to remove from
ADDON_JSON=$(oc --context=ossm-kiali-hub get clustermanagementaddon \
  multicluster-observability-addon -o json 2>/dev/null || true)

if [ -n "${ADDON_JSON}" ]; then
  remove_mcoa_ref() {
    local group=$1 resource=$2 name=$3
    local patch
    patch=$(echo "${ADDON_JSON}" | jq -c \
      --arg g "${group}" --arg r "${resource}" --arg n "${name}" \
      --arg ns "open-cluster-management-observability" \
      '[(.spec.installStrategy.placements // []) | to_entries[] as $p |
        ($p.value.configs // []) | to_entries[] |
        select(.value.group == $g and .value.resource == $r and
               .value.name == $n and .value.namespace == $ns) |
        {op:"remove",
         path:("/spec/installStrategy/placements/"+($p.key|tostring)+"/configs/"+(.key|tostring))}]
      | sort_by(.path) | reverse')
    [ "${patch}" = "[]" ] || \
      oc --context=ossm-kiali-hub patch clustermanagementaddon \
        multicluster-observability-addon --type=json -p="${patch}"
    ADDON_JSON=$(oc --context=ossm-kiali-hub get clustermanagementaddon \
      multicluster-observability-addon -o json 2>/dev/null || echo "${ADDON_JSON}")
  }

  # Remove in reverse order: the PrometheusRule first, then the platform and shared ScrapeConfigs
  remove_mcoa_ref monitoring.coreos.com prometheusrules kiali-istio-aggregation
  remove_mcoa_ref monitoring.rhobs scrapeconfigs kiali-istio-platform-federation
  remove_mcoa_ref monitoring.rhobs scrapeconfigs kiali-istio-federation
fi

# Delete the hub-side MCOA configuration resources
oc --context=ossm-kiali-hub delete prometheusrule kiali-istio-aggregation \
  -n open-cluster-management-observability --ignore-not-found
oc --context=ossm-kiali-hub delete scrapeconfig kiali-istio-platform-federation \
  -n open-cluster-management-observability --ignore-not-found
oc --context=ossm-kiali-hub delete scrapeconfig kiali-istio-federation \
  -n open-cluster-management-observability --ignore-not-found

Step 3 — Remove ACM Observability from the hub. Delete the MCO first and wait for it to be gone before removing SeaweedFS:

oc --context=ossm-kiali-hub delete mco observability --ignore-not-found
oc --context=ossm-kiali-hub wait mco observability --for=delete --timeout=120s 2>/dev/null || true
oc --context=ossm-kiali-hub delete deployment seaweedfs -n open-cluster-management-observability --ignore-not-found
oc --context=ossm-kiali-hub delete service seaweedfs -n open-cluster-management-observability --ignore-not-found
oc --context=ossm-kiali-hub delete secret thanos-object-storage -n open-cluster-management-observability --ignore-not-found
oc --context=ossm-kiali-hub delete namespace open-cluster-management-observability --ignore-not-found

Step 4 — Detach the spoke from ACM:

# Delete the Klusterlet on the spoke and wait for it to be gone
oc --context=ossm-kiali-spoke delete klusterlet klusterlet --ignore-not-found --wait=false 2>/dev/null || true
until ! oc --context=ossm-kiali-spoke get klusterlet klusterlet &>/dev/null 2>&1; do
  echo "Waiting for Klusterlet removal..."
  sleep 10
done

# Remove ACM agent namespaces from spoke (may already be gone after Klusterlet removal)
oc --context=ossm-kiali-spoke delete namespace \
  open-cluster-management-agent \
  open-cluster-management-agent-addon \
  open-cluster-management-policies \
  --ignore-not-found --wait=false 2>/dev/null || true

# Delete the ManagedCluster on the hub
oc --context=ossm-kiali-hub delete managedcluster "${SPOKE_CLUSTER_NAME}" \
  --ignore-not-found --wait=false
until ! oc --context=ossm-kiali-hub get managedcluster "${SPOKE_CLUSTER_NAME}" &>/dev/null 2>&1; do
  echo "Waiting for ManagedCluster removal..."
  sleep 10
done
oc --context=ossm-kiali-hub delete namespace "${SPOKE_CLUSTER_NAME}" --ignore-not-found

Step 5 — Clean up spoke ACM residue (objects that can linger if hub-side deletion completed after the Klusterlet stopped):

# Remove residual AppliedManifestWork objects
if oc --context=ossm-kiali-spoke get crd appliedmanifestworks.work.open-cluster-management.io &>/dev/null; then
  AMWS=$(oc --context=ossm-kiali-spoke get appliedmanifestwork -o name 2>/dev/null || true)
  if [ -n "${AMWS}" ]; then
    echo "${AMWS}" | xargs oc --context=ossm-kiali-spoke delete --ignore-not-found
  fi
fi

# Release orphaned ConfigurationPolicy finalizers on the spoke. The stranded
# policies live in the namespace named after the spoke cluster. Scope to that
# namespace, the cluster-name label, terminating state, and the specific ACM
# finalizer to avoid touching unrelated policies.
SPOKE_NAME="ossm-kiali-spoke"   # substitute your SPOKE_CLUSTER_NAME if different
if oc --context=ossm-kiali-spoke get crd \
    configurationpolicies.policy.open-cluster-management.io &>/dev/null 2>&1; then
  oc --context=ossm-kiali-spoke get configurationpolicy \
    -n "${SPOKE_NAME}" -o json 2>/dev/null | \
    jq -r --arg cluster "${SPOKE_NAME}" '.items[] |
      select(.metadata.deletionTimestamp != null) |
      select(.metadata.labels["policy.open-cluster-management.io/cluster-name"] == $cluster) |
      select(any(.metadata.finalizers[]?;
        . == "policy.open-cluster-management.io/delete-related-objects")) |
      .metadata.name' | \
  while read -r policy; do
    echo "Releasing finalizer: ${SPOKE_NAME}/${policy}"
    oc --context=ossm-kiali-spoke patch configurationpolicy "${policy}" \
      -n "${SPOKE_NAME}" \
      --type=merge -p '{"metadata":{"finalizers":[]}}' 2>/dev/null || true
  done
fi

# Remove ACM platform recording rules left in openshift-monitoring
PROM_RULES=$(oc --context=ossm-kiali-spoke get prometheusrule \
  -n openshift-monitoring -o name 2>/dev/null | grep '/acm-rs-' || true)
if [ -n "${PROM_RULES}" ]; then
  echo "${PROM_RULES}" | xargs oc --context=ossm-kiali-spoke \
    -n openshift-monitoring delete --ignore-not-found
fi

# Remove residual ACM RBAC from spoke
RBAC=$(oc --context=ossm-kiali-spoke get clusterrole,clusterrolebinding -o name 2>/dev/null | \
  grep -E '/(ocm:|.*open-cluster-management|.*multicluster-observability|.*observability.*mco)' || true)
if [ -n "${RBAC}" ]; then echo "${RBAC}" | xargs oc --context=ossm-kiali-spoke delete --ignore-not-found; fi

# Remove residual ACM admission webhooks from spoke
WEBHOOKS=$(oc --context=ossm-kiali-spoke get \
  validatingwebhookconfiguration,mutatingwebhookconfiguration -o name 2>/dev/null | \
  grep -E '/.*(open-cluster-management|multicluster|observability)' || true)
if [ -n "${WEBHOOKS}" ]; then echo "${WEBHOOKS}" | xargs oc --context=ossm-kiali-spoke delete --ignore-not-found; fi

# Remove residual ACM and Observatorium APIServices from spoke
APIS=$(oc --context=ossm-kiali-spoke get apiservice -o name 2>/dev/null | \
  grep -E '\.(open-cluster-management\.io|multicluster\.openshift\.io|multicluster\.x-k8s\.io|observatorium\.io)$' || true)
if [ -n "${APIS}" ]; then echo "${APIS}" | xargs oc --context=ossm-kiali-spoke delete --ignore-not-found; fi

# Remove residual ACM-installed Hive APIServices from spoke (only when no live Hive workloads)
HIVE_WORKLOADS=$(oc --context=ossm-kiali-spoke get deploy,statefulset,daemonset,pod \
  -n hive -o name 2>/dev/null || true)
if ! oc --context=ossm-kiali-spoke get namespace hive &>/dev/null || \
   [ -z "${HIVE_WORKLOADS}" ]; then
  HIVE_APIS=$(oc --context=ossm-kiali-spoke get apiservice -o name 2>/dev/null | \
    grep -E '\.(hive\.openshift\.io|hiveinternal\.openshift\.io)$' || true)
  if [ -n "${HIVE_APIS}" ]; then echo "${HIVE_APIS}" | xargs oc --context=ossm-kiali-spoke delete --ignore-not-found; fi
fi

# Remove residual ACM, Observatorium, and Hive CRDs from spoke (last — the cleanup above still needs their APIs)
ACM_CRDS=$(oc --context=ossm-kiali-spoke get crd -o name 2>/dev/null | \
  grep -E '\.(open-cluster-management\.io|multicluster\.openshift\.io|multicluster\.x-k8s\.io|observatorium\.io)$' || true)
if [ -n "${ACM_CRDS}" ]; then echo "${ACM_CRDS}" | xargs oc --context=ossm-kiali-spoke delete --ignore-not-found; fi

HIVE_WORKLOADS=$(oc --context=ossm-kiali-spoke get deploy,statefulset,daemonset,pod \
  -n hive -o name 2>/dev/null || true)
if ! oc --context=ossm-kiali-spoke get namespace hive &>/dev/null || \
   [ -z "${HIVE_WORKLOADS}" ]; then
  HIVE_CRDS=$(oc --context=ossm-kiali-spoke get crd -o name 2>/dev/null | \
    grep -E '\.(hive\.openshift\.io|hiveinternal\.openshift\.io)$' || true)
  if [ -n "${HIVE_CRDS}" ]; then echo "${HIVE_CRDS}" | xargs oc --context=ossm-kiali-spoke delete --ignore-not-found; fi
fi

Step 6 — Remove ACM from the hub (deleting MultiClusterHub cascades all ACM components; this takes 5–15 minutes):

oc --context=ossm-kiali-hub delete multiclusterhub multiclusterhub \
  -n open-cluster-management --ignore-not-found

echo "Waiting for MultiClusterHub deletion (5–15 minutes)..."
while oc --context=ossm-kiali-hub get multiclusterhub multiclusterhub \
  -n open-cluster-management &>/dev/null 2>&1; do
  echo "  Still deleting..."
  sleep 15
done
echo "MultiClusterHub deleted"

oc --context=ossm-kiali-hub delete subscriptions.operators.coreos.com acm-operator-subscription \
  -n open-cluster-management --ignore-not-found
oc --context=ossm-kiali-hub delete csv \
  -n open-cluster-management --all --ignore-not-found
oc --context=ossm-kiali-hub delete namespace open-cluster-management --ignore-not-found --timeout=300s

Step 6b — Clean up hub-side ACM/MCE/Hive/Observatorium residue (CRDs, RBAC, webhooks, and APIService registrations that survive after the ACM namespace and operators are removed):

# Hub RBAC
HUB_RBAC=$(oc --context=ossm-kiali-hub get clusterrole,clusterrolebinding -o name 2>/dev/null | \
  grep -E '/(ocm:|.*open-cluster-management|.*multiclusterengine|.*multicluster-observability|.*observability.*mco)' || true)
[ -n "${HUB_RBAC}" ] && echo "${HUB_RBAC}" | xargs oc --context=ossm-kiali-hub delete --ignore-not-found

# Hub admission webhooks
HUB_WEBHOOKS=$(oc --context=ossm-kiali-hub get \
  validatingwebhookconfiguration,mutatingwebhookconfiguration -o name 2>/dev/null | \
  grep -E '/.*(open-cluster-management|multicluster|observability)' || true)
[ -n "${HUB_WEBHOOKS}" ] && echo "${HUB_WEBHOOKS}" | xargs oc --context=ossm-kiali-hub delete --ignore-not-found

# Hub APIServices: ACM/MCE + Observatorium; Hive only when no live workloads
HUB_APIS=$(oc --context=ossm-kiali-hub get apiservice -o name 2>/dev/null | \
  grep -E '\.(open-cluster-management\.io|multicluster\.openshift\.io|multicluster\.x-k8s\.io|observatorium\.io)$' || true)
[ -n "${HUB_APIS}" ] && echo "${HUB_APIS}" | xargs oc --context=ossm-kiali-hub delete --ignore-not-found
HIVE_WORKLOADS=$(oc --context=ossm-kiali-hub get deploy,statefulset,daemonset,pod \
  -n hive -o name 2>/dev/null || true)
if ! oc --context=ossm-kiali-hub get namespace hive &>/dev/null || \
   [ -z "${HIVE_WORKLOADS}" ]; then
  HIVE_APIS=$(oc --context=ossm-kiali-hub get apiservice -o name 2>/dev/null | \
    grep -E '\.(hive\.openshift\.io|hiveinternal\.openshift\.io)$' || true)
  [ -n "${HIVE_APIS}" ] && echo "${HIVE_APIS}" | xargs oc --context=ossm-kiali-hub delete --ignore-not-found
fi

# Hub CRDs last — the cleanup above still needs their APIs
HUB_CRDS=$(oc --context=ossm-kiali-hub get crd -o name 2>/dev/null | \
  grep -E '\.(open-cluster-management\.io|multicluster\.openshift\.io|multicluster\.x-k8s\.io|observatorium\.io)$' || true)
[ -n "${HUB_CRDS}" ] && echo "${HUB_CRDS}" | xargs oc --context=ossm-kiali-hub delete --ignore-not-found
HIVE_WORKLOADS=$(oc --context=ossm-kiali-hub get deploy,statefulset,daemonset,pod \
  -n hive -o name 2>/dev/null || true)
if ! oc --context=ossm-kiali-hub get namespace hive &>/dev/null || \
   [ -z "${HIVE_WORKLOADS}" ]; then
  HIVE_CRDS=$(oc --context=ossm-kiali-hub get crd -o name 2>/dev/null | \
    grep -E '\.(hive\.openshift\.io|hiveinternal\.openshift\.io)$' || true)
  [ -n "${HIVE_CRDS}" ] && echo "${HIVE_CRDS}" | xargs oc --context=ossm-kiali-hub delete --ignore-not-found
  # Remove the empty hive namespace ACM installed Hive into
  oc --context=ossm-kiali-hub delete namespace hive \
    --ignore-not-found --timeout=120s 2>/dev/null || true
fi

Step 7 — Remove OSSM and Kiali operators from the spoke. Skip this block if other workloads on the cluster use these operators:

# Remove Subscriptions
oc --context=ossm-kiali-spoke delete subscriptions.operators.coreos.com \
  kiali-ossm openshift-service-mesh-operator \
  -n openshift-operators --ignore-not-found

# Delete this tutorial's pending InstallPlans before removing CSVs — otherwise OLM may recreate CSVs from in-flight plans
for SUBSCRIPTION_LABEL in \
  operators.coreos.com/kiali-ossm.openshift-operators \
  operators.coreos.com/servicemeshoperator3.openshift-operators
do
  oc --context=ossm-kiali-spoke delete installplan -n openshift-operators \
    -l "${SUBSCRIPTION_LABEL}" --ignore-not-found
done

# Remove ALL CSVs — delete the CSV in the operator namespace only; OLM cascades deletion to all copied namespaces automatically
CSV=$(oc --context=ossm-kiali-spoke get csv -n openshift-operators \
  -l operators.coreos.com/kiali-ossm.openshift-operators \
  --no-headers -o custom-columns=NAME:.metadata.name 2>/dev/null | head -1)
if [ -n "${CSV}" ]; then oc --context=ossm-kiali-spoke delete csv "${CSV}" -n openshift-operators --ignore-not-found; fi
CSV=$(oc --context=ossm-kiali-spoke get csv -n openshift-operators \
  -l operators.coreos.com/servicemeshoperator3.openshift-operators \
  --no-headers -o custom-columns=NAME:.metadata.name 2>/dev/null | head -1)
if [ -n "${CSV}" ]; then oc --context=ossm-kiali-spoke delete csv "${CSV}" -n openshift-operators --ignore-not-found; fi

# Remove ALL CRDs — you must remove every CRD installed by the OSSM and Kiali operators or reinstallation will conflict
for suffix in sailoperator.io istio.io kiali.io; do
  CRDS=$(oc --context=ossm-kiali-spoke get crd \
    --no-headers -o custom-columns=NAME:.metadata.name 2>/dev/null \
    | grep "\.${suffix}$")
  if [ -n "${CRDS}" ]; then
    echo "${CRDS}" | xargs oc --context=ossm-kiali-spoke delete crd --ignore-not-found
  fi
done

# Remove cluster-scoped and cross-namespace resources left by Istio
oc --context=ossm-kiali-spoke delete gatewayclass \
  istio istio-remote istio-waypoint istio-east-west --ignore-not-found
for NS in $(oc --context=ossm-kiali-spoke get configmap -A \
  -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name' --no-headers 2>/dev/null \
  | grep -E 'istio-ca-root-cert|istio-ca-crl' | awk '{print $1}' | sort -u); do
  oc --context=ossm-kiali-spoke delete configmap istio-ca-root-cert istio-ca-crl \
    -n "${NS}" --ignore-not-found
done
CRDS=$(oc --context=ossm-kiali-spoke get crd -o name 2>/dev/null \
  | grep -E '\.(gateway\.networking\.k8s\.io|inference\.networking\.(k8s|x-k8s)\.io)$' || true)
[ -z "${CRDS}" ] || echo "${CRDS}" \
  | xargs oc --context=ossm-kiali-spoke delete --ignore-not-found

# Remove the ClusterRole installed by the OSSM operator
oc --context=ossm-kiali-spoke delete clusterrole servicemeshoperator3-metrics-reader --ignore-not-found

Step 8 — Remove UWM configuration (only if the tutorial created it). The automation script labels the ConfigMap kiali.io/tutorial-owned=true when it creates it from scratch, and skips deletion when that label is absent. Reproduce that behavior manually:

# Remove the spoke ConfigMap only if the tutorial created it
OWNED=$(oc --context=ossm-kiali-spoke get configmap cluster-monitoring-config \
  -n openshift-monitoring \
  -o jsonpath='{.metadata.labels.kiali\.io/tutorial-owned}' 2>/dev/null || true)
[ "${OWNED}" = "true" ] && oc --context=ossm-kiali-spoke delete configmap \
  cluster-monitoring-config -n openshift-monitoring --ignore-not-found

# The hub ConfigMap is never created by this tutorial — do not touch it.

Notes and Considerations

1. discoverySelectors Require Explicit Namespace Labeling

If the Istio CR uses meshConfig.discoverySelectors, every namespace that should be part of the mesh — including app namespaces and the ztunnel namespace — must carry the matching label (e.g., istio-discovery: enabled). Without it, istiod ignores the namespace: sidecar injection won’t work and ztunnel won’t route ambient traffic.

The ztunnel namespace must be labeled with istio-discovery: enabled. Even though it is referenced via pilot.trustedZtunnelNamespace, istiod still needs to discover the namespace via discoverySelectors in order to distribute the istio-ca-root-cert ConfigMap there — without which ztunnel pods fail to start with a MountVolume.SetUp failed error.

2. Sidecar PodMonitors Must Be Per-Namespace

OpenShift’s User Workload Monitoring does not honor namespaceSelector in PodMonitor resources. A separate PodMonitor named istio-proxies-monitor must be created in every namespace that has sidecar-injected pods. Forgetting this is the most common reason Kiali shows an empty traffic graph for sidecar-mode namespaces.

3. Restricting Kiali’s Visible Namespaces with Discovery Selectors

By default, the Kiali CR in this guide uses cluster_wide_access: true, which gives Kiali access to — and makes visible — every namespace on the cluster. In an environment with many namespaces this can be noisy and affect performance.

Kiali has its own deployment.discovery_selectors that control which namespaces appear in the UI. These are independent of Istio’s meshConfig.discoverySelectors — Kiali does not read Istio’s selectors automatically. If you want Kiali to show only mesh namespaces, configure matching selectors in the Kiali CR.

For example, to restrict Kiali to namespaces labeled istio-discovery: enabled (the same label used by the Istio CR in this guide):

spec:
  deployment:
    cluster_wide_access: true   # keep ClusterRole for performance
    discovery_selectors:
      default:
      - matchLabels:
          istio-discovery: enabled

With cluster_wide_access: true and discovery_selectors set, Kiali retains efficient cluster-wide watches but only surfaces matching namespaces to users in the UI. To apply this, patch the Kiali CR on the spoke:

oc --context=ossm-kiali-spoke patch kiali kiali -n istio-system --type=merge -p '{
  "spec": {
    "deployment": {
      "discovery_selectors": {
        "default": [{"matchLabels": {"istio-discovery": "enabled"}}]
      }
    }
  }
}'

See the Namespace Management documentation for the full set of options, including per-cluster selectors for multi-cluster deployments.

Docs: Deployment Options

Mon, 01 Jan 0001 00:00:00 +0000

There are other, more complex deployment settings, described in dedicated pages:

All examples on this page are focused on the Kiali CR (when installing via the Kiali operator). Remember that Helm charts mirror all these configurations.

Kiali and Istio installation namespaces

By default, the Kiali operator installs Kiali in the same namespace where the Kiali CR is created. However, it is possible to specify a different namespace for installation:

spec:
  deployment:
    namespace: "custom-kiali-namespace"

Log level and format

By default, Kiali will print up to INFO-level messages in simple text format. You can change the log level, output format, and time format as in the following example:

spec:
  deployment:
    logger:
      # Supported values are "trace", "debug", "info", "warn", "error" and "fatal"
      log_level: error  
      # Supported values are "text" and "json".
      log_format: json  
      time_field_format: "2006-01-02T15:04:05Z07:00"

The syntax for the time_field_format is the same as the Time.Format function of the Go language.

The json format is useful if you are parsing logs of your applications for further processing.

In Kiali, there are some special logs called audit logs that are emitted each time a user creates, updates or deletes a resource through Kiali. Audit logs are INFO-level messages and are enabled by default. If audit logs are too verbose, you can disable them without reducing the log level as follows:

spec:
  server:
    audit_log: false

Kiali instance name

If you plan to install more than one Kiali instance on the same cluster, you may need to configure an instance name to avoid conflicts on created resources:

spec:
  deployment:
    instance_name: "secondary"

The instance_name will be used as a prefix for all created Kiali resources. The exception is the kiali-signing-key secret which will always have the same name and will be shared on all deployments of the same namespace, unless you specify a custom secret name.

Since the instance_name will be used as a name prefix in resources, it must follow Kubernetes naming constraints.

Since Kubernetes resources cannot be renamed, you cannot change the instance_name of an existing Kiali installation. The workaround is to uninstall Kiali and re-install with the desired instance_name.

Resource requests and limits

You can set the amount of resources available to Kiali using the spec.deployment.resources attribute, like in the following example:

spec:
  deployment:
    resources:
      requests:
        memory: "128Mi"
        cpu: "100m"
      limits:
        memory: "1Gi"
        cpu: "500m"

Please, read the Managing Resources for Containers section in the Kubernetes documentation for more details about possible configurations.

Custom labels and annotations on the Kiali pod and service

Although some labels and annotations are set on the Kiali pod and on its service (depending on configurations), you can add additional ones. For the pod, use the spec.deployment.pod_labels and spec.deployment.pod_annotations attributes. For the service, you can only add annotations using the spec.deployment.service_annotations attribute. For example:

spec:
  deployment:
    pod_annotations:
      a8r.io/repository: "https://github.com/kiali/kiali"
    pod_labels:
      sidecar.istio.io/inject: "true"
    service_annotations:
      a8r.io/documentation: "https://kiali.io/docs/installation/deployment-configuration"

Kiali page title (browser title bar)

If you have several Kiali installations and you are using them at the same time, there are good chances that you will want to identify each Kiali by simply looking at the browser’s title bar. You can set a custom text in the title bar with the following configuration:

spec:
  installation_tag: "Kiali West"

The installation_tag is any human readable text of your desire.

Kubernetes scheduler settings

Replicas and automatic scaling

By default, only one replica of Kiali is deployed. If needed, you can change the replica count like in the following example:

spec:
  deployment:
    replicas: 2

If you prefer automatic scaling, creation of an HorizontalPodAutoscaler resource is supported. For example, the following configuration automatically scales Kiali based on CPU utilization:

spec:
  deployment:
    hpa:
      api_version: "autoscaling/v1"
      spec:
        minReplicas: 1
        maxReplicas: 2
        targetCPUUtilizationPercentage: 80

You must omit the scaleTargetRef field of the HPA spec, because this field will be populated by the Kiali operator (or by Helm) depending on other configuration.

Read the Kubernetes Horizontal Pod Autoscaler documentation to learn more about the HPA.

Allocating the Kiali pod to specific nodes of the cluster

You can constrain the Kiali pod to run on a specific set of nodes by using some of the standard Kubernetes scheduler configurations.

The simplest option is to use the nodeSelector configuration which you can configure like in the following example:

spec:
  deployment:
    node_selector:
      worker-type: infra

You can also use the affinity/anti-affinity native Kubernetes feature if you prefer its more expressive syntax, or if you need more complex matching rules. The following is an example for configuring node affinity:

spec:
  deployment:
    affinity:
      node:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: worker-type
              operator: In
              values:
              - infra

Similarly, you can also configure pod affinity and pod anti-affinity using the spec.deployment.affinity.pod and spec.deployment.affinity.pod_anti attributes.

Finally, if you want to run Kiali in a node with taints, the following is an example to configure tolerations:

spec:
  deployment:
    tolerations: # Allow to run Kiali in a tainted master node
    - key: "node-role.kubernetes.io/master"
      operator: "Exists"
      effect: "NoSchedule"

Read the following Kubernetes documentation to learn more about these configurations:

Priority class of the Kiali pod

If you are using priority classes in your cluster, you can specify the priority class that will be set on the Kiali pod. For example:

spec:
  deployment:
    priority_class_name: high-priority

For more information about priority classes, read Pod Priority and Preemption in the Kubernetes documentation.

Adding host aliases to the Kiali pod

If you need to provide some static hostname resolution in the Kiali pod, you can use HostAliases to add entries to the /etc/hosts file of the Kiali pod, like in the following example:

spec:
  deployment:
   host_aliases:
   - ip: 192.168.1.100
     hostnames:
     - "foo.local"
     - "bar.local"

HTTP server

Kiali is served over HTTP. You can configure a few options of the HTTP server. The following are the defaults, but you can change them to suit your needs.

spec:
  server:
    # Listen/bind address of the HTTP server. By default it is empty, which means to
    # listen on all interfaces.
    address: ""

    # Listening port of the HTTP server. If you change it, also Kiali's Kubernetes
    # Service is affected to use this port.
    port: 20001

    # Use GZip compression for responses larger than 1400 bytes. You may want to disable
    # compression if you are exposing Kiali via a reverse proxy that is already
    # doing compression.
    gzip_enabled: true

    # For development purposes only. Controls if "Cross-Origin Resourse Sharing" is
    # enabled. 
    cors_allow_all: false

There is one additional spec.server.web_root option that affects the HTTP server, but that one is described in the Specifying route settings section of the Instalation guide.

Metrics server

Kiali emits metrics that can be collected by Prometheus. Most of these metrics are performance measurements.

The metrics server is enabled by default and listens on port 9090:

spec:
  server:
    metrics_enabled: true
    metrics_port: 9090

The bind address is the same as the HTTP server. Thus, make sure that the HTTP Server and the metrics server are not configured to the same port.

Docs: Install via OperatorHub

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

The OperatorHub is a website that contains a catalog of Kubernetes Operators. Its aim is to be the central location to find Operators.

The OperatorHub relies in the Operator Lifecycle Manager (OLM) to install, manage and update Operators on any Kubernetes cluster.

The Kiali Operator is being published to the OperatorHub. So, you can use the OLM to install and manage the Kiali Operator installation.

Installing the Kiali Operator using the OLM

Go to the Kiali Operator page in the OperatorHub: https://operatorhub.io/operator/kiali.

You will see an Install button at the right of the page. Press it and you will be presented with the installation instructions. Follow these instructions to install and manage the Kiali Operator installation using OLM.

Afterwards, you can create the Kiali CR to install Kiali.

Installing the Kiali Operator in OpenShift

The OperatorHub is bundled in the OpenShift console. To install the Kiali Operator, simply go to the OperatorHub in the OpenShift console and search for the Kiali Operator. Then, click on the Install button and follow the instruction on the screen.

Afterwards, you can create the Kiali CR to install Kiali.

Docs: Multi-Primary Mesh

Mon, 01 Jan 0001 00:00:00 +0000

Multi-Primary Mesh on OpenShift with ACM

This guide extends the result of the MultiCluster on OpenShift guide into a two-primary Istio mesh spanning two spoke clusters. The ACM hub cluster is not changed — it remains dedicated to fleet management and centralized metrics collection.

What this guide adds:

A second spoke cluster (ossm-kiali-spoke-two) is imported into ACM and gets OSSM 3 installed
Both spoke clusters are updated to share a single Istio mesh via East-West gateways and cross-cluster endpoint discovery
ACM Observability is extended to collect metrics from the second spoke
Kiali on the first spoke gains API access to the second spoke’s workloads via a remote cluster secret
A cross-cluster demo app demonstrates traffic flowing between the two spokes

Cluster roles after this guide:

ossm-kiali-hub — ACM hub: fleet management + centralized Thanos metrics (unchanged)
ossm-kiali-spoke — Istio primary cluster 1: existing mesh + Kiali
ossm-kiali-spoke-two — Istio primary cluster 2: new, added to the mesh

The diagram below shows the environment after this guide completes. Components from Guide 1 are marked prior; badges such as G2:P5.1-5.3 mark which guide section(s) add each piece (Guide 2, §§5.1–5.3). Click the diagram to open a full-size SVG in a new tab.

This guide requires OSSM 3.4+ (Istio 1.30+) on OpenShift 4.19+. The East-West gateway configuration for cross-cluster sidecar traffic relies on Gateway API behavior introduced in Istio 1.30. Istio 1.30’s oldest supported Kubernetes version is 1.32, which corresponds to OpenShift 4.19.

Prerequisites

The MultiCluster on OpenShift guide completed successfully — ACM, OSSM 3, and Kiali must already be running on ossm-kiali-hub and ossm-kiali-spoke.
A second fresh OpenShift 4.19+ cluster accessible via kubeconfig context ossm-kiali-spoke-two (same minimum as the hub/spoke guide).
istioctl installed locally (required to create Istio cross-cluster endpoint discovery secrets). Version must match ${ISTIO_VERSION}.

All three kubeconfig contexts reachable:

oc --context=ossm-kiali-hub       whoami --show-server
oc --context=ossm-kiali-spoke     whoami --show-server
oc --context=ossm-kiali-spoke-two whoami --show-server

The Istio CA root certificate from the hub/spoke guide must still be available at /tmp/istio-certs/root-cert.pem and /tmp/istio-certs/root-key.pem. The second spoke’s intermediate CA must be signed by the same root to establish mutual trust.

Environment Setup

Set these variables in the same shell session used for the hub/spoke guide, or re-export the variables from that guide first.

# Must match the hub/spoke guide values exactly
export SPOKE_CLUSTER_NAME="spoke"
export ISTIO_VERSION="1.30.1"
export MESH_ID="mesh1"

# Network names — each cluster must be on a different network
# (no direct pod-to-pod connectivity between clusters)
export SPOKE_NETWORK="network1"
export SPOKE_TWO_NETWORK="network2"

# Spoke-two ACM cluster name — also used as the Istio cluster identity and
# Kiali cluster name so that naming is consistent across all components
export SPOKE_TWO_CLUSTER_NAME="spoke-two"

Phase 1: Generate Spoke-Two CA Certificate

The second spoke needs its own intermediate CA certificate, signed by the same root CA used for the first spoke in the hub/spoke guide. This shared root establishes mutual mTLS trust between the two control planes.

mkdir -p /tmp/istio-certs/spoke-two && cd /tmp/istio-certs

cat > spoke-two/intermediate.conf <<'CONF'
[ req ]
encrypt_key = no
prompt = no
utf8 = yes
default_md = sha256
default_bits = 4096
req_extensions = req_ext
x509_extensions = req_ext
distinguished_name = req_dn
[ req_ext ]
subjectKeyIdentifier = hash
basicConstraints = critical, CA:true, pathlen:0
keyUsage = critical, digitalSignature, nonRepudiation, keyEncipherment, keyCertSign
subjectAltName=@san
[ san ]
DNS.1 = istiod.istio-system.svc
[ req_dn ]
O = Istio
CN = Intermediate CA
L = spoke-two
CONF

openssl genrsa -out spoke-two/ca-key.pem 4096 2>/dev/null

openssl req -new \
  -config spoke-two/intermediate.conf \
  -key spoke-two/ca-key.pem \
  -out spoke-two/cluster-ca.csr 2>/dev/null

openssl x509 -req -sha256 -days 3650 \
  -CA root-cert.pem \
  -CAkey root-key.pem -CAcreateserial \
  -extensions req_ext -extfile spoke-two/intermediate.conf \
  -in spoke-two/cluster-ca.csr \
  -out spoke-two/ca-cert.pem 2>/dev/null

cat spoke-two/ca-cert.pem root-cert.pem > spoke-two/cert-chain.pem
cp root-cert.pem spoke-two/root-cert.pem

echo "Spoke-two intermediate CA: $(openssl x509 -noout -subject -in spoke-two/ca-cert.pem)"

cd -

Phase 2: Import Spoke-Two into ACM

On the hub cluster, create a ManagedCluster resource and import ossm-kiali-spoke-two using the same auto-import approach as the hub/spoke guide.

oc --context=ossm-kiali-hub create namespace "${SPOKE_TWO_CLUSTER_NAME}" 2>/dev/null || true

oc --context=ossm-kiali-hub apply -f - <<EOF
apiVersion: cluster.open-cluster-management.io/v1
kind: ManagedCluster
metadata:
  name: ${SPOKE_TWO_CLUSTER_NAME}
  labels:
    cloud: auto-detect
    vendor: auto-detect
spec:
  hubAcceptsClient: true
EOF

oc --context=ossm-kiali-hub apply -f - <<EOF
apiVersion: agent.open-cluster-management.io/v1
kind: KlusterletAddonConfig
metadata:
  name: ${SPOKE_TWO_CLUSTER_NAME}
  namespace: ${SPOKE_TWO_CLUSTER_NAME}
spec:
  applicationManager:
    enabled: true
  certPolicyController:
    enabled: true
  policyController:
    enabled: true
  searchCollector:
    enabled: true
EOF

Extract the spoke-two kubeconfig and create the auto-import secret:

oc config view --context=ossm-kiali-spoke-two --minify --flatten \
  > /tmp/spoke-two-kubeconfig.yaml

oc --kubeconfig=/tmp/spoke-two-kubeconfig.yaml whoami --show-server

oc --context=ossm-kiali-hub create secret generic auto-import-secret \
  -n "${SPOKE_TWO_CLUSTER_NAME}" \
  --from-file=kubeconfig=/tmp/spoke-two-kubeconfig.yaml

rm -f /tmp/spoke-two-kubeconfig.yaml

Wait for spoke-two to join:

echo "Waiting for ${SPOKE_TWO_CLUSTER_NAME} to join..."
while true; do
  STATUS=$(oc --context=ossm-kiali-hub get managedcluster "${SPOKE_TWO_CLUSTER_NAME}" \
    -o jsonpath='{range .status.conditions[*]}{.type}={.status}{" "}{end}' 2>/dev/null)
  echo "  Status: ${STATUS}"
  if echo "${STATUS}" | grep -q "ManagedClusterJoined=True" && \
     echo "${STATUS}" | grep -q "ManagedClusterConditionAvailable=True"; then
    echo "${SPOKE_TWO_CLUSTER_NAME} is joined and available"
    break
  fi
  sleep 15
done

Phase 3: Install OSSM 3 on Spoke-Two

3.1 Enable User Workload Monitoring

Check if UWM is already enabled on the spoke cluster:

oc --context=ossm-kiali-spoke-two get configmap cluster-monitoring-config \
  -n openshift-monitoring \
  -o jsonpath='{.data.config\.yaml}' 2>/dev/null | \
  grep -q "enableUserWorkload: true" && \
  echo "Already enabled" || echo "Not enabled"

If UWM is not yet enabled, enable it following the instructions below. The following enables UWM safely — merge enableUserWorkload: true without clobbering other monitoring settings. If you created the ConfigMap from scratch, label it so cleanup knows it is safe to remove:

if oc --context=ossm-kiali-spoke-two get configmap cluster-monitoring-config \
    -n openshift-monitoring &>/dev/null 2>&1; then
  # ConfigMap already exists — patch only the enableUserWorkload key
  EXISTING=$(oc --context=ossm-kiali-spoke-two get configmap cluster-monitoring-config \
    -n openshift-monitoring -o jsonpath='{.data.config\.yaml}' 2>/dev/null || true)
  if echo "${EXISTING}" | grep -q "enableUserWorkload:"; then
    oc --context=ossm-kiali-spoke-two get configmap cluster-monitoring-config \
      -n openshift-monitoring -o json \
      | jq '.data["config.yaml"] |= sub("enableUserWorkload:\\s*\\w+"; "enableUserWorkload: true")' \
      | oc --context=ossm-kiali-spoke-two apply -f -
  else
    oc --context=ossm-kiali-spoke-two get configmap cluster-monitoring-config \
      -n openshift-monitoring -o json \
      | jq '.data["config.yaml"] = ((.data["config.yaml"] // "") + "\nenableUserWorkload: true\n")' \
      | oc --context=ossm-kiali-spoke-two apply -f -
  fi
else
  # ConfigMap does not exist — create it and label it as tutorial-owned
  oc --context=ossm-kiali-spoke-two create configmap cluster-monitoring-config \
    -n openshift-monitoring \
    --from-literal=config.yaml="enableUserWorkload: true"
  oc --context=ossm-kiali-spoke-two label configmap cluster-monitoring-config \
    -n openshift-monitoring kiali.io/tutorial-owned=true
fi

Wait for the User Workload Monitoring Prometheus pods to be created and become ready. This can take a few minutes after UWM is enabled.

until oc --context=ossm-kiali-spoke-two get pods \
  -l app.kubernetes.io/name=prometheus \
  -n openshift-user-workload-monitoring \
  --no-headers 2>/dev/null | grep -q .; do
  sleep 5
done
oc --context=ossm-kiali-spoke-two wait pod \
  --for=condition=Ready \
  -l app.kubernetes.io/name=prometheus \
  -n openshift-user-workload-monitoring \
  --timeout=300s

3.2 Install OSSM 3 Operator

# Merge this key into the existing ConfigMap without replacing unrelated UWM settings.
oc --context=ossm-kiali-spoke-two apply -f - <<'EOF'
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: openshift-service-mesh-operator
  namespace: openshift-operators
spec:
  channel: stable
  installPlanApproval: Automatic
  name: servicemeshoperator3
  source: redhat-operators
  sourceNamespace: openshift-marketplace
EOF

until oc --context=ossm-kiali-spoke-two get pods \
  -l app.kubernetes.io/created-by=servicemeshoperator3 \
  -n openshift-operators \
  --no-headers 2>/dev/null | grep -q .; do
  sleep 5
done
oc --context=ossm-kiali-spoke-two wait pod \
  --for=condition=Ready \
  -l app.kubernetes.io/created-by=servicemeshoperator3 \
  -n openshift-operators \
  --timeout=300s

3.3 Create Namespaces and Label ztunnel for Discovery

oc --context=ossm-kiali-spoke-two create namespace istio-system 2>/dev/null || true
oc --context=ossm-kiali-spoke-two create namespace istio-cni 2>/dev/null || true
oc --context=ossm-kiali-spoke-two create namespace ztunnel 2>/dev/null || true

# ztunnel namespace must be labeled so istiod distributes the CA cert ConfigMap there
oc --context=ossm-kiali-spoke-two label namespace ztunnel istio-discovery=enabled

# istio-system must be labeled for two reasons:
# 1. topology.istio.io/network - tells istiod which network the cluster is on (required for multi-network routing)
# 2. istio-discovery: enabled - required for the Gateway API controller to process Gateways in this namespace
oc --context=ossm-kiali-spoke-two label namespace istio-system \
  topology.istio.io/network="${SPOKE_TWO_NETWORK}" \
  istio-discovery=enabled

3.4 Load CA Certificates

oc --context=ossm-kiali-spoke-two get secret cacerts -n istio-system &>/dev/null || \
oc --context=ossm-kiali-spoke-two create secret generic cacerts -n istio-system \
  --from-file=ca-cert.pem=/tmp/istio-certs/spoke-two/ca-cert.pem \
  --from-file=ca-key.pem=/tmp/istio-certs/spoke-two/ca-key.pem \
  --from-file=root-cert.pem=/tmp/istio-certs/spoke-two/root-cert.pem \
  --from-file=cert-chain.pem=/tmp/istio-certs/spoke-two/cert-chain.pem

3.5 Install IstioCNI

oc --context=ossm-kiali-spoke-two apply -f - <<EOF
apiVersion: sailoperator.io/v1
kind: IstioCNI
metadata:
  name: default
spec:
  namespace: istio-cni
  profile: openshift-ambient
  version: v${ISTIO_VERSION}
EOF

oc --context=ossm-kiali-spoke-two wait istiocni default \
  --for=condition=Ready \
  --timeout=300s

3.6 Install Istio Control Plane (with Multi-Cluster Config)

The Istio CR for spoke-two sets the cluster identity, network identity, and enables AMBIENT_ENABLE_MULTI_NETWORK for ambient cross-network routing. The clusterName must match the value in the ZTunnel CR exactly — istiod uses it to identify the local cluster, and ztunnel uses it when authenticating to istiod. A mismatch causes ztunnel pods to fail with authentication failure:

oc --context=ossm-kiali-spoke-two apply -f - <<EOF
apiVersion: sailoperator.io/v1
kind: Istio
metadata:
  name: default
spec:
  namespace: istio-system
  profile: openshift-ambient
  updateStrategy:
    type: InPlace
  values:
    global:
      meshID: ${MESH_ID}
      multiCluster:
        clusterName: ${SPOKE_TWO_CLUSTER_NAME}
      network: ${SPOKE_TWO_NETWORK}
    meshConfig:
      discoverySelectors:
      - matchLabels:
          istio-discovery: enabled
    pilot:
      trustedZtunnelNamespace: ztunnel
      env:
        AMBIENT_ENABLE_MULTI_NETWORK: "true"
  version: v${ISTIO_VERSION}
EOF

oc --context=ossm-kiali-spoke-two wait istio default \
  --for=condition=Ready \
  --timeout=300s

3.7 Install ZTunnel (with Cluster and Network Identity)

oc --context=ossm-kiali-spoke-two apply -f - <<EOF
apiVersion: sailoperator.io/v1
kind: ZTunnel
metadata:
  name: default
spec:
  namespace: ztunnel
  version: v${ISTIO_VERSION}
  values:
    ztunnel:
      multiCluster:
        clusterName: ${SPOKE_TWO_CLUSTER_NAME}
      network: ${SPOKE_TWO_NETWORK}
EOF

oc --context=ossm-kiali-spoke-two wait ztunnel default \
  --for=condition=Ready \
  --timeout=300s

oc --context=ossm-kiali-spoke-two get pods -n ztunnel -l app=ztunnel

3.8 Configure Istio Metrics Collection on Spoke-Two

The hub-side MCOA configuration resources from Guide 1 are already registered with the placement. When spoke-two joins the cluster set, MCOA propagates the shared recording rule into mesh-observability on spoke-two so UWM can aggregate workload:istio_* series from all scraped namespaces there.

Ensure the aggregation namespace exists and is exempt from UWM label enforcement on spoke-two before MCOA propagates the recording rule:

oc --context=ossm-kiali-spoke-two create namespace mesh-observability --dry-run=client -o yaml | \
  oc --context=ossm-kiali-spoke-two apply -f -
oc --context=ossm-kiali-spoke-two apply -f - <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    namespacesWithoutLabelEnforcement:
    - mesh-observability
EOF
for NS in istio-system ztunnel ambient-demo bookinfo; do
  oc --context=ossm-kiali-spoke-two create namespace "${NS}" --dry-run=client -o yaml | \
    oc --context=ossm-kiali-spoke-two apply -f -
done

Create the UWM monitors so UWM Prometheus scrapes the raw Istio metrics that MCOA will then federate to hub Thanos:

oc --context=ossm-kiali-spoke-two apply -f - <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: istiod-monitor
  namespace: istio-system
spec:
  targetLabels:
  - app
  selector:
    matchLabels:
      istio: pilot
  endpoints:
  - port: http-monitoring
    interval: 30s
EOF

oc --context=ossm-kiali-spoke-two apply -f - <<EOF
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: ztunnel-monitor
  namespace: ztunnel
spec:
  selector:
    matchExpressions:
    - key: istio-prometheus-ignore
      operator: DoesNotExist
  podMetricsEndpoints:
  - path: /stats/prometheus
    interval: 30s
    relabelings:
    - action: keep
      sourceLabels: ["__meta_kubernetes_pod_container_name"]
      regex: "istio-proxy"
    - action: keep
      sourceLabels: ["__meta_kubernetes_pod_annotationpresent_prometheus_io_scrape"]
    - action: replace
      regex: (\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})
      replacement: '[\$2]:\$1'
      sourceLabels: ["__meta_kubernetes_pod_annotation_prometheus_io_port","__meta_kubernetes_pod_ip"]
      targetLabel: "__address__"
    - action: replace
      regex: (\d+);((([0-9]+?)(\.|$)){4})
      replacement: '\$2:\$1'
      sourceLabels: ["__meta_kubernetes_pod_annotation_prometheus_io_port","__meta_kubernetes_pod_ip"]
      targetLabel: "__address__"
    - sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_name","__meta_kubernetes_pod_label_app"]
      separator: ";"
      targetLabel: "app"
      action: replace
      regex: "(.+);.*|.*;(.+)"
      replacement: "\${1}\${2}"
    - sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_version","__meta_kubernetes_pod_label_version"]
      separator: ";"
      targetLabel: "version"
      action: replace
      regex: "(.+);.*|.*;(.+)"
      replacement: "\${1}\${2}"
    - action: replace
      regex: "(.+)"
      replacement: "\${1}"
      sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
      targetLabel: "app_kubernetes_io_name"
    - action: replace
      regex: "(.+)"
      replacement: "\${1}"
      sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_version"]
      targetLabel: "app_kubernetes_io_version"
    - sourceLabels: ["__meta_kubernetes_namespace"]
      action: replace
      targetLabel: namespace
    - action: replace
      replacement: "${MESH_ID}"
      targetLabel: mesh_id
EOF

Phase 4: Update Spoke (Cluster 1) for Multi-Primary

The first spoke’s Istio and ZTunnel CRs from the hub/spoke guide did not include cluster identity or multi-network settings. Patch them now.

4.1 Label Spoke istio-system with Network Identity

oc --context=ossm-kiali-spoke label namespace istio-system \
  topology.istio.io/network="${SPOKE_NETWORK}" \
  istio-discovery=enabled

4.2 Update Spoke Istio CR

oc --context=ossm-kiali-spoke patch istio default --type=merge -p "{
  \"spec\": {
    \"values\": {
      \"global\": {
        \"multiCluster\": {\"clusterName\": \"${SPOKE_CLUSTER_NAME}\"},
        \"network\": \"${SPOKE_NETWORK}\"
      },
      \"pilot\": {
        \"env\": {
          \"AMBIENT_ENABLE_MULTI_NETWORK\": \"true\"
        }
      }
    }
  }
}"

oc --context=ossm-kiali-spoke wait istio default \
  --for=condition=Ready \
  --timeout=300s

4.3 Update Spoke ZTunnel CR

Setting clusterName in the ZTunnel CR must match the value in the Istio CR above — both istiod and ztunnel must agree on the local cluster identity (see Notes #5):

oc --context=ossm-kiali-spoke patch ztunnel default --type=merge -p "{
  \"spec\": {
    \"values\": {
      \"ztunnel\": {
        \"multiCluster\": {\"clusterName\": \"${SPOKE_CLUSTER_NAME}\"},
        \"network\": \"${SPOKE_NETWORK}\"
      }
    }
  }
}"

oc --context=ossm-kiali-spoke wait ztunnel default \
  --for=condition=Ready \
  --timeout=300s

4.4 Update Kiali with the Cluster Name

Kiali was installed in the hub/spoke guide without a cluster name because only one cluster existed. Now that spoke has multiCluster.clusterName: spoke configured, Kiali must be updated to match so it correctly identifies the home cluster — without this, the Kiali login page will redirect in an infinite loop:

oc --context=ossm-kiali-spoke patch kiali kiali -n istio-system --type=merge -p "{
  \"spec\": {
    \"kubernetes_config\": {
      \"cluster_name\": \"${SPOKE_CLUSTER_NAME}\"
    }
  }
}"

oc --context=ossm-kiali-spoke rollout status deployment/kiali \
  -n istio-system --timeout=120s

Wait until the injector ConfigMap reflects ${SPOKE_CLUSTER_NAME}, then restart the sidecar-injected pods in bookinfo so they pick up the new cluster identity. The OSSM/Sail operator publishes injection settings in the istio-sidecar-injector ConfigMap in istio-system. The first loop below polls that ConfigMap until global.multiCluster.clusterName equals ${SPOKE_CLUSTER_NAME} (see Notes #5 for why you must wait). Only then restart bookinfo so new sidecars pick up the identity. After the rollout, the final check reads ISTIO_META_CLUSTER_ID from a sample pod’s istio-proxy container to confirm injection used the new name — not Kubernetes:

echo "Waiting for sidecar injector to publish clusterName=${SPOKE_CLUSTER_NAME}..."
while true; do
  INJECTOR_CLUSTER=$(oc --context=ossm-kiali-spoke get configmap istio-sidecar-injector \
    -n istio-system \
    -o jsonpath='{.data.values}' | \
    jq -r '.global.multiCluster.clusterName // empty')
  echo "  injector clusterName=${INJECTOR_CLUSTER:-<empty>}"
  if [ "${INJECTOR_CLUSTER}" = "${SPOKE_CLUSTER_NAME}" ]; then
    echo "Sidecar injector is ready"
    break
  fi
  sleep 5
done

oc --context=ossm-kiali-spoke rollout restart deployment -n bookinfo

# Wait on Deployments — not `oc wait pods --all`, which races with Terminating pods during the restart
oc --context=ossm-kiali-spoke wait deployment --all \
  -n bookinfo \
  --for=condition=Available \
  --timeout=180s

# Confirm sidecars were injected with the new cluster identity (must not be "Kubernetes")
SAMPLE_POD=$(oc --context=ossm-kiali-spoke get pod -n bookinfo \
  -l app=productpage \
  -o jsonpath='{.items[0].metadata.name}')
PROXY_CLUSTER=$(oc --context=ossm-kiali-spoke get pod "${SAMPLE_POD}" -n bookinfo -o json | \
  jq -r '[.spec.containers[]?, .spec.initContainers[]?
    | select(.name == "istio-proxy")
    | .env[]? | select(.name == "ISTIO_META_CLUSTER_ID")
    | .value] | first // empty')
echo "productpage ISTIO_META_CLUSTER_ID=${PROXY_CLUSTER}"
if [ "${PROXY_CLUSTER}" != "${SPOKE_CLUSTER_NAME}" ]; then
  echo "ERROR: expected ISTIO_META_CLUSTER_ID=${SPOKE_CLUSTER_NAME}, got '${PROXY_CLUSTER}'."
  echo "Re-check the injector ConfigMap above, then re-run the rollout restart."
fi

Phase 5: East-West Gateways

East-West gateways bridge the two cluster networks. Two types of cross-network traffic need separate gateways because they use different protocols:

Ambient traffic (ztunnel → ztunnel): uses HBONE on port 15008, handled by the istio-east-west gatewayClassName
Sidecar traffic (sidecar → sidecar): uses mTLS auto-passthrough on port 15443, handled by a standard istio gatewayClassName gateway with TLS Passthrough

5.1 HBONE Gateway (Ambient Traffic)

Deploy the HBONE east-west gateway on each cluster. This handles cross-cluster traffic for ambient-mode workloads:

for CTX_AND_NET in "ossm-kiali-spoke:${SPOKE_NETWORK}" "ossm-kiali-spoke-two:${SPOKE_TWO_NETWORK}"; do
  CTX="${CTX_AND_NET%%:*}"
  NET="${CTX_AND_NET##*:}"
  oc --context="${CTX}" apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: istio-eastwestgateway
  namespace: istio-system
  labels:
    topology.istio.io/network: ${NET}
spec:
  gatewayClassName: istio-east-west
  listeners:
  - name: mesh
    port: 15008
    protocol: HBONE
    tls:
      mode: Terminate
      options:
        gateway.istio.io/tls-terminate-mode: ISTIO_MUTUAL
EOF
  oc --context="${CTX}" wait gateway/istio-eastwestgateway \
    -n istio-system \
    --for=condition=Programmed=True \
    --timeout=180s
done

5.2 Sidecar Gateway (mTLS Auto-Passthrough)

Sidecar proxies route cross-network traffic via mTLS on port 15443 — a different protocol than HBONE. The istio-east-west gatewayClassName only handles HBONE, so a second gateway using the standard istio gatewayClassName is needed.

A Gateway API resource with protocol: TLS and tls.mode: Passthrough on port 15443 is sufficient — Istio’s Gateway API implementation automatically generates SNI-based passthrough filter chains for all discovered mesh services.

Deploy the sidecar east-west gateway on each cluster:

for CTX_AND_NET in "ossm-kiali-spoke:${SPOKE_NETWORK}" "ossm-kiali-spoke-two:${SPOKE_TWO_NETWORK}"; do
  CTX="${CTX_AND_NET%%:*}"
  NET="${CTX_AND_NET##*:}"
  oc --context="${CTX}" apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: istio-eastwestgateway-sidecar
  namespace: istio-system
  labels:
    topology.istio.io/network: ${NET}
spec:
  gatewayClassName: istio
  listeners:
  - name: tls
    port: 15443
    protocol: TLS
    tls:
      mode: Passthrough
    allowedRoutes:
      namespaces:
        from: Same
EOF
  oc --context="${CTX}" wait gateway/istio-eastwestgateway-sidecar \
    -n istio-system \
    --for=condition=Programmed=True \
    --timeout=180s
done

5.3 Configure meshNetworks

The meshNetworks configuration tells each istiod where the East-West gateways are for each network. Each network entry has two gateways: the HBONE gateway (port 15008) for ambient workloads, and the sidecar gateway (port 15443) for sidecar-injected workloads. Istiod selects the correct gateway based on the proxy type of the requesting workload.

HBONE_IP1=$(oc --context=ossm-kiali-spoke get svc istio-eastwestgateway \
  -n istio-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
SIDECAR_IP1=$(oc --context=ossm-kiali-spoke get svc istio-eastwestgateway-sidecar-istio \
  -n istio-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
HBONE_IP2=$(oc --context=ossm-kiali-spoke-two get svc istio-eastwestgateway \
  -n istio-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
SIDECAR_IP2=$(oc --context=ossm-kiali-spoke-two get svc istio-eastwestgateway-sidecar-istio \
  -n istio-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo "Spoke:     HBONE=${HBONE_IP1}  Sidecar=${SIDECAR_IP1}"
echo "Spoke-two: HBONE=${HBONE_IP2}  Sidecar=${SIDECAR_IP2}"

for CTX in ossm-kiali-spoke ossm-kiali-spoke-two; do
  oc --context="${CTX}" patch istio default --type=merge -p "{
    \"spec\": {
      \"values\": {
        \"global\": {
          \"meshNetworks\": {
            \"${SPOKE_NETWORK}\": {
              \"endpoints\": [{\"fromRegistry\": \"${SPOKE_CLUSTER_NAME}\"}],
              \"gateways\": [
                {\"address\": \"${HBONE_IP1}\", \"port\": 15008},
                {\"address\": \"${SIDECAR_IP1}\", \"port\": 15443}
              ]
            },
            \"${SPOKE_TWO_NETWORK}\": {
              \"endpoints\": [{\"fromRegistry\": \"${SPOKE_TWO_CLUSTER_NAME}\"}],
              \"gateways\": [
                {\"address\": \"${HBONE_IP2}\", \"port\": 15008},
                {\"address\": \"${SIDECAR_IP2}\", \"port\": 15443}
              ]
            }
          }
        }
      }
    }
  }"
done

Phase 6: Cross-Cluster Endpoint Discovery

Each istiod must be able to watch the other cluster’s Kubernetes API server for services and endpoints. This requires a service account with cluster-reader permissions on each cluster and a cross-cluster secret generated via istioctl.

6.1 Create istio-reader Service Accounts

oc --context=ossm-kiali-spoke create serviceaccount istio-reader-service-account \
  -n istio-system 2>/dev/null || true
oc --context=ossm-kiali-spoke adm policy add-cluster-role-to-user cluster-reader \
  -z istio-reader-service-account -n istio-system

oc --context=ossm-kiali-spoke-two create serviceaccount istio-reader-service-account \
  -n istio-system 2>/dev/null || true
oc --context=ossm-kiali-spoke-two adm policy add-cluster-role-to-user cluster-reader \
  -z istio-reader-service-account -n istio-system

6.2 Install Remote Secret on Cluster 1 (from Cluster 2)

This lets cluster 1’s istiod discover cluster 2’s endpoints:

istioctl create-remote-secret \
  --context=ossm-kiali-spoke-two \
  --name="${SPOKE_TWO_CLUSTER_NAME}" \
  --create-service-account=false | \
  oc --context=ossm-kiali-spoke apply -f -

6.3 Install Remote Secret on Cluster 2 (from Cluster 1)

This lets cluster 2’s istiod discover cluster 1’s endpoints:

istioctl create-remote-secret \
  --context=ossm-kiali-spoke \
  --name="${SPOKE_CLUSTER_NAME}" \
  --create-service-account=false | \
  oc --context=ossm-kiali-spoke-two apply -f -

Verify remote secrets are present on both clusters:

oc --context=ossm-kiali-spoke get secrets \
  -n istio-system -l istio/multiCluster=true

oc --context=ossm-kiali-spoke-two get secrets \
  -n istio-system -l istio/multiCluster=true

Phase 7: Update Kiali for Multi-Cluster Access

Kiali runs on ossm-kiali-spoke and already queries ACM Thanos for metrics (set up in the hub/spoke guide). It now also needs API access to ossm-kiali-spoke-two to show workloads, services, and Istio config from that cluster.

7.1 Install Kiali Operator on Spoke-Two

The Kiali Operator on spoke-two creates the service account and RBAC that Kiali on spoke uses to access the cluster. No Kiali server is deployed on spoke-two.

oc --context=ossm-kiali-spoke-two apply -f - <<'EOF'
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: kiali-ossm
  namespace: openshift-operators
spec:
  channel: stable
  installPlanApproval: Automatic
  name: kiali-ossm
  source: redhat-operators
  sourceNamespace: openshift-marketplace
EOF

until oc --context=ossm-kiali-spoke-two get pods \
  -l app.kubernetes.io/name=kiali-operator \
  -n openshift-operators \
  --no-headers 2>/dev/null | grep -q .; do
  sleep 5
done
oc --context=ossm-kiali-spoke-two wait pod \
  --for=condition=Ready \
  -l app.kubernetes.io/name=kiali-operator \
  -n openshift-operators \
  --timeout=300s

7.2 Install Kiali CR on Spoke-Two (Remote Resources Only)

Create the Kiali CR with remote_cluster_resources_only: true. This creates the kiali-service-account ServiceAccount and RBAC but no Kiali server. The tutorial explicitly disables impersonation for compatibility with Kiali releases that do not support it.

The redirect URI points back to the home Kiali route and is required for the non-impersonation per-cluster OpenShift OAuth flow. Get the home Kiali route first:

KIALI_HOST=$(oc --context=ossm-kiali-spoke get route kiali -n istio-system \
  -o jsonpath='{.spec.host}')
echo "Kiali host: ${KIALI_HOST}"

oc --context=ossm-kiali-spoke-two apply -f - <<EOF
apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
  name: kiali
  namespace: istio-system
spec:
  auth:
    openshift:
      impersonation:
        enabled: false
      redirect_uris:
      - "https://${KIALI_HOST}/api/auth/callback/${SPOKE_TWO_CLUSTER_NAME}"
  deployment:
    logger:
      log_level: debug
    namespace: istio-system
    remote_cluster_resources_only: true
EOF

Wait for the CR to reconcile successfully, then verify the SA and ClusterRoleBinding were created:

until oc --context=ossm-kiali-spoke-two get kiali kiali -n istio-system \
  -o jsonpath='{.status.conditions[?(@.type=="Successful")].status}' 2>/dev/null | grep -q "True"; do
  echo "Waiting for Kiali CR reconciliation..."
  sleep 10
done

oc --context=ossm-kiali-spoke-two get sa kiali-service-account -n istio-system
oc --context=ossm-kiali-spoke-two get clusterrolebinding kiali -o name

7.3 Create the Kiali Multi-Cluster Secret

Kiali reads remote cluster credentials from a secret named kiali-multi-cluster-secret in the Kiali deployment namespace. Each key in the secret is a cluster name; its value is a kubeconfig that grants Kiali API access to that cluster. This single-secret pattern scales naturally — to add more clusters later, add more keys to the same secret.

Labeling the secret with kiali.io/kiali-multi-cluster-secret: "true" tells the Kiali Operator to watch it and automatically roll out a new Kiali pod whenever the secret changes (no manual trigger needed).

In Kubernetes 1.24+, service account token secrets are not created automatically. Create one for kiali-service-account on spoke-two so that a long-lived token is available:

The kubernetes.io/service-account-token Secret created below produces a token with no expiration. For production environments, consider using oc create token kiali-service-account -n istio-system --duration=720h instead to create a time-bounded token. You will need to rotate the token (and update the kiali-multi-cluster-secret on the home cluster) before it expires. Kiali detects secret changes and reloads credentials automatically without a pod restart.

oc --context=ossm-kiali-spoke-two apply -f - <<'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: "kiali-service-account"
  namespace: "istio-system"
  annotations:
    kubernetes.io/service-account.name: "kiali-service-account"
type: kubernetes.io/service-account-token
EOF

until oc --context=ossm-kiali-spoke-two get secret kiali-service-account \
  -n istio-system \
  -o jsonpath='{.data.token}' 2>/dev/null | grep -q .; do
  sleep 3
done
echo "SA token secret is ready"

Collect spoke-two’s connection details and write a kubeconfig to a temporary file:

SPOKE_TWO_SERVER=$(oc --context=ossm-kiali-spoke-two whoami --show-server | tr -d '[:space:]') && \
SPOKE_TWO_CA=$(oc --context=ossm-kiali-spoke-two get secret kiali-service-account \
  -n istio-system -o jsonpath='{.data.ca\.crt}') && \
SPOKE_TWO_TOKEN=$(oc --context=ossm-kiali-spoke-two get secret kiali-service-account \
  -n istio-system -o jsonpath='{.data.token}' | base64 -d) && \
cat > /tmp/cluster2-kubeconfig.yaml <<EOF
apiVersion: v1
kind: Config
current-context: ${SPOKE_TWO_CLUSTER_NAME}
contexts:
- name: ${SPOKE_TWO_CLUSTER_NAME}
  context:
    cluster: ${SPOKE_TWO_CLUSTER_NAME}
    user: ${SPOKE_TWO_CLUSTER_NAME}
clusters:
- name: ${SPOKE_TWO_CLUSTER_NAME}
  cluster:
    server: ${SPOKE_TWO_SERVER}
    certificate-authority-data: ${SPOKE_TWO_CA}
users:
- name: ${SPOKE_TWO_CLUSTER_NAME}
  user:
    token: ${SPOKE_TWO_TOKEN}
EOF
echo "Kubeconfig written — server: ${SPOKE_TWO_SERVER}"

Create kiali-multi-cluster-secret on spoke. The file key name must match the cluster name (${SPOKE_TWO_CLUSTER_NAME}):

oc --context=ossm-kiali-spoke create secret generic kiali-multi-cluster-secret \
  -n istio-system \
  --from-file="${SPOKE_TWO_CLUSTER_NAME}=/tmp/cluster2-kubeconfig.yaml"

oc --context=ossm-kiali-spoke label secret kiali-multi-cluster-secret \
  -n istio-system \
  "kiali.io/kiali-multi-cluster-secret=true"

rm -f /tmp/cluster2-kubeconfig.yaml

Verify the secret and confirm the kubeconfig can reach spoke-two:

oc --context=ossm-kiali-spoke get secret kiali-multi-cluster-secret \
  -n istio-system --show-labels

# Extract and test the embedded kubeconfig
oc --context=ossm-kiali-spoke get secret kiali-multi-cluster-secret \
  -n istio-system \
  -o jsonpath="{.data.${SPOKE_TWO_CLUSTER_NAME}}" | base64 -d > /tmp/verify-kubeconfig.yaml

oc --kubeconfig=/tmp/verify-kubeconfig.yaml whoami --show-server
# Expected: spoke-two's API server URL

rm -f /tmp/verify-kubeconfig.yaml

The kiali.io/kiali-multi-cluster-secret: "true" label causes the Kiali Operator to automatically detect the secret and roll out a new Kiali pod with the spoke-two credentials mounted.

Wait for the rollout to complete (triggered by the multi-cluster secret being detected):

oc --context=ossm-kiali-spoke rollout status deployment/kiali \
  -n istio-system --timeout=120s

Phase 8: Demo Applications (Spoke-Two)

8.1 Ambient Demo App — Helloworld

Deploy helloworld v1 and v2 on spoke-two’s ambient-demo namespace. Both clusters now have the same service name — Istio’s multi-cluster federation merges them into a single virtual service with endpoints from both clusters. Label the service istio.io/global=true so spoke’s istiod discovers spoke-two’s endpoints.

Once deployed, the existing traffic-gen on spoke (from the hub/spoke guide) will automatically start load-balancing requests across pods on both clusters through the East-West gateway.

oc --context=ossm-kiali-spoke-two create namespace ambient-demo 2>/dev/null || true
oc --context=ossm-kiali-spoke-two label namespace ambient-demo \
  istio.io/dataplane-mode=ambient \
  istio-discovery=enabled

ISTIO_MINOR=$(echo "${ISTIO_VERSION}" | cut -d. -f1-2)

# Download the helloworld manifest once to avoid GitHub rate limits on repeated requests
curl -sL "https://raw.githubusercontent.com/openshift-service-mesh/istio/release-${ISTIO_MINOR}/samples/helloworld/helloworld.yaml" \
  -o /tmp/helloworld.yaml

oc --context=ossm-kiali-spoke-two apply -n ambient-demo -l service=helloworld -f /tmp/helloworld.yaml
oc --context=ossm-kiali-spoke-two apply -n ambient-demo -l version=v1 -f /tmp/helloworld.yaml
oc --context=ossm-kiali-spoke-two apply -n ambient-demo -l version=v2 -f /tmp/helloworld.yaml
rm -f /tmp/helloworld.yaml

oc --context=ossm-kiali-spoke-two wait deployment/helloworld-v1 \
  -n ambient-demo --for=condition=Available --timeout=120s
oc --context=ossm-kiali-spoke-two wait deployment/helloworld-v2 \
  -n ambient-demo --for=condition=Available --timeout=120s

# Label service global on BOTH clusters so each istiod discovers the other's endpoints
oc --context=ossm-kiali-spoke label svc helloworld \
  -n ambient-demo \
  istio.io/global=true
oc --context=ossm-kiali-spoke-two label svc helloworld \
  -n ambient-demo \
  istio.io/global=true

oc --context=ossm-kiali-spoke-two get pods -n ambient-demo
# Each pod should show 1/1 READY — ambient mode, no sidecars

8.1.1 Deploy a Waypoint for L7 Metrics

Without a waypoint, ztunnel only produces TCP metrics. Deploy one to get full L7 HTTP visibility on spoke-two’s side of the cross-cluster traffic. See the hub/spoke guide’s section 5.1.1 for the explanation of why istio.io/waypoint-for: service is required.

oc --context=ossm-kiali-spoke-two apply -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: waypoint
  namespace: ambient-demo
  labels:
    istio.io/waypoint-for: service
spec:
  gatewayClassName: istio-waypoint
  listeners:
  - name: mesh
    port: 15008
    protocol: HBONE
EOF

oc --context=ossm-kiali-spoke-two wait gateway/waypoint \
  -n ambient-demo \
  --for=condition=Programmed=True \
  --timeout=120s

oc --context=ossm-kiali-spoke-two label namespace ambient-demo \
  istio.io/use-waypoint=waypoint

Add a PodMonitor to scrape waypoint Envoy metrics:

oc --context=ossm-kiali-spoke-two apply -f - <<EOF
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: istio-proxies-monitor
  namespace: ambient-demo
spec:
  selector:
    matchExpressions:
    - key: istio-prometheus-ignore
      operator: DoesNotExist
  podMetricsEndpoints:
  - path: /stats/prometheus
    interval: 30s
    relabelings:
    - action: keep
      sourceLabels: ["__meta_kubernetes_pod_container_name"]
      regex: "istio-proxy"
    - action: keep
      sourceLabels: ["__meta_kubernetes_pod_annotationpresent_prometheus_io_scrape"]
    - action: replace
      regex: (\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})
      replacement: '[\$2]:\$1'
      sourceLabels: ["__meta_kubernetes_pod_annotation_prometheus_io_port","__meta_kubernetes_pod_ip"]
      targetLabel: "__address__"
    - action: replace
      regex: (\d+);((([0-9]+?)(\.|$)){4})
      replacement: '\$2:\$1'
      sourceLabels: ["__meta_kubernetes_pod_annotation_prometheus_io_port","__meta_kubernetes_pod_ip"]
      targetLabel: "__address__"
    - sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_name","__meta_kubernetes_pod_label_app"]
      separator: ";"
      targetLabel: "app"
      action: replace
      regex: "(.+);.*|.*;(.+)"
      replacement: "\${1}\${2}"
    - sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_version","__meta_kubernetes_pod_label_version"]
      separator: ";"
      targetLabel: "version"
      action: replace
      regex: "(.+);.*|.*;(.+)"
      replacement: "\${1}\${2}"
    - action: replace
      regex: "(.+)"
      replacement: "\${1}"
      sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
      targetLabel: "app_kubernetes_io_name"
    - action: replace
      regex: "(.+)"
      replacement: "\${1}"
      sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_version"]
      targetLabel: "app_kubernetes_io_version"
    - sourceLabels: ["__meta_kubernetes_namespace"]
      action: replace
      targetLabel: namespace
    - action: replace
      replacement: "${MESH_ID}"
      targetLabel: mesh_id
EOF

8.2 Sidecar Demo — Split Bookinfo

Spoke-one already has the full Bookinfo application running (from the hub/spoke guide). Here we extend it by deploying a ratings-v2 workload on spoke-two. Because the ratings Service is federated across the mesh, reviews-v2 and reviews-v3 on spoke will occasionally route their ratings calls to spoke-two via the East-West gateway — creating cross-cluster L7 traffic visible in Kiali.

No changes to spoke’s existing bookinfo are needed. Sidecars produce full L7 HTTP metrics natively so no waypoint is required.

Create the bookinfo namespace on spoke-two with sidecar injection enabled:

oc --context=ossm-kiali-spoke-two create namespace bookinfo 2>/dev/null || true
oc --context=ossm-kiali-spoke-two label namespace bookinfo \
  istio-injection=enabled \
  istio-discovery=enabled

Deploy ratings-v2 on spoke-two. This uses the same container image as ratings-v1 but with a version: v2 label so it appears as a distinct versioned workload in Kiali:

oc --context=ossm-kiali-spoke-two apply -f - <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bookinfo-ratings
  namespace: bookinfo
---
apiVersion: v1
kind: Service
metadata:
  name: ratings
  namespace: bookinfo
  labels:
    app: ratings
    service: ratings
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: ratings
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ratings-v2
  namespace: bookinfo
  labels:
    app: ratings
    version: v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ratings
      version: v2
  template:
    metadata:
      labels:
        app: ratings
        version: v2
    spec:
      serviceAccountName: bookinfo-ratings
      containers:
      - name: ratings
        image: docker.io/istio/examples-bookinfo-ratings-v1:1.20.3
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9080
EOF

Wait for ratings-v2 to be ready. Each pod should show 2/2 (app container + istio-proxy sidecar):

oc --context=ossm-kiali-spoke-two wait deployment/ratings-v2 \
  -n bookinfo --for=condition=Available --timeout=120s

oc --context=ossm-kiali-spoke-two get pods -n bookinfo
# Should show 2/2 READY

Label the ratings Service as global on both clusters so each istiod discovers the other’s endpoints:

oc --context=ossm-kiali-spoke label svc ratings \
  -n bookinfo \
  istio.io/global=true
oc --context=ossm-kiali-spoke-two label svc ratings \
  -n bookinfo \
  istio.io/global=true

Add a PodMonitor for the bookinfo namespace on spoke-two so its sidecar metrics reach hub Thanos:

oc --context=ossm-kiali-spoke-two apply -f - <<EOF
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: istio-proxies-monitor
  namespace: bookinfo
spec:
  selector:
    matchExpressions:
    - key: istio-prometheus-ignore
      operator: DoesNotExist
  podMetricsEndpoints:
  - path: /stats/prometheus
    interval: 30s
    relabelings:
    - action: keep
      sourceLabels: ["__meta_kubernetes_pod_container_name"]
      regex: "istio-proxy"
    - action: keep
      sourceLabels: ["__meta_kubernetes_pod_annotationpresent_prometheus_io_scrape"]
    - action: replace
      regex: (\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})
      replacement: '[\$2]:\$1'
      sourceLabels: ["__meta_kubernetes_pod_annotation_prometheus_io_port","__meta_kubernetes_pod_ip"]
      targetLabel: "__address__"
    - action: replace
      regex: (\d+);((([0-9]+?)(\.|$)){4})
      replacement: '\$2:\$1'
      sourceLabels: ["__meta_kubernetes_pod_annotation_prometheus_io_port","__meta_kubernetes_pod_ip"]
      targetLabel: "__address__"
    - sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_name","__meta_kubernetes_pod_label_app"]
      separator: ";"
      targetLabel: "app"
      action: replace
      regex: "(.+);.*|.*;(.+)"
      replacement: "\${1}\${2}"
    - sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_version","__meta_kubernetes_pod_label_version"]
      separator: ";"
      targetLabel: "version"
      action: replace
      regex: "(.+);.*|.*;(.+)"
      replacement: "\${1}\${2}"
    - action: replace
      regex: "(.+)"
      replacement: "\${1}"
      sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
      targetLabel: "app_kubernetes_io_name"
    - action: replace
      regex: "(.+)"
      replacement: "\${1}"
      sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_version"]
      targetLabel: "app_kubernetes_io_version"
    - sourceLabels: ["__meta_kubernetes_namespace"]
      action: replace
      targetLabel: namespace
    - action: replace
      replacement: "${MESH_ID}"
      targetLabel: mesh_id
EOF

The cross-cluster traffic flow is: traffic-gen → productpage → reviews-v2/v3 → ratings (load-balanced: sometimes ratings-v1 on cluster1, sometimes ratings-v2 on cluster2 via the East-West gateway). Kiali will show this as a cross-cluster edge from the reviews workloads to ratings with the cluster label distinguishing which instance served each request.

Phase 9: Verification

9.1 Verify Both Control Planes

oc --context=ossm-kiali-spoke     get istio default
oc --context=ossm-kiali-spoke     get ztunnel default
oc --context=ossm-kiali-spoke-two get istio default
oc --context=ossm-kiali-spoke-two get ztunnel default

9.2 Verify East-West Gateways

Both the HBONE (ambient) and sidecar gateways should be Programmed=True with external IPs:

oc --context=ossm-kiali-spoke     get gateway -n istio-system
oc --context=ossm-kiali-spoke-two get gateway -n istio-system
# Each cluster should show istio-eastwestgateway (HBONE) and istio-eastwestgateway-sidecar (mTLS)

9.3 Verify Remote Secrets

oc --context=ossm-kiali-spoke     get secrets -n istio-system -l istio/multiCluster=true
oc --context=ossm-kiali-spoke-two get secrets -n istio-system -l istio/multiCluster=true
# Each cluster should show a secret for the other cluster

9.4 Verify ACM Sees Both Spokes

oc --context=ossm-kiali-hub get managedclusters
# Both spoke and spoke-two should show JOINED=True, AVAILABLE=True

9.4.1 Verify MCOA Propagation on Spoke-Two

Confirm the MCOA add-on is available and the federation recording rules have propagated:

# MCOA add-on must be Available on spoke-two
oc --context=ossm-kiali-hub get managedclusteraddon \
  multicluster-observability-addon -n "${SPOKE_TWO_CLUSTER_NAME}"

# PrometheusAgent should exist on spoke-two
oc --context=ossm-kiali-spoke-two get prometheusagent \
  -n open-cluster-management-agent-addon

# The shared aggregation rule should be propagated to the exempt namespace
oc --context=ossm-kiali-spoke-two get prometheusrule \
  kiali-istio-aggregation -n mesh-observability

9.5 Verify meshNetworks Configuration

Confirm that the meshNetworks settings from the Istio CR were applied to the istio ConfigMap on both clusters. Each cluster should show both network gateways with their correct external IPs:

oc --context=ossm-kiali-spoke get configmap istio \
  -n istio-system \
  -o jsonpath='{.data.meshNetworks}'

oc --context=ossm-kiali-spoke-two get configmap istio \
  -n istio-system \
  -o jsonpath='{.data.meshNetworks}'

Expected output on each cluster — each network should have two gateway entries (HBONE on 15008, sidecar on 15443):

networks:
  network1:
    endpoints:
    - fromRegistry: spoke
    gateways:
    - address: <spoke-hbone-gw-ip>
      port: 15008
    - address: <spoke-sidecar-gw-ip>
      port: 15443
  network2:
    endpoints:
    - fromRegistry: spoke-two
    gateways:
    - address: <spoke-two-hbone-gw-ip>
      port: 15008
    - address: <spoke-two-sidecar-gw-ip>
      port: 15443

9.6 Verify Cross-Cluster Traffic

The helloworld response includes the pod instance name. First, record the pod names on each cluster so you know what to look for in the logs:

echo "=== Spoke-one helloworld pods ==="
oc --context=ossm-kiali-spoke get pods -n ambient-demo -l app=helloworld \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'

echo "=== Spoke-two helloworld pods ==="
oc --context=ossm-kiali-spoke-two get pods -n ambient-demo -l app=helloworld \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'

Now check the traffic-gen logs — you should see instance names from both clusters appearing:

oc --context=ossm-kiali-spoke logs -n ambient-demo \
  deployment/traffic-gen --tail=20
# Expect lines like:
# Hello version: v1, instance: helloworld-v1-<spoke-suffix>
# Hello version: v1, instance: helloworld-v1-<spoke-two-suffix>   ← different pod name = cross-cluster

9.7 Verify Kiali Sees Both Clusters

oc --context=ossm-kiali-spoke get route kiali -n istio-system \
  -o jsonpath='https://{.spec.host}{"\n"}'

Open the URL and log in. In the top-right cluster dropdown you should see both spoke and spoke-two. After logging into spoke-two via the Kiali OAuth flow:

Overview: namespaces from both clusters visible
Traffic Graph: navigate to the Traffic Graph page and select ambient-demo and bookinfo from the namespace dropdown — namespaces from both clusters appear in the same list. Traffic edges should appear showing traffic-gen → helloworld in ambient-demo and the bookinfo topology in bookinfo (allow 10–15 minutes for ACM metrics pipeline)
Mesh page: both control planes represented

Hub metrics for both clusters should include distinct cluster labels. Verify hub Thanos contains metrics from both spokes:

# Query raw and recorded traffic metrics independently on each spoke
for CONTEXT in ossm-kiali-spoke ossm-kiali-spoke-two; do
  echo "=== ${CONTEXT}: edge UWM ==="
  PROM_POD=$(oc --context="${CONTEXT}" \
    -n openshift-user-workload-monitoring \
    get pods -l app.kubernetes.io/name=prometheus \
    -o jsonpath='{.items[0].metadata.name}')

  # Raw counters scraped from Istio proxies
  oc --context="${CONTEXT}" \
    -n openshift-user-workload-monitoring \
    exec -c prometheus "${PROM_POD}" -- \
    wget -qO- \
    'http://localhost:9090/api/v1/query?query=sum%28istio_requests_total%29' \
    | jq '.data.result'

  # Aggregates produced by the recording rules
  oc --context="${CONTEXT}" \
    -n openshift-user-workload-monitoring \
    exec -c prometheus "${PROM_POD}" -- \
    wget -qO- \
    'http://localhost:9090/api/v1/query?query=sum%28workload%3Aistio_requests_total%29' \
    | jq '.data.result'

  # Edge UWM metric names, including workload:* recording-rule series
  oc --context="${CONTEXT}" \
    -n openshift-user-workload-monitoring \
    exec -c prometheus "${PROM_POD}" -- \
    wget -qO- \
    'http://localhost:9090/api/v1/label/__name__/values' \
    | jq -r '
    [.data[] | select(test("istio|envoy"; "i"))] | unique
    | .[] ,
    "===\nTOTAL COUNT OF istio_ AND envoy_ METRICS: \(length)"'
done

# Hub Thanos: federated and relabeled totals, separated by managed cluster
oc --context=ossm-kiali-hub get --raw \
  "/api/v1/namespaces/open-cluster-management-observability/services/http:observability-thanos-query-frontend:9090/proxy/api/v1/query?query=sum%20by%20%28cluster%29%20%28istio_requests_total%29" \
  | jq '.data.result'
# Should return non-empty values for both ${SPOKE_CLUSTER_NAME} and
# ${SPOKE_TWO_CLUSTER_NAME}.

# Hub metric names; workload:* is absent after federation relabeling
oc --context=ossm-kiali-hub get --raw \
  "/api/v1/namespaces/open-cluster-management-observability/services/http:observability-thanos-query-frontend:9090/proxy/api/v1/label/__name__/values" \
  | jq -r '
  [.data[] | select(test("istio|envoy"; "i"))] | unique
  | .[] ,
  "===\nTOTAL COUNT OF istio_ AND envoy_ METRICS: \(length)"'

Generate traffic on both spokes before running these commands. The edge raw and recorded queries should be non-empty on each cluster. Allow at least one five-minute MCOA collection interval for hub Thanos. Hub values lag the edge, so compare presence and approximately corresponding counter values rather than expecting exact point-in-time equality. Each edge should list workload:* metrics and substantially more metric names than hub Thanos, where the workload: prefix has been removed.

Cleanup

Remove cross-cluster additions from spoke:

oc --context=ossm-kiali-spoke delete gateway istio-eastwestgateway -n istio-system --ignore-not-found
oc --context=ossm-kiali-spoke delete gateway istio-eastwestgateway-sidecar -n istio-system --ignore-not-found
oc --context=ossm-kiali-spoke delete secrets -n istio-system -l istio/multiCluster=true --ignore-not-found

Remove OSSM, Kiali, and demo apps from spoke-two:

oc --context=ossm-kiali-spoke-two delete gateway istio-eastwestgateway-sidecar -n istio-system --ignore-not-found
oc --context=ossm-kiali-spoke-two adm policy remove-cluster-role-from-user cluster-reader \
  -z istio-reader-service-account -n istio-system 2>/dev/null || true
oc --context=ossm-kiali-spoke-two delete namespace ambient-demo bookinfo --ignore-not-found
oc --context=ossm-kiali-spoke-two delete kiali kiali -n istio-system --ignore-not-found
oc --context=ossm-kiali-spoke-two delete ztunnel default --ignore-not-found
oc --context=ossm-kiali-spoke-two delete istio default --ignore-not-found
oc --context=ossm-kiali-spoke-two delete istiocni default --ignore-not-found
oc --context=ossm-kiali-spoke-two delete namespace ztunnel istio-system istio-cni --ignore-not-found

# Remove Subscriptions
oc --context=ossm-kiali-spoke-two delete subscriptions.operators.coreos.com \
  kiali-ossm openshift-service-mesh-operator \
  -n openshift-operators --ignore-not-found

# Delete pending install plans before removing CSVs — otherwise OLM may recreate CSVs from in-flight plans
oc --context=ossm-kiali-spoke-two delete installplan -n openshift-operators --all --ignore-not-found

# Remove ALL CSVs — delete the CSV in the operator namespace only; OLM cascades deletion to all copied namespaces automatically
CSV=$(oc --context=ossm-kiali-spoke-two get csv -n openshift-operators \
  -l operators.coreos.com/kiali-ossm.openshift-operators \
  --no-headers -o custom-columns=NAME:.metadata.name 2>/dev/null | head -1)
if [ -n "${CSV}" ]; then oc --context=ossm-kiali-spoke-two delete csv "${CSV}" -n openshift-operators --ignore-not-found; fi
CSV=$(oc --context=ossm-kiali-spoke-two get csv -n openshift-operators \
  -l operators.coreos.com/servicemeshoperator3.openshift-operators \
  --no-headers -o custom-columns=NAME:.metadata.name 2>/dev/null | head -1)
if [ -n "${CSV}" ]; then oc --context=ossm-kiali-spoke-two delete csv "${CSV}" -n openshift-operators --ignore-not-found; fi

# Remove ALL CRDs — you must remove every CRD installed by the OSSM and Kiali operators or reinstallation will conflict
for suffix in sailoperator.io istio.io kiali.io; do
  CRDS=$(oc --context=ossm-kiali-spoke-two get crd \
    --no-headers -o custom-columns=NAME:.metadata.name 2>/dev/null \
    | grep "\.${suffix}$")
  [ -n "${CRDS}" ] && echo "${CRDS}" | xargs oc --context=ossm-kiali-spoke-two delete crd --ignore-not-found
done

# Remove cluster-scoped and cross-namespace resources left by Istio
oc --context=ossm-kiali-spoke-two delete gatewayclass \
  istio istio-remote istio-waypoint istio-east-west --ignore-not-found
for NS in $(oc --context=ossm-kiali-spoke-two get configmap -A \
  -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name' --no-headers 2>/dev/null \
  | grep -E 'istio-ca-root-cert|istio-ca-crl' | awk '{print $1}' | sort -u); do
  oc --context=ossm-kiali-spoke-two delete configmap istio-ca-root-cert istio-ca-crl \
    -n "${NS}" --ignore-not-found
done
CRDS=$(oc --context=ossm-kiali-spoke-two get crd -o name 2>/dev/null \
  | grep -E '\.(gateway\.networking\.k8s\.io|inference\.networking\.(k8s|x-k8s)\.io)$' || true)
[ -z "${CRDS}" ] || echo "${CRDS}" \
  | xargs oc --context=ossm-kiali-spoke-two delete --ignore-not-found

# Remove the ClusterRole installed by the OSSM operator
oc --context=ossm-kiali-spoke-two delete clusterrole servicemeshoperator3-metrics-reader --ignore-not-found

Detach spoke-two from ACM — tear down the Klusterlet on spoke-two before waiting for the hub namespace, then clean up all ACM/MCOA/COO/UWM residue:

oc --context=ossm-kiali-hub delete klusterletaddonconfig "${SPOKE_TWO_CLUSTER_NAME}" \
  -n "${SPOKE_TWO_CLUSTER_NAME}" --ignore-not-found

# Tear down the Klusterlet agent on spoke-two
oc --context=ossm-kiali-spoke-two delete klusterlet klusterlet \
  --ignore-not-found --wait=false 2>/dev/null || true
until ! oc --context=ossm-kiali-spoke-two get klusterlet klusterlet &>/dev/null 2>&1; do
  echo "Waiting for Klusterlet removal on spoke-two..."
  sleep 10
done
oc --context=ossm-kiali-spoke-two delete namespace \
  open-cluster-management-agent open-cluster-management-agent-addon \
  open-cluster-management-policies \
  --ignore-not-found --wait=false 2>/dev/null || true

oc --context=ossm-kiali-hub delete managedcluster "${SPOKE_TWO_CLUSTER_NAME}" \
  --ignore-not-found --wait=false
until ! oc --context=ossm-kiali-hub get managedcluster "${SPOKE_TWO_CLUSTER_NAME}" &>/dev/null 2>&1; do
  echo "Waiting for ManagedCluster removal..."
  sleep 10
done
oc --context=ossm-kiali-hub delete namespace "${SPOKE_TWO_CLUSTER_NAME}" --ignore-not-found

# Clean up spoke-two ACM residue (objects that linger after Klusterlet teardown)
if oc --context=ossm-kiali-spoke-two get crd appliedmanifestworks.work.open-cluster-management.io &>/dev/null; then
  AMWS=$(oc --context=ossm-kiali-spoke-two get appliedmanifestwork -o name 2>/dev/null || true)
  [ -n "${AMWS}" ] && echo "${AMWS}" | xargs oc --context=ossm-kiali-spoke-two delete --ignore-not-found
fi

# Release orphaned ConfigurationPolicy finalizers — scoped to the spoke-two
# namespace, cluster-name label, terminating state, and the specific finalizer
SPOKE_TWO_NS="${SPOKE_TWO_CLUSTER_NAME}"
if oc --context=ossm-kiali-spoke-two get crd \
    configurationpolicies.policy.open-cluster-management.io &>/dev/null 2>&1; then
  oc --context=ossm-kiali-spoke-two get configurationpolicy \
    -n "${SPOKE_TWO_NS}" -o json 2>/dev/null | \
    jq -r --arg cluster "${SPOKE_TWO_NS}" '.items[] |
      select(.metadata.deletionTimestamp != null) |
      select(.metadata.labels["policy.open-cluster-management.io/cluster-name"] == $cluster) |
      select(any(.metadata.finalizers[]?;
        . == "policy.open-cluster-management.io/delete-related-objects")) |
      .metadata.name' | \
  while read -r policy; do
    echo "Releasing finalizer: ${SPOKE_TWO_NS}/${policy}"
    oc --context=ossm-kiali-spoke-two patch configurationpolicy "${policy}" \
      -n "${SPOKE_TWO_NS}" \
      --type=merge -p '{"metadata":{"finalizers":[]}}' 2>/dev/null || true
  done
fi

PROM_RULES=$(oc --context=ossm-kiali-spoke-two get prometheusrule \
  -n openshift-monitoring -o name 2>/dev/null | grep '/acm-rs-' || true)
[ -n "${PROM_RULES}" ] && echo "${PROM_RULES}" | xargs oc --context=ossm-kiali-spoke-two \
  -n openshift-monitoring delete --ignore-not-found

RBAC=$(oc --context=ossm-kiali-spoke-two get clusterrole,clusterrolebinding -o name 2>/dev/null | \
  grep -E '/(ocm:|.*open-cluster-management|.*multicluster-observability|.*observability.*mco)' || true)
[ -n "${RBAC}" ] && echo "${RBAC}" | xargs oc --context=ossm-kiali-spoke-two delete --ignore-not-found

WEBHOOKS=$(oc --context=ossm-kiali-spoke-two get \
  validatingwebhookconfiguration,mutatingwebhookconfiguration -o name 2>/dev/null | \
  grep -E '/.*(open-cluster-management|multicluster|observability)' || true)
[ -n "${WEBHOOKS}" ] && echo "${WEBHOOKS}" | xargs oc --context=ossm-kiali-spoke-two delete --ignore-not-found

# APIServices: ACM/MCE + Observatorium; Hive only when no live workloads
APIS=$(oc --context=ossm-kiali-spoke-two get apiservice -o name 2>/dev/null | \
  grep -E '\.(open-cluster-management\.io|multicluster\.openshift\.io|multicluster\.x-k8s\.io|observatorium\.io)$' || true)
[ -n "${APIS}" ] && echo "${APIS}" | xargs oc --context=ossm-kiali-spoke-two delete --ignore-not-found
HIVE_WORKLOADS=$(oc --context=ossm-kiali-spoke-two get deploy,statefulset,daemonset,pod \
  -n hive -o name 2>/dev/null || true)
if ! oc --context=ossm-kiali-spoke-two get namespace hive &>/dev/null || \
   [ -z "${HIVE_WORKLOADS}" ]; then
  HIVE_APIS=$(oc --context=ossm-kiali-spoke-two get apiservice -o name 2>/dev/null | \
    grep -E '\.(hive\.openshift\.io|hiveinternal\.openshift\.io)$' || true)
  [ -n "${HIVE_APIS}" ] && echo "${HIVE_APIS}" | xargs oc --context=ossm-kiali-spoke-two delete --ignore-not-found
fi

# CRDs last — the cleanup above still needs their APIs
ACM_CRDS=$(oc --context=ossm-kiali-spoke-two get crd -o name 2>/dev/null | \
  grep -E '\.(open-cluster-management\.io|multicluster\.openshift\.io|multicluster\.x-k8s\.io|observatorium\.io)$' || true)
[ -n "${ACM_CRDS}" ] && echo "${ACM_CRDS}" | xargs oc --context=ossm-kiali-spoke-two delete --ignore-not-found
HIVE_WORKLOADS=$(oc --context=ossm-kiali-spoke-two get deploy,statefulset,daemonset,pod \
  -n hive -o name 2>/dev/null || true)
if ! oc --context=ossm-kiali-spoke-two get namespace hive &>/dev/null || \
   [ -z "${HIVE_WORKLOADS}" ]; then
  HIVE_CRDS=$(oc --context=ossm-kiali-spoke-two get crd -o name 2>/dev/null | \
    grep -E '\.(hive\.openshift\.io|hiveinternal\.openshift\.io)$' || true)
  [ -n "${HIVE_CRDS}" ] && echo "${HIVE_CRDS}" | xargs oc --context=ossm-kiali-spoke-two delete --ignore-not-found
fi

# Remove UWM config from spoke-two — only if the tutorial created it
# (skip if cluster-monitoring-config was pre-existing or not labeled as tutorial-owned)
OWNED=$(oc --context=ossm-kiali-spoke-two get configmap cluster-monitoring-config \
  -n openshift-monitoring \
  -o jsonpath='{.metadata.labels.kiali\.io/tutorial-owned}' 2>/dev/null || true)
[ "${OWNED}" = "true" ] && oc --context=ossm-kiali-spoke-two delete configmap \
  cluster-monitoring-config -n openshift-monitoring --ignore-not-found

Remove Kiali remote access and endpoint discovery resources from spoke:

# Remove Kiali remote cluster secret
oc --context=ossm-kiali-spoke delete secret kiali-multi-cluster-secret -n istio-system --ignore-not-found

# Remove istio-reader SA — unbind the role first while the SA still exists
oc --context=ossm-kiali-spoke adm policy remove-cluster-role-from-user cluster-reader \
  -z istio-reader-service-account -n istio-system 2>/dev/null || true
oc --context=ossm-kiali-spoke delete serviceaccount istio-reader-service-account -n istio-system --ignore-not-found

Revert spoke Istio, ZTunnel, and Kiali CRs to single-cluster configuration:

oc --context=ossm-kiali-spoke patch istio default --type=json \
  -p '[{"op":"remove","path":"/spec/values/global/multiCluster"},{"op":"remove","path":"/spec/values/global/network"},{"op":"remove","path":"/spec/values/global/meshNetworks"},{"op":"remove","path":"/spec/values/pilot/env"}]'

oc --context=ossm-kiali-spoke patch ztunnel default --type=json \
  -p '[{"op":"remove","path":"/spec/values/ztunnel/multiCluster"},{"op":"remove","path":"/spec/values/ztunnel/network"}]'

# Remove the multi-cluster network and discovery labels from istio-system
oc --context=ossm-kiali-spoke label namespace istio-system \
  topology.istio.io/network- \
  istio-discovery-

# Revert Kiali cluster name to default
oc --context=ossm-kiali-spoke patch kiali kiali -n istio-system --type=json \
  -p '[{"op":"remove","path":"/spec/kubernetes_config/cluster_name"}]'

Notes and Considerations

1. Hub Stays ACM-Only

The hub cluster (ossm-kiali-hub) is not changed by this guide. It remains dedicated to fleet management and centralized Thanos metrics collection. Istio is not installed on the hub. All mesh control planes run on the spoke clusters.

2. Two East-West Gateways: Ambient and Sidecar Use Different Cross-Network Protocols

Cross-cluster traffic uses different protocols depending on the proxy type:

Ambient workloads (ztunnel): use HBONE on port 15008. The istio-east-west gatewayClassName handles this.
Sidecar workloads (envoy sidecar proxy): use mTLS auto-passthrough on port 15443. A standard istio gatewayClassName gateway with protocol: TLS, tls.mode: Passthrough on port 15443 provides this — Istio’s Gateway API implementation automatically generates the required SNI-based passthrough filter chains.

The istio-east-west gatewayClassName only processes HBONE — it does not create a 15443 listener. That is why this guide deploys two east-west gateways per cluster: one for ambient (HBONE, port 15008) and one for sidecar (TLS passthrough, port 15443). Both gateways are registered in meshNetworks so istiod can route each proxy type to the correct gateway.

3. East-West Gateways Require External Load Balancers

Both east-west gateways (istio-eastwestgateway and istio-eastwestgateway-sidecar) create LoadBalancer-type Services. On bare-metal or on-premise OpenShift without cloud load balancers, install and configure the MetalLB Operator to assign external IPs. Alternatively, you can patch the Services to use NodePort and provide the node IP manually — Istio will use spec.externalIPs or the Service status address for routing, so as long as ports 15008 and 15443 are reachable from the remote cluster, either approach works.

4. `istio.io/global=true` Required for Cross-Cluster Service Discovery

In multi-primary multi-network mode, a Service must be labeled istio.io/global=true on both clusters for its endpoints to be discoverable across the mesh. This applies to both ambient mode services (such as helloworld in ambient-demo) and sidecar mode services (such as ratings in bookinfo). Without this label on a given cluster, the remote cluster’s istiod will not import that cluster’s endpoints and cross-cluster traffic will not be routed to it.

5. Cluster Naming Must Be Consistent Across All Components

In a multi-primary setup, the same cluster name must be used in every place that references cluster identity. A mismatch between any of these will cause failures ranging from Kiali redirect loops to Istio endpoint discovery silently not working.

The table below shows where each cluster’s name appears. Each cluster (spoke and spoke-two) has its own name applied in all the same locations:

Component	Field / location	Cluster name value
ACM `ManagedCluster` resource	`metadata.name` on the hub	`spoke` or `spoke-two`
Istio CR	`spec.values.global.multiCluster.clusterName`	`spoke` or `spoke-two`
ZTunnel CR	`spec.values.ztunnel.multiCluster.clusterName`	must match the Istio CR on the same cluster
Istio CR	`spec.values.global.meshNetworks.<network>.endpoints[].fromRegistry`	`spoke` and `spoke-two` (one per network entry)
Istio remote secrets	`--name=` passed to `istioctl create-remote-secret`	name of the remote cluster being described
Kiali CR (home cluster)	`spec.kubernetes_config.cluster_name`	`spoke` (spoke's own name)
Kiali CR (remote cluster)	`spec.auth.openshift.redirect_uris` — the `/api/auth/callback/<name>` path segment used by the non-impersonation per-cluster OAuth flow	`spoke-two`
`kiali-multi-cluster-secret` key	secret key name in the secret on spoke	`spoke-two` (the remote cluster being added)
OTEL collector `resource` processor (tracing only)	`k8s.cluster.name` attribute stamped on spans — only required if setting up distributed tracing (see the Dashboards and Tracing guide)	`spoke` or `spoke-two`

Critical rule for Istio and ZTunnel CRs: clusterName must be set in both the Istio CR and the ZTunnel CR with the same value on each cluster. istiod uses the Istio CR value to register the local cluster identity; ztunnel uses the ZTunnel CR value when authenticating to istiod. If they differ, istiod rejects ztunnel connections with:

client claims to be in cluster "spoke-two", but we only know about local cluster "Kubernetes"

The same error appears for sidecar proxies when ISTIO_META_CLUSTER_ID does not match istiod’s local cluster name. The sidecar injector template sets that env var from global.multiCluster.clusterName and defaults to Kubernetes when the value is missing. After you patch clusterName on an existing mesh (Phase 4.2), wait until istio-sidecar-injector’s .data.values shows the new name before restarting sidecar-injected workloads (Phase 4.4). Istio Ready alone is not enough — Sail updates the injector ConfigMap asynchronously. Restarting too early permanently bakes Kubernetes into the pod spec until another restart runs against the updated injector. Confirm a sample pod’s istio-proxy env shows ISTIO_META_CLUSTER_ID=<your-cluster-name> (for example spoke), not Kubernetes.

6. Kiali OAuth Redirect / Optional Kiali Impersonation Mode

Impersonation is disabled by default in this tutorial.

When not using impersonation, logging into spoke-two through Kiali’s multi-cluster UI will cause Kiali to redirect to spoke-two’s OpenShift OAuth endpoint. The redirect URI must therefore be reachable from the user’s browser. If spoke-two’s OAuth route is on a different domain, ensure the redirect back to the Kiali URL is reachable.

If your installed Kiali version supports OpenShift impersonation and you want users to only need to authenticate once to the home cluster, set spec.auth.openshift.impersonation.enabled to true on both Kiali CRs. Kiali then accesses remote clusters on the user’s behalf, subject to the user’s RBAC permissions. Keep the remote redirect_uris entry as a fallback if you later disable impersonation, but it is not needed when impersonation is enabled. See the Kiali impersonation documentation for configuration and security considerations.

Docs: OpenID Connect strategy

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

The openid authentication strategy lets you integrate Kiali to an external identity provider that implements OpenID Connect, and allows users to login to Kiali using their existing accounts of a third-party system.

If your Kubernetes cluster is also integrated with your OpenId provider, then Kiali’s openid strategy can offer namespace access control.

Kiali only supports the authorization code flow of the OpenId Connect spec.

Requirements

The Kiali’s signing key needs to be 16, 24 or 32 byte long. If you install Kiali via the operator and don’t set a custom signing key, the operator should create a 16 byte long signing key.

If you don’t need namespace access control support, you can use any working OpenId Server where Kiali can be configured as a client application.

If you do need namespace access control support, you need either:

A Kubernetes cluster configured with OpenID connect integration, which results in the API server accepting tokens issued by your identity provider.
A replacement or reverse proxy for the Kubernetes cluster API capable of handling the OIDC authentication.

The first option is preferred if you can manipulate your cluster API server startup flags, which will result in your cluster to also be integrated with the external OpenID provider.

The second option is provided for cases where you are using a managed Kubernetes and your cloud provider does not support configuring OpenID integration. Kiali assumes an implementation of a Kubernetes API server. For example, a community user has reported to successfully configure Kiali’s OpenID strategy by using kube-oidc-proxy which is a reverse proxy that handles the OpenID authentication and forwards the authenticated requests to the Kubernetes API.

Set-up with namespace access control support

Assuming you already have a working Kubernetes cluster with OpenId integration (or a working alternative like kube-oidc-proxy), you should already had configured an application or a client in your OpenId server (some cloud providers configure this app/client automatically for you). You must re-use this existing application/client by adding the root path of your Kiali instance as an allowed/authorized callback URL. If the OpenID server provided you a client secret for the application/client, or if you had manually set a client secret, issue the following command to create a Kubernetes secret holding the OpenId client secret:

kubectl create secret generic kiali --from-literal="oidc-secret=$CLIENT_SECRET" -n $NAMESPACE

where $NAMESPACE is the namespace where you installed Kiali and $CLIENT_SECRET is the secret you configured or provided by your OpenId Server. If Kiali is already running, you may need to restart the Kiali pod so that the secret is mounted in Kiali.

It’s worth emphasizing that to configure OpenID integration you must re-use the OpenID application/client that you created for your Kubernetes cluster. If you create a new application/client for Kiali in your OpenId server, Kiali will fail to properly authenticate users.

Then, to enable the OpenID Connect strategy, the minimal configuration you need to set in the Kiali CR is like the following:

spec:
  auth:
    strategy: openid
    openid:
      client_id: "kiali-client"
      issuer_uri: "https://openid.issuer.com"

This assumes that your Kubernetes cluster is configured with OpenID Connect integration. In this case, the client-id and issuer_uri attributes must match the --oidc-client-id and --oidc-issuer-url flags used to start the cluster API server. If these values don’t match, users will fail to login to Kiali.

If you are using a replacement or a reverse proxy for the Kubernetes API server, the minimal configuration is like the following:

spec:
  auth:
    strategy: openid
    openid:
      api_proxy: "https://proxy.domain.com:port"
      api_proxy_ca_data: "..."
      client_id: "kiali-client"
      issuer_uri: "https://openid.issuer.com"

The value of client-id and issuer_uri must match the values of the configuration of your reverse proxy or cluster API replacement. The api_proxy attribute is the URI of the reverse proxy or cluster API replacement (only HTTPS is allowed). The api_proxy_ca_data is the public certificate authority file encoded in a base64 string, to trust the secure connection.

Set-up with no namespace access control support

Register Kiali as a client application in your OpenId Server. Use the root path of your Kiali instance as the callback URL. If the OpenId Server provides you a client secret, or if you manually set a client secret, issue the following command to create a Kubernetes secret holding the OpenId client secret:

kubectl create secret generic kiali --from-literal="oidc-secret=$CLIENT_SECRET" -n $NAMESPACE

Then, to enable the OpenID Connect strategy, the minimal configuration you need to set in the Kiali CR is like the following:

spec:
  auth:
    strategy: openid
    openid:
      client_id: "kiali-client"
      disable_rbac: true
      issuer_uri: "https://openid.issuer.com"

As namespace access control is disabled, all users logging into Kiali will share the same cluster-wide privileges.

Additional configurations

Configuring the displayed user name

The Kiali front-end will, by default, retrieve the string of the sub claim of the OpenID token and display it as the user name. You can customize which field to display as the user name by setting the username_claim attribute of the Kiali CR. For example:

spec:
  auth:
    openid:
      username_claim: "email"

If you enabled namespace access control, you will want the username_claim attribute to match the --oidc-username-claim flag used to start the Kubernetes API server, or the equivalent option if you are using a replacement or reverse proxy of the API server. Else, any user-friendly claim will be OK as it is purely informational.

Configuring requested scopes

By default, Kiali will request access to the openid, profile and email standard scopes. If you need a different set of scopes, you can set the scopes attribute in the Kiali CR. For example:

spec:
  auth:
    openid:
      scopes:
      - "openid"
      - "email"
      - "groups"

The openid scope is forced. If you don’t add it to the list of scopes to request, Kiali will still request it from the identity provider.

Configuring authentication timeout

When the user is redirected to the external authentication system, by default Kiali will wait at most 5 minutes for the user to authenticate. After that time has elapsed, Kiali will reject authentication. You can adjust this timeout by setting the authentication_timeout with the number of seconds that Kiali should wait at most. For example:

spec:
  auth:
    openid:
      authentication_timeout: 60 # Wait only one minute.

Configuring allowed domains

Some identity providers use a shared login and regardless of configuring your own application under your domain (or organization account), login can succeed even if the user that is logging in does not belong to your account or organization. Google is an example of this kind of provider.

To prevent foreign users from logging into your Kiali instance, you can configure a list of allowed domains:

spec:
  auth:
    openid:
      allowed_domains:
      - example.com
      - foo.com

The e-mail reported by the identity provider is used for the validation. Login will be allowed if the domain part of the e-mail is listed as an allowed domain; else, the user will be rejected. Naturally, you will need to configure the email scope to be requested.

There is a special case: some identity providers include a hd claim in the id_token. If this claim is present, this is used instead of extracting the domain from the user e-mail. For example, Google Workspace (aka G Suite) includes this hd claim for hosted domains.

Using an OpenID provider with a self-signed certificate

If your OpenID provider is using a self-signed certificate, you can disable certificate validation by setting the insecure_skip_verify_tls to true in the Kiali CR:

spec:
  auth:
    openid:
      insecure_skip_verify_tls: true

You should use self-signed certificates only for testing purposes.

However, if your organization or internal network has an internal trusted certificate authority (CA), and your OpenID server is using a certificate issued by this CA, you can configure Kiali to trust certificates from this CA rather than disabling verification.

See the TLS Configuration page for detailed instructions on configuring custom CA certificates. You can use either the global additional-ca-bundle.pem key (which makes the CA trusted for all HTTPS connections) or the OpenID-specific openid-server-ca.crt key in the kiali-cabundle ConfigMap.

Using an HTTP/HTTPS Proxy

In some network configurations, there is the need to use proxies to connect to the outside world. OpenID requires outside world connections to get metadata and do key validation, so you can configure it by setting the http_proxy and https_proxy keys in the Kiali CR. They use the same format as the HTTP_PROXY and HTTPS_PROXY environment variables.

spec:
  auth:
    openid:
      http_proxy: http://USERNAME:PASSWORD@10.0.1.1:8080/
      https_proxy: https://USERNAME:PASSWORD@10.0.0.1:8080/

Passing additional options to the identity provider

When users click on the Login button on Kiali, a redirection occurs to the authentication page of the external identity provider. Kiali sends a fixed set of parameters to the identity provider to enable authentication. If you need to add an additional set of parameters to your identity provider, you can use the additional_request_params setting of the Kiali CR, which accepts key-value pairs. For example:

spec:
  auth:
    openid:
      additional_request_params:
        prompt: login

The prompt parameter is a standard OpenID parameter. When the login value is passed in this parameter, the identity provider is instructed to ask for user credentials regardless if the user already has an active session because of a previous login in some other system.

If your OpenId provider supports other non-standard parameters, you can specify the ones you need in this additional_request_params setting.

Take into account that you should not add the client_id, response_type, redirect_uri, scope, nonce nor state parameters to this list. These are already in use by Kiali and some already have a dedicated setting.

Provider-specific instructions

Using with Keycloak

When using OpenId with Keycloak, you will need to enable the Standard Flow Enabled option on the Client (in the Administration Console):

The Standard Flow described on the options is the same as the authorization code flow from the rest of the documentation.

Using with Google Cloud Platform / GKE OAuth2

If you are using Google Cloud Platform (GCP) and its products such as Google Kubernetes Engine (GKE), it should be straightforward to configure Kiali’s OpenID strategy to authenticate using your Google credentials.

First, you’ll need to go to your GCP Project and to the Credentials screen which is available at (Menu Icon) > APIs & Services > Credentials.

On the Credentials screen you can select to create a new OAuth client ID.

If you’ve never setup the OAuth consent screen you will need to do that before you can create an OAuth client ID. On screen you’ll have multiple warnings and prompts to walk you through this.

On the Create OAuth client ID screen, set the Application type to Web Application and enter a name for your key.

Then enter in the Authorized Javascript origins and Authorized redirect URIs for your project. You can enter in localhost as appropriate during testing. You can also enter multiple URIs as appropriate.

After clicking Create you’ll be shown your newly minted client id and secret. These are important and needed for your Kiali CR yaml and Kiali secrets files.

You’ll need to update your Kiali CR file to include the following auth block.

spec:
  auth:
    strategy: "openid"
    openid:
      client_id: "<your client id from GCP>"
      disable_rbac: true
      issuer_uri: "https://accounts.google.com"
      scopes: ["openid", "email"]
      username_claim: "email"

Don’t get creative here. The issuer_uri should be https://accounts.google.com.

Finally you will need to create a secret, if you don’t have one already, that sets the oidc-secret for the openid flow.

apiVersion: v1
kind: Secret
metadata:
  name: kiali
  namespace: istio-system
  labels:
    app: kiali
type: Opaque
data:
  oidc-secret: "<base64 encode your client secret from GCP and enter here>"

Once all these settings are complete just set your Kiali CR and the Kiali secret to your cluster. You may need to refresh your Kiali Pod to set the Secret if you add the Secret after the Kiali pod is created.

Using with OpenShift and an external OIDC provider

Starting with OpenShift 4.20, you can configure your OpenShift cluster to authenticate users against an external OpenID Connect (OIDC) provider instead of using the built-in OAuth server. This is sometimes called “Bring Your Own OIDC” (BYO OIDC). When OpenShift is configured this way, Kiali can use the openid authentication strategy with full namespace access control support.

This section applies when you want to use an external OIDC provider (such as Keycloak, Okta, Auth0, or others) with OpenShift. If you want to use OpenShift’s built-in OAuth server, use the openshift authentication strategy instead.

Prerequisites

OpenShift 4.20 or later
An external OIDC provider configured and accessible from your OpenShift cluster
The OIDC provider must be configured as an authentication source for both OpenShift and Kiali (they share the same provider)
A certificate-based kubeconfig or long-lived service account token for emergency cluster access (the built-in OAuth will be disabled)

Step 1: Configure OpenShift for external OIDC authentication

First, configure your OpenShift cluster to use your external OIDC provider. This involves modifying the cluster’s Authentication resource to specify your OIDC provider details.

Refer to the official OpenShift documentation for detailed instructions: Enabling direct authentication with an external OIDC identity provider.

The key configuration elements include:

Issuer URL: The URL of your OIDC provider
Client ID: The OAuth2 client ID registered with your OIDC provider
Audiences: The list of acceptable audiences for tokens (must include your Kiali client ID)
Username claim mapping: How the OIDC token claims map to Kubernetes usernames
Username prefix: Optional prefix to distinguish OIDC users (e.g., oidc:)
CA certificate: If your OIDC provider uses a private CA
webhookTokenAuthenticator: null: Must be set when type is OIDC

When OpenShift is configured with external OIDC authentication, the built-in OAuth server is disabled. This means:

Users cannot log in using OpenShift’s standard login page
The OAuthClient API becomes unavailable
Keep a certificate-based kubeconfig (or other long-lived admin credentials) available for emergency access because the normal OAuth login paths and OAuth APIs are unavailable in this mode

Ensure your OIDC provider is properly configured and you have set up RBAC policies before enabling external OIDC authentication.

Step 2: Configure user RBAC

When using external OIDC with OpenShift, user identities in Kubernetes are derived from the OIDC token claims. OpenShift typically adds a configurable prefix to the username (e.g., oidc:) to distinguish OIDC users from other identity sources.

For example, if your OIDC provider returns user@example.com in the email claim and you configured a prefix of oidc:, the Kubernetes username becomes oidc:user@example.com.

Create RBAC resources to grant users access to the namespaces they need. See Namespace access control for details on the required privileges.

Example Role and RoleBinding to grant a user access to the istio-system namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kiali-user-access
  namespace: istio-system
rules:
- apiGroups: [""]
  resources:
  - namespaces
  - pods/log
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kiali-user-access
  namespace: istio-system
subjects:
- kind: User
  name: "oidc:user@example.com"  # Use the prefixed username
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: kiali-user-access
  apiGroup: rbac.authorization.k8s.io

The username prefix (e.g., oidc:) is configured in OpenShift’s Authentication resource under spec.oidcProviders[].claimMappings.username.prefix.prefixString. Make sure your RBAC resources use the same prefixed username format.

In multi-tenant environments where users are granted broad LIST permission on namespaces but should only see specific namespaces, you can enable kiali_feature_flags.authz.require_namespace_get to enforce per-namespace GET checks. This ensures that LIST alone is not sufficient for a namespace to be visible.

Step 3: Create the OIDC client secret

If your OIDC provider requires a client secret, create a Kubernetes secret to store it:

oc create secret generic kiali --from-literal="oidc-secret=$CLIENT_SECRET" -n istio-system

Replace $CLIENT_SECRET with the client secret from your OIDC provider.

Step 4: Configure the CA certificate (if needed)

If your OIDC provider uses a certificate issued by a private CA (not a public CA), you need to configure Kiali to trust it. Create a ConfigMap with the CA certificate:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kiali-cabundle
  namespace: istio-system
data:
  openid-server-ca.crt: |
    -----BEGIN CERTIFICATE-----
    MIIDxTCCAq2gAwIBAgIQAqxcJmoLQ...
    ... (your OIDC provider's CA certificate) ...
    -----END CERTIFICATE-----

See TLS Configuration for more details on configuring custom CA certificates.

Step 5: Configure the Kiali CR

Configure Kiali to use the openid authentication strategy. The client_id and issuer_uri must match the values configured in OpenShift’s Authentication resource:

spec:
  auth:
    strategy: openid
    openid:
      client_id: "kiali-client"
      issuer_uri: "https://your-oidc-provider.example.com"
      scopes:
      - "openid"
      - "email"
      username_claim: "email"

The client_id must be listed in the audiences array of your OpenShift OIDC configuration, and the issuer_uri must exactly match the issuer URL configured in OpenShift. If these don’t match, authentication will fail.

Important configuration notes:

username_claim: Should match the claim mapping configured in OpenShift (commonly email or preferred_username)
scopes: Request the scopes that provide the claims you need (typically openid and email)
disable_rbac: Do not set this to true if you want per-user namespace access control. When disable_rbac is false (the default), Kiali uses the user’s OIDC token for Kubernetes API calls, enabling per-user RBAC.

Complete example

Here’s a complete example of the Kiali CR configuration for OpenShift with an external OIDC provider:

apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
  name: kiali
  namespace: istio-system
spec:
  auth:
    strategy: openid
    openid:
      client_id: "kiali-client"
      issuer_uri: "https://your-oidc-provider.example.com"
      scopes:
      - "openid"
      - "email"
      username_claim: "email"

With the supporting resources:

# OIDC client secret (if required by your provider)
apiVersion: v1
kind: Secret
metadata:
  name: kiali
  namespace: istio-system
type: Opaque
stringData:
  oidc-secret: "your-client-secret-here"
---
# CA certificate (if using a private CA)
apiVersion: v1
kind: ConfigMap
metadata:
  name: kiali-cabundle
  namespace: istio-system
data:
  openid-server-ca.crt: |
    -----BEGIN CERTIFICATE-----
    ... (your CA certificate) ...
    -----END CERTIFICATE-----    
---
# RBAC for a user (repeat for each user/namespace combination)
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kiali-user-access
  namespace: istio-system
rules:
- apiGroups: [""]
  resources:
  - namespaces
  - pods/log
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kiali-user-access
  namespace: istio-system
subjects:
- kind: User
  name: "oidc:user@example.com"
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: kiali-user-access
  apiGroup: rbac.authorization.k8s.io

Using with Azure: AKS and AAD

The OpenID authentication strategy can be used with Azure Kubernetes Service (AKS) and Azure Active Directory (AAD) with Kiali versions 1.33 and later. Prior Kiali versions do not support namespace access control on Azure.

AKS has support for a feature named AKS-managed Azure Active Directory, which enables integration between AKS and AAD. This has the advantage that users can use their AAD credentials to access AKS clusters and can also use Kubernetes RBAC features to assign privileges to AAD users.

However, Azure is implementing this integration via the Kubernetes Webhook Token Authentication rather than via the Kubernetes OpenID Connect Tokens authentication (see the Azure AD integration section in AKS Concepts documentation). Because of this difference, authentication in AKS behaves slightly different from a standard OpenID setup, but Kiali’s OpenID authentication strategy can still be used with namespace access control support by following the next steps.

First, enable the AAD integration on your AKS cluster. See the official AKS documentation to learn how. Once it is enabled, your AKS panel should show the following:

Create a web application for Kiali in your Azure AD panel:

Go to AAD > App Registration, create an application with a redirect url like https://<your-kiali-url>/kiali
Go to Certificates & secrets and create a client secret.
1. After creating the client secret, take note of the provided secret. Create a Kubernetes secret in your cluster as mentioned in the Set-up with namespace access control support section. Please, note that the suggested name for the Kubernetes Secret is kiali. If you want to customize the secret name, you will have to specify your custom name in the Kiali CR. See: secret_name in Kial CR Reference.
Go to API Permissions and press the Add a permission button. In the new page that appears, switch to the APIs my organization uses tab.
Type the following ID in the search field: 6dae42f8-4368-4678-94ff-3960e28e3630 (this is a shared ID for all Azure clusters). And select the resulting entry.
Select the Delegated permissions square.
Select the user.read permission.
Go to Authentication and make sure that the Access tokens checkbox is ticked.

Then, create or modify your Kiali CR and include the following settings:

spec:
  auth:
    strategy: "openid"
    openid:
      client_id: "<your Kiali application client id from Azure>"
      issuer_uri: "https://sts.windows.net/<your AAD tenant id>/"
      username_claim: preferred_username
      api_token: access_token
      additional_request_params:
        resource: "6dae42f8-4368-4678-94ff-3960e28e3630"

You can find your client_id and tenant_id in the Overview page of the Kiali App registration that you just created. See this documentation for more information.

Docs: TLS Configuration

Mon, 01 Jan 0001 00:00:00 +0000

Overview

When Kiali connects to external services (Prometheus, Grafana, Jaeger/Tempo, Perses) over HTTPS, it needs to verify the TLS certificates presented by those services. By default, Kiali trusts the system certificate authorities (CAs) that are built into the container image.

If your external services use certificates issued by a private CA (such as an internal corporate CA, a service mesh CA, or self-signed certificates), you need to configure Kiali to trust those additional CAs.

Adding Custom Certificate Authorities

Kiali uses a global CA bundle mechanism to trust additional certificate authorities. All custom CAs are added to a single certificate pool that applies to all HTTPS connections Kiali makes to external services.

On Kubernetes

To add custom CAs, create a ConfigMap named <kiali-instance-name>-cabundle in the Kiali namespace. The default instance name is kiali, so the ConfigMap would be named kiali-cabundle:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kiali-cabundle
  namespace: istio-system  # Or your Kiali namespace
data:
  additional-ca-bundle.pem: |
    -----BEGIN CERTIFICATE-----
    MIIDxTCCAq2gAwIBAgIQAqxcJmoLQ...
    ... (your CA certificate) ...
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    MIIDyTCCArGgAwIBAgIRAJ4K...
    ... (additional CA certificates if needed) ...
    -----END CERTIFICATE-----

Key name: The key must be additional-ca-bundle.pem. You can include multiple CA certificates in PEM format in the same file.

Alternative keys: You can also use openid-server-ca.crt or (on OpenShift) oauth-server-ca.crt as key names. While these names suggest specific purposes, all CAs are loaded into Kiali’s global certificate pool and trusted for all TLS connections. Using additional-ca-bundle.pem is recommended for clarity.

For OpenShift OAuth authentication: On OpenShift, you can alternatively create a separate ConfigMap named <instance-name>-oauth-cabundle with the key oauth-server-ca.crt. See the OpenShift authentication documentation for details. However, adding your CA to kiali-cabundle under additional-ca-bundle.pem achieves the same result.

On OpenShift

On OpenShift, the Kiali Operator automatically creates a ConfigMap named <kiali-instance-name>-cabundle-openshift (e.g., kiali-cabundle-openshift) with the annotation service.beta.openshift.io/inject-cabundle: "true". This tells OpenShift to automatically inject the cluster’s service CA into the ConfigMap.

This means that by default, Kiali on OpenShift already trusts:

The system CAs
The OpenShift service CA (used by services with serving certificates)

If you need to add additional CAs beyond the OpenShift service CA, create a separate ConfigMap named <kiali-instance-name>-cabundle (e.g., kiali-cabundle):

apiVersion: v1
kind: ConfigMap
metadata:
  name: kiali-cabundle
  namespace: istio-system  # Or your Kiali namespace
data:
  additional-ca-bundle.pem: |
    -----BEGIN CERTIFICATE-----
    MIIDxTCCAq2gAwIBAgIQAqxcJmoLQ...
    ... (your CA certificate) ...
    -----END CERTIFICATE-----

The operator uses a projected volume that automatically combines both ConfigMaps, so your custom CAs work alongside the OpenShift service CA.

How It Works

When Kiali starts, it loads certificates from:

System certificate pool: The default trusted CAs from the container’s operating system
Additional CA bundle: Certificates from /kiali-cabundle/additional-ca-bundle.pem (if present)
OpenShift service CA (OpenShift only): Certificates from /kiali-cabundle/service-ca.crt (automatically injected from the <instance-name>-cabundle-openshift ConfigMap)
OpenID server CA (OpenID auth only): Certificates from /kiali-cabundle/openid-server-ca.crt (if present)
OAuth CA bundle (OpenShift with OAuth auth): Certificates from /kiali-cabundle/oauth-server-ca.crt (if the <instance-name>-oauth-cabundle ConfigMap exists)

All these certificates are combined into a single certificate pool used for all HTTPS connections to external services.

On OpenShift: The operator uses a projected volume that automatically combines multiple ConfigMap sources (<instance-name>-cabundle-openshift, <instance-name>-cabundle, and <instance-name>-oauth-cabundle) into the /kiali-cabundle mount path. This means you don’t need to manually merge ConfigMaps - each ConfigMap can be managed independently.

Automatic refresh: Kiali watches CA bundle files for changes using filesystem notifications (fsnotify) and automatically refreshes the certificate pool without requiring a pod restart. When you update the ConfigMap, Kubernetes propagates the changes to the mounted volume based on the kubelet’s sync interval (default: 60 seconds). Once the files are updated on disk, Kiali detects and applies them immediately. Total propagation time is typically 0-90 seconds after the ConfigMap update.

Skipping Certificate Verification

If you need to temporarily skip certificate verification (for testing purposes only), you can set insecure_skip_verify: true in the authentication configuration for each external service:

spec:
  external_services:
    prometheus:
      auth:
        insecure_skip_verify: true
    grafana:
      auth:
        insecure_skip_verify: true
    tracing:
      auth:
        insecure_skip_verify: true

Security warning: Disabling certificate verification makes Kiali vulnerable to man-in-the-middle attacks. Only use this option for testing purposes, never in production.

Common Scenarios

Internal Corporate CA

If your organization has an internal CA that issues certificates for internal services:

Obtain the root CA certificate (public part only) from your security team
Create the ConfigMap with the CA certificate as shown above

Self-Signed Certificates

For development or testing environments using self-signed certificates:

Export the certificate from your service (usually the same certificate that was generated)
Create the ConfigMap with that certificate

Istio Service Mesh mTLS

If your external services are part of the Istio service mesh and use Istio’s mTLS:

Kiali typically accesses these services through their Kubernetes service names, which may bypass the sidecar
If you need to go through the mesh, you may need to add Istio’s root CA to the bundle

cert-manager Issued Certificates

If you use cert-manager with a private CA:

The CA certificate is typically stored in a Secret (e.g., my-ca-secret with key ca.crt)
Extract the CA and add it to the ConfigMap:

kubectl get secret my-ca-secret -n cert-manager -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt
kubectl create configmap kiali-cabundle -n istio-system --from-file=additional-ca-bundle.pem=ca.crt

Troubleshooting

Certificate Errors in Logs

If you see errors like x509: certificate signed by unknown authority in Kiali logs:

Verify the ConfigMap exists and has the correct name
Check that the key is exactly additional-ca-bundle.pem
Ensure the certificate is in valid PEM format
Verify the CA certificate is the correct one (the root or intermediate CA that signed the service’s certificate)

Verifying the ConfigMap is Mounted

Check that the ConfigMap is properly mounted in the Kiali pod:

kubectl exec -n istio-system deploy/kiali -- ls -la /kiali-cabundle/

You should see your CA bundle file listed.

Testing Certificate Chain

To verify your CA certificate is correct, you can test it outside of Kiali:

# Get the server's certificate chain
openssl s_client -connect prometheus.istio-system:9090 -showcerts

# Verify against your CA
openssl verify -CAfile your-ca.pem server-cert.pem

Docs: Dashboards and Tracing

Mon, 01 Jan 0001 00:00:00 +0000

This guide extends the result of the Multi-Primary Mesh guide by adding two observability tools that operate across both spoke clusters. Both tools are delivered through the Red Hat Cluster Observability Operator (COO), which manages their deployment and integrates them into the OpenShift console:

Phase 1 — Perses: Deploys a Perses metrics dashboard server and OpenShift console plugin via COO. The datasource is pointed at hub Thanos so it queries aggregated metrics from both spokes, giving a true multi-cluster view.
Phase 2 — Tempo: Deploys a distributed tracing server and OpenShift console plugin via COO. A central Tempo instance on spoke collects traces from both clusters via OpenTelemetry collectors. Kiali links to the console tracing UI and can filter traces by cluster. Neither ACM Observability nor any other component already in place aggregates traces — that requires an explicit OTEL pipeline, which this guide sets up.

The diagram below shows the environment after this guide completes. Components from Guides 1–2 are marked prior; badges such as G3:P2.4-2.5 mark which guide section(s) add each piece (Guide 3, §§2.4–2.5). Click the diagram to open a full-size SVG in a new tab.

Prerequisites

The Multi-Primary Mesh guide must be complete:

OpenShift 4.19+ on all clusters used in the series
ACM hub with both spoke and spoke-two managed clusters
OSSM 3 running on both spokes with East-West gateways
Kiali running on spoke
ACM Observability (Thanos) running on the hub
Demo apps (ambient-demo, bookinfo) deployed on both spokes

Environment Setup

These variables extend the multi-primary environment. Re-export all variables from that guide first, then add:

# Namespace where the TempoStack and SeaweedFS are deployed
export TEMPO_NAMESPACE="tempo"

# Development credentials for the in-cluster SeaweedFS S3 endpoint
export SEAWEEDFS_ACCESS_KEY="seaweedfs"
export SEAWEEDFS_SECRET_KEY="seaweedfs123"

# Name of the TempoStack CR — becomes part of service names
# (e.g. tempo-${TEMPO_STACK_NAME}-gateway). Use one stack per independent
# Tempo deployment; if you run multiple meshes, each mesh can share this
# stack by using a separate tenant.
export TEMPO_STACK_NAME="istio"

# Tempo tenant name — a logical partition within the TempoStack that
# isolates one mesh's traces from another's. Matches MESH_ID from the
# hub/spoke guide so the tenant name reflects the mesh it belongs to.
# If you add a second mesh later, create a second tenant rather than
# a second TempoStack.
export TEMPO_TENANT="mesh1"

Phase 1: Perses — Multi-Cluster Metrics Dashboards

Perses is a metrics dashboard platform. On OpenShift it is managed by the Cluster Observability Operator (COO), which provides native CRDs (Perses, PersesDatasource, PersesDashboard) and integrates with the OpenShift console as a UI plugin. Kiali links to Perses dashboards from its workload and service metrics pages.

The key multi-cluster aspect: Perses is configured with a datasource pointing at hub Thanos (the ACM Observatorium endpoint) rather than a local Prometheus. This gives Perses visibility into metrics from both spoke clusters without any additional aggregation work.

1.1 Install or Verify the Cluster Observability Operator

MCOA installs its own managed-cluster Prometheus component for metric federation; that component is not the full Cluster Observability Operator. Perses requires the full COO product, so install it here unless a COO Subscription is already present.

Check whether a full COO Subscription is already installed. Do not use the ScrapeConfig CRD for this check because MCOA also installs that API:

COO_SUB=$(oc --context=ossm-kiali-spoke get subscriptions.operators.coreos.com -A -o json 2>/dev/null | \
  jq -r '[.items[] | select(.spec.name == "cluster-observability-operator")][0] |
    select(. != null) | "\(.metadata.namespace)/\(.metadata.name)"')
[ -n "${COO_SUB}" ] && echo "COO Subscription found: ${COO_SUB}" || \
  echo "COO Subscription is not found"

If COO is not yet installed, install it:

oc --context=ossm-kiali-spoke apply -f - <<'EOF'
apiVersion: v1
kind: Namespace
metadata:
  labels:
    openshift.io/cluster-monitoring: "true"
  name: openshift-cluster-observability-operator
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: cluster-observability-operator
  namespace: openshift-cluster-observability-operator
spec: {}
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: cluster-observability-operator
  namespace: openshift-cluster-observability-operator
spec:
  channel: stable
  installPlanApproval: Automatic
  name: cluster-observability-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
EOF

Wait for COO and its CRDs to be ready (skip if COO was already installed):

until oc --context=ossm-kiali-spoke get crd perses.perses.dev &>/dev/null; do
  echo "Waiting for Perses CRDs..."
  sleep 10
done
echo "COO CRDs ready"

until oc --context=ossm-kiali-spoke get pods \
  -l app.kubernetes.io/name=perses-operator \
  -n openshift-cluster-observability-operator \
  --no-headers 2>/dev/null | grep -q .; do sleep 5; done
oc --context=ossm-kiali-spoke wait pod \
  --for=condition=Ready \
  -l app.kubernetes.io/name=perses-operator \
  -n openshift-cluster-observability-operator \
  --timeout=300s
echo "COO Perses operator ready"

1.2 Create the Monitoring UIPlugin to Deploy the Perses Server

The Perses server is deployed by a UIPlugin CR with spec.monitoring.perses.enabled: true. This also adds the Observe > Dashboards (Perses) menu to the OpenShift console. The Perses CR that COO creates under the covers is a configuration holder — the UIPlugin is what triggers the actual Perses server pod:

oc --context=ossm-kiali-spoke apply -f - <<'EOF'
apiVersion: observability.openshift.io/v1alpha1
kind: UIPlugin
metadata:
  name: monitoring
spec:
  type: Monitoring
  monitoring:
    perses:
      enabled: true
EOF

Wait for the Perses server pod. The COO creates it as a StatefulSet in the openshift-cluster-observability-operator namespace:

until oc --context=ossm-kiali-spoke get pods \
  -l app.kubernetes.io/name=perses \
  -n openshift-cluster-observability-operator \
  --no-headers 2>/dev/null | grep -q .; do
  echo "Waiting for Perses pod..."
  sleep 5
done
oc --context=ossm-kiali-spoke wait pod \
  --for=condition=Ready \
  -l app.kubernetes.io/name=perses \
  -n openshift-cluster-observability-operator \
  --timeout=300s
echo "Perses ready"

1.3 Create the Hub Thanos Datasource

The Observatorium API on the hub requires mutual TLS (mTLS): Perses must verify the server’s certificate (using a custom ACM CA, not the OCP service CA) and also present a client certificate to authenticate. The Perses server handles this through two mechanisms:

Server CA — mounted into /etc/ssl/certs/ so Go’s system cert pool trusts the hub Thanos route’s certificate
Client cert — the PersesDatasource CR’s spec.client.tls.userCert field tells the Perses operator which K8s Secret holds the client cert. The operator reads it and creates a Perses-internal secret. The spec.config.plugin.spec.proxy.spec.secret field then wires that internal secret to the HTTP proxy transport, completing the mTLS handshake

Get the Observatorium URL:

export OBSERVATORIUM_URL=$(oc --context=ossm-kiali-hub get route observatorium-api \
  -n open-cluster-management-observability \
  -o jsonpath='https://{.spec.host}/api/metrics/v1/default')
echo "Observatorium URL: ${OBSERVATORIUM_URL}"

Two credentials need to be prepared before creating the datasource:

Server CA — the hub Observatorium API route’s TLS certificate is signed by a custom ACM CA (observability-server-ca-certificate), not the standard OCP service CA. Extract it from the hub and store it in a ConfigMap in openshift-cluster-observability-operator on spoke so it can be mounted into the Perses pod.
Client cert — Perses will use the same mTLS client certificate that Kiali uses to authenticate to hub Thanos. That certificate is the acm-observability-certs secret in istio-system on spoke. Copy it to openshift-cluster-observability-operator under a new name so it can be mounted into the Perses pod as a volume (the Perses pod runs in openshift-cluster-observability-operator and can only mount secrets from its own namespace).

# Server CA — extracted from the hub, stored as a ConfigMap on spoke
oc --context=ossm-kiali-hub get secret observability-server-ca-certs \
  -n open-cluster-management-observability \
  -o jsonpath='{.data.ca\.crt}' | base64 -d > /tmp/obs-server-ca.crt

oc --context=ossm-kiali-spoke create configmap perses-acm-server-ca \
  -n openshift-cluster-observability-operator \
  --from-file=ca.crt=/tmp/obs-server-ca.crt \
  --dry-run=client -o yaml | oc --context=ossm-kiali-spoke apply -f -
rm -f /tmp/obs-server-ca.crt

# Client cert — copied from istio-system to openshift-cluster-observability-operator
oc --context=ossm-kiali-spoke get secret acm-observability-certs \
  -n istio-system -o json | \
  jq 'del(.metadata.namespace, .metadata.resourceVersion, .metadata.uid, .metadata.creationTimestamp, .metadata.ownerReferences) | .metadata.namespace = "openshift-cluster-observability-operator" | .metadata.name = "perses-acm-client-certs"' | \
  oc --context=ossm-kiali-spoke apply -f -

Mount the CA ConfigMap into /etc/ssl/certs/ in the Perses pod using the Perses CR’s spec.volumes and spec.volumeMounts. Placing it there adds it to Go’s system certificate pool, so the Perses server trusts the hub Thanos route’s TLS certificate for all outgoing connections:

oc --context=ossm-kiali-spoke patch perses perses \
  -n openshift-cluster-observability-operator --type=merge -p '{
  "spec": {
    "volumes": [
      {
        "name": "acm-server-ca",
        "configMap": {"name": "perses-acm-server-ca"}
      }
    ],
    "volumeMounts": [
      {
        "name": "acm-server-ca",
        "mountPath": "/etc/ssl/certs/acm-thanos-ca.crt",
        "subPath": "ca.crt",
        "readOnly": true
      }
    ]
  }
}'

oc --context=ossm-kiali-spoke wait pod \
  --for=condition=Ready \
  -l app.kubernetes.io/name=perses \
  -n openshift-cluster-observability-operator \
  --timeout=120s

Create the perses namespace if it does not yet exist. This is where PersesDatasource and PersesDashboard CRs live — the Perses Operator maps this namespace to a Perses project, making dashboards available in the console under the perses project:

oc --context=ossm-kiali-spoke create namespace perses

Create the PersesDatasource CR. The client.tls.userCert field references the perses-acm-client-certs secret created above. The Perses operator reads that K8s Secret and creates a Perses-internal secret automatically named <datasource-name>-secret — in this case acm-thanos-secret. The proxy.spec.secret: acm-thanos-secret field tells the Perses server to use that internal secret (with the client cert and key) when building the mTLS transport for this datasource:

oc --context=ossm-kiali-spoke apply -f - <<EOF
apiVersion: perses.dev/v1alpha2
kind: PersesDatasource
metadata:
  name: acm-thanos
  namespace: perses
spec:
  config:
    display:
      name: "ACM Hub Thanos"
    default: false
    plugin:
      kind: PrometheusDatasource
      spec:
        proxy:
          kind: HTTPProxy
          spec:
            url: "${OBSERVATORIUM_URL}"
            secret: acm-thanos-secret
  client:
    tls:
      enable: true
      caCert:
        type: file
        certPath: /etc/ssl/certs/acm-thanos-ca.crt
      userCert:
        type: secret
        name: perses-acm-client-certs
        namespace: openshift-cluster-observability-operator
        certPath: tls.crt
        privateKeyPath: tls.key
EOF

Verify the datasource was accepted:

oc --context=ossm-kiali-spoke get persesdatasource acm-thanos \
  -n perses \
  -o jsonpath='{.status.conditions[?(@.type=="Available")].message}{"\n"}'
# Expected: Datasource (acm-thanos) created successfully

1.4 Create Istio Dashboards

Create three Istio dashboards, each focusing on a different layer of observability:

Istio Mesh Overview — cluster-wide HTTP request rates, 5xx error rates, and ambient L4 TCP throughput. Use this to get a top-down view of overall mesh health across both spoke clusters.
Istio Workload Dashboard — per-workload inbound and outbound HTTP request rates, p99 latency, and success rate. Accepts $workload and $namespace variables to filter to a specific workload.
Istio Ztunnel Dashboard — ambient-mode L4 TCP connection counts and byte throughput, showing traffic handled by ztunnel across the mesh.

The PersesDashboard resources below contain the full Prometheus queries for each panel. You can modify the queries, add panels, or create entirely new dashboards to suit your environment — these are a starting point. All panels use the acm-thanos datasource so data from both spoke clusters is included.

The rate windows below are selected for the ACM federation cadence. Before changing the scrape, federation, or query intervals, review Scrape Intervals so the windows remain compatible with the rate at which samples arrive in hub Thanos.

Istio Mesh Overview — cluster-wide HTTP and TCP traffic rates:

oc --context=ossm-kiali-spoke apply -f - <<'EOF'
apiVersion: perses.dev/v1alpha2
kind: PersesDashboard
metadata:
  name: istio-mesh-overview
  namespace: perses
spec:
  config:
    display:
      name: "Istio Mesh Overview"
    duration: 30m
    layouts:
    - kind: Grid
      spec:
        display:
          title: "HTTP Traffic"
        items:
        - content:
            $ref: '#/spec/panels/request_rate'
          height: 8
          width: 12
          x: 0
          "y": 0
        - content:
            $ref: '#/spec/panels/error_rate'
          height: 8
          width: 12
          x: 12
          "y": 0
    - kind: Grid
      spec:
        display:
          title: "TCP Traffic (Ambient / Ztunnel)"
        items:
        - content:
            $ref: '#/spec/panels/tcp_sent'
          height: 8
          width: 12
          x: 0
          "y": 0
        - content:
            $ref: '#/spec/panels/tcp_received'
          height: 8
          width: 12
          x: 12
          "y": 0
    panels:
      request_rate:
        kind: Panel
        spec:
          display:
            name: "Request Rate (rps)"
          plugin:
            kind: TimeSeriesChart
            spec:
              legend:
                mode: list
                position: bottom
          queries:
          - kind: TimeSeriesQuery
            spec:
              plugin:
                kind: PrometheusTimeSeriesQuery
                spec:
                  datasource:
                    kind: PrometheusDatasource
                    name: acm-thanos
                  query: sum(rate(istio_requests_total{reporter=~"destination|waypoint"}[10m])) by (destination_service_name)
      error_rate:
        kind: Panel
        spec:
          display:
            name: "5xx Error Rate"
          plugin:
            kind: TimeSeriesChart
            spec:
              legend:
                mode: list
                position: bottom
          queries:
          - kind: TimeSeriesQuery
            spec:
              plugin:
                kind: PrometheusTimeSeriesQuery
                spec:
                  datasource:
                    kind: PrometheusDatasource
                    name: acm-thanos
                  query: sum(rate(istio_requests_total{reporter=~"destination|waypoint",response_code=~"5.*"}[10m])) by (destination_service_name)
      tcp_sent:
        kind: Panel
        spec:
          display:
            name: "TCP Bytes Sent"
          plugin:
            kind: TimeSeriesChart
            spec:
              legend:
                mode: list
                position: bottom
          queries:
          - kind: TimeSeriesQuery
            spec:
              plugin:
                kind: PrometheusTimeSeriesQuery
                spec:
                  datasource:
                    kind: PrometheusDatasource
                    name: acm-thanos
                  query: sum(rate(istio_tcp_sent_bytes_total[10m])) by (destination_workload)
      tcp_received:
        kind: Panel
        spec:
          display:
            name: "TCP Bytes Received"
          plugin:
            kind: TimeSeriesChart
            spec:
              legend:
                mode: list
                position: bottom
          queries:
          - kind: TimeSeriesQuery
            spec:
              plugin:
                kind: PrometheusTimeSeriesQuery
                spec:
                  datasource:
                    kind: PrometheusDatasource
                    name: acm-thanos
                  query: sum(rate(istio_tcp_received_bytes_total[10m])) by (destination_workload)
EOF

Istio Workload Dashboard — per-workload inbound/outbound HTTP rates and latency, with $workload and $namespace variables:

oc --context=ossm-kiali-spoke apply -f - <<'EOF'
apiVersion: perses.dev/v1alpha2
kind: PersesDashboard
metadata:
  name: istio-workload-dashboard
  namespace: perses
spec:
  config:
    display:
      name: "Istio Workload Dashboard"
    duration: 30m
    variables:
    - kind: TextVariable
      spec:
        name: workload
        display:
          name: Workload
        value: ""
    - kind: TextVariable
      spec:
        name: namespace
        display:
          name: Namespace
        value: ""
    layouts:
    - kind: Grid
      spec:
        display:
          title: "Inbound HTTP"
        items:
        - content:
            $ref: '#/spec/panels/inbound_rps'
          height: 8
          width: 12
          x: 0
          "y": 0
        - content:
            $ref: '#/spec/panels/inbound_latency'
          height: 8
          width: 12
          x: 12
          "y": 0
    - kind: Grid
      spec:
        display:
          title: "Outbound HTTP"
        items:
        - content:
            $ref: '#/spec/panels/outbound_rps'
          height: 8
          width: 12
          x: 0
          "y": 0
        - content:
            $ref: '#/spec/panels/success_rate'
          height: 8
          width: 12
          x: 12
          "y": 0
    panels:
      inbound_rps:
        kind: Panel
        spec:
          display:
            name: "Inbound Request Rate"
          plugin:
            kind: TimeSeriesChart
            spec:
              legend:
                mode: list
                position: bottom
          queries:
          - kind: TimeSeriesQuery
            spec:
              plugin:
                kind: PrometheusTimeSeriesQuery
                spec:
                  datasource:
                    kind: PrometheusDatasource
                    name: acm-thanos
                  query: sum(rate(istio_requests_total{reporter=~"destination|waypoint",destination_workload="$workload",destination_workload_namespace=~"$namespace"}[10m])) by (source_app)
      inbound_latency:
        kind: Panel
        spec:
          display:
            name: "Inbound Request Latency (p99)"
          plugin:
            kind: TimeSeriesChart
            spec:
              legend:
                mode: list
                position: bottom
          queries:
          - kind: TimeSeriesQuery
            spec:
              plugin:
                kind: PrometheusTimeSeriesQuery
                spec:
                  datasource:
                    kind: PrometheusDatasource
                    name: acm-thanos
                  query: histogram_quantile(0.99, sum(rate(istio_request_duration_milliseconds_bucket{reporter=~"destination|waypoint",destination_workload="$workload",destination_workload_namespace=~"$namespace"}[10m])) by (le))
      outbound_rps:
        kind: Panel
        spec:
          display:
            name: "Outbound Request Rate"
          plugin:
            kind: TimeSeriesChart
            spec:
              legend:
                mode: list
                position: bottom
          queries:
          - kind: TimeSeriesQuery
            spec:
              plugin:
                kind: PrometheusTimeSeriesQuery
                spec:
                  datasource:
                    kind: PrometheusDatasource
                    name: acm-thanos
                  query: sum(rate(istio_requests_total{reporter=~"source|waypoint",source_workload="$workload",source_workload_namespace=~"$namespace"}[10m])) by (destination_service_name)
      success_rate:
        kind: Panel
        spec:
          display:
            name: "Success Rate (non-5xx)"
          plugin:
            kind: TimeSeriesChart
            spec:
              legend:
                mode: list
                position: bottom
          queries:
          - kind: TimeSeriesQuery
            spec:
              plugin:
                kind: PrometheusTimeSeriesQuery
                spec:
                  datasource:
                    kind: PrometheusDatasource
                    name: acm-thanos
                  query: sum(rate(istio_requests_total{reporter=~"destination|waypoint",destination_workload="$workload",destination_workload_namespace=~"$namespace",response_code!~"5.*"}[10m])) / sum(rate(istio_requests_total{reporter=~"destination|waypoint",destination_workload="$workload",destination_workload_namespace=~"$namespace"}[10m]))
EOF

Istio Ztunnel Dashboard — ambient L4 TCP connection and throughput metrics:

oc --context=ossm-kiali-spoke apply -f - <<'EOF'
apiVersion: perses.dev/v1alpha2
kind: PersesDashboard
metadata:
  name: istio-ztunnel-dashboard
  namespace: perses
spec:
  config:
    display:
      name: "Istio Ztunnel Dashboard"
    duration: 30m
    layouts:
    - kind: Grid
      spec:
        display:
          title: "Ambient L4 Traffic"
        items:
        - content:
            $ref: '#/spec/panels/tcp_connections_opened'
          height: 8
          width: 12
          x: 0
          "y": 0
        - content:
            $ref: '#/spec/panels/tcp_bytes'
          height: 8
          width: 12
          x: 12
          "y": 0
    panels:
      tcp_connections_opened:
        kind: Panel
        spec:
          display:
            name: "TCP Connections Opened"
          plugin:
            kind: TimeSeriesChart
            spec:
              legend:
                mode: list
                position: bottom
          queries:
          - kind: TimeSeriesQuery
            spec:
              plugin:
                kind: PrometheusTimeSeriesQuery
                spec:
                  datasource:
                    kind: PrometheusDatasource
                    name: acm-thanos
                  query: sum(rate(istio_tcp_connections_opened_total[10m])) by (destination_workload)
      tcp_bytes:
        kind: Panel
        spec:
          display:
            name: "TCP Throughput (bytes/s)"
          plugin:
            kind: TimeSeriesChart
            spec:
              legend:
                mode: list
                position: bottom
          queries:
          - kind: TimeSeriesQuery
            spec:
              plugin:
                kind: PrometheusTimeSeriesQuery
                spec:
                  datasource:
                    kind: PrometheusDatasource
                    name: acm-thanos
                  query: sum(rate(istio_tcp_sent_bytes_total[10m]) + rate(istio_tcp_received_bytes_total[10m])) by (destination_workload)
EOF

Confirm all three dashboards are registered:

oc --context=ossm-kiali-spoke get persesdashboard -n perses | grep istio
# Expected: istio-mesh-overview, istio-workload, istio-ztunnel

1.5 Configure Kiali for Perses

Get the OpenShift console URL — this is where the COO Perses plugin lives:

CONSOLE_URL=$(oc --context=ossm-kiali-spoke get route console \
  -n openshift-console \
  -o jsonpath='https://{.spec.host}')
echo "Console URL: ${CONSOLE_URL}"

Patch the Kiali CR on spoke. Key settings:

url_format: "openshift" — Kiali builds dashboard links pointing to the OpenShift console Perses plugin. In this mode internal_url must not be set — COO Perses uses OpenShift OAuth, so Kiali cannot authenticate to its API. Without internal_url, Kiali skips dashboard validation and generates the links unconditionally
health_check_url — points at the in-cluster Perses /api/v1/health endpoint so Kiali can verify reachability. This endpoint does not require authentication. Without it, Kiali’s cluster status shows Perses as “Unreachable” because the health check falls back to external_url (the OCP console), which is not a Perses API
project — the Perses project name, which COO sets to the Kubernetes namespace where the dashboards live (perses)
variables — maps Kiali’s semantic names to the exact Perses variable names defined in the dashboards created in the previous step.

oc --context=ossm-kiali-spoke patch kiali kiali -n istio-system --type=merge -p "{
  \"spec\": {
    \"external_services\": {
      \"perses\": {
        \"enabled\": true,
        \"url_format\": \"openshift\",
        \"external_url\": \"${CONSOLE_URL}\",
        \"health_check_url\": \"https://perses.openshift-cluster-observability-operator.svc.cluster.local:8080/api/v1/health\",
        \"project\": \"perses\",
        \"dashboards\": [
          {\"name\": \"Istio Mesh Overview\"},
          {\"name\": \"Istio Workload Dashboard\",
           \"variables\": {
             \"namespace\": \"namespace\",
             \"workload\": \"workload\"}},
          {\"name\": \"Istio Ztunnel Dashboard\"}
        ]
      }
    }
  }
}"

oc --context=ossm-kiali-spoke wait kiali kiali \
  -n istio-system \
  --for=condition=Successful \
  --timeout=300s

oc --context=ossm-kiali-spoke rollout status deployment/kiali \
  -n istio-system --timeout=120s

1.6 Verify Perses

# Confirm Perses pod is running
oc --context=ossm-kiali-spoke get pods \
  -l app.kubernetes.io/name=perses \
  -n openshift-cluster-observability-operator

# Check the Istio dashboards are registered
oc --context=ossm-kiali-spoke get persesdashboard -n perses | grep istio

# Confirm Kiali's Perses config is applied
oc --context=ossm-kiali-spoke get kiali kiali -n istio-system \
  -o jsonpath='{.spec.external_services.perses.enabled}{"\n"}'
# Expected: true

The Perses links appear in specific Kiali pages depending on the dashboard:

Istio Workload Dashboard — navigate to Workloads, select any workload (e.g. productpage-v1 in bookinfo), and open the Inbound Metrics or Outbound Metrics tab. The View in Perses link appears in the metrics toolbar.
Istio Ztunnel Dashboard — navigate to Workloads, select the ztunnel workload in the ztunnel namespace, and open its Metrics tab found in the ZTunnel subtab.
Istio Mesh Overview — this dashboard is not linked automatically from any Kiali page. Access it directly from the OpenShift console at Observe > Dashboards (Perses) and select it from the list.

Clicking a View in Perses link opens the OpenShift console Perses plugin on spoke. If you are not already logged in, the console will prompt for credentials.

Due to a known Kiali bug (kiali/kiali#10021), the link lands on the dashboard list rather than navigating directly to the specific dashboard with variables pre-filled. Until the bug is fixed: click the dashboard by name from the list, then manually fill in the Workload and Namespace fields for the Istio Workload Dashboard (e.g. productpage-v1 and bookinfo). When the bug is fixed, the link will open the correct dashboard with the workload already selected.

Phase 2: Tempo — Multi-Cluster Distributed Tracing

Tempo is a distributed tracing backend. The architecture deployed here:

A central TempoStack runs on spoke, backed by in-cluster SeaweedFS
A local OTEL collector on spoke receives traces from spoke’s Istio proxies and forwards them to Tempo
A remote OTEL collector on spoke receives traces from spoke-two’s OTEL collector over a passthrough mTLS Route
A remote forwarder on spoke-two receives traces from spoke-two’s Istio proxies and forwards them to spoke via the mTLS Route
Each cluster’s Istio is patched with an extensionProvider and a Telemetry CR to send traces to the local OTEL collector

Kiali on spoke queries the single Tempo endpoint. Traces from both clusters land in the same Tempo tenant. Each OTEL collector stamps a k8s.cluster.name resource attribute on every span using its resource processor — spoke for the spoke collector and spoke-two for the spoke-two forwarder. This attribute is set at collector deploy time and can be used to filter traces by cluster in the Kiali tracing view.

2.1 Install Tempo and OpenTelemetry Operators

Install the Tempo Operator and OpenTelemetry Operator on spoke, and the OpenTelemetry Operator on spoke-two. Both operators go into openshift-operators — the same namespace used by OSSM and Kiali:

# Tempo Operator on spoke only
oc --context=ossm-kiali-spoke apply -f - <<'EOF'
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: tempo-product
  namespace: openshift-operators
spec:
  channel: stable
  installPlanApproval: Automatic
  name: tempo-product
  source: redhat-operators
  sourceNamespace: openshift-marketplace
EOF

# OpenTelemetry Operator on both clusters
for CTX in ossm-kiali-spoke ossm-kiali-spoke-two; do
  oc --context="${CTX}" apply -f - <<'EOF'
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: opentelemetry-product
  namespace: openshift-operators
spec:
  channel: stable
  installPlanApproval: Automatic
  name: opentelemetry-product
  source: redhat-operators
  sourceNamespace: openshift-marketplace
EOF
done

Wait for the CRDs to be established:

until oc --context=ossm-kiali-spoke get crd tempostacks.tempo.grafana.com &>/dev/null; do
  echo "Waiting for TempoStack CRD on spoke..."
  sleep 10
done
echo "Tempo CRD ready"

until oc --context=ossm-kiali-spoke get crd opentelemetrycollectors.opentelemetry.io &>/dev/null; do
  echo "Waiting for OpenTelemetryCollector CRD on spoke..."
  sleep 10
done
echo "OTel CRD ready on spoke"

until oc --context=ossm-kiali-spoke-two get crd opentelemetrycollectors.opentelemetry.io &>/dev/null; do
  echo "Waiting for OpenTelemetryCollector CRD on spoke-two..."
  sleep 10
done
echo "OTel CRD ready on spoke-two"

2.2 Deploy SeaweedFS and TempoStack on Spoke

Deploy SeaweedFS mini mode as a single-pod S3-compatible object store for Tempo, then create the TempoStack. The stack runs in multi-tenant OpenShift mode with tenant ${TEMPO_TENANT} for trace writes and reads.

oc --context=ossm-kiali-spoke create namespace "${TEMPO_NAMESPACE}" 2>/dev/null || true

# SeaweedFS deployment and PVC
oc --context=ossm-kiali-spoke apply -n "${TEMPO_NAMESPACE}" -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: seaweedfs-pv-claim
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 256Mi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: seaweedfs
spec:
  selector:
    matchLabels:
      app: seaweedfs
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: seaweedfs
    spec:
      volumes:
      - name: storage
        persistentVolumeClaim:
          claimName: seaweedfs-pv-claim
      containers:
      - name: seaweedfs
        image: ghcr.io/chrislusf/seaweedfs:4.47
        args:
        - mini
        - -dir=/data
        - -admin.port=12646
        - -master.telemetry=false
        env:
        - name: AWS_ACCESS_KEY_ID
          value: "${SEAWEEDFS_ACCESS_KEY}"
        - name: AWS_SECRET_ACCESS_KEY
          value: "${SEAWEEDFS_SECRET_KEY}"
        - name: S3_BUCKET
          value: "tempo-data"
        ports:
        - containerPort: 8333
          name: s3
        volumeMounts:
        - name: storage
          mountPath: "/data"
        readinessProbe:
          tcpSocket:
            port: 8333
          initialDelaySeconds: 10
          periodSeconds: 5
        livenessProbe:
          tcpSocket:
            port: 8333
          initialDelaySeconds: 10
          periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: seaweedfs
spec:
  type: ClusterIP
  ports:
  - port: 8333
    targetPort: 8333
    protocol: TCP
    name: s3
  selector:
    app: seaweedfs
EOF

oc --context=ossm-kiali-spoke rollout status deployment/seaweedfs \
  -n "${TEMPO_NAMESPACE}" --timeout=120s

Create the SeaweedFS object storage secret and the TempoStack CR:

oc --context=ossm-kiali-spoke create secret generic tempostack-dev-seaweedfs \
  -n "${TEMPO_NAMESPACE}" \
  --from-literal=bucket="tempo-data" \
  --from-literal=endpoint="http://seaweedfs.${TEMPO_NAMESPACE}.svc.cluster.local:8333" \
  --from-literal=access_key_id="${SEAWEEDFS_ACCESS_KEY}" \
  --from-literal=access_key_secret="${SEAWEEDFS_SECRET_KEY}" \
  --dry-run=client -o yaml | oc --context=ossm-kiali-spoke apply -f -

oc --context=ossm-kiali-spoke apply -f - <<EOF
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: ${TEMPO_STACK_NAME}
  namespace: ${TEMPO_NAMESPACE}
spec:
  managementState: Managed
  storageSize: 1Gi
  storage:
    secret:
      type: s3
      name: tempostack-dev-seaweedfs
  tenants:
    mode: openshift
    authentication:
    - tenantName: ${TEMPO_TENANT}
      tenantId: a1b2c3d4-e5f6-7890-abcd-ef1234567890
  template:
    gateway:
      enabled: true
EOF

echo "Waiting for TempoStack gateway service..."
until oc --context=ossm-kiali-spoke get svc \
  "tempo-${TEMPO_STACK_NAME}-gateway" \
  -n "${TEMPO_NAMESPACE}" &>/dev/null; do sleep 5; done
echo "TempoStack ready"

2.3 RBAC for Tempo Gateway

Grant the OTEL collectors permission to write and read traces through the Tempo gateway:

REMOTE_COLLECTOR_NAME="otel-remote-${SPOKE_TWO_CLUSTER_NAME}"

# Write role — used by both the local and remote collectors
oc --context=ossm-kiali-spoke apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: tempostack-traces-write
rules:
- apiGroups:
  - tempo.grafana.com
  resources:
  - ${TEMPO_TENANT}
  resourceNames:
  - traces
  verbs:
  - create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tempostack-traces-write-collectors
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: tempostack-traces-write
subjects:
- kind: ServiceAccount
  name: otel-collector
  namespace: istio-system
- kind: ServiceAccount
  name: ${REMOTE_COLLECTOR_NAME}-collector
  namespace: istio-system
EOF

# Read role — allows authenticated users (including Kiali) to query traces
oc --context=ossm-kiali-spoke apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: tempostack-traces-reader-${TEMPO_TENANT}
rules:
- apiGroups:
  - tempo.grafana.com
  resources:
  - ${TEMPO_TENANT}
  resourceNames:
  - traces
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tempostack-traces-reader-${TEMPO_TENANT}
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: tempostack-traces-reader-${TEMPO_TENANT}
subjects:
- kind: Group
  apiGroup: rbac.authorization.k8s.io
  name: system:authenticated
EOF
echo "RBAC applied"

2.4 Local OTEL Collector on Spoke

Deploy the local collector in istio-system. It receives OTLP traces from spoke’s Istio proxies and forwards them to the Tempo gateway using the pod’s service account token for authentication:

oc --context=ossm-kiali-spoke apply -f - <<EOF
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otel
  namespace: istio-system
spec:
  mode: deployment
  replicas: 1
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    extensions:
      bearertokenauth:
        filename: /var/run/secrets/kubernetes.io/serviceaccount/token
    processors:
      batch: {}
      resource:
        attributes:
        - key: k8s.cluster.name
          action: upsert
          value: ${SPOKE_CLUSTER_NAME}
    exporters:
      otlp_http/tempo:
        auth:
          authenticator: bearertokenauth
        endpoint: https://tempo-${TEMPO_STACK_NAME}-gateway.${TEMPO_NAMESPACE}.svc.cluster.local:8080/api/traces/v1/${TEMPO_TENANT}
        headers:
          X-Scope-OrgID: ${TEMPO_TENANT}
        tls:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
    service:
      extensions:
      - bearertokenauth
      pipelines:
        traces:
          receivers:
          - otlp
          processors:
          - resource
          - batch
          exporters:
          - otlp_http/tempo
EOF

until oc --context=ossm-kiali-spoke get deployment otel-collector \
  -n istio-system &>/dev/null; do sleep 5; done

oc --context=ossm-kiali-spoke rollout status deployment/otel-collector \
  -n istio-system --timeout=300s
echo "Local OTEL collector ready"

2.5 Remote Receiver on Spoke with mTLS

Traces from spoke-two arrive at spoke via an mTLS-secured passthrough Route. This step creates the remote receiver collector, the Route, and the mTLS certificates (generated with openssl — no additional operators required).

First, create the remote receiver collector (it will stay in CrashLoopBackOff until the certs are ready — that is expected):

REMOTE_COLLECTOR_NAME="otel-remote-${SPOKE_TWO_CLUSTER_NAME}"
REMOTE_MTLS_CA_SECRET="${REMOTE_COLLECTOR_NAME}-ca"
REMOTE_MTLS_SERVER_SECRET="${REMOTE_COLLECTOR_NAME}-server-tls"
REMOTE_MTLS_CLIENT_SECRET="${REMOTE_COLLECTOR_NAME}-client-tls"
REMOTE_MTLS_CLIENT_BUNDLE="${REMOTE_COLLECTOR_NAME}-client-bundle"

oc --context=ossm-kiali-spoke apply -f - <<EOF
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: ${REMOTE_COLLECTOR_NAME}
  namespace: istio-system
spec:
  mode: deployment
  replicas: 1
  volumeMounts:
  - name: server-certs
    mountPath: /etc/otel/server
    readOnly: true
  - name: ca-cert
    mountPath: /etc/otel/ca
    readOnly: true
  volumes:
  - name: server-certs
    secret:
      secretName: ${REMOTE_MTLS_SERVER_SECRET}
  - name: ca-cert
    secret:
      secretName: ${REMOTE_MTLS_CA_SECRET}
  config:
    receivers:
      otlp:
        protocols:
          http:
            endpoint: 0.0.0.0:4318
            tls:
              cert_file: /etc/otel/server/tls.crt
              key_file: /etc/otel/server/tls.key
              client_ca_file: /etc/otel/ca/tls.crt
    processors:
      batch: {}
      resource:
        attributes:
        - key: receiving.cluster
          action: upsert
          value: ${SPOKE_CLUSTER_NAME}
        - key: source.cluster
          action: upsert
          value: ${SPOKE_TWO_CLUSTER_NAME}
    extensions:
      bearertokenauth:
        filename: /var/run/secrets/kubernetes.io/serviceaccount/token
    exporters:
      otlp_http/tempo:
        auth:
          authenticator: bearertokenauth
        endpoint: https://tempo-${TEMPO_STACK_NAME}-gateway.${TEMPO_NAMESPACE}.svc.cluster.local:8080/api/traces/v1/${TEMPO_TENANT}
        headers:
          X-Scope-OrgID: ${TEMPO_TENANT}
        tls:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
    service:
      extensions:
      - bearertokenauth
      pipelines:
        traces:
          receivers:
          - otlp
          processors:
          - resource
          - batch
          exporters:
          - otlp_http/tempo
EOF

Create the passthrough Route that exposes the remote receiver:

oc --context=ossm-kiali-spoke apply -f - <<EOF
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: ${REMOTE_COLLECTOR_NAME}
  namespace: istio-system
spec:
  to:
    kind: Service
    name: ${REMOTE_COLLECTOR_NAME}-collector
    weight: 100
  port:
    targetPort: otlp-http
  tls:
    termination: passthrough
  wildcardPolicy: None
EOF

# Wait for the route hostname to be assigned
REMOTE_ROUTE_HOST=""
until [ -n "${REMOTE_ROUTE_HOST}" ]; do
  REMOTE_ROUTE_HOST=$(oc --context=ossm-kiali-spoke get route "${REMOTE_COLLECTOR_NAME}" \
    -n istio-system \
    -o jsonpath='{.spec.host}' 2>/dev/null || true)
  sleep 3
done
echo "Remote receiver Route: https://${REMOTE_ROUTE_HOST}"

Generate the mTLS certificates with openssl. The same approach used for the Istio CA in the hub/spoke guide. The server certificate includes the Route hostname as a SAN so the spoke-two forwarder can verify it:

OTEL_CERT_DIR="/tmp/otel-mtls"
mkdir -p "${OTEL_CERT_DIR}"

# Self-signed CA
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 \
  -days 3650 -nodes \
  -keyout "${OTEL_CERT_DIR}/ca.key" \
  -out "${OTEL_CERT_DIR}/ca.crt" \
  -subj "/CN=${REMOTE_COLLECTOR_NAME}-ca" 2>/dev/null

# Server cert (signed by CA) — SANs cover both the in-cluster service and the Route hostname
openssl req -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
  -keyout "${OTEL_CERT_DIR}/server.key" \
  -out "${OTEL_CERT_DIR}/server.csr" \
  -subj "/CN=${REMOTE_COLLECTOR_NAME}-server" 2>/dev/null

printf "subjectAltName=DNS:%s,DNS:%s.istio-system.svc,DNS:%s.istio-system.svc.cluster.local,DNS:%s" \
  "${REMOTE_COLLECTOR_NAME}-collector" \
  "${REMOTE_COLLECTOR_NAME}-collector" \
  "${REMOTE_COLLECTOR_NAME}-collector" \
  "${REMOTE_ROUTE_HOST}" > "${OTEL_CERT_DIR}/server-ext.cnf"

openssl x509 -req -days 3650 \
  -in "${OTEL_CERT_DIR}/server.csr" \
  -CA "${OTEL_CERT_DIR}/ca.crt" \
  -CAkey "${OTEL_CERT_DIR}/ca.key" -CAcreateserial \
  -extfile "${OTEL_CERT_DIR}/server-ext.cnf" \
  -out "${OTEL_CERT_DIR}/server.crt" 2>/dev/null

# Client cert (signed by same CA)
openssl req -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
  -keyout "${OTEL_CERT_DIR}/client.key" \
  -out "${OTEL_CERT_DIR}/client.csr" \
  -subj "/CN=${REMOTE_COLLECTOR_NAME}-client" 2>/dev/null

openssl x509 -req -days 3650 \
  -in "${OTEL_CERT_DIR}/client.csr" \
  -CA "${OTEL_CERT_DIR}/ca.crt" \
  -CAkey "${OTEL_CERT_DIR}/ca.key" -CAcreateserial \
  -out "${OTEL_CERT_DIR}/client.crt" 2>/dev/null

echo "Certs generated: $(openssl x509 -noout -subject -in ${OTEL_CERT_DIR}/server.crt)"

# Load certs into Kubernetes Secrets on spoke
oc --context=ossm-kiali-spoke create secret generic "${REMOTE_MTLS_CA_SECRET}" \
  -n istio-system \
  --from-file=tls.crt="${OTEL_CERT_DIR}/ca.crt" \
  --from-file=tls.key="${OTEL_CERT_DIR}/ca.key" \
  --dry-run=client -o yaml | oc --context=ossm-kiali-spoke apply -f -

oc --context=ossm-kiali-spoke create secret tls "${REMOTE_MTLS_SERVER_SECRET}" \
  -n istio-system \
  --cert="${OTEL_CERT_DIR}/server.crt" \
  --key="${OTEL_CERT_DIR}/server.key" \
  --dry-run=client -o yaml | oc --context=ossm-kiali-spoke apply -f -

oc --context=ossm-kiali-spoke create secret tls "${REMOTE_MTLS_CLIENT_SECRET}" \
  -n istio-system \
  --cert="${OTEL_CERT_DIR}/client.crt" \
  --key="${OTEL_CERT_DIR}/client.key" \
  --dry-run=client -o yaml | oc --context=ossm-kiali-spoke apply -f -

echo "Certificates ready"

# Kubernetes may take a moment to restart the crashlooping pod after secrets appear.
# Wait until the deployment has at least one ready replica before checking rollout status.
until oc --context=ossm-kiali-spoke get deployment \
  "${REMOTE_COLLECTOR_NAME}-collector" \
  -n istio-system &>/dev/null; do sleep 3; done

oc --context=ossm-kiali-spoke rollout restart \
  "deployment/${REMOTE_COLLECTOR_NAME}-collector" \
  -n istio-system

oc --context=ossm-kiali-spoke rollout status \
  "deployment/${REMOTE_COLLECTOR_NAME}-collector" \
  -n istio-system --timeout=300s
echo "Remote receiver collector ready"

2.6 Remote Forwarder on Spoke-Two

Sync the mTLS client certificate bundle from spoke to spoke-two, then deploy the forwarder collector on spoke-two. The forwarder receives traces from spoke-two’s Istio proxies and ships them to spoke’s Route with mTLS client authentication:

# Extract the client bundle from spoke
CA_BUNDLE=$(oc --context=ossm-kiali-spoke get secret "${REMOTE_MTLS_CA_SECRET}" \
  -n istio-system -o jsonpath='{.data.tls\.crt}' | base64 -d)
CLIENT_CRT=$(oc --context=ossm-kiali-spoke get secret "${REMOTE_MTLS_CLIENT_SECRET}" \
  -n istio-system -o jsonpath='{.data.tls\.crt}' | base64 -d)
CLIENT_KEY=$(oc --context=ossm-kiali-spoke get secret "${REMOTE_MTLS_CLIENT_SECRET}" \
  -n istio-system -o jsonpath='{.data.tls\.key}' | base64 -d)

# Create the bundle secret on spoke-two
oc --context=ossm-kiali-spoke-two create secret generic "${REMOTE_MTLS_CLIENT_BUNDLE}" \
  -n istio-system \
  --from-literal=ca.crt="${CA_BUNDLE}" \
  --from-literal=tls.crt="${CLIENT_CRT}" \
  --from-literal=tls.key="${CLIENT_KEY}" \
  --dry-run=client -o yaml | oc --context=ossm-kiali-spoke-two apply -f -

# Deploy the forwarder on spoke-two
oc --context=ossm-kiali-spoke-two apply -f - <<EOF
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otel
  namespace: istio-system
spec:
  mode: deployment
  replicas: 1
  volumeMounts:
  - name: mtls-bundle
    mountPath: /etc/otel/mtls
    readOnly: true
  volumes:
  - name: mtls-bundle
    secret:
      secretName: ${REMOTE_MTLS_CLIENT_BUNDLE}
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    processors:
      batch: {}
      resource:
        attributes:
        - key: k8s.cluster.name
          action: upsert
          value: ${SPOKE_TWO_CLUSTER_NAME}
    exporters:
      otlp_http/central:
        endpoint: https://${REMOTE_ROUTE_HOST}
        tls:
          cert_file: /etc/otel/mtls/tls.crt
          key_file: /etc/otel/mtls/tls.key
          ca_file: /etc/otel/mtls/ca.crt
    service:
      pipelines:
        traces:
          receivers:
          - otlp
          processors:
          - resource
          - batch
          exporters:
          - otlp_http/central
EOF

until oc --context=ossm-kiali-spoke-two get deployment otel-collector \
  -n istio-system &>/dev/null; do sleep 5; done

oc --context=ossm-kiali-spoke-two rollout status deployment/otel-collector \
  -n istio-system --timeout=300s
echo "Spoke-two remote forwarder ready"

2.7 Configure Istio to Send Traces

Patch both clusters’ Istio CRs to add the OTEL collector as a tracing extension provider, then apply a mesh-level Telemetry CR to enable 100% sampling. Also apply per-namespace Telemetry CRs in the demo namespaces to ensure trace generation from both ambient and sidecar workloads:

# Patch meshConfig on spoke
oc --context=ossm-kiali-spoke patch istio default --type=merge -p '{
  "spec": {
    "values": {
      "meshConfig": {
        "enableTracing": true,
        "extensionProviders": [
          {
            "name": "otel-tracing",
            "opentelemetry": {
              "service": "otel-collector.istio-system.svc.cluster.local",
              "port": 4317
            }
          }
        ]
      }
    }
  }
}'

# Patch meshConfig on spoke-two
oc --context=ossm-kiali-spoke-two patch istio default --type=merge -p '{
  "spec": {
    "values": {
      "meshConfig": {
        "enableTracing": true,
        "extensionProviders": [
          {
            "name": "otel-tracing",
            "opentelemetry": {
              "service": "otel-collector.istio-system.svc.cluster.local",
              "port": 4317
            }
          }
        ]
      }
    }
  }
}'

# Mesh-level Telemetry on both clusters
for CTX in ossm-kiali-spoke ossm-kiali-spoke-two; do
  oc --context="${CTX}" apply -f - <<'EOF'
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: otel-tracing
  namespace: istio-system
spec:
  tracing:
  - providers:
    - name: otel-tracing
    randomSamplingPercentage: 100
EOF
done

# Per-namespace Telemetry in demo namespaces
for CTX in ossm-kiali-spoke ossm-kiali-spoke-two; do
  for NS in ambient-demo bookinfo; do
    if oc --context="${CTX}" get namespace "${NS}" &>/dev/null; then
      oc --context="${CTX}" apply -f - <<EOF
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: ${NS}-tracing
  namespace: ${NS}
spec:
  tracing:
  - providers:
    - name: otel-tracing
    randomSamplingPercentage: 100
EOF
    fi
  done
done
echo "Telemetry CRs applied"

# Restart waypoints so they pick up the new tracing config
for NS in ambient-demo; do
  for CTX in ossm-kiali-spoke ossm-kiali-spoke-two; do
    if oc --context="${CTX}" get deployment waypoint -n "${NS}" &>/dev/null; then
      oc --context="${CTX}" rollout restart deployment/waypoint -n "${NS}"
    fi
  done
done

2.8 Enable the Distributed Tracing Console Plugin

The COO distributed tracing plugin adds Observe > Traces to the OpenShift console, providing a UI for browsing and filtering traces from the TempoStack. Kiali’s url_format: "openshift" links go directly to this console page. COO discovers the multi-tenant TempoStack automatically — no additional configuration is needed beyond creating the UIPlugin:

oc --context=ossm-kiali-spoke apply -f - <<'EOF'
apiVersion: observability.openshift.io/v1alpha1
kind: UIPlugin
metadata:
  name: distributed-tracing
spec:
  type: DistributedTracing
EOF

until oc --context=ossm-kiali-spoke get uiplugin distributed-tracing \
  -o jsonpath='{.status.conditions[?(@.type=="Reconciled")].status}' 2>/dev/null | grep -q "True"; do
  echo "Waiting for distributed tracing plugin..."
  sleep 5
done
echo "Distributed tracing plugin ready"

2.9 Configure Kiali for Tempo

Patch the Kiali CR on spoke to enable mesh tracing with Tempo and to emit Kiali server self-traces through the local OTEL collector from §2.4 (otel-collector.istio-system.svc:4317, OTLP gRPC). Self-traces flow through the same collector pipeline as Istio mesh traces and appear on the Kiali workload Traces tab once external_services.tracing is configured.

The url_format: "openshift" and tempo_config settings cause Kiali to use the OpenShift console distributed tracing plugin for UI links. When tenant is set in tempo_config, Kiali automatically appends the tenant API path to internal_url — so providing just the gateway base URL is sufficient.

With url_format: "openshift", Kiali appends /observe/traces?... to external_url, so external_url must be the OCP console base URL — not the Tempo route:

CONSOLE_URL=$(oc --context=ossm-kiali-spoke get route console \
  -n openshift-console \
  -o jsonpath='https://{.spec.host}')
echo "Console URL: ${CONSOLE_URL}"

oc --context=ossm-kiali-spoke patch kiali kiali -n istio-system --type=merge -p "{
  \"spec\": {
    \"server\": {
      \"observability\": {
        \"tracing\": {
          \"enabled\": true,
          \"collector_type\": \"otel\",
          \"collector_url\": \"otel-collector.istio-system.svc:4317\",
          \"otel\": {
            \"protocol\": \"grpc\"
          }
        }
      }
    },
    \"external_services\": {
      \"tracing\": {
        \"enabled\": true,
        \"provider\": \"tempo\",
        \"use_grpc\": false,
        \"internal_url\": \"https://tempo-${TEMPO_STACK_NAME}-gateway.${TEMPO_NAMESPACE}.svc.cluster.local:8080/\",
        \"external_url\": \"${CONSOLE_URL}\",
        \"auth\": {
          \"type\": \"bearer\",
          \"use_kiali_token\": true
        },
        \"tempo_config\": {
          \"name\": \"${TEMPO_STACK_NAME}\",
          \"namespace\": \"${TEMPO_NAMESPACE}\",
          \"tenant\": \"${TEMPO_TENANT}\",
          \"org_id\": \"${TEMPO_TENANT}\",
          \"url_format\": \"openshift\"
        }
      }
    }
  }
}"

oc --context=ossm-kiali-spoke wait kiali kiali \
  -n istio-system \
  --for=condition=Successful \
  --timeout=300s

oc --context=ossm-kiali-spoke rollout status deployment/kiali \
  -n istio-system --timeout=120s
echo "Kiali updated with Tempo config"

2.10 Verify Tempo

# Confirm collectors are running on both clusters
echo "=== Spoke collectors ==="
oc --context=ossm-kiali-spoke get pods -n istio-system \
  -l app.kubernetes.io/managed-by=opentelemetry-operator

echo "=== Spoke-two forwarder ==="
oc --context=ossm-kiali-spoke-two get pods -n istio-system \
  -l app.kubernetes.io/managed-by=opentelemetry-operator

# Confirm Telemetry CRs are in place on both clusters
echo "=== Telemetry CRs ==="
oc --context=ossm-kiali-spoke get telemetry -A
oc --context=ossm-kiali-spoke-two get telemetry -A

# Confirm TempoStack is healthy
oc --context=ossm-kiali-spoke get tempostack "${TEMPO_STACK_NAME}" \
  -n "${TEMPO_NAMESPACE}"

# Confirm the distributed tracing console plugin is reconciled
oc --context=ossm-kiali-spoke get uiplugin distributed-tracing \
  -o jsonpath='{.status.conditions[?(@.type=="Reconciled")].status}{"\n"}'
# Expected: True

Allow a few minutes for traces to accumulate, then verify. Both the OpenShift console and Kiali are running on the spoke cluster — use the ossm-kiali-spoke context to get their URLs:

echo "OpenShift console: $(oc --context=ossm-kiali-spoke get route console -n openshift-console -o jsonpath='https://{.spec.host}')"
echo "Kiali:             $(oc --context=ossm-kiali-spoke get route kiali -n istio-system -o jsonpath='https://{.spec.host}')"

In the OpenShift console, navigate to Observe > Traces — the distributed tracing plugin should show the TempoStack and allow browsing traces by service and duration. To identify which cluster a trace came from, open a trace and look for the k8s.cluster.name resource attribute in the span details, or filter using TraceQL: { resource.k8s.cluster.name = "spoke" }
In the Kiali UI, navigate to Distributed Tracing in the left nav. Confirm:

Traces are visible from both spoke and spoke-two clusters (open a trace and check the k8s.cluster.name resource attribute in the span details to identify the cluster)
Cross-cluster traces appear when bookinfo traffic routes from spoke’s productpage through the East-West gateway to spoke-two’s ratings-v2
The View in Tracing link from a workload detail page opens the OCP console Observe > Traces page scoped to that workload
On the Kiali workload in the istio-system namespace, the Traces tab lists Kiali server spans (service name matches the Kiali instance, for example kiali.istio-system)

Cleanup

Remove Perses

MCOA does not require the full COO subscription. After removing the Perses and Tempo resources below, you can remove the COO subscription installed by this guide. Do not delete the shared monitoring.rhobs CRDs: MCOA owns and uses those APIs independently.

# Revert Kiali Perses config
oc --context=ossm-kiali-spoke patch kiali kiali -n istio-system --type=json \
  -p '[{"op":"remove","path":"/spec/external_services/perses"}]' 2>/dev/null || true

# Remove UIPlugin (disables the Dashboards (Perses) console menu)
oc --context=ossm-kiali-spoke delete uiplugin monitoring --ignore-not-found

# Delete dashboard and datasource resources from the perses namespace
oc --context=ossm-kiali-spoke delete persesdashboard --all -n perses --ignore-not-found
oc --context=ossm-kiali-spoke delete persesdatasource --all -n perses --ignore-not-found

# Delete the Perses server instance (lives in openshift-cluster-observability-operator namespace)
oc --context=ossm-kiali-spoke delete perses perses \
  -n openshift-cluster-observability-operator --ignore-not-found

# Remove cert resources created for Perses mTLS
oc --context=ossm-kiali-spoke delete configmap perses-acm-server-ca \
  -n openshift-cluster-observability-operator --ignore-not-found
oc --context=ossm-kiali-spoke delete secret perses-acm-client-certs \
  -n openshift-cluster-observability-operator --ignore-not-found

# Delete the perses namespace
oc --context=ossm-kiali-spoke delete namespace perses --ignore-not-found

# Remove Perses CRDs only (do NOT remove monitoring.rhobs or observability.openshift.io CRDs —
# those are needed by COO for MCOA and are removed in Guide 1 cleanup)
for suffix in perses.dev; do
  CRDS=$(oc --context=ossm-kiali-spoke get crd \
    --no-headers -o custom-columns=NAME:.metadata.name 2>/dev/null \
    | grep "\.${suffix}$")
  [ -n "${CRDS}" ] && echo "${CRDS}" | xargs oc --context=ossm-kiali-spoke delete crd --ignore-not-found
done

Remove Tempo

# Re-export variables if running cleanup in a fresh shell
export SPOKE_TWO_CLUSTER_NAME="spoke-two"
export TEMPO_NAMESPACE="tempo"
export TEMPO_STACK_NAME="istio"
export TEMPO_TENANT="mesh1"
REMOTE_COLLECTOR_NAME="otel-remote-${SPOKE_TWO_CLUSTER_NAME}"
REMOTE_MTLS_CLIENT_BUNDLE="${REMOTE_COLLECTOR_NAME}-client-bundle"

# Revert Kiali mesh and self-tracing config
oc --context=ossm-kiali-spoke patch kiali kiali -n istio-system --type=json \
  -p '[{"op":"remove","path":"/spec/external_services/tracing"},{"op":"remove","path":"/spec/server/observability/tracing"}]' 2>/dev/null || true

# Remove the distributed tracing console plugin
oc --context=ossm-kiali-spoke delete uiplugin distributed-tracing --ignore-not-found

# Remove Telemetry CRs and meshConfig tracing from both clusters
for CTX in ossm-kiali-spoke ossm-kiali-spoke-two; do
  oc --context="${CTX}" delete telemetry otel-tracing -n istio-system --ignore-not-found
  for NS in ambient-demo bookinfo; do
    oc --context="${CTX}" delete telemetry "${NS}-tracing" -n "${NS}" --ignore-not-found
  done
  oc --context="${CTX}" patch istio default --type=json \
    -p '[{"op":"remove","path":"/spec/values/meshConfig/extensionProviders"},{"op":"remove","path":"/spec/values/meshConfig/enableTracing"}]' \
    2>/dev/null || true
done

# Remove spoke-two forwarder and client bundle secret
oc --context=ossm-kiali-spoke-two delete opentelemetrycollector otel \
  -n istio-system --ignore-not-found
oc --context=ossm-kiali-spoke-two delete secret "${REMOTE_MTLS_CLIENT_BUNDLE}" \
  -n istio-system --ignore-not-found

# Remove spoke collectors, Route, certs, and RBAC
oc --context=ossm-kiali-spoke delete opentelemetrycollector otel \
  "${REMOTE_COLLECTOR_NAME}" -n istio-system --ignore-not-found
oc --context=ossm-kiali-spoke delete route "${REMOTE_COLLECTOR_NAME}" \
  -n istio-system --ignore-not-found
oc --context=ossm-kiali-spoke delete secret \
  "${REMOTE_COLLECTOR_NAME}-ca" \
  "${REMOTE_COLLECTOR_NAME}-server-tls" \
  "${REMOTE_COLLECTOR_NAME}-client-tls" \
  -n istio-system --ignore-not-found
oc --context=ossm-kiali-spoke delete clusterrolebinding \
  tempostack-traces-write-collectors \
  "tempostack-traces-reader-${TEMPO_TENANT}" --ignore-not-found
oc --context=ossm-kiali-spoke delete clusterrole \
  tempostack-traces-write \
  "tempostack-traces-reader-${TEMPO_TENANT}" --ignore-not-found

# Remove TempoStack, SeaweedFS, and namespace
oc --context=ossm-kiali-spoke delete tempostack "${TEMPO_STACK_NAME}" \
  -n "${TEMPO_NAMESPACE}" --ignore-not-found
oc --context=ossm-kiali-spoke delete deployment seaweedfs -n "${TEMPO_NAMESPACE}" --ignore-not-found
oc --context=ossm-kiali-spoke delete pvc seaweedfs-pv-claim -n "${TEMPO_NAMESPACE}" --ignore-not-found
oc --context=ossm-kiali-spoke delete secret tempostack-dev-seaweedfs \
  -n "${TEMPO_NAMESPACE}" --ignore-not-found
oc --context=ossm-kiali-spoke delete namespace "${TEMPO_NAMESPACE}" --ignore-not-found

# Remove Subscriptions (OpenTelemetry on both clusters, Tempo on spoke only)
for CTX in ossm-kiali-spoke ossm-kiali-spoke-two; do
  oc --context="${CTX}" delete subscriptions.operators.coreos.com opentelemetry-product \
    -n openshift-operators --ignore-not-found
done
oc --context=ossm-kiali-spoke delete subscriptions.operators.coreos.com tempo-product \
  -n openshift-operators --ignore-not-found

# Delete pending install plans before removing CSVs — otherwise OLM may recreate CSVs from in-flight plans
for CTX in ossm-kiali-spoke ossm-kiali-spoke-two; do
  oc --context="${CTX}" delete installplan -n openshift-operators --all --ignore-not-found
done

# Remove ALL CSVs — delete the CSV in the operator namespace only; OLM cascades deletion to all copied namespaces automatically
for CTX in ossm-kiali-spoke ossm-kiali-spoke-two; do
  CSV=$(oc --context="${CTX}" get csv -n openshift-operators \
    -l operators.coreos.com/opentelemetry-product.openshift-operators \
    --no-headers -o custom-columns=NAME:.metadata.name 2>/dev/null | head -1)
  if [ -n "${CSV}" ]; then oc --context="${CTX}" delete csv "${CSV}" -n openshift-operators --ignore-not-found; fi
done
CSV=$(oc --context=ossm-kiali-spoke get csv -n openshift-operators \
  -l operators.coreos.com/tempo-product.openshift-operators \
  --no-headers -o custom-columns=NAME:.metadata.name 2>/dev/null | head -1)
if [ -n "${CSV}" ]; then oc --context=ossm-kiali-spoke delete csv "${CSV}" -n openshift-operators --ignore-not-found; fi

# Remove ALL CRDs — you must remove every CRD installed by the Tempo and OpenTelemetry operators or reinstallation will conflict
for CTX in ossm-kiali-spoke ossm-kiali-spoke-two; do
  CRDS=$(oc --context="${CTX}" get crd \
    --no-headers -o custom-columns=NAME:.metadata.name 2>/dev/null \
    | grep "\.opentelemetry\.io$")
  [ -n "${CRDS}" ] && echo "${CRDS}" | xargs oc --context="${CTX}" delete crd --ignore-not-found
done
CRDS=$(oc --context=ossm-kiali-spoke get crd \
  --no-headers -o custom-columns=NAME:.metadata.name 2>/dev/null \
  | grep "\.tempo\.grafana\.com$")
[ -n "${CRDS}" ] && echo "${CRDS}" | xargs oc --context=ossm-kiali-spoke delete crd --ignore-not-found

# Remove the ClusterRole installed by the OpenTelemetry Operator
for CTX in ossm-kiali-spoke ossm-kiali-spoke-two; do
  oc --context="${CTX}" delete clusterrole opentelemetry-operator-metrics-reader --ignore-not-found
done

Remove COO

After the Perses resources have been removed, remove the full COO installation created in Phase 1.1. Leave the monitoring.rhobs CRDs in place because they are managed by MCOA for metric federation:

COO_CSV=$(oc --context=ossm-kiali-spoke get subscription cluster-observability-operator \
  -n openshift-cluster-observability-operator \
  -o jsonpath='{.status.installedCSV}' 2>/dev/null || true)
oc --context=ossm-kiali-spoke delete subscription cluster-observability-operator \
  -n openshift-cluster-observability-operator --ignore-not-found
[ -z "${COO_CSV}" ] || oc --context=ossm-kiali-spoke delete csv "${COO_CSV}" \
  -n openshift-cluster-observability-operator --ignore-not-found
oc --context=ossm-kiali-spoke delete namespace openshift-cluster-observability-operator \
  --ignore-not-found --wait=false