Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 13 additions & 6 deletions .wordlist.txt
Original file line number Diff line number Diff line change
Expand Up @@ -39,10 +39,10 @@ anattama
anonymized
anonymizer
ansible
api's
apicast
apicurito
apis
api's
apiversion
appdev
applicati
Expand Down Expand Up @@ -75,11 +75,11 @@ awsregion
awx
azs
backend
backend's
backends
backend's
baldessari
baremetal
baremetal integrations
baremetal
baseos
baz
bcb
Expand Down Expand Up @@ -514,8 +514,10 @@ kafdrop
kafkasource
kafkatopic
kafkatopics
kairos
kam
kamelet
kaoto
kasten
kastendr
kata
Expand All @@ -538,8 +540,10 @@ kno
koppa
koqh
kqhdzjvisxrurtnwackiemb
kserve
kservice
kstreams
kuadrant
kube
kubeadmin
kubeconfig
Expand Down Expand Up @@ -587,6 +591,7 @@ machineconfigs
machineset
macos
macosx
mailpit
mailto
maistra
makefile
Expand Down Expand Up @@ -708,8 +713,8 @@ opendatahub
openid
openjdk
openshift
openshift's
openshiftpullsecret
openshift's
openshiftsdn
openshiftversion
openssl
Expand Down Expand Up @@ -831,9 +836,9 @@ renderers
replicaset
replicasets
repo
repo's
repolist
repos
repo's
repourl
reranked
reranking
Expand Down Expand Up @@ -911,6 +916,7 @@ sigstore
siteadmin
skipdryrunonmissingresource
skopeo
skupper
sla
slas
sme
Expand Down Expand Up @@ -1037,8 +1043,8 @@ unsealvault
untrusted
updatingconfig
updatingversion
upstream's
upstreaming
upstream's
ure
uri
usecsv
Expand Down Expand Up @@ -1117,4 +1123,5 @@ zh
zj
zja
zk
ztunnel
zwkq
138 changes: 138 additions & 0 deletions content/patterns/hybrid-mesh-platform/_index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
---
title: Hybrid Mesh Platform
date: 2026-06-15
tier: sandbox
summary: Hub-spoke multi-cluster GitOps on OpenShift with ACM, ambient Service Mesh, Skupper, Industrial Edge, and centralized observability.
rh_products:
- Red Hat OpenShift Container Platform
- Red Hat Advanced Cluster Management
- Red Hat OpenShift GitOps
- Red Hat Advanced Cluster Security for Kubernetes
- Red Hat OpenShift Service Mesh
- Red Hat Connectivity Link
- Red Hat OpenShift AI
- Red Hat AMQ Streams
- Red Hat build of Apache Camel
- Red Hat OpenShift Pipelines
- Red Hat Developer Hub
- Red Hat Service Interconnect
industries:
- General
- Industrial
focus_areas:
- Edge
- DevSecOps
- AI
- Observability
aliases: /hybrid-mesh-platform/
links:
github: https://github.com/maximilianoPizarro/hybrid-mesh-platform
install: getting-started
bugs: https://github.com/maximilianoPizarro/hybrid-mesh-platform/issues
feedback: https://docs.google.com/forms/d/e/1FAIpQLScI76b6tD1WyPu2-d_9CCVDr3Fu5jYERthqLKJDUGwqBg7Vcg/viewform
tested_on:
platform: AWS
ocp_version: "4.17+"
topology: "3 clusters (hub + east spoke + west spoke)"
contributor:
name: Maximiliano Pizarro
contact: mailto:maximilianopizarro5@gmail.com
git: https://github.com/maximilianoPizarro
---

# Hybrid Mesh Platform

**Maintainer:** Maximiliano Pizarro, Specialist Solution Architect at Red Hat

> **Your journey:** Install via the Validated Patterns framework (`./pattern.sh install`), connect three OpenShift clusters (hub + east + west) through ACM managedClusterGroups, and observe IoT sensor data across Grafana and Developer Hub. The pages below follow one continuous story — concept, install, operate, scaffold — so you can read straight through or jump to any chapter.

## What is Hybrid Mesh Platform?

**Hybrid Mesh Platform** is a production-grade, multi-cluster GitOps reference architecture that mirrors how Red Hat customers run hybrid cloud on OpenShift. It implements a **hub-spoke topology** where:

- A **hub cluster** centralizes fleet governance with **ACM**, deploys via **OpenShift GitOps** (Argo CD), hosts **Developer Hub**, runs **ACS Central**, aggregates observability in **Grafana**, and exposes cross-cluster services through a **Gateway API** hub gateway.
- Two **spoke clusters** (east and west) execute **Industrial Edge** factory workloads — MQTT sensors, Kafka pipelines, ML inference, and dashboards — connected to the hub via a **Skupper Virtual Application Network** (no VPN or firewall changes).
- **OpenShift Service Mesh 3** in **ambient mode** provides ztunnel-based L4 encryption and optional waypoint L7 policy across all clusters.
- **Connectivity Link (Kuadrant)** layers API-aware ingress policies — rate limiting, auth, DNS/TLS automation — on top of Gateway API.

**Tested on:** Red Hat OpenShift Container Platform **4.17+** on **AWS** (hub + east spoke + west spoke). See [Cluster sizing](cluster-sizing) for recommended instance types.

**Implementation repo:** [hybrid-mesh-platform](https://github.com/maximilianoPizarro/hybrid-mesh-platform) — Validated Patterns layout (`clustergroup`, Vault + External Secrets, ACM managedClusterGroups).

Read **concept → mechanics → operations**: start with [Architecture](architecture), install via [Getting Started](getting-started), explore the [Demo scenario](demo-scenario), scaffold workloads via [Scaffolding](scaffolding), then use platform chapters (**Hub Gateway**, **Observability**, **Industrial Edge**).

[![Hybrid Mesh Platform — hub-spoke architecture](/images/hybrid-mesh-platform/workshop-hybrid-mesh.png)](/images/hybrid-mesh-platform/workshop-hybrid-mesh.png)

_Hub cluster aggregates observability and Developer Hub; east and west spokes run Industrial Edge workloads connected via Service Interconnect (Skupper)._

## Hub-spoke architecture at a glance

| Cluster | Role | Key components |
| --- | --- | --- |
| **Hub** | Fleet governance and centralized services | ACM, OpenShift GitOps, Developer Hub, OpenShift AI, Service Mesh control plane, Skupper listeners, Kuadrant, ACS Central, Grafana, Kafka Console, Kubecost |
| **East spoke** | Factory workloads and developer tools | Industrial Edge, DevSpaces, Kairos SmartScaling, spoke-local GitOps |
| **West spoke** | Workload replicas and cross-cluster validation | Industrial Edge replicas, MirrorMaker replication to hub, Skupper connectors |

Industrial Edge components exist **only** on spokes. The hub aggregates metrics and provides gateway access — it does not host factory sensor workloads.

[![Platform architecture overview](/images/hybrid-mesh-platform/arch-overview.png)](/images/hybrid-mesh-platform/arch-overview.png)

_Detailed architecture showing Git repo structure, ACM placement, Skupper VAN, and sync-wave delivery to east/west spokes._

## Quick links

| Topic | Page |
| --- | --- |
| Architecture deep dive | [Architecture](architecture) |
| Install flow | [Getting Started](getting-started) |
| Cluster sizing | [Cluster sizing](cluster-sizing) |
| Demo scenario and showroom | [Demo scenario](demo-scenario) |
| Hub Gateway and Connectivity Link | [Hub Gateway](hub-gateway) |
| Observability | [Observability](observability) |
| Industrial Edge (multi-cluster) | [Industrial Edge](industrial-edge) |
| Scaffolding | [Scaffolding](scaffolding) |
| Customization ideas | [Ideas for customization](ideas-for-customization) |

## Recommended reading order

1. [Architecture](architecture) — mental model of hub, spokes, GitOps, Skupper, and observability
2. [Getting Started](getting-started) — bring clusters under GitOps (ACM + ApplicationSet)
3. [Cluster sizing](cluster-sizing) — hub and spoke minimum requirements
4. [Demo scenario](demo-scenario) — what the workshop showroom demonstrates
5. [Scaffolding](scaffolding) — deploy Industrial Edge instances from Developer Hub
6. [Hub Gateway](hub-gateway) — weighted ingress and circuit breaking across spokes
7. [Observability](observability) — Grafana, Kiali, Kafka Console
8. [Industrial Edge](industrial-edge) — factory data pipeline on multiple spokes

**Next →** [Architecture](architecture)

## Workshop Showroom

A **Hybrid Mesh AI Workshop Showroom** provides an explanatory, navigable view of the same product surfaces after deployment — hub-spoke diagrams, ACM fleet, mesh, Industrial Edge, observability, ACS, and OpenShift AI.

| Resource | Link |
| --- | --- |
| What the demo shows (on this site) | [Demo scenario](demo-scenario) |
| Showroom content repository | [showroom-hybrid-mesh-ai](https://github.com/maximilianoPizarro/showroom-hybrid-mesh-ai) |
| Extended pattern docs (RHDP, GitOps chain, troubleshooting) | [GitHub Pages documentation](https://maximilianopizarro.github.io/hybrid-mesh-platform/) |

Hands-on lab modules and registration flows remain in the showroom repository and deployed workshop environment — not duplicated here.

## Support

This is a **Sandbox tier** Validated Pattern with community best-effort support. See [SUPPORT.md](https://github.com/maximilianoPizarro/hybrid-mesh-platform/blob/main/SUPPORT.md) in the pattern repository.

## Red Hat products used

- Red Hat OpenShift Container Platform
- Red Hat Advanced Cluster Management for Kubernetes
- Red Hat OpenShift GitOps (Argo CD)
- Red Hat Advanced Cluster Security for Kubernetes
- Red Hat OpenShift Service Mesh
- Red Hat Connectivity Link (Kuadrant, Gateway API)
- Red Hat OpenShift AI
- Red Hat AMQ Streams (Apache Kafka)
- Red Hat build of Apache Camel / Camel K
- Red Hat OpenShift Pipelines (Tekton)
- Red Hat Developer Hub (Backstage)
- Red Hat Service Interconnect (Skupper)
121 changes: 121 additions & 0 deletions content/patterns/hybrid-mesh-platform/architecture.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
---
title: Architecture
weight: 20
aliases: /hybrid-mesh-platform/architecture/
---

# Architecture

## Hub-spoke theory in multi-cluster Kubernetes

In multi-cluster Kubernetes, a **hub-spoke** model designates one administrative cluster (the **hub**) and one or more workload clusters (**spokes**). The hub owns fleet APIs: cluster inventory, policy placement, credentials for spoke registration, and often centralized GitOps controllers that fan out desired state.

Spokes remain the execution venues for application namespaces, data-plane components (Kafka, MQTT bridges, mesh dataplane), and regional isolation for latency, data residency, or blast-radius control.

## Why hub-spoke?

| Benefit | Description |
| --------|------------- |
| **Centralized management** | One control plane for membership, RBAC patterns, and bulk upgrades. |
| **Policy enforcement** | Kubernetes policies, compliance checks, and security baselines propagate from the hub. |
| **Observability** | Aggregated metrics, logging, and tracing strategies start at the hub and uniform dashboards span spokes. |
| **GitOps consistency** | A single Git revision (`main`) with region paths drives spoke drift correction. |

## Platform architecture overview

![Hub-spoke platform — Git paths, ApplicationSet, Skupper VAN, and per-cluster components](/images/hybrid-mesh-platform/arch-hub-spoke-flow.png)
*Single `main` branch: hub at `charts/region/hub`, spokes at `charts/region/east` and `charts/region/west`, shared charts under `charts/all/`.*
## Follow the request — one temperature reading end to end

When a machine sensor on the **east** spoke publishes a temperature sample, the path is: **MQTT** (`messaging` broker) → **Camel K** (`mqtt-to-kafka` integration) → **Kafka** (`dev-cluster` topic) → optional **ML scoring** (KServe) → **line-dashboard** WebSocket consumer. In parallel, **Thanos Querier** on east scrapes Istio and Kafka metrics; a **Skupper Connector** (`prometheus-east`) tunnels HTTP to the hub, where **Grafana** datasource `prometheus-east` plots the series. The **Hub Gateway** can route browser traffic to the east line-dashboard via **spoke-gateway** and Skupper listener `ie-gateway-east`. Developer Hub **Topology** shows the same pods when the catalog entity carries `backstage.io/kubernetes-cluster: east` and spoke API tokens are synced.

## Components on the hub vs spokes

| Area | Hub | Spokes | Config path |
| -----|-----|--------|-------------|
| ACM hub operator & APIs | yes | | `charts/region/hub/values.yaml` |
| ArgoCD / clustergroup root | yes | yes | `charts/region/hub` / `charts/region/east` / `charts/region/west` |
| ApplicationSet (spoke apps) | yes | | `charts/region/hub/values.yaml` |
| ACS Central | yes | | `charts/region/hub/values.yaml` |
| ACS Secured Cluster | | yes | `charts/region/east|west/values.yaml` |
| Developer Hub | yes | | `charts/region/hub/values.yaml` |
| Hub Gateway (Gateway API) | yes | | `charts/region/hub/values.yaml` |
| Spoke Gateway (Gateway API) | | yes | `charts/region/east|west/values.yaml` |
| Industrial Edge workloads | | yes | `charts/region/east|west/values.yaml` |
| Kafka brokers (regional) | | yes | `charts/region/east|west/values.yaml` |
| Service Mesh ambient / ztunnel | yes | yes | both |
| Istio CNI (`profile: ambient`) | yes | yes | both |
| Skupper Site (hub listeners) | yes | | `charts/region/hub/values.yaml` |
| Skupper Site (spoke connectors) | | yes | `charts/region/east|west/values.yaml` |
| Grafana (multi-cluster dashboards) | yes | | `charts/region/hub/values.yaml` |
| Grafana (local metrics) | | yes | `charts/region/east|west/values.yaml` |
| Kiali + OSSM Console plugin | yes | yes | both |
| Connectivity Link (RHCL) | yes | yes | both |
| Kubecost (primary aggregator) | yes | | `charts/region/hub/values.yaml` |
| Kubecost (agent) | | yes | `charts/region/east|west/values.yaml` |
| Kafka Console (all clusters) | yes | | `charts/region/hub/values.yaml` |

## GitOps application delivery flow

See **[GitOps deployment chain](https://maximilianopizarro.github.io/hybrid-mesh-platform/validatedpatterns-docs/gitops-deployment-chain.html)** for the full encadenamiento (hub `field-content-*` → ApplicationSet `fleet-spoke-push` → `*-spoke-components` → spoke `*-east` / `*-west` apps) with copy-paste YAML fragments.

![GitOps sequence — hub Argo CD, ApplicationSet, remote spoke sync](/images/hybrid-mesh-platform/arch-gitops-sync-sequence.png)
*Hub syncs first; ApplicationSet pushes per-spoke charts; each spoke Argo CD reconciles child Applications locally.*
## Sync wave ordering

Components deploy in strict order via ArgoCD sync waves:

![Argo CD sync wave ordering from bootstrap through dashboards](/images/hybrid-mesh-platform/arch-sync-waves.png)
*Sync waves prevent operators from racing workloads — mesh and namespaces land before Industrial Edge and gateways.*
### Spoke sync-wave reference

Matches ebook Ch.4 ordering (`charts/region/east/values.yaml`, `charts/region/west/values.yaml`):

| Wave | What deploys | Why this order |
| ---- | ------------ | -------------- |
| 1 | Namespaces (no ambient label yet) | Names must exist before operators and workloads |
| 2 | OLM Subscriptions | CRDs and operators installed |
| 3 | Service Mesh 3 (Istio + ZTunnel + ambient labels wave 2 inside chart) | Mesh dataplane before application pods |
| 4 | Observability, ACS secured cluster | Scraping and security after mesh |
| 5 | Industrial Edge (Kafka, sensors, dashboard) | Pods enroll in ambient with HBONE ready |
| 6 | Spoke gateway + Skupper interconnect | Routing after backends exist |

Hub chart uses a similar pattern; ApplicationSet for spokes runs at hub wave **5** after ACM placement is healthy.

## Service Interconnect (Skupper) topology

Red Hat Service Interconnect creates a Virtual Application Network (VAN) that bridges services across clusters without VPN or direct network connectivity.

![Skupper VAN — hub Listeners, spoke Connectors, AccessGrant and AccessToken](/images/hybrid-mesh-platform/arch-skupper-topology.png)
*Connectors expose spoke-gateway and prometheus-auth-proxy; Listeners materialize ClusterIP services on the hub.*
## Spoke gateway aggregation

Each spoke runs a **Gateway API gateway** that fronts all Industrial Edge services, providing a single entry point for Skupper to expose to the hub.

![Spoke gateway aggregates Industrial Edge HTTP routes for Skupper](/images/hybrid-mesh-platform/arch-spoke-gateway.png)
*One Connector per spoke exposes the gateway instead of every microservice individually.*
## Multi-cluster observability pipeline

![Multi-cluster observability — spoke metrics via Skupper into hub Grafana](/images/hybrid-mesh-platform/arch-observability-pipeline.png)
*Spoke Thanos Querier is reached through nginx auth-proxy Connectors; hub Grafana uses HTTP datasources without bearer tokens.*
## Data flow (sensors to dashboard)

![Industrial Edge data flow — sensors through MQTT, Camel, Kafka to Grafana and data lake](/images/hybrid-mesh-platform/arch-data-flow.png)
*Telemetry path on each spoke; MirrorMaker replicates to the hub-centralized MinIO data lake.*
## Comparison with Red Hat Validated Patterns

The **[Multicloud GitOps](https://validatedpatterns.io/patterns/multicloud-gitops)** validated pattern demonstrates fleet GitOps with OpenShift GitOps and ACM patterns that resemble this repository's hub-push model: a declarative root Application, cluster grouping, and progressive rollout.

This platform extends that idea with **Industrial Edge** workloads, **Service Mesh ambient**, **Connectivity Link**, optional **OpenShift AI**, **ACS** depth, and **Service Interconnect** for cross-cluster service exposure -- tuned for factory-style telemetry and east-west observability rather than only infrastructure provisioning.

---

**Next →** translate diagrams into installs via **[Getting Started](getting-started)**, scaffold new edge instances via **[Scaffolding](scaffolding)**, then follow **[Observability](observability)** once workloads expose Prometheus signals. For ACM placement detail and additional reference pages, see the [pattern documentation site](https://maximilianopizarro.github.io/hybrid-mesh-platform/validatedpatterns-docs/).

## Official documentation

- [ACM Architecture](https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.16/html/about/welcome-to-red-hat-advanced-cluster-management-for-kubernetes)
- [Multicloud GitOps Pattern](https://validatedpatterns.io/patterns/multicloud-gitops)
- [Red Hat Service Interconnect](https://docs.redhat.com/en/documentation/red_hat_service_interconnect/2.1)
- [Kubernetes Gateway API](https://gateway-api.sigs.k8s.io/)
- [Argo CD ApplicationSet Generators](https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators/)
22 changes: 22 additions & 0 deletions content/patterns/hybrid-mesh-platform/cluster-sizing.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
---
title: Cluster sizing
weight: 30
aliases: /hybrid-mesh-platform/cluster-sizing/
---

:toc:
:imagesdir: /images
:_content-type: ASSEMBLY
include::modules/comm-attributes.adoc[]
include::modules/hybrid-mesh-platform/metadata-hybrid-mesh-platform.adoc[]

include::modules/cluster-sizing-template.adoc[]

[id="hybrid-mesh-platform-additional-requirements"]
== Additional requirements

The Hybrid Mesh Platform pattern deploys a **hub** cluster plus **two spoke** clusters (east and west). Plan for three OpenShift clusters at the recommended sizes above.

Optional features such as OpenShift AI workbenches and MaaS-backed inference require additional hub capacity. Workshop Showroom content is maintained in a [separate repository](https://github.com/maximilianoPizarro/showroom-hybrid-mesh-ai) and does not change minimum cluster sizing for the core pattern.

For RHDP catalog provisioning details and validation steps, see the [pattern documentation site](https://maximilianopizarro.github.io/hybrid-mesh-platform/).
Loading