diff --git a/.wordlist.txt b/.wordlist.txt index 46ca445a7..2332cdcb0 100644 --- a/.wordlist.txt +++ b/.wordlist.txt @@ -39,10 +39,10 @@ anattama anonymized anonymizer ansible -api's apicast apicurito apis +api's apiversion appdev applicati @@ -75,11 +75,11 @@ awsregion awx azs backend -backend's backends +backend's baldessari -baremetal baremetal integrations +baremetal baseos baz bcb @@ -514,8 +514,10 @@ kafdrop kafkasource kafkatopic kafkatopics +kairos kam kamelet +kaoto kasten kastendr kata @@ -538,8 +540,10 @@ kno koppa koqh kqhdzjvisxrurtnwackiemb +kserve kservice kstreams +kuadrant kube kubeadmin kubeconfig @@ -587,6 +591,7 @@ machineconfigs machineset macos macosx +mailpit mailto maistra makefile @@ -708,8 +713,8 @@ opendatahub openid openjdk openshift -openshift's openshiftpullsecret +openshift's openshiftsdn openshiftversion openssl @@ -831,9 +836,9 @@ renderers replicaset replicasets repo -repo's repolist repos +repo's repourl reranked reranking @@ -911,6 +916,7 @@ sigstore siteadmin skipdryrunonmissingresource skopeo +skupper sla slas sme @@ -1037,8 +1043,8 @@ unsealvault untrusted updatingconfig updatingversion -upstream's upstreaming +upstream's ure uri usecsv @@ -1117,4 +1123,5 @@ zh zj zja zk +ztunnel zwkq diff --git a/content/patterns/hybrid-mesh-platform/_index.md b/content/patterns/hybrid-mesh-platform/_index.md new file mode 100644 index 000000000..3e597f720 --- /dev/null +++ b/content/patterns/hybrid-mesh-platform/_index.md @@ -0,0 +1,152 @@ +--- +title: Hybrid Mesh Platform +date: 2026-06-15 +tier: sandbox +summary: Hub-spoke multi-cluster GitOps on OpenShift combining Service Mesh, AI-assisted operations (OpenShift Lightspeed + MCP), Industrial Edge, and a Red Hat Device Edge extension path via MicroShift. +rh_products: + - Red Hat OpenShift Container Platform + - Red Hat Advanced Cluster Management + - Red Hat OpenShift GitOps + - Red Hat Advanced Cluster Security for Kubernetes + - Red Hat OpenShift Service Mesh + - Red Hat Connectivity Link + - Red Hat OpenShift AI + - Red Hat AMQ Streams + - Red Hat build of Apache Camel + - Red Hat OpenShift Pipelines + - Red Hat Developer Hub + - Red Hat Service Interconnect +industries: + - General + - Industrial +focus_areas: + - Edge + - DevSecOps + - AI + - Observability +aliases: /hybrid-mesh-platform/ +links: + github: https://github.com/maximilianoPizarro/hybrid-mesh-platform + install: getting-started + bugs: https://github.com/maximilianoPizarro/hybrid-mesh-platform/issues + feedback: https://docs.google.com/forms/d/e/1FAIpQLScI76b6tD1WyPu2-d_9CCVDr3Fu5jYERthqLKJDUGwqBg7Vcg/viewform +tested_on: + platform: AWS + ocp_version: "4.17+" + topology: "3 clusters (hub + east spoke + west spoke)" +contributor: + name: Maximiliano Pizarro + contact: mailto:maximilianopizarro5@gmail.com + git: https://github.com/maximilianoPizarro +--- + +# Hybrid Mesh Platform + +**Maintainer:** Maximiliano Pizarro, Specialist Solution Architect at Red Hat + +## The problem this pattern solves + +Operating a multi-cluster OpenShift fleet creates three compounding challenges that a Service Mesh alone cannot address: + +| Challenge | Without this pattern | With Hybrid Mesh Platform | +| --- | --- | --- | +| **Cross-cluster connectivity** | Site-to-site VPNs, manual firewall rules per pair of clusters | Skupper Virtual Application Network — outbound-only mTLS, no inbound firewall changes | +| **Fleet governance drift** | Each cluster managed independently; configurations diverge over time | Single `main` branch drives hub + east + west via ACM + dual GitOps (PUSH ApplicationSet + PULL clustergroup) | +| **AI-assisted operations** | Operators react to incidents by parsing dashboards and YAML | OpenShift Lightspeed + MCP Gateway let operators act on platform state in natural language, reducing MTTA on infrastructure incidents | + +**Goal:** This pattern combines Red Hat Service Mesh for secure inter-service connectivity with OpenShift AI (MaaS + vLLM) and OpenShift Lightspeed + MCP for natural-language platform operations — giving teams centralized GitOps governance, secure cross-cluster communication, and AI-assisted incident response in a single deployable reference architecture. For physically constrained factory devices, the architecture extends naturally to **Red Hat Device Edge** with **MicroShift**, managed by the same ACM placement rules without hub-side changes. + +> **Your journey:** Install via the Validated Patterns framework (`./pattern.sh install`), connect three OpenShift clusters (hub + east + west) through ACM `managedClusterGroups`, and observe IoT sensor data across Grafana and Developer Hub. The pages below follow one continuous story — concept, install, operate, scaffold. + +## What is Hybrid Mesh Platform? + +**Hybrid Mesh Platform** is a production-grade, multi-cluster GitOps reference architecture that mirrors how Red Hat customers run hybrid cloud on OpenShift. It implements a **hub-spoke topology** where: + +- A **hub cluster** centralizes fleet governance with **ACM**, deploys via **OpenShift GitOps** (Argo CD), hosts **Developer Hub**, runs **ACS Central**, aggregates observability in **Grafana**, and exposes cross-cluster services through a **Gateway API** hub gateway. +- Two **spoke clusters** (east and west) execute **Industrial Edge** factory workloads — MQTT sensors, Kafka pipelines, ML inference, and dashboards — connected to the hub via a **Skupper Virtual Application Network** (no VPN or firewall changes). +- **OpenShift Service Mesh 3** in **ambient mode** provides ztunnel-based L4 encryption and optional waypoint L7 policy across all clusters. +- **Connectivity Link (Kuadrant)** layers API-aware ingress policies — rate limiting, auth, DNS/TLS automation — on top of Gateway API. + +**Tested on:** Red Hat OpenShift Container Platform **4.17+** on **AWS** (hub + east spoke + west spoke, 3 workers each). + +**Multi-cluster topology:** this is a **hub + two spokes** pattern (not single-cluster). All three clusters are required; standalone single-cluster deployment is not supported by default. See [Cluster sizing](cluster-sizing) for minimum instance types per role. + +**Implementation repo:** [hybrid-mesh-platform](https://github.com/maximilianoPizarro/hybrid-mesh-platform) — Validated Patterns layout (`clustergroup`, Vault + External Secrets, ACM managedClusterGroups). + +Read **concept → mechanics → operations**: start with [Architecture](architecture), install via [Getting Started](getting-started), explore the [Demo scenario](demo-scenario), scaffold workloads via [Scaffolding](scaffolding), then use platform chapters (**Hub Gateway**, **Observability**, **Industrial Edge**). + +[![Hybrid Mesh Platform — hub-spoke architecture](/images/hybrid-mesh-platform/workshop-hybrid-mesh.png)](/images/hybrid-mesh-platform/workshop-hybrid-mesh.png) + +_Hub cluster aggregates observability and Developer Hub; east and west spokes run Industrial Edge workloads connected via Service Interconnect (Skupper)._ + +## Hub-spoke architecture at a glance + +| Cluster | Role | Key components | +| --- | --- | --- | +| **Hub** | Fleet governance and centralized services | ACM, OpenShift GitOps, Developer Hub, OpenShift AI, Service Mesh control plane, Skupper listeners, Kuadrant, ACS Central, Grafana, Kafka Console, Kubecost | +| **East spoke** | Factory workloads and developer tools | Industrial Edge, DevSpaces, Kairos SmartScaling, spoke-local GitOps | +| **West spoke** | Workload replicas and cross-cluster validation | Industrial Edge replicas, MirrorMaker replication to hub, Skupper connectors | + +Industrial Edge components exist **only** on spokes. The hub aggregates metrics and provides gateway access — it does not host factory sensor workloads. + +[![Platform architecture overview](/images/hybrid-mesh-platform/arch-overview.png)](/images/hybrid-mesh-platform/arch-overview.png) + +_Detailed architecture showing Git repo structure, ACM placement, Skupper VAN, and sync-wave delivery to east/west spokes._ + +## Quick links + +| Topic | Page | +| --- | --- | +| Architecture deep dive | [Architecture](architecture) | +| Install flow | [Getting Started](getting-started) | +| Cluster sizing | [Cluster sizing](cluster-sizing) | +| Demo scenario and showroom | [Demo scenario](demo-scenario) | +| Hub Gateway and Connectivity Link | [Hub Gateway](hub-gateway) | +| Observability | [Observability](observability) | +| Industrial Edge (multi-cluster) | [Industrial Edge](industrial-edge) | +| Scaffolding | [Scaffolding](scaffolding) | +| Customization ideas | [Ideas for customization](ideas-for-customization) | + +## Recommended reading order + +1. [Architecture](architecture) — mental model of hub, spokes, GitOps, Skupper, and observability +2. [Getting Started](getting-started) — bring clusters under GitOps (ACM + ApplicationSet) +3. [Cluster sizing](cluster-sizing) — hub and spoke minimum requirements +4. [Demo scenario](demo-scenario) — what the workshop showroom demonstrates +5. [Scaffolding](scaffolding) — deploy Industrial Edge instances from Developer Hub +6. [Hub Gateway](hub-gateway) — weighted ingress and circuit breaking across spokes +7. [Observability](observability) — Grafana, Kiali, Kafka Console +8. [Industrial Edge](industrial-edge) — factory data pipeline on multiple spokes + +**Next →** [Architecture](architecture) + +## Workshop Showroom + +A **Hybrid Mesh AI Workshop Showroom** provides an explanatory, navigable view of the same product surfaces after deployment — hub-spoke diagrams, ACM fleet, mesh, Industrial Edge, observability, ACS, and OpenShift AI. + +| Resource | Link | +| --- | --- | +| What the demo shows (on this site) | [Demo scenario](demo-scenario) | +| Showroom content repository | [showroom-hybrid-mesh-ai](https://github.com/maximilianoPizarro/showroom-hybrid-mesh-ai) | +| Extended pattern docs (RHDP, GitOps chain, troubleshooting) | [GitHub Pages documentation](https://maximilianopizarro.github.io/hybrid-mesh-platform/) | + +Hands-on lab modules and registration flows remain in the showroom repository and deployed workshop environment — not duplicated here. + +## Support + +This is a **Sandbox tier** Validated Pattern with community best-effort support. See [SUPPORT.md](https://github.com/maximilianoPizarro/hybrid-mesh-platform/blob/main/SUPPORT.md) in the pattern repository. + +## Red Hat products used + +- Red Hat OpenShift Container Platform +- Red Hat Advanced Cluster Management for Kubernetes +- Red Hat OpenShift GitOps (Argo CD) +- Red Hat Advanced Cluster Security for Kubernetes +- Red Hat OpenShift Service Mesh +- Red Hat Connectivity Link (Kuadrant, Gateway API) +- Red Hat OpenShift AI +- Red Hat AMQ Streams (Apache Kafka) +- Red Hat build of Apache Camel / Camel K +- Red Hat OpenShift Pipelines (Tekton) +- Red Hat Developer Hub (Backstage) +- Red Hat Service Interconnect (Skupper) diff --git a/content/patterns/hybrid-mesh-platform/architecture.md b/content/patterns/hybrid-mesh-platform/architecture.md new file mode 100644 index 000000000..509189611 --- /dev/null +++ b/content/patterns/hybrid-mesh-platform/architecture.md @@ -0,0 +1,156 @@ +--- +title: Architecture +weight: 20 +aliases: /hybrid-mesh-platform/architecture/ +--- + +# Architecture + +## Hub-spoke theory in multi-cluster Kubernetes + +In multi-cluster Kubernetes, a **hub-spoke** model designates one administrative cluster (the **hub**) and one or more workload clusters (**spokes**). The hub owns fleet APIs: cluster inventory, policy placement, credentials for spoke registration, and often centralized GitOps controllers that fan out desired state. + +Spokes remain the execution venues for application namespaces, data-plane components (Kafka, MQTT bridges, mesh dataplane), and regional isolation for latency, data residency, or blast-radius control. + +## Why hub-spoke? + +| Benefit | Description | +| --------|------------- | +| **Centralized management** | One control plane for membership, RBAC patterns, and bulk upgrades. | +| **Policy enforcement** | Kubernetes policies, compliance checks, and security baselines propagate from the hub. | +| **Observability** | Aggregated metrics, logging, and tracing strategies start at the hub and uniform dashboards span spokes. | +| **GitOps consistency** | A single Git revision (`main`) with region paths drives spoke drift correction. | + +## Multi-cluster topology: three required clusters + +This pattern requires **three OpenShift clusters** — not a single-cluster deployment. The topology follows the Validated Patterns hub-spoke model: one **hub** for fleet governance and two **spokes** for factory workloads. Hub and spokes communicate over a **Skupper Virtual Application Network** (outbound-only mTLS — no inbound firewall rule changes needed). The hub's `ApplicationSet` pushes spoke charts; each spoke's local Argo CD pulls its `clusterGroup` from Git autonomously. + +[![Hybrid Mesh Platform — hub-spoke component map](/images/hybrid-mesh-platform/workshop-hybrid-mesh-arch.png)](/images/hybrid-mesh-platform/workshop-hybrid-mesh-arch.png) + +_Platform component map: hub hosts fleet governance and AI services; east and west spokes host Industrial Edge workloads. Skupper VAN bridges cross-cluster service traffic without VPN._ + +## Platform architecture overview + +[![Hub-spoke platform — Git paths, ApplicationSet, Skupper VAN, and per-cluster components](/images/hybrid-mesh-platform/arch-hub-spoke-flow.png)](/images/hybrid-mesh-platform/arch-hub-spoke-flow.png) + +*Single `main` branch: hub at `charts/region/hub`, spokes at `charts/region/east` and `charts/region/west`, shared charts under `charts/all/`.* + +## Follow the request — one temperature reading end to end + +When a machine sensor on the **east** spoke publishes a temperature sample, the path is: **MQTT** (`messaging` broker) → **Camel K** (`mqtt-to-kafka` integration) → **Kafka** (`dev-cluster` topic) → optional **ML scoring** (KServe) → **line-dashboard** WebSocket consumer. In parallel, **Thanos Querier** on east scrapes Istio and Kafka metrics; a **Skupper Connector** (`prometheus-east`) tunnels HTTP to the hub, where **Grafana** datasource `prometheus-east` plots the series. The **Hub Gateway** can route browser traffic to the east line-dashboard via **spoke-gateway** and Skupper listener `ie-gateway-east`. Developer Hub **Topology** shows the same pods when the catalog entity carries `backstage.io/kubernetes-cluster: east` and spoke API tokens are synced. + +## Components on the hub vs spokes + +| Area | Hub | Spokes | Config path | +| -----|-----|--------|-------------| +| ACM hub operator & APIs | yes | | `charts/region/hub/values.yaml` | +| ArgoCD / clustergroup root | yes | yes | `charts/region/hub` / `charts/region/east` / `charts/region/west` | +| ApplicationSet (spoke apps) | yes | | `charts/region/hub/values.yaml` | +| ACS Central | yes | | `charts/region/hub/values.yaml` | +| ACS Secured Cluster | | yes | `charts/region/east|west/values.yaml` | +| Developer Hub | yes | | `charts/region/hub/values.yaml` | +| Hub Gateway (Gateway API) | yes | | `charts/region/hub/values.yaml` | +| Spoke Gateway (Gateway API) | | yes | `charts/region/east|west/values.yaml` | +| Industrial Edge workloads | | yes | `charts/region/east|west/values.yaml` | +| Kafka brokers (regional) | | yes | `charts/region/east|west/values.yaml` | +| Service Mesh ambient / ztunnel | yes | yes | both | +| Istio CNI (`profile: ambient`) | yes | yes | both | +| Skupper Site (hub listeners) | yes | | `charts/region/hub/values.yaml` | +| Skupper Site (spoke connectors) | | yes | `charts/region/east|west/values.yaml` | +| Grafana (multi-cluster dashboards) | yes | | `charts/region/hub/values.yaml` | +| Grafana (local metrics) | | yes | `charts/region/east|west/values.yaml` | +| Kiali + OSSM Console plugin | yes | yes | both | +| Connectivity Link (RHCL) | yes | yes | both | +| Kubecost (primary aggregator) | yes | | `charts/region/hub/values.yaml` | +| Kubecost (agent) | | yes | `charts/region/east|west/values.yaml` | +| Kafka Console (all clusters) | yes | | `charts/region/hub/values.yaml` | + +## GitOps application delivery flow + +See **[GitOps deployment chain](https://maximilianopizarro.github.io/hybrid-mesh-platform/validatedpatterns-docs/gitops-deployment-chain.html)** for the full encadenamiento (hub `field-content-*` → ApplicationSet `fleet-spoke-push` → `*-spoke-components` → spoke `*-east` / `*-west` apps) with copy-paste YAML fragments. + +![GitOps sequence — hub Argo CD, ApplicationSet, remote spoke sync](/images/hybrid-mesh-platform/arch-gitops-sync-sequence.png) +*Hub syncs first; ApplicationSet pushes per-spoke charts; each spoke Argo CD reconciles child Applications locally.* +## Sync wave ordering + +Components deploy in strict order via ArgoCD sync waves: + +![Argo CD sync wave ordering from bootstrap through dashboards](/images/hybrid-mesh-platform/arch-sync-waves.png) +*Sync waves prevent operators from racing workloads — mesh and namespaces land before Industrial Edge and gateways.* +### Spoke sync-wave reference + +Matches ebook Ch.4 ordering (`charts/region/east/values.yaml`, `charts/region/west/values.yaml`): + +| Wave | What deploys | Why this order | +| ---- | ------------ | -------------- | +| 1 | Namespaces (no ambient label yet) | Names must exist before operators and workloads | +| 2 | OLM Subscriptions | CRDs and operators installed | +| 3 | Service Mesh 3 (Istio + ZTunnel + ambient labels wave 2 inside chart) | Mesh dataplane before application pods | +| 4 | Observability, ACS secured cluster | Scraping and security after mesh | +| 5 | Industrial Edge (Kafka, sensors, dashboard) | Pods enroll in ambient with HBONE ready | +| 6 | Spoke gateway + Skupper interconnect | Routing after backends exist | + +Hub chart uses a similar pattern; ApplicationSet for spokes runs at hub wave **5** after ACM placement is healthy. + +## Service Interconnect (Skupper) topology + +Red Hat Service Interconnect creates a Virtual Application Network (VAN) that bridges services across clusters without VPN or direct network connectivity. + +![Skupper VAN — hub Listeners, spoke Connectors, AccessGrant and AccessToken](/images/hybrid-mesh-platform/arch-skupper-topology.png) +*Connectors expose spoke-gateway and prometheus-auth-proxy; Listeners materialize ClusterIP services on the hub.* +## Spoke gateway aggregation + +Each spoke runs a **Gateway API gateway** that fronts all Industrial Edge services, providing a single entry point for Skupper to expose to the hub. + +![Spoke gateway aggregates Industrial Edge HTTP routes for Skupper](/images/hybrid-mesh-platform/arch-spoke-gateway.png) +*One Connector per spoke exposes the gateway instead of every microservice individually.* +## Multi-cluster observability pipeline + +![Multi-cluster observability — spoke metrics via Skupper into hub Grafana](/images/hybrid-mesh-platform/arch-observability-pipeline.png) +*Spoke Thanos Querier is reached through nginx auth-proxy Connectors; hub Grafana uses HTTP datasources without bearer tokens.* +## Data flow (sensors to dashboard) + +![Industrial Edge data flow — sensors through MQTT, Camel, Kafka to Grafana and data lake](/images/hybrid-mesh-platform/arch-data-flow.png) +*Telemetry path on each spoke; MirrorMaker replicates to the hub-centralized MinIO data lake.* +## Relationship to Red Hat Validated Patterns + +The **[Multicloud GitOps](https://validatedpatterns.io/patterns/multicloud-gitops)** validated pattern demonstrates fleet GitOps with OpenShift GitOps and ACM — a declarative root Application, cluster grouping, and progressive rollout. Hybrid Mesh Platform forks that foundation and extends it with: + +- **Industrial Edge** factory telemetry (MQTT → Kafka → ML → dashboards) on east/west spokes +- **Service Mesh 3 ambient mode** — L4 encryption without sidecars +- **Service Interconnect (Skupper)** — cross-cluster service exposure without VPN +- **Connectivity Link (Kuadrant)** — API-aware Gateway API ingress with rate limits and API keys +- **OpenShift AI (RHOAI)** — hub-resident MaaS inference (Qwen3 / Granite via vLLM) for factory AI pipelines +- **OpenShift Lightspeed + MCP Gateway** — natural-language platform operations via `OLSConfig` and a dual MCP server (Quarkus 19 tools + Go SDK 21 tools), enabling operators to query and act on cluster state without raw YAML + +## Red Hat Device Edge extension path + +The current spokes run **full OpenShift clusters** (3 workers, cloud or bare-metal). For physically constrained factory devices — industrial controllers, single-board computers, ruggedized appliances — the natural extension is **Red Hat Device Edge** with **MicroShift**. + +[**Red Hat Device Edge**](https://www.redhat.com/en/technologies/device-edge) combines **Red Hat Enterprise Linux** with **MicroShift**, a minimal OpenShift-compatible Kubernetes runtime optimized for edge hardware with as little as 2 CPU cores and 2 GiB RAM. MicroShift exposes the same Kubernetes API surface as OpenShift, so ACM can manage it as a `ManagedCluster` alongside full OpenShift spokes. + +In a Hybrid Mesh Platform extension: + +| Layer | Current spokes | Far-edge extension | +| --- | --- | --- | +| Runtime | OpenShift 4.17+ (3 workers) | MicroShift on RHEL (single node) | +| Management | ACM ManagedCluster | ACM ManagedCluster (same placement rules) | +| Workloads | Industrial Edge — Kafka, Camel K, ML | Lightweight sensors, MQTT bridge, edge inference | +| Connectivity | Skupper VAN (full) | Skupper VAN (single-pod footprint) | +| GitOps | clusterGroup PULL via ACM | clusterGroup PULL via ACM | + +This extension path is not deployed in the current sandbox tier implementation, but the ACM placement model and Skupper VAN are designed to accommodate MicroShift spokes with no hub-side changes. + +--- + +**Next →** translate diagrams into installs via **[Getting Started](getting-started)**, scaffold new edge instances via **[Scaffolding](scaffolding)**, then follow **[Observability](observability)** once workloads expose Prometheus signals. For ACM placement detail and additional reference pages, see the [pattern documentation site](https://maximilianopizarro.github.io/hybrid-mesh-platform/validatedpatterns-docs/). + +## Official documentation + +- [ACM Architecture](https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.16/html/about/welcome-to-red-hat-advanced-cluster-management-for-kubernetes) +- [Red Hat Device Edge](https://www.redhat.com/en/technologies/device-edge) +- [MicroShift documentation](https://docs.redhat.com/en/documentation/red_hat_build_of_microshift) +- [Multicloud GitOps Pattern](https://validatedpatterns.io/patterns/multicloud-gitops) +- [Red Hat Service Interconnect](https://docs.redhat.com/en/documentation/red_hat_service_interconnect/2.1) +- [Kubernetes Gateway API](https://gateway-api.sigs.k8s.io/) +- [Argo CD ApplicationSet Generators](https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators/) diff --git a/content/patterns/hybrid-mesh-platform/cluster-sizing.adoc b/content/patterns/hybrid-mesh-platform/cluster-sizing.adoc new file mode 100644 index 000000000..cde0daf19 --- /dev/null +++ b/content/patterns/hybrid-mesh-platform/cluster-sizing.adoc @@ -0,0 +1,27 @@ +--- +title: Cluster sizing +weight: 30 +aliases: /hybrid-mesh-platform/cluster-sizing/ +--- + +:toc: +:imagesdir: /images +:_content-type: ASSEMBLY +include::modules/comm-attributes.adoc[] +include::modules/hybrid-mesh-platform/metadata-hybrid-mesh-platform.adoc[] + +[NOTE] +==== +The values in this page are generated from link:https://github.com/maximilianoPizarro/hybrid-mesh-platform/blob/main/pattern-metadata.yaml[pattern-metadata.yaml] in the implementation repository using `utils/flatten_yaml.rb`. To verify consistency, compare `modules/hybrid-mesh-platform/metadata-hybrid-mesh-platform.adoc` in this PR with the source file. +==== + +include::modules/cluster-sizing-template.adoc[] + +[id="hybrid-mesh-platform-additional-requirements"] +== Additional requirements + +The Hybrid Mesh Platform pattern deploys a **hub** cluster plus **two spoke** clusters (east and west). Plan for three OpenShift clusters at the recommended sizes above. + +Optional features such as OpenShift AI workbenches and MaaS-backed inference require additional hub capacity. Workshop Showroom content is maintained in a link:https://github.com/maximilianoPizarro/showroom-hybrid-mesh-ai[separate repository] and does not change minimum cluster sizing for the core pattern. + +For RHDP catalog provisioning details and validation steps, see the link:https://maximilianopizarro.github.io/hybrid-mesh-platform/[pattern documentation site]. diff --git a/content/patterns/hybrid-mesh-platform/demo-scenario.md b/content/patterns/hybrid-mesh-platform/demo-scenario.md new file mode 100644 index 000000000..da4489fcb --- /dev/null +++ b/content/patterns/hybrid-mesh-platform/demo-scenario.md @@ -0,0 +1,74 @@ +--- +title: Demo scenario +weight: 80 +aliases: /hybrid-mesh-platform/demo-scenario/ +--- + +# Demo scenario — Hybrid Mesh AI Workshop Showroom + +> **Note:** This page describes what the workshop showroom demonstrates and which product surfaces it exercises. It is explanatory content for architects and facilitators — not a hands-on lab guide. For step-by-step lab modules, user registration, and the embedded terminal, see the [showroom-hybrid-mesh-ai](https://github.com/maximilianoPizarro/showroom-hybrid-mesh-ai) repository and the deployed showroom environment. + +## What the showroom shows + +The showroom mirrors how Red Hat customers run hybrid cloud on OpenShift: a **hub cluster** managing **east** and **west** spokes through ACM, with ambient service mesh, GitOps, Industrial Edge factory telemetry, and an AI inference layer on OpenShift AI. + +[![Hybrid Mesh Platform — hub-spoke architecture](/images/hybrid-mesh-platform/workshop-hybrid-mesh.png)](/images/hybrid-mesh-platform/workshop-hybrid-mesh.png) + +_Hub cluster aggregates observability and Developer Hub; east and west spokes run Industrial Edge workloads connected via Service Interconnect (Skupper)._ + +[![Platform component map — hub vs spokes](/images/hybrid-mesh-platform/workshop-hybrid-mesh-arch.png)](/images/hybrid-mesh-platform/workshop-hybrid-mesh-arch.png) + +_Component placement across hub and spoke clusters — fleet governance centralized, factory workloads at the edge._ + +## Dual-track experience + +The workshop content is organized in two tracks: + +| Track | Audience | Focus | +| --- | --- | --- | +| **Part A (modules 01–05)** | Executives and architects | Hybrid cloud strategy, ROSA architecture, security at scale, cloud AI services, customer cases | +| **Part B (modules 10–28)** | Practitioners | ACM fleet, mesh, GitOps, Industrial Edge, observability, ACS, Connectivity Link, OpenShift AI, Kuadrant API gateway | + +On validatedpatterns.io we document the **platform architecture and VP install path**. The showroom provides a rich, navigable view of the same product surfaces after deployment. + +## Product surfaces highlighted + +The showroom hero images and module structure align with the pattern chapters on this site: + +| Product area | Showroom illustration | Pattern chapter | +| --- | --- | --- | +| ACM multicluster fleet | ![ACM fleet](/images/hybrid-mesh-platform/workshop-acm-multicluster.png) | [Getting Started](getting-started) | +| Ambient Service Mesh | ![Service Mesh](/images/hybrid-mesh-platform/workshop-service-mesh.png) | [Architecture](architecture) | +| Developer Hub templates | ![Software templates](/images/hybrid-mesh-platform/workshop-software-templates.png) | [Scaffolding](scaffolding) | +| Industrial Edge factory | ![Industrial Edge](/images/hybrid-mesh-platform/workshop-industrial-edge.png) | [Industrial Edge](industrial-edge) | +| Observability stack | ![Observability](/images/hybrid-mesh-platform/workshop-observability.png) | [Observability](observability) | +| ACS and Connectivity Link | ![ACS and Kuadrant](/images/hybrid-mesh-platform/workshop-acs-kuadrant.png) | [Hub Gateway](hub-gateway) | +| OpenShift AI / MaaS | ![OpenShift AI](/images/hybrid-mesh-platform/workshop-openshift-ai.png) | [_index](.) | +| Kairos SmartScaling | ![Kairos scaling](/images/hybrid-mesh-platform/workshop-kairos-scaling.png) | [Observability](observability) | + +## Hub-spoke topology in the demo + +The demo fleet uses the same three-cluster layout documented in [Architecture](architecture): + +- **Hub:** ACM, Argo CD, Developer Hub, OpenShift AI, Service Mesh control plane, Skupper listeners, Kuadrant, ACS Central, Grafana, Kubecost +- **East spoke:** Industrial Edge workloads, DevSpaces, Kairos, spoke-local GitOps +- **West spoke:** Workload replicas, cross-cluster traffic via Skupper + +Traffic crosses **OpenShift Service Mesh 3 ambient mode** (ztunnels, optional waypoints) and **Skupper** tunnels exposed through **Gateway API** ingress on the hub. + +## OpenShift AI and API gateway (conceptual) + +The showroom illustrates a shared **Model as a Service (MaaS)** endpoint on the hub. Applications that speak the OpenAI REST API can consume inference without code changes by pointing to the in-cluster service. Spoke factory pipelines reach MaaS through Skupper connectors. + +**Red Hat Connectivity Link (RHCL)** with **Kuadrant** exposes managed API products — for example HTTP utilities, REST catalog APIs, and LLM chat completion routes — with rate limits and API key plans. See [Hub Gateway](hub-gateway) for the platform ingress design. + +## External resources + +| Resource | URL | +| --- | --- | +| Pattern implementation | [github.com/maximilianoPizarro/hybrid-mesh-platform](https://github.com/maximilianoPizarro/hybrid-mesh-platform) | +| Full pattern documentation (GitHub Pages) | [maximilianopizarro.github.io/hybrid-mesh-platform](https://maximilianopizarro.github.io/hybrid-mesh-platform/) | +| Workshop showroom content repo | [showroom-hybrid-mesh-ai](https://github.com/maximilianoPizarro/showroom-hybrid-mesh-ai) | +| Live showroom (when deployed) | [Hybrid Mesh AI Workshop Showroom](https://showroom-showroom.apps.cluster-22jv2.dynamic2.redhatworkshops.io/en/index.html) | + +**Next →** [Getting Started](getting-started) to deploy the Validated Patterns implementation, or [Architecture](architecture) for the technical deep dive. diff --git a/content/patterns/hybrid-mesh-platform/getting-started.md b/content/patterns/hybrid-mesh-platform/getting-started.md new file mode 100644 index 000000000..0836f0234 --- /dev/null +++ b/content/patterns/hybrid-mesh-platform/getting-started.md @@ -0,0 +1,462 @@ +--- +title: Getting Started +weight: 10 +aliases: /hybrid-mesh-platform/getting-started/ +--- + +# Getting Started + +Follow these steps to bootstrap the Hybrid Mesh Platform hub-spoke GitOps environment from the [hybrid-mesh-platform](https://github.com/maximilianoPizarro/hybrid-mesh-platform) Validated Patterns repository (fork of [multicloud-gitops](https://github.com/validatedpatterns/multicloud-gitops)). + +## You'll have when finished + +After a successful hub deploy and spoke registration, expect: + +| Component | Verification | +| --- | --- | +| ACM | `east` and `west` in `ManagedCluster` **Available** | +| Argo CD | Hub `clustergroup` **Synced**; east/west spokes pull their clusterGroup from Git via ACM | +| Industrial Edge | Sensors, MQTT, Kafka, `line-dashboard` on each spoke | +| Skupper | Hub `sitesInNetwork: 3`; listeners **Ready** in `service-interconnect` | +| Grafana | Hub dashboards with `prometheus-east` / `prometheus-west` datasources | +| Developer Hub | Industrial Edge catalog + software templates under **Create** | +| Gitea | Route `gitea-gitea.`; orgs `ws-platformadmin`, `app-of-apps` | +| Quay | Route `quay-registry.` (optional image catalog) | + +Then continue with [Scaffolding](scaffolding) to deploy a new edge instance on east or west. + +## Platform operators (reference) + +The hub chart deploys **ACM**, **OpenShift GitOps**, **ACS**, Service Mesh, Skupper, and related operators before application workloads. + +[![OpenShift GitOps — Argo CD Applications on the hub](/images/hybrid-mesh-platform/product-argocd-openshift-gitops.png)](/images/hybrid-mesh-platform/product-argocd-openshift-gitops.png) + +### Advanced Cluster Management (ACM) + +ACM must show **`east`** and **`west`** as **Available** managed clusters before the ApplicationSet can push spoke charts. + +[![ACM multicluster fleet management](/images/hybrid-mesh-platform/workshop-acm-multicluster.png)](/images/hybrid-mesh-platform/workshop-acm-multicluster.png) + +_ACM fleet overview: ManagedCluster registration, placement rules, and GitOpsCluster binding for hub-spoke delivery._ + +[![ACM fleet view — east and west registered on the hub](/images/hybrid-mesh-platform/ACM.png)](/images/hybrid-mesh-platform/ACM.png) + +_OpenShift Console — ACM All Clusters view showing east and west as Available managed clusters._ + +Verify: + +```bash +oc get managedcluster +oc get multiclusterhub -n open-cluster-management +``` + +Spoke names must match repository folders (`east`, `west`). Placement labels drive ApplicationSet targeting — see [Step 4](#step-4-import-managed-clusters-in-acm) and [Deploy with ACM and GitOps](#deploy-with-acm-and-gitops). + +### Advanced Cluster Security (ACS) + +ACS Central runs on the hub; **SecuredCluster** agents install on hub and both spokes. All three clusters appear in the Central UI when init bundles are applied. + +[![ACS Central — hub, east, and west clusters](/images/hybrid-mesh-platform/ACS.png)](/images/hybrid-mesh-platform/ACS.png) + +_ACS Central showing all three clusters registered. SecuredCluster agents report compliance and runtime events._ + +[![ACS Central — policies and vulnerability views](/images/hybrid-mesh-platform/ACS-2.png)](/images/hybrid-mesh-platform/ACS-2.png) + +_ACS vulnerability management and policy enforcement across Industrial Edge container images._ + +#### Generating SecuredCluster init bundles + +Generate one init bundle per cluster from Central (do not commit secrets to Git): + +```bash +roxctl -e central.stackrox:443 --password "$ROX_ADMIN_PASSWORD" --insecure-skip-tls-verify \ + central init-bundles generate --output-secrets - | oc apply -n stackrox -f - +``` + +Use cluster names **`hub`**, **`east`**, and **`west`**. Namespace **`stackrox`** must stay **off** Service Mesh ambient — see [Architecture — ACS](architecture#advanced-cluster-security-acs). + +## Prerequisites + +### OpenShift clusters + +| Requirement | Value | +| --- | --- | +| **OpenShift version** | 4.17+ (tested on 4.20 on AWS) | +| **Topology** | **3 clusters** — one hub, one east spoke, one west spoke (single-cluster deployment is not supported by default) | +| **Storage class** | Dynamic provisioner required on all clusters (AWS gp3-csi or equivalent). Kafka, Gitea, Quay, and Vault all require `PersistentVolumeClaim`. | +| **Network** | All clusters must reach your Git fork (GitHub by default) and public container registries, or mirrored equivalents. | + +### Hub cluster — additional requirements + +| Operator / feature | Requirement | +| --- | --- | +| **OpenShift AI (RHOAI)** | Required for MaaS / vLLM inference. Needs Node Feature Discovery and GPU operator **only** if you enable GPU-accelerated models. CPU-based inference (Qwen3 / Granite on CPU) works without GPU. | +| **GPU (optional)** | NVIDIA or AMD GPU node for accelerated vLLM. Without GPU, enable `modelServing.cpuOnly: true` in `charts/region/hub/values.yaml`. | +| **OpenShift Lightspeed** | Requires `OLSConfig` CRD and an OpenAI-compatible endpoint (MaaS on hub or external). | +| **Vault** | HashiCorp Vault is deployed by the pattern operator as the secrets backend. | + +### Workstation + +- **`oc` CLI** logged into the hub as `cluster-admin` for ACM import +- **Helm 3** (`helm version`) +- **Git** client and a GitHub (or Gitea) account + +### Network requirements (connected environments) + +1. Access to public container registries (or mirrored equivalents) from all clusters +2. Access to your Git fork from all clusters + +### Cluster sizing (AWS — OpenShift 4.20) + +Tested on **demo.redhat.com** with the following provisioning parameters: + +| Cluster | Workers | vCPU/worker | Memory/worker | Total capacity | +| --- | --- | --- | --- | --- | +| **Hub** | 3 | 8 | 32 GiB | 24 vCPU / 96 GiB | +| **East spoke** | 3 | 4 | 16 GiB | 12 vCPU / 48 GiB | +| **West spoke** | 3 | 4 | 16 GiB | 12 vCPU / 48 GiB | + +Hub estimated workload: ~12.5 CPU / ~29 GiB (ACM, ACS Central, Developer Hub, data lake Kafka 3-replica, Service Mesh, OpenShift AI, hub gateway). + +Spoke estimated workload: ~5.5 CPU / ~11 GiB (Industrial Edge, factory Kafka, ACS SecuredCluster, Service Mesh ambient). + +For constrained environments, use `values-lite.yaml` on the hub to disable heavy components (OpenShift AI, ACS, Grafana dashboards, hub gateway). + +See [Cluster sizing](cluster-sizing) for metadata-driven minimum and recommended sizes across cloud providers. + +## Repository layout + +``` +charts/all/ → shared component Helm charts (50+ charts) +charts/region/hub/ → hub bootstrap + hub clusterGroup values +charts/region/east/ → east spoke bootstrap + clusterGroup values +charts/region/west/ → west spoke bootstrap + clusterGroup values +values-global.yaml → pattern name and shared Git URL +values.yaml → root chart values (clustergroup subscription) +values-secret.yaml → secrets (from values-secret.yaml.template; Vault + ESO) +pattern.sh → Validated Patterns install wrapper +``` + +The pattern uses a **dual GitOps strategy**: +- **PUSH** — hub `ApplicationSet` (`fleet-spoke-push`) deploys `charts/all/spoke-meta-push` to each spoke cluster via ACM placement +- **PULL** — each spoke's local Argo CD syncs its own `charts/region/east/` or `charts/region/west/` from Git autonomously + +## Step 1: Fork the repository + +Fork [hybrid-mesh-platform](https://github.com/maximilianoPizarro/hybrid-mesh-platform) so you can customize domains, secrets, and Git URLs. + +Update `main.gitops.repoURL` in `values-global.yaml` and cluster domains in `overrides/values-aws-{hub,east,west}.yaml` (or your cloud-specific overrides). + +## Step 2: Configure secrets and cluster domains + +Copy the secrets template and edit before installation. **Do not commit `values-secret.yaml` to Git.** + +```bash +cp values-secret.yaml.template values-secret.yaml +``` + +### Required secrets (values-secret.yaml) + +The Validated Patterns secrets framework (Vault + External Secrets Operator) reads `values-secret.yaml` at install time and populates Vault. The following secrets are defined in the template: + +| Secret name | Fields | When required | +| --- | --- | --- | +| `config-demo` | `secret` | Always (auto-generated if left as `onMissingValue: generate`) | +| `kairos-ai-credentials` | `api-key` | When Kairos SmartScaling AI features are enabled on spokes | +| `openshift-ai-maas-credentials` | `OPENAI_API_KEY`, `OPENAI_API_BASE` | When OpenShift AI / MaaS inference is enabled on the hub | +| `mcp-gateway-argocd` | `token` | When MCP Gateway is enabled (OpenShift Lightspeed integration) | +| `workshop-users` | `defaultPassword` | Workshop Showroom (demo only; use OAuth in production) | +| AWS credentials | `aws_access_key_id`, `aws_secret_access_key` | Only if using ACM ClusterPools to provision new clusters | + +Fields marked `onMissingValue: generate` are auto-generated by Vault for demo environments. For production, set them to `onMissingValue: error` and provide values explicitly. + +Set hub and spoke cluster domains in override files before install: + +```bash +# Edit your cloud provider override (AWS example): +vi overrides/values-aws-hub.yaml +vi overrides/values-aws-east.yaml +vi overrides/values-aws-west.yaml +``` + +See [MIGRATION.md](https://github.com/maximilianoPizarro/hybrid-mesh-platform/blob/main/MIGRATION.md) for the mapping from legacy RHDP-injected secrets. + +## Step 3: Install with Validated Patterns + +From the hub cluster (logged in with `oc`): + +```bash +./pattern.sh install +``` + +The Pattern operator (or utility container) deploys the root `clustergroup` Application on the hub. ACM `managedClusterGroups` in `charts/region/hub/values.yaml` register east and west spokes once they join the fleet. + +## Step 4: Import managed clusters in ACM + +From the hub, import east and west using ACM **Import cluster** or automated klusterlet flows. + +Apply labels used by placement rules: + +```bash +# Example — adjust cluster names to match your environment +oc label managedcluster east region=east +oc label managedcluster west region=west +oc label managedcluster east cluster.open-cluster-management.io/clusterset=global +oc label managedcluster west cluster.open-cluster-management.io/clusterset=global +``` + +Ensure spoke credentials are stored per ACM requirements. + +## Step 5: Register spokes as Argo CD cluster secrets + +The ApplicationSet deploys spoke charts remotely. Register each spoke on the hub Argo CD instance: + +```bash +helm upgrade field-content . \ + --set clusters.east.token=sha256~... \ + --set clusters.west.token=sha256~... +``` + +Or create secrets with label `argocd.argoproj.io/secret-type: cluster`. + +Spoke cluster names in ACM and Argo CD must match folder names: **`east`** and **`west`**. + +## Step 6: Verify ApplicationSet generates spoke applications + +Confirm the ACM → GitOps chain: + +1. `Placement` `hub-spoke-placement` selects `region=east` and `region=west` +2. `PlacementDecision` lists those clusters in `openshift-gitops` +3. `GitOpsCluster` `hub-spoke-gitops` registers clusters in Argo CD +4. ApplicationSet creates `east-spoke-components` and `west-spoke-components` +5. Each spoke Argo CD syncs child Applications (namespaces, operators, Industrial Edge, and others) + +**Hub:** + +```bash +oc get applications -n openshift-gitops +# Expect: east-spoke-components, west-spoke-components (Synced / Healthy) +``` + +**Each spoke:** + +```bash +oc get applications -n openshift-gitops +# Expect: east-namespaces, east-operators, east-industrial-edge-tst, etc. +``` + +**PlacementDecision:** + +```bash +oc get placementdecisions.cluster.open-cluster-management.io -n openshift-gitops \ + -l cluster.open-cluster-management.io/placement=hub-spoke-placement -o yaml +``` + +Healthy sync waves progress: **namespaces → operators → platform → observability → Industrial Edge workloads**. + +## Deploy with ACM and GitOps + +### ManagedClusterSet + +Groups clusters for RBAC and placement. Downstream objects reference set membership — not hard-coded cluster names in Git. + +### Placement + +Selects clusters from a ManagedClusterSet using label selectors (for example `region=east`). ACM recomputes decisions as clusters join, leave, or change labels. + +### PlacementDecision + +Publishes the concrete cluster names that satisfied a Placement at a given time. The ApplicationSet `clusterDecisionResource` generator watches these decisions. + +### GitOpsCluster + +Associates Argo CD (`openshift-gitops`) with clusters chosen by placement — bridging ACM fleet selection and Argo CD cluster secrets. + +### ApplicationSet with clusterDecisionResource + +For each cluster in the PlacementDecision, the hub ApplicationSet (`fleet-spoke-push`) creates an Application that: + +- Uses repository path `charts/all/spoke-meta-push` (the PUSH meta chart) deployed to the spoke cluster +- The spoke's local Argo CD then pulls its own `charts/region/east/` or `charts/region/west/` from Git autonomously +- Deploys to remote cluster `{{name}}` via hub-stored cluster credentials + +Adding a spoke with correct labels and a new `charts/region/south/` folder automatically generates a new Application when the Placement matches. + +### Troubleshooting: no spoke Applications + +| Check | Command / expectation | +| --- | --- | +| Managed clusters | `oc get managedcluster` — **Available** | +| Cluster names | Must be `east` and `west` (match repo folders) | +| Placement | `hub-spoke-placement` includes both regions | +| PlacementDecision | Decisions list `east`, `west` | +| GitOpsCluster | `hub-spoke-gitops` reconciled; clusters visible in Argo CD UI | +| RBAC | Role `applicationset-placementdecisions` for ApplicationSet controller | +| Spoke charts | `charts/region/east/` and `charts/region/west/` valid Helm charts with `Chart.yaml` | + +## Step 7: Kiali multi-cluster (hub) + +Hub Kiali shows mesh topology from east and west **without** Istio multi-cluster trust federation. Each spoke keeps its own control plane; hub Kiali uses remote secrets and links to spoke Kiali UIs. + +### Automated token sync (default) + +With `multiCluster.automateTokens: true` (hub) and `exportTokenForHub: true` (spokes): + +1. Spoke PostSync Job exports `kiali-hub-export` ConfigMap with token +2. Hub PostSync Job reads ACM `ManagedCluster` apiUrl/caBundle and creates `kiali-multi-cluster-secret` +3. CronJobs renew tokens every 6 hours + +### Manual tokens (optional) + +```bash +oc create token kiali-service-account -n openshift-cluster-observability-operator --duration=8760h +helm upgrade field-content . \ + --set multiCluster.automateTokens=false \ + --set clusters.east.kialiToken=sha256~... \ + --set clusters.east.kialiCaData=LS0tLS1... \ + --reuse-values +``` + +With `auth.strategy: openshift`, users **Log in** per remote cluster the first time they open Kiali for that cluster. + +## Step 8: Developer Hub (Keycloak OIDC) + +Developer Hub uses cluster Keycloak (`sso.`) realm `backstage` — not GitHub OAuth. + +```bash +SECRET=$(openssl rand -base64 24) +helm upgrade field-content . \ + --set keycloakOidcClientSecret="$SECRET" \ + --reuse-values +``` + +Or patch after deploy: + +```bash +oc create secret generic developer-hub-oidc-auth \ + --from-literal=OIDC_CLIENT_SECRET="$SECRET" \ + -n developer-hub --dry-run=client -o yaml | oc apply -f - +``` + +Log in at `https://developer-hub.` (for example `platformadmin` / workshop password from your values). + +## Step 9: Continue AI (DevSpaces + Kaoto) + +Do not commit MaaS API keys to Git. Create the DevSpaces secret after deploy: + +```bash +oc create secret generic continue-ai-config -n devspaces \ + --from-literal=CONTINUE_API_KEY='' \ + --from-literal=CONTINUE_API_BASE='https://litellm-prod.apps.maas.redhatworkshops.io/v1' \ + --from-literal=CONTINUE_MODEL='deepseek-r1-distill-qwen-14b' \ + --dry-run=client -o yaml | oc apply -f - +``` + +Industrial Edge catalog loads from an in-cluster ConfigMap. Software templates ship as static assets in the pattern repository under `docs/assets/backstage/software-templates/`. + +### Developer Hub multi-cluster Topology + +Spoke workloads appear in **Topology** / **Kubernetes** only when: + +1. `developer-hub-spoke-tokens` Secret exists (token sync Job completed) +2. Catalog entities include `backstage.io/kubernetes-cluster: east` or `west` + +```bash +oc get secret developer-hub-spoke-tokens -n developer-hub +oc get job -n developer-hub -l job-name=developer-hub-spoke-token-sync-hook +``` + +### Optional: Quay credentials (hub) + +Never commit registry passwords to Git. Generate dockerconfig and pass via Helm at upgrade time if you enable Quay push from pipelines. + +## Verify the installation + +Run these checks after all sync waves complete to confirm the pattern is healthy. Run each command from the hub cluster unless noted. + +### Fleet and GitOps + +```bash +# ACM: east and west must be Available +oc get managedcluster + +# Hub clustergroup: all Applications Synced +oc get applications -n openshift-gitops + +# ApplicationSet: spoke apps generated +oc get applicationset fleet-spoke-push -n openshift-gitops +oc get applications -n openshift-gitops | grep spoke-components +``` + +### Service Interconnect (Skupper) + +```bash +# Hub VAN: sitesInNetwork must equal 3 (hub + east + west) +oc get site hub -n service-interconnect \ + -o jsonpath='sitesInNetwork={.status.sitesInNetwork}{"\n"}' + +# Skupper listeners ready on hub +oc get listeners -n service-interconnect +``` + +### OpenShift Lightspeed (MCP Gateway) + +```bash +# OLSConfig must be Available +oc get olsconfig cluster -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}{"\n"}' + +# OpenShift Lightspeed pods running on hub +oc get pods -n openshift-lightspeed + +# MCP Gateway pods running on hub +oc get pods -n mcp-gateway +``` + +### OpenShift AI / RHOAI + +```bash +# DataScienceCluster components: all Managed / Ready +oc get datasciencecluster default-dsc \ + -o jsonpath='{range .status.installedComponents[*]}{.name}: {.managementState}{"\n"}{end}' + +# Model serving pods (if enabled) +oc get pods -n rhods-model-mesh-serving 2>/dev/null || \ +oc get pods -n modelmesh-serving 2>/dev/null +``` + +### Industrial Edge (run on each spoke) + +```bash +# Switch to east or west context +oc config use-context east # adjust to your kubeconfig context name + +# Sensors, Kafka, and Camel K running +oc get pods -n industrial-edge-tst +oc get kafka dev-cluster -n industrial-edge-tst -o jsonpath='{.status.conditions[0].type}{"\n"}' + +# Line dashboard accessible +oc get route line-dashboard -n industrial-edge-tst +``` + +### Console links (hub — 19 expected) + +```bash +# Log in first so OAuth-protected links get a bearer token +oc login --token= --server= + +# Expect 19 OK (HTTP 200) on a full hub install +MIN_OK_CODE=200 bash scripts/verify-console-links.sh +``` + +If any check fails, see the [extended troubleshooting guide](https://maximilianopizarro.github.io/hybrid-mesh-platform/validatedpatterns-docs/troubleshooting.html) and the [Validation Guide](https://maximilianopizarro.github.io/hybrid-mesh-platform/validation-guide.html). + +## References + +- [ACM documentation](https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.16/html/about/welcome-to-red-hat-advanced-cluster-management-for-kubernetes) +- [OpenShift AI documentation](https://docs.redhat.com/en/documentation/red_hat_openshift_ai_self-managed) +- [OpenShift Lightspeed documentation](https://docs.redhat.com/en/documentation/openshift_lightspeed) +- [Multicloud GitOps Validated Pattern](/patterns/multicloud-gitops) +- [RHDP install playbook and extended docs](https://maximilianopizarro.github.io/hybrid-mesh-platform/validatedpatterns-docs/) +- [ApplicationSet Generators](https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators/) + +**Next →** [Scaffolding](scaffolding) to deploy Industrial Edge instances from Developer Hub · [Architecture](architecture) for diagrams · [Observability](observability) once metrics are flowing. diff --git a/content/patterns/hybrid-mesh-platform/hub-gateway.md b/content/patterns/hybrid-mesh-platform/hub-gateway.md new file mode 100644 index 000000000..28493771c --- /dev/null +++ b/content/patterns/hybrid-mesh-platform/hub-gateway.md @@ -0,0 +1,164 @@ +--- +title: Hub Gateway +weight: 40 +aliases: /hybrid-mesh-platform/hub-gateway/ +--- + +# Hub Gateway + +The **hub gateway** pattern provides centralized HTTP ingress on the hub cluster with behaviors similar to an **F5 BIG-IP ADC**: VIP-style routing, TLS termination at the edge, and **weighted traffic splits** across backend services or spoke-derived routes. + +## Gateway API theory + +Kubernetes **Gateway API** separates concerns: + +- **`Gateway`** — listens on addresses and ports; attaches TLS and listener policies. +- **`HTTPRoute`** — attaches to a `Gateway` and selects backend `Service` resources with matches, filters, and weighted backends. + +Weighted rules enable **canary** or **active-active** distributions between service versions or regions when paired with mesh or multi-cluster DNS strategies. + +## Cross-cluster routing architecture + +[![ACS and Connectivity Link — security and gateway policy](/images/hybrid-mesh-platform/workshop-acs-kuadrant.png)](/images/hybrid-mesh-platform/workshop-acs-kuadrant.png) + +_ACS and Connectivity Link: security policies and API gateway management across hub and spokes._ + +[![Gateway API policy topology — hub HTTPRoute and route rules](/images/hybrid-mesh-platform/connectivity-link-hub.png)](/images/hybrid-mesh-platform/connectivity-link-hub.png) + +_OpenShift Console — Gateway API resources on the hub: Gateway, HTTPRoute rules, and backend service references._ + +[![Hub cluster Gateway API and HTTPRoute](/images/hybrid-mesh-platform/connectivity-link-hub-gateway.png)](/images/hybrid-mesh-platform/connectivity-link-hub-gateway.png) + +_Hub Gateway — HTTPRoute with weighted backends to east and west spoke ExternalName services._ + +[![Spoke cluster Gateway API and backends](/images/hybrid-mesh-platform/connectivity-link-spoke.png)](/images/hybrid-mesh-platform/connectivity-link-spoke.png) + +_Spoke cluster — Gateway API and backend services registered in the mesh._ + +[![Spoke gateway aggregating Industrial Edge services](/images/hybrid-mesh-platform/connectivity-link-spoke-gateway.png)](/images/hybrid-mesh-platform/connectivity-link-spoke-gateway.png) + +_Spoke-gateway HTTPRoute: routes `/api`, `/dashboard`, and catch-all paths to Industrial Edge services._ + +The hub gateway routes traffic to spoke cluster OpenShift Routes via `ExternalName` services: + +```mermaid +flowchart LR + Browser["Browser"] --> Router["Hub OpenShift
Router (HTTPS)"] + Router --> IGW["Istio Gateway
(port 8080)"] + IGW --> WP["Waypoint Proxy
(circuit breaking)"] + WP --> EXT_E["ExternalName
Service (east)"] + WP --> EXT_W["ExternalName
Service (west)"] + EXT_E --> SPOKE_E["East OpenShift
Router (HTTP:80)"] + EXT_W --> SPOKE_W["West OpenShift
Router (HTTP:80)"] + SPOKE_E --> POD_E["Backend Pod"] + SPOKE_W --> POD_W["Backend Pod"] +``` + +### Front / API split + +Traffic is split into separate `Service` objects per cluster and traffic type to give Kiali and Grafana finer-grained visibility: + +| Service | Purpose | +| ------- | ------- | +| `industrial-edge-east-front` | Static frontend assets for east spoke | +| `industrial-edge-east-api` | Socket.IO / API backend for east spoke | +| `industrial-edge-west-front` | Static frontend assets for west spoke | +| `industrial-edge-west-api` | Socket.IO / API backend for west spoke | + +All four services use `ExternalName` pointing to the same spoke Route hostname, but Istio tracks them as distinct destinations. In Kiali's traffic graph, each appears as a separate node. + +The `HTTPRoute` uses two rules: + +```mermaid +flowchart TB + REQ["Incoming Request"] --> MATCH{Path?} + MATCH -->|"/api/*"| API["API rule
(session affinity)"] + MATCH -->|"/*"| FRONT["Frontend rule
(weighted split)"] + API --> API_E["east-api
weight: 100"] + API --> API_W["west-api
weight: 0"] + FRONT --> FRONT_E["east-front
weight: 50"] + FRONT --> FRONT_W["west-front
weight: 50"] +``` + +1. **`/api` prefix** → routed to `*-api` services (defaults to east 100%, west 0% for Socket.IO session affinity). +2. **Catch-all** → routed to `*-front` services (split by `gateway.weights.east` / `west`). + +Override API weights with `gateway.apiWeights` in values when your application supports cross-cluster API load balancing. + +### Key requirements + +1. **HTTP port 80, not HTTPS** — Istio ambient mode gateways do not apply `DestinationRule` TLS settings. Using HTTPS causes `CERTIFICATE_VERIFY_FAILED` errors. Spoke Routes must set `insecureEdgeTerminationPolicy: Allow`. + +2. **ServiceEntry for each external host** — without a `ServiceEntry`, Envoy has no cluster definition for the external hostname and returns HTTP 500. + +3. **Per-backend Host header rewrite** — the spoke OpenShift router routes by `Host` header. Use `RequestHeaderModifier` filters on each `backendRef` in the HTTPRoute. + +4. **Session affinity for Socket.IO** — when load-balancing across multiple backends, Socket.IO polling and WebSocket upgrade must reach the same backend. The `/api` rule pins to a single spoke by default. + +## Circuit breaking + +Each `ExternalName` service gets a `DestinationRule` with `outlierDetection` and `connectionPool` settings, enforced by the `hub-gateway-system-waypoint` proxy (Istio ambient L7). + +### Default configuration (aggressive / demo) + +| Parameter | Default | Purpose | +| --------- | ------- | ------- | +| `connectionPool.tcp.maxConnections` | 100 | Max concurrent TCP connections | +| `connectionPool.http.h2UpgradePolicy` | `DO_NOT_UPGRADE` | Spokes expect HTTP/1.1 | +| `connectionPool.http.maxRequestsPerConnection` | 10 | Force connection recycling | +| `outlierDetection.consecutive5xxErrors` | 3 | Eject after 3 consecutive 5xx | +| `outlierDetection.interval` | 10s | Health check interval | +| `outlierDetection.baseEjectionTime` | 30s | Minimum ejection duration | +| `outlierDetection.maxEjectionPercent` | 100 | Allow ejecting the only endpoint | + +`maxEjectionPercent: 100` is required because each ExternalName service resolves to a single endpoint; without it, Istio refuses to eject the last remaining host. + +### Overriding circuit breaker values + +Set `gateway.circuitBreaking.*` in the hub-gateway values: + +```yaml +gateway: + circuitBreaking: + enabled: true + outlierDetection: + consecutive5xxErrors: 5 + baseEjectionTime: 60s +``` + +Set `enabled: false` to disable circuit breaking entirely. + +## Relationship to Connectivity Link + +Connectivity Link (Kuadrant) layers DNS automation, TLS policies, and advanced controls atop Gateway API. Start with plain HTTPRoutes if you need incremental adoption; enable Kuadrant policies when teams are ready. + +## IoT Dashboard integration + +The Industrial Edge `line-dashboard` (iot-frontend) requires an `iot-consumer` sidecar to display sensor data: + +- **iot-consumer** bridges MQTT to WebSocket via Socket.IO +- Mount a ConfigMap `config.json` with `websocketHost: ""` (empty = same origin) +- Path-based Route `/api` to port 3000 (iot-consumer) +- The hub gateway proxies `/api` requests to the correct spoke backend where iot-consumer handles the Socket.IO connection + +## Operational notes + +- Keep **hostnames aligned** with `deployer.domain` and corporate DNS wildcard records. +- Combine with **Service Mesh ambient** when east-west encryption between gateway hops and workloads matters. +- Monitor the gateway Envoy proxy metrics at port 15020 `/stats/prometheus`. + +--- + +**Next:** attach HTTPRoutes per Connectivity Link policies when Kuadrant enforcement tightens, then verify backends inside Service Mesh ambient namespaces. See the [pattern documentation](https://maximilianopizarro.github.io/hybrid-mesh-platform/validatedpatterns-docs/hub-gateway.html) for extended RHCL detail. + +## References + +- [Kubernetes Gateway API](https://gateway-api.sigs.k8s.io/) +- [Connectivity Link documentation](https://docs.redhat.com/en/documentation/red_hat_connectivity_link/) +- [Istio DestinationRule](https://istio.io/latest/docs/reference/config/networking/destination-rule/) + +Implementation chart: `charts/all/hub-gateway`. + +--- + +**Next →** [Observability](observability) for Grafana dashboards, Kiali traffic graphs, and Kafka Console across clusters. diff --git a/content/patterns/hybrid-mesh-platform/ideas-for-customization.md b/content/patterns/hybrid-mesh-platform/ideas-for-customization.md new file mode 100644 index 000000000..3b33b3c14 --- /dev/null +++ b/content/patterns/hybrid-mesh-platform/ideas-for-customization.md @@ -0,0 +1,107 @@ +--- +title: Ideas for customization +weight: 90 +aliases: /hybrid-mesh-platform/ideas-for-customization/ +--- + +# Ideas for customization + +Adapt Hybrid Mesh Platform to your fleet size, regions, and operational constraints. The pattern repository follows a **single-branch, multi-directory** model so all clusters stay on `main` while Helm values and folders isolate hub vs spoke configuration. + +## Branch strategy + +All configuration lives on branch **`main`**. Cluster-specific settings use directories — not long-lived environment branches. + +### Repository layout + +``` +charts/region/hub/ → Hub bootstrap + hub clusterGroup values +charts/region/east/ → East spoke bootstrap + clusterGroup values +charts/region/west/ → West spoke bootstrap + clusterGroup values +charts/all/ → Shared component charts (50+ charts) +values-global.yaml → Pattern name and shared Git URL +values.yaml → Root chart values (clustergroup subscription) +``` + +Each `charts/region//` directory is an independent Helm chart with its own `Chart.yaml` and `values.yaml`. Component charts under `charts/all/` are referenced by hub and spoke clusterGroup Applications. + +### How values and ApplicationSet interact + +| Path / File | Purpose | +| --- | --- | +| `values-global.yaml` | Pattern name and shared Git URL | +| `charts/region/hub/values.yaml` | Hub clusterGroup — operators, observability, gateway, managedClusterGroups | +| `charts/region/east/values.yaml` | East spoke clusterGroup — subscriptions, domains, Applications | +| `charts/region/west/values.yaml` | West spoke clusterGroup — subscriptions, domains, Applications | +| `charts/all/` | Shared component charts referenced by all clusters | + +The pattern uses **dual GitOps**. The hub ApplicationSet (`charts/all/acm-hub-spoke`) uses a `clusterDecisionResource` generator for the PUSH leg: + +```yaml +# Hub ApplicationSet (fleet-spoke-push) — actual path in the pattern repo +source: + path: charts/all/spoke-meta-push # PUSH: meta chart deploys to spoke cluster +destination: + name: '{{name}}' # east or west, via hub Argo CD cluster secret +``` + +The spoke's local Argo CD then autonomously **PULL**s from `charts/region/east/` or `charts/region/west/` — the hub does not apply spoke workloads directly. + +## Adding a new spoke cluster + +1. Provision OpenShift and import the cluster into ACM. +2. Label `ManagedCluster`: `region=`, `cluster.open-cluster-management.io/clusterset=global`. +3. Copy `charts/region/east/` to `charts/region/south/` and update `clusterName`, `deployer.domain`, `clusters.hub.domain` inside the new chart. +4. Add `clusters.south.domain` (or equivalent) to `charts/region/hub/values.yaml`. +5. Register the spoke as an Argo CD cluster secret on the hub (name must match the folder name: `south`). +6. Validate: `helm lint charts/region/south/` and `helm template charts/region/south/`. +7. Confirm Placement selects the cluster and ApplicationSet creates `south-spoke-components`. + +## Minimal profiles + +For constrained environments (labs, CI, edge PoC), use `values-lite.yaml` on the hub: + +```bash +helm template test . -f values-lite.yaml +``` + +Disables heavy subscriptions while preserving GitOps bootstrap and ApplicationSet paths. + +## Customization ideas + +| Area | Customization | +| --- | --- | +| Regions | Add spokes beyond east/west — new `charts/region//` folder + ACM labels + hub gateway weights | +| Gateway weights | `gateway.weights.east` / `west` for canary or active-active front traffic | +| API affinity | `gateway.apiWeights` when Socket.IO can span spokes | +| Circuit breaking | Tune `gateway.circuitBreaking` per environment (see [Hub Gateway](hub-gateway)) | +| Industrial Edge | Scaffold additional factory instances per spoke via [Scaffolding](scaffolding) | +| Observability | Custom Grafana dashboards; additional Prometheus federation | +| Mesh | Waypoints for L7 policy on selected namespaces; keep data-lake off ambient if using MinIO | +| Skupper | Additional listeners/connectors for new spoke services | +| Secrets | Vault or External Secrets for multi-cluster credentials (never commit tokens) | +| OpenShift version | Validate on 4.20+; align operator channels per release | +| Developer Hub | Extend software templates under `docs/assets/backstage/software-templates/` | + +## Red Hat Device Edge with MicroShift + +The current spokes require full OpenShift clusters (3 workers, cloud or bare-metal). For physically constrained factory locations — industrial controllers, single-board computers, ruggedized appliances — the pattern extends to **Red Hat Device Edge** with **MicroShift**. + +[**Red Hat Device Edge**](https://www.redhat.com/en/technologies/device-edge) bundles RHEL with **MicroShift**, a minimal OpenShift-compatible runtime that runs on hardware with as little as 2 CPU cores and 2 GiB RAM. Because MicroShift exposes the Kubernetes API, ACM manages it as a `ManagedCluster` with the same placement rules used for full OpenShift spokes — no hub changes required. + +### Extending the pattern for MicroShift spokes + +1. Provision a RHEL device with MicroShift installed (`rpm-ostree install microshift`). +2. Import the device into ACM as a `ManagedCluster` with labels `region=far-edge`. +3. Add `charts/region/far-edge/` in the pattern repo with a lightweight values profile — omit Kafka 3-replica and DevSpaces; keep MQTT bridge, Camel K lite, and Skupper. +4. Register a Skupper `AccessToken` on the device (same outbound-only mTLS as cloud spokes). +5. Hub Grafana aggregates metrics from the far-edge site the same way it does from east/west. + +This extension is documented as a future topology in [Architecture — Red Hat Device Edge extension path](architecture#red-hat-device-edge-extension-path). It is not deployed in the current sandbox tier. + +## Related patterns + +- [Multicloud GitOps](/patterns/multicloud-gitops) — fleet GitOps and ACM placement patterns +- [Industrial Edge](/patterns/industrial-edge/) — underlying factory/OT workload pattern (maintained tier) + +**Pattern repository:** [hybrid-mesh-platform](https://github.com/maximilianoPizarro/hybrid-mesh-platform) (Validated Patterns). Legacy App-of-Apps: [platform-hub-spoke-config](https://github.com/maximilianoPizarro/platform-hub-spoke-config). diff --git a/content/patterns/hybrid-mesh-platform/industrial-edge.md b/content/patterns/hybrid-mesh-platform/industrial-edge.md new file mode 100644 index 000000000..892527424 --- /dev/null +++ b/content/patterns/hybrid-mesh-platform/industrial-edge.md @@ -0,0 +1,72 @@ +--- +title: Industrial Edge +weight: 60 +aliases: /hybrid-mesh-platform/industrial-edge/ +--- + +# Industrial Edge + +## What problem does it solve? + +Manufacturing and OT environments generate sensor telemetry (temperature, vibration, pressure) that must flow from **shop floor MQTT brokers** into **Kafka pipelines**, through **ML anomaly scoring**, and into **operations dashboards** — often at the edge where latency and connectivity are constrained. Running that stack on Kubernetes at each factory site, while keeping GitOps alignment with a central hub, is the core Industrial Edge problem. + +This pattern deploys a **repeatable factory stack** on east and west spokes: AMQ Broker → Camel K → Kafka → optional KServe → line-dashboard, plus Tekton CI for edge software updates. The hub aggregates metrics (Skupper + Grafana) and exposes unified gateway access without duplicating brokers on the hub. + +The **Industrial Edge** pattern models discrete manufacturing and operational technology (OT) connectivity to Kubernetes: sensors, MQTT, Kafka-centric pipelines, CI/CD for edge software, ML-assisted anomaly detection, and centralized data lakes. + +After operators discover Kafka clusters and mesh namespaces receive ambient labels, use this page as the **business narrative** tying workloads together. If you haven't yet installed, start with **[Getting Started](getting-started)** and scaffold new instances via **[Scaffolding](scaffolding)**. + +## Factory pattern + +Factories emit telemetry through MQTT brokers and Camel integrations. Kubernetes namespaces isolate teams while GitOps keeps spoke clusters aligned with approved revisions. Each spoke runs an identical stack independently — the hub aggregates metrics and provides gateway access. + +## Data stack + +| Stage | Component | Namespace | +| ----- | ---------- | --------- | +| Ingress | AMQ Broker (MQTT acceptors), machine sensors | `industrial-edge-tst-all` | +| Integration | Camel K routes (MQTT→Kafka, Kafka→S3) | `industrial-edge-tst-all` | +| Streaming | AMQ Streams / Kafka dev-cluster topics | `industrial-edge-tst-all` | +| Replication | Strimzi MirrorMaker (factory→data-lake) | `industrial-edge-stormshift-messaging` | +| Data lake | prod-cluster Kafka + Camel S3 archiver | `industrial-edge-data-lake` | +| Processing | OpenShift AI (KServe InferenceService) | `industrial-edge-tst-all` | +| CI/CD | Tekton pipelines (buildah + deploy) | `industrial-edge-ci` | +| Visualization | line-dashboard (WebSocket consumer) | `industrial-edge-tst-all` | + +## Camel K integrations + +The `mqtt-to-kafka` integration bridges sensor data from the AMQ Broker to Kafka topics. It runs as a Camel K `Integration` CR deployed by the `industrial-edge-tst` chart. The **Camel Kaoto** software template in Developer Hub scaffolds additional Camel routes with a DevSpaces-ready project including: + +- MQTT→Kafka route (temperature, vibration) +- Kafka→S3/MinIO archiver (data lake) +- Alert→Mailpit route (anomaly notifications) +- Kaoto visual editor + Continue AI code assistant + +To scaffold a Camel route: **Developer Hub → Create → Industrial Edge — Camel Routes (Kaoto + Continue AI)** → choose east or west. + +## Spoke components (per cluster) + +Each spoke (east/west) deploys the following Argo CD Applications: + +| Application | Chart | Purpose | +| ----------- | ----- | ------- | +| `industrial-edge-tst` | `charts/all/industrial-edge-tst` | Sensors, broker, Kafka, Camel, dashboard, ML | +| `industrial-edge-stormshift` | `charts/all/industrial-edge-stormshift` | Factory Kafka + MirrorMaker | +| `industrial-edge-data-lake` | `charts/all/industrial-edge-data-lake` | prod-cluster + S3 archiver | +| `industrial-edge-pipelines` | `charts/all/industrial-edge-pipelines` | Tekton build-and-test pipeline | + +## Service mesh (ambient mode) + +Industrial Edge namespaces run under **OSSM3 ambient** mesh with ztunnel L4 metrics. The namespace `industrial-edge-tst-all` carries the label `istio.io/dataplane-mode: ambient`. Ztunnel captures `istio_tcp_connections_opened_total` and related L4 series; L7 `istio_requests_total` flows through waypoint gateways where deployed. + +**Exception:** `spoke-gateway-system` and `industrial-edge-data-lake` stay **off** ambient to avoid TLS conflicts with MinIO and WebSocket routing. + +## Links + +- [Industrial Edge Validated Pattern](https://validatedpatterns.io/patterns/industrial-edge) + +Related charts: `industrial-edge-tst`, `industrial-edge-stormshift`, `industrial-edge-pipelines`, hub-side data lake charts. + +--- + +**Next →** [Red Hat Products](products/) for per-operator deep dives (Service Mesh, ACM, AMQ Streams, Developer Hub, etc.). diff --git a/content/patterns/hybrid-mesh-platform/observability.md b/content/patterns/hybrid-mesh-platform/observability.md new file mode 100644 index 000000000..e864232f0 --- /dev/null +++ b/content/patterns/hybrid-mesh-platform/observability.md @@ -0,0 +1,319 @@ +--- +title: Observability +weight: 50 +aliases: /hybrid-mesh-platform/observability/ +--- + +# Observability + +Observability ties together **metrics**, **logs**, **traces**, and **mesh visualization** so operators can compare east and west Industrial Edge clusters from the hub. It sits mid-flight between **[architecture](architecture)** (why telemetry crosses Skupper) and **[Industrial Edge](industrial-edge)** (what applications emit). **Screenshots** below open full-screen when clicked — useful for reading dense Grafana legends. + +[![Observability overview](/images/hybrid-mesh-platform/workshop-observability.png)](/images/hybrid-mesh-platform/workshop-observability.png) + +_Observability stack overview: Grafana for multi-cluster dashboards, Kiali for mesh topology, Kafka Console for streaming health, and OpenTelemetry for distributed traces._ + +## Grafana — multi-cluster dashboards + +[![Grafana multi-cluster dashboards](/images/hybrid-mesh-platform/product-grafana-observability.png)](/images/hybrid-mesh-platform/product-grafana-observability.png) + +_Hub Grafana landing — fleet dashboards organized by cluster and workload type._ + +[![Grafana — east-west traffic and Service Mesh](/images/hybrid-mesh-platform/product-grafana-observability-2.png)](/images/hybrid-mesh-platform/product-grafana-observability-2.png) + +_East-west traffic: broker state gauges, leader distribution, and API request bargauges per cluster._ + +[![Grafana — multi-cluster Istio metrics (ztunnel L4)](/images/hybrid-mesh-platform/product-grafana-observability-3.png)](/images/hybrid-mesh-platform/product-grafana-observability-3.png) + +_Multi-cluster Istio: ztunnel TCP connections, bytes sent/received, error rate per cluster._ + +[![Grafana — extended fleet KPI panels](/images/hybrid-mesh-platform/product-grafana-observability-4.png)](/images/hybrid-mesh-platform/product-grafana-observability-4.png) + +_Fleet KPI: combined Kafka + mesh health across all clusters._ + +## Kiali — mesh topology visualization + +[![Kiali service mesh](/images/hybrid-mesh-platform/product-kiali-service-mesh.png)](/images/hybrid-mesh-platform/product-kiali-service-mesh.png) + +_Kiali traffic graph: L4 ztunnel connections between hub gateway and spoke services._ + +[![Kiali — service mesh traffic graph](/images/hybrid-mesh-platform/product-kiali-service-mesh-2.png)](/images/hybrid-mesh-platform/product-kiali-service-mesh-2.png) + +_Kiali detail view: per-service traffic rates, error percentages, and response time distributions._ + +## Kafka Console — streaming health + +[![Kafka Console](/images/hybrid-mesh-platform/product-kafka-console-amq-streams.png)](/images/hybrid-mesh-platform/product-kafka-console-amq-streams.png) + +_Kafka Console landing: five registered clusters (hub + east/west × dev/factory)._ + +[![Kafka Console — multi-cluster clusters and topics](/images/hybrid-mesh-platform/product-kafka-console-amq-streams-2.png)](/images/hybrid-mesh-platform/product-kafka-console-amq-streams-2.png) + +_Cluster detail: topics, partitions, and replicas per spoke Kafka cluster over Skupper._ + +[![Kafka Console — broker and topic detail over Skupper](/images/hybrid-mesh-platform/product-kafka-console-amq-streams-3.png)](/images/hybrid-mesh-platform/product-kafka-console-amq-streams-3.png) + +_Broker and topic metrics: producer/consumer rates, lag, and partition leadership distribution._ + +## AI-assisted operations (Kairos) + +Kairos integrates AI-driven operational intelligence through event correlation, anomaly detection, and human-in-the-loop workflows: + +[![Kairos — worker scaling and SmartScalingPolicy](/images/hybrid-mesh-platform/workshop-kairos-scaling.png)](/images/hybrid-mesh-platform/workshop-kairos-scaling.png) + +_Kairos SmartScalingPolicy: AI-driven recommendations for scaling Industrial Edge sensor workers based on Kafka lag and CPU pressure._ + +[![Kairos — event correlation and alerting](/images/hybrid-mesh-platform/kairos-events.png)](/images/hybrid-mesh-platform/kairos-events.png) + +_Event correlation: Kairos aggregates Kubernetes events, Prometheus alerts, and mesh signals to surface actionable incidents._ + +[![Kairos — historical analysis and trends](/images/hybrid-mesh-platform/kairos-history.png)](/images/hybrid-mesh-platform/kairos-history.png) + +_Historical analysis: trend detection across factory shifts, identifying degradation patterns before hard failures._ + +[![Kairos — human-in-the-loop decision workflows](/images/hybrid-mesh-platform/kairos-human-in-the-loop.png)](/images/hybrid-mesh-platform/kairos-human-in-the-loop.png) + +_Human-in-the-loop: Kairos proposes actions (scale, restart, reroute) that require operator approval before execution._ + +[![Kairos — AI agents for autonomous remediation](/images/hybrid-mesh-platform/kairos-ia-agents.png)](/images/hybrid-mesh-platform/kairos-ia-agents.png) + +_AI agents: autonomous remediation for pre-approved scenarios (auto-scale on Kafka consumer lag threshold)._ + +[![Kairos — observability-driven insights](/images/hybrid-mesh-platform/kairos-observability.png)](/images/hybrid-mesh-platform/kairos-observability.png) + +_Observability-driven insights: Kairos correlates Grafana metrics with mesh topology to pinpoint root cause across clusters._ +## Observability architecture + +```mermaid +flowchart TB + subgraph Hub["Hub Cluster"] + direction TB + GRAFANA_H["Grafana
(multi-cluster dashboards)"] + KIALI_H["Kiali + OSSM Console"] + KAFKA_C["Kafka Console
(4 remote clusters)"] + OTEL_H["OpenTelemetry Collector"] + PROM_H["Prometheus / Thanos"] + ZT_H["ztunnel + hub-gateway"] + DS_LOCAL["Datasource: Hub"] + DS_EAST["Datasource: Prometheus-East"] + DS_WEST["Datasource: Prometheus-West"] + GRAFANA_H --> DS_LOCAL --> PROM_H + GRAFANA_H --> DS_EAST + GRAFANA_H --> DS_WEST + KIALI_H --> PROM_H + ZT_H --> PROM_H + end + + subgraph East["East Spoke"] + GRAFANA_E["Grafana (local)"] + KIALI_E["Kiali + OSSM Console"] + PROM_E["Prometheus / Thanos"] + ZT_E["ztunnel"] + SM_E["PodMonitor / ServiceMonitor"] + SM_E --> PROM_E + ZT_E --> PROM_E + GRAFANA_E --> PROM_E + KIALI_E --> PROM_E + end + + subgraph West["West Spoke"] + GRAFANA_W["Grafana (local)"] + KIALI_W["Kiali + OSSM Console"] + PROM_W["Prometheus / Thanos"] + ZT_W["ztunnel"] + SM_W["PodMonitor / ServiceMonitor"] + SM_W --> PROM_W + ZT_W --> PROM_W + GRAFANA_W --> PROM_W + KIALI_W --> PROM_W + end + + PROM_E -->|"Skupper + auth proxy
prometheus-east:9091"| DS_EAST + PROM_W -->|"Skupper + auth proxy
prometheus-west:9091"| DS_WEST + East -->|"Kafka :9092 Skupper"| KAFKA_C + West -->|"Kafka :9092 Skupper"| KAFKA_C +``` + +## Components + +| Layer | Technology | Role | +| ----- | ----------- | ---- | +| Metrics | User Workload Monitoring / Prometheus | RED/USE signals, Kafka lag, mesh L4/L7 stats | +| Dashboards (hub) | Grafana + multi-cluster datasources | Fleet and factory KPI views (`charts/all/grafana-dashboards`) | +| Dashboards (spoke) | Grafana local | Per-cluster ztunnel L4, Kafka, workloads (`charts/all/spoke-dashboards`) | +| Mesh UI | Kiali + OSSM Console plugin | Traffic graphs in OpenShift Console | +| Kafka UI | Streams for Apache Kafka Console | Hub UI for all spoke Kafka clusters (`charts/all/kafka-console`) | +| Cross-cluster metrics | Skupper + GrafanaDatasource | Prometheus metrics via VAN | +| Tracing | OpenTelemetry Collector | Distributed traces | + +## Service Mesh metrics (OSSM3 GA + ztunnel) + +Use **`stable-3.2`** for the Service Mesh operator. Tech Preview (`candidates` / 3.0.0-tp.2) does not deploy ztunnel. + +| Metric | Producer | Notes | +| ------ | -------- | ----- | +| `istio_tcp_connections_opened_total` | ztunnel | Primary spoke/hub L4 signal | +| `istio_tcp_sent_bytes_total` / `received` | ztunnel | Bytes per workload namespace | +| `istio_requests_total` | Waypoints, ingress gateways | L7; hub `hub-gateway-istio` always has some traffic | +| `kafka_server_kafkaserver_brokerstate` | Strimzi JMX | `3` = Running; use in Grafana **gauge** panels | +| `kafka_network_requestmetrics_requestspersec_total` | Strimzi JMX | API activity; use in **bargauge** panels | +| `kafka_server_replicamanager_leadercount` / `partitioncount` | Strimzi JMX | **piechart** / **bargauge** on hub fleet view | + +`charts/all/istio-monitoring` scrapes istiod, gateways/waypoints, ztunnel, and Kafka. Grant UWM RoleBindings in `istio-system`, `ztunnel`, `hub-gateway-system`, and Industrial Edge namespaces. + +**Prerequisite for L4 mesh metrics:** `IstioCNI` CR must include `profile: ambient` (not namespace-only). Without it, ztunnel never becomes Ready and `istio_tcp_*` are absent. See [Service Mesh 3 — troubleshooting](products/service-mesh.md#troubleshooting-ztunnel-ztunnelnothealthy). + +## Skupper Network Observer + +**Skupper Network Observer** provides a real-time web UI of the Virtual Application Network — sites, connected services, process topology, and traffic metrics — deployed on the hub via `charts/all/skupper-network-observer`. + +[![Skupper Network Observer — VAN overview](/images/hybrid-mesh-platform/service-interconnect-console.png)](/images/hybrid-mesh-platform/service-interconnect-console.png) + +_Skupper Network Observer main view: three sites (hub, east, west) connected in the VAN, with site status and listener/connector counts._ + +[![Skupper Network Observer — service topology](/images/hybrid-mesh-platform/service-interconnect-console-topology.png)](/images/hybrid-mesh-platform/service-interconnect-console-topology.png) + +_Topology view: services exposed via Skupper across hub and spoke sites — Kafka brokers, Prometheus auth proxies, and Industrial Edge gateways._ + +[![Skupper Network Observer — topology with processes](/images/hybrid-mesh-platform/service-interconnect-console-topology-process.png)](/images/hybrid-mesh-platform/service-interconnect-console-topology-process.png) + +_Topology with processes overlay: shows pod-level endpoints backing each Skupper connector and listener._ + +[![Skupper Network Observer — process detail](/images/hybrid-mesh-platform/service-interconnect-console-process.png)](/images/hybrid-mesh-platform/service-interconnect-console-process.png) + +_Process view: per-process byte throughput and connection counts across the VAN. Useful to verify that Grafana datasources (`prometheus-east`, `prometheus-west`) are actively receiving metrics._ + +[![Skupper Network Observer — traffic metrics](/images/hybrid-mesh-platform/service-interconnect-console-metrics.png)](/images/hybrid-mesh-platform/service-interconnect-console-metrics.png) + +_Metrics view: application-level traffic rates per service. Validate that Kafka Console is reaching spoke bootstrap services and that Industrial Edge gateway traffic flows through the VAN._ + +Access the Network Observer at `https://skupper-network-observer-service-interconnect.apps./`. Verify the route exists after `service-interconnect` Argo CD app syncs: + +```bash +oc get route -n service-interconnect skupper-network-observer +``` + +## Kiali and OSSM Console plugin + +Each cluster (hub and spokes) runs **Kiali** with an **OSSMConsole** CR. The dynamic plugin adds **Service Mesh** to the OpenShift Console. + +### Fixing 401 Unauthorized on the plugin + +The plugin proxies API calls to Kiali, which queries **Thanos Querier** (`:9091`). Kiali's service account needs cluster monitoring read access. + +**GitOps** (`charts/all/kiali/templates/all.yaml`): + +```yaml +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: kiali-monitoring-rbac +roleRef: + kind: ClusterRole + name: cluster-monitoring-view +subjects: +- kind: ServiceAccount + name: kiali-service-account + namespace: openshift-cluster-observability-operator +``` + +Kiali CR `external_services.prometheus`: + +```yaml +prometheus: + auth: + type: bearer + use_kiali_token: true + thanos_proxy: + enabled: true + url: https://thanos-querier.openshift-monitoring.svc.cluster.local:9091 +``` + +With **ztunnel** running, Kiali shows L4 traffic graphs; L7 graphs appear for HTTP routed through waypoints. + +## Grafana + Thanos (dashboards with data) + +Hub Grafana uses a ServiceAccount token for local Thanos and **HTTP** URLs for remote spokes (Skupper auth proxy — no bearer token from hub). + +Spoke Grafana uses the **default** Prometheus datasource (local Thanos). Do not point spoke dashboards at hub Skupper listener names unless intentionally cross-querying. + +**Metric panels:** + +| Dashboard | Visualizations | Data sources | +| --------- | -------------- | -------------- | +| `east-west-traffic` | Gauges (broker state), donut pie (leaders East/West), bargauge (partitions, Kafka API req/s) | Hub + Prometheus-East/West | +| `multi-cluster-istio` | Timeseries + L4 **bargauge** per cluster | Mixed datasources | +| `local-metrics` | ztunnel readiness **gauge**, Kafka bargauge/piechart, L4 timeseries | Local Thanos | + +- Hub gateway / Istio HTTP panels may show **no data** until clients generate traffic through `hub-gateway-istio` or waypoints. +- Kafka panels use `kafka_network_requestmetrics_*` and `kafka_server_replicamanager_*` — not `brokertopicmetrics` with `_objectname` filters. + +Enable **User Workload Monitoring** on spokes (`cluster-monitoring-config` → `enableUserWorkload: true`). + +**Quick validation:** + +```bash +oc get ds -n ztunnel +oc logs -n istio-cni $(oc get pods -n istio-cni -o name | head -1) | grep AmbientEnabled +# Expect: AmbientEnabled: true +``` + +## Multi-cluster metrics via Skupper + +Spoke Thanos is exposed through an **Nginx auth proxy** on each spoke (injects bearer token), then a Skupper **Connector**. Hub **Listeners** `prometheus-east` / `prometheus-west` feed Grafana datasources. + +See [Service Interconnect](https://maximilianopizarro.github.io/hybrid-mesh-platform/validatedpatterns-docs/service-interconnect.html) for the full VAN diagram. + +## Kafka Console (hub) + +The **Streams for Apache Kafka Console** on the hub registers five clusters: `prod-cluster` (hub, full metrics) + dev/factory × east/west via Skupper bootstrap services. + +**Metrics configuration:** The `metricsSources` type `openshift-monitoring` is broken in Console operator 0.12.x (logs: `Prometheus URL is not configured`). Use `type: standalone` with: +- URL: `https://thanos-querier.openshift-monitoring.svc.cluster.local:9091` +- Bearer token via `kubernetes.io/service-account-token` Secret +- TrustStore: `openshift-service-ca.crt` ConfigMap (PEM) +- `ClusterRoleBinding` for `cluster-monitoring-view` + +Each `kafkaCluster` entry **must include `namespace`** — without it, logs show `namespace is required for metrics retrieval`. + +Only the hub `prod-cluster` (namespace `industrial-edge-data-lake`) displays full metrics. Spoke clusters via Skupper show topics and nodes but no metrics (their Prometheus data is not federated to hub Thanos). + +**Common error:** `Timed out waiting for a node assignment` / `listNodes` — the console reaches bootstrap over Skupper but broker **advertised DNS** from spokes does not resolve on the hub. + +**Fix:** + +1. Spokes: Strimzi `advertisedHost` per broker with `clusterName` suffix (`dev-cluster-broker-0-east`, etc.) +2. Hub: headless Services + **`EndpointSlice`** in `charts/all/kafka-console/templates/broker-dns.yaml` (Helm `lookup` of Skupper ClusterIPs per broker) + +Argo CD **excludes `Endpoints`** from managed resources — use `EndpointSlice` so broker DNS syncs via GitOps. + +Re-sync the `kafka-console` Argo CD application after Skupper listeners are healthy. Confirm `listNodes` returns 200 in the Console UI. + +## Grafana dashboard inventory + +| Dashboard | Scope | Datasources | +| --------- | ----- | ----------- | +| **`platform-overview`** | Hub | Hub, Prometheus-East, Prometheus-West — mesh, hub-gateway, Kafka fleet KPIs | +| `local-metrics` | Each spoke | Local Prometheus (UWM/Thanos) — ztunnel + Kafka + workloads | + +Legacy doc names `east-west-traffic` / `multi-cluster-istio` are consolidated into **`platform-overview`** (`charts/all/grafana-dashboards/templates/platform-overview.yaml`). + +### Fleet metrics checklist + +1. Skupper VAN complete — `sitesInNetwork: 3` +2. Hub listeners `prometheus-east`, `prometheus-west` **Ready** +3. Spoke Connectors + `prometheus-auth-proxy` Running (`spoke-interconnect`) +4. GrafanaDatasource CRs present on hub +5. Generate traffic (Industrial Edge / hub-gateway) for mesh panels + +## References + +- [OpenShift Observability](https://docs.redhat.com/en/documentation/openshift_container_platform/latest/html/monitoring/) +- [Red Hat Service Interconnect](https://docs.redhat.com/en/documentation/red_hat_service_interconnect/2.1) +- [OSSM 3.2 ambient mode](https://docs.redhat.com/en/documentation/red_hat_openshift_service_mesh/3.2/html/installing/ossm-istio-ambient-mode) +- [Kiali on OSSM 3.2](https://docs.redhat.com/en/documentation/red_hat_openshift_service_mesh/3.2/html/observability/kiali-operator-provided-by-red-hat) + +Charts: `charts/all/observability`, `charts/all/grafana-dashboards`, `charts/all/spoke-dashboards`, `charts/all/kiali`, `charts/all/kafka-console`, `charts/all/opentelemetry`, `charts/all/istio-monitoring`, `charts/all/service-interconnect`, `charts/all/spoke-interconnect`. + +--- + +**Next →** [Industrial Edge](industrial-edge) for the factory data pipeline pattern, then see the [Red Hat Products index](https://maximilianopizarro.github.io/hybrid-mesh-platform/validatedpatterns-docs/products/) for per-operator specifics. diff --git a/content/patterns/hybrid-mesh-platform/scaffolding.md b/content/patterns/hybrid-mesh-platform/scaffolding.md new file mode 100644 index 000000000..dbf8ef6df --- /dev/null +++ b/content/patterns/hybrid-mesh-platform/scaffolding.md @@ -0,0 +1,108 @@ +--- +title: Scaffolding +weight: 70 +aliases: /hybrid-mesh-platform/scaffolding/ +--- + +# Scaffolding Industrial Edge on East and West + +Developer Hub **Create** templates deploy new Industrial Edge instances to **east** or **west** spokes. This chapter is the operational follow-up to [Architecture](architecture) and [Getting Started](getting-started). + +## What you need before scaffolding + +| Requirement | How to verify | +| ----------- | ------------- | +| Developer Hub reachable | `https://developer-hub.` | +| Signed in as `platformadmin` (or your catalog user) | Keycloak OIDC; user in `catalog-users.yaml` | +| Gitea org `ws-platformadmin` exists | PostSync Job `gitea-admin-setup` in namespace `gitea` | +| Spoke tokens synced | `oc get secret developer-hub-spoke-tokens -n developer-hub` | +| Templates catalog loaded | **Create** shows templates (after GitHub Pages deploy) | +| Dev Spaces on target spoke | `oc get checluster -n devspaces` on east/west | + +`platformadmin` is the Gitea root-equivalent user (`gitea_admin` is the admin account). Workshop users get orgs `ws-user1`, `ws-user2`, …; `platformadmin` uses **`ws-platformadmin`**. + +## Software templates + +Templates are static files under `docs/assets/backstage/software-templates/` and load from GitHub Pages: + +```text +https://maximilianopizarro.github.io/hybrid-mesh-platform/assets/backstage/software-templates/templates-catalog.yaml +``` + +| Template | Target | Result | +| -------- | ------ | ------ | +| **Industrial Edge** | east or west | IoT namespace, sensors, Kafka, deployment, Tekton pipeline | +| **Camel Kaoto** | east or west | Camel routes, DevSpaces devfile, Continue AI config | +| **Camel CDC (Kaoto + Continue AI)** | east or west | Standalone CDC route; DevSpaces on **spoke** (`spokeAppsDomain`) | +| **Industrial Edge Delete** | east or west | Removes ArgoCD Application + Gitea repo + notification | +| **CNV VM Workshop** | hub | Virtual machine manifests in Gitea | + +## Step-by-step — deploy on east + +1. Open Developer Hub → **Create**. +2. Choose **Industrial Edge: IoT Manufacturing (Multi-Cluster)**. +3. Set **Instance Name** (e.g. `edge-factory-1`), **Owner** `platformadmin`, **Target Cluster** `east`. +4. Set **Hub cluster apps domain** (e.g. `apps.cluster-xqg4c.dynamic2.redhatworkshops.io`). +5. Run the template. + +Behind the scenes: + +1. `fetch:template` — skeleton from GitHub Pages (`maximilianopizarro.github.io` integration). +2. `publish:github` — repo `ws-platformadmin/edge-factory-1-east` on Gitea. +3. `catalog:register` — new Component in the catalog. +4. `http:backstage:request` — ArgoCD Application `gen-platformadmin-edge-factory-1-east` on cluster **east**. +5. Notification to `user:default/platformadmin`. + +## Verify Topology and Kubernetes tabs + +1. Open the new catalog entity (or **line-dashboard-east** for the platform stack). +2. **Topology** tab — requires `backstage.io/kubernetes-cluster: east` and `backstage.io/kubernetes-id` matching deployment labels. +3. **Kubernetes** tab — pods in `industrial-edge-tst-all` on cluster east. + +If Topology is empty: + +```bash +oc get secret developer-hub-spoke-tokens -n developer-hub -o jsonpath='{.data.EAST_SA_TOKEN}' | wc -c +oc get managedserviceaccount -n east +oc logs -n developer-hub -l app.kubernetes.io/name=backstage --tail=20 | grep -i kubernetes +``` + +## Gitea organizations + +| Org | Purpose | +| --- | ------- | +| `developer-hub` | Platform-owned repos | +| `ws-` | Per-user scaffold repos (e.g. `ws-platformadmin`) | +| `app-of-apps` | ApplicationSet-managed GitOps repos — delete repo to trigger ArgoCD prune | + +The `app-of-apps` org is created by the Gitea PostSync Job. Use it when wiring an **ApplicationSet** with a Gitea generator: each generated repo maps to one Argo CD Application; removing the repo lets prune clean up spoke resources. + +## Quay vs internal registry + +| Stage | Registry | +| ----- | -------- | +| Tekton **buildah** push | `image-registry.openshift-image-registry.svc:5000//:latest` | +| Deployment pull | Same internal image (no pull secret on OpenShift) | +| Catalog display | `quay.io/maximilianopizarro/` annotation only | + +On-prem **Quay** (`quay-registry.`) is for public catalog metadata and optional mirror; pipelines do not require Quay credentials unless you add an explicit push step. + +## DevSpaces and Continue AI + +DevSpaces runs on **spokes only** — not the hub. Template output links use: + +```text +https://devspaces./#https://gitea-gitea./ws-//raw/branch/main/devfile.yaml +``` + +Continue AI credentials are synced into `{username}-devspaces` by PostSync job `devspaces-continue-ai-sync` (reads `kairos-system/kairos-ai-credentials` on the spoke). Devfile `setup-continue` substitutes `CONTINUE_API_KEY` from the auto-mounted secret. + +## Delete an instance + +1. **Create** → **Industrial Edge — Delete Instance**. +2. Enter the same name, owner, and target cluster. +3. Template deletes the ArgoCD Application and Gitea repo; unregister the catalog entity manually if it still appears. + +--- + +**Next →** [Hub Gateway](hub-gateway) for cross-cluster HTTP routing, or [Observability](observability) to confirm metrics after workloads are running. diff --git a/modules/hybrid-mesh-platform/metadata-hybrid-mesh-platform.adoc b/modules/hybrid-mesh-platform/metadata-hybrid-mesh-platform.adoc new file mode 100644 index 000000000..30e6e58d6 --- /dev/null +++ b/modules/hybrid-mesh-platform/metadata-hybrid-mesh-platform.adoc @@ -0,0 +1,56 @@ +// This file has been generated automatically from the pattern-metadata.yaml file +// Do not edit manually! +:metadata_version: 1.0 +:name: hybrid-mesh-platform +:description: Hub-spoke multi-cluster GitOps on OpenShift with ACM, ambient Service Mesh, Service Interconnect (Skupper), Red Hat Connectivity Link (RHCL/Kuadrant), Industrial Edge, OpenShift AI, ACS, and observability — Validated Patterns layout. +:pattern_version: 1.4.0 +:display_name: Hybrid Mesh Platform +:repo_url: https://github.com/maximilianoPizarro/hybrid-mesh-platform +:docs_repo_url: https://github.com/validatedpatterns/docs +:issues_url: https://github.com/maximilianoPizarro/hybrid-mesh-platform/issues +:docs_url: https://maximilianopizarro.github.io/hybrid-mesh-platform/ +:tier: sandbox +:owners: maximilianoPizarro +:requirements_hub_minimum_workers: 3 +:requirements_hub_minimum_vcpu: 4 +:requirements_hub_minimum_memory_gib: 16 +:requirements_hub_minimum_openshift: 4.17 +:requirements_hub_recommended_workers: 3 +:requirements_hub_recommended_vcpu: 8 +:requirements_hub_recommended_memory_gib: 32 +:requirements_hub_recommended_openshift: 4.17+ +:requirements_hub_compute_platform_aws_replicas: 3 +:requirements_hub_compute_platform_aws_type: m5.2xlarge +:requirements_hub_compute_platform_azure_replicas: 3 +:requirements_hub_compute_platform_azure_type: Standard_D8s_v3 +:requirements_hub_compute_platform_gcp_replicas: 3 +:requirements_hub_compute_platform_gcp_type: n1-standard-8 +:requirements_hub_controlPlane_platform_aws_replicas: 3 +:requirements_hub_controlPlane_platform_aws_type: m5.xlarge +:requirements_hub_controlPlane_platform_azure_replicas: 3 +:requirements_hub_controlPlane_platform_azure_type: Standard_D4s_v3 +:requirements_hub_controlPlane_platform_gcp_replicas: 3 +:requirements_hub_controlPlane_platform_gcp_type: n1-standard-4 +:requirements_spoke_minimum_workers: 3 +:requirements_spoke_minimum_vcpu: 4 +:requirements_spoke_minimum_memory_gib: 16 +:requirements_spoke_minimum_openshift: 4.17 +:requirements_spoke_recommended_workers: 3 +:requirements_spoke_recommended_vcpu: 4 +:requirements_spoke_recommended_memory_gib: 16 +:requirements_spoke_recommended_openshift: 4.17+ +:requirements_spoke_compute_platform_aws_replicas: 3 +:requirements_spoke_compute_platform_aws_type: m5.xlarge +:requirements_spoke_compute_platform_azure_replicas: 3 +:requirements_spoke_compute_platform_azure_type: Standard_D4s_v3 +:requirements_spoke_compute_platform_gcp_replicas: 3 +:requirements_spoke_compute_platform_gcp_type: n1-standard-4 +:requirements_spoke_controlPlane_platform_aws_replicas: 3 +:requirements_spoke_controlPlane_platform_aws_type: m5.xlarge +:requirements_spoke_controlPlane_platform_azure_replicas: 3 +:requirements_spoke_controlPlane_platform_azure_type: Standard_D4s_v3 +:requirements_spoke_controlPlane_platform_gcp_replicas: 3 +:requirements_spoke_controlPlane_platform_gcp_type: n1-standard-4 +:extra_features_hypershift_support: False +:extra_features_spoke_support: True +:external_requirements: ['MaaS endpoint or OpenShift AI on hub for Kairos/Lightspeed optional features', 'Workshop Showroom content remains in showroom-hybrid-mesh-ai (separate repo)'] diff --git a/static/images/hybrid-mesh-platform/ACM.png b/static/images/hybrid-mesh-platform/ACM.png new file mode 100644 index 000000000..b205514e9 Binary files /dev/null and b/static/images/hybrid-mesh-platform/ACM.png differ diff --git a/static/images/hybrid-mesh-platform/ACS-2.png b/static/images/hybrid-mesh-platform/ACS-2.png new file mode 100644 index 000000000..794851c51 Binary files /dev/null and b/static/images/hybrid-mesh-platform/ACS-2.png differ diff --git a/static/images/hybrid-mesh-platform/ACS.png b/static/images/hybrid-mesh-platform/ACS.png new file mode 100644 index 000000000..2e347b8e2 Binary files /dev/null and b/static/images/hybrid-mesh-platform/ACS.png differ diff --git a/static/images/hybrid-mesh-platform/arch-data-flow.png b/static/images/hybrid-mesh-platform/arch-data-flow.png new file mode 100644 index 000000000..6d1fb223e Binary files /dev/null and b/static/images/hybrid-mesh-platform/arch-data-flow.png differ diff --git a/static/images/hybrid-mesh-platform/arch-gitops-sync-sequence.png b/static/images/hybrid-mesh-platform/arch-gitops-sync-sequence.png new file mode 100644 index 000000000..1bc93b31b Binary files /dev/null and b/static/images/hybrid-mesh-platform/arch-gitops-sync-sequence.png differ diff --git a/static/images/hybrid-mesh-platform/arch-hub-spoke-flow.png b/static/images/hybrid-mesh-platform/arch-hub-spoke-flow.png new file mode 100644 index 000000000..e93c3ae9e Binary files /dev/null and b/static/images/hybrid-mesh-platform/arch-hub-spoke-flow.png differ diff --git a/static/images/hybrid-mesh-platform/arch-observability-pipeline.png b/static/images/hybrid-mesh-platform/arch-observability-pipeline.png new file mode 100644 index 000000000..b2cc2ff8a Binary files /dev/null and b/static/images/hybrid-mesh-platform/arch-observability-pipeline.png differ diff --git a/static/images/hybrid-mesh-platform/arch-overview.png b/static/images/hybrid-mesh-platform/arch-overview.png new file mode 100644 index 000000000..a50ea54fd Binary files /dev/null and b/static/images/hybrid-mesh-platform/arch-overview.png differ diff --git a/static/images/hybrid-mesh-platform/arch-skupper-topology.png b/static/images/hybrid-mesh-platform/arch-skupper-topology.png new file mode 100644 index 000000000..8209b06b9 Binary files /dev/null and b/static/images/hybrid-mesh-platform/arch-skupper-topology.png differ diff --git a/static/images/hybrid-mesh-platform/arch-spoke-gateway.png b/static/images/hybrid-mesh-platform/arch-spoke-gateway.png new file mode 100644 index 000000000..27e20a748 Binary files /dev/null and b/static/images/hybrid-mesh-platform/arch-spoke-gateway.png differ diff --git a/static/images/hybrid-mesh-platform/arch-sync-waves.png b/static/images/hybrid-mesh-platform/arch-sync-waves.png new file mode 100644 index 000000000..c9f90e010 Binary files /dev/null and b/static/images/hybrid-mesh-platform/arch-sync-waves.png differ diff --git a/static/images/hybrid-mesh-platform/connectivity-link-hub-gateway.png b/static/images/hybrid-mesh-platform/connectivity-link-hub-gateway.png new file mode 100644 index 000000000..c3eaf8157 Binary files /dev/null and b/static/images/hybrid-mesh-platform/connectivity-link-hub-gateway.png differ diff --git a/static/images/hybrid-mesh-platform/connectivity-link-hub.png b/static/images/hybrid-mesh-platform/connectivity-link-hub.png new file mode 100644 index 000000000..3ea74718f Binary files /dev/null and b/static/images/hybrid-mesh-platform/connectivity-link-hub.png differ diff --git a/static/images/hybrid-mesh-platform/connectivity-link-spoke-gateway.png b/static/images/hybrid-mesh-platform/connectivity-link-spoke-gateway.png new file mode 100644 index 000000000..36fa6f8de Binary files /dev/null and b/static/images/hybrid-mesh-platform/connectivity-link-spoke-gateway.png differ diff --git a/static/images/hybrid-mesh-platform/connectivity-link-spoke.png b/static/images/hybrid-mesh-platform/connectivity-link-spoke.png new file mode 100644 index 000000000..df6b48cd3 Binary files /dev/null and b/static/images/hybrid-mesh-platform/connectivity-link-spoke.png differ diff --git a/static/images/hybrid-mesh-platform/kairos-events.png b/static/images/hybrid-mesh-platform/kairos-events.png new file mode 100644 index 000000000..d405784fd Binary files /dev/null and b/static/images/hybrid-mesh-platform/kairos-events.png differ diff --git a/static/images/hybrid-mesh-platform/kairos-history.png b/static/images/hybrid-mesh-platform/kairos-history.png new file mode 100644 index 000000000..4a5e96905 Binary files /dev/null and b/static/images/hybrid-mesh-platform/kairos-history.png differ diff --git a/static/images/hybrid-mesh-platform/kairos-human-in-the-loop.png b/static/images/hybrid-mesh-platform/kairos-human-in-the-loop.png new file mode 100644 index 000000000..92787de8c Binary files /dev/null and b/static/images/hybrid-mesh-platform/kairos-human-in-the-loop.png differ diff --git a/static/images/hybrid-mesh-platform/kairos-ia-agents.png b/static/images/hybrid-mesh-platform/kairos-ia-agents.png new file mode 100644 index 000000000..9ab09f5ad Binary files /dev/null and b/static/images/hybrid-mesh-platform/kairos-ia-agents.png differ diff --git a/static/images/hybrid-mesh-platform/kairos-observability.png b/static/images/hybrid-mesh-platform/kairos-observability.png new file mode 100644 index 000000000..1b358267d Binary files /dev/null and b/static/images/hybrid-mesh-platform/kairos-observability.png differ diff --git a/static/images/hybrid-mesh-platform/product-argocd-openshift-gitops.png b/static/images/hybrid-mesh-platform/product-argocd-openshift-gitops.png new file mode 100644 index 000000000..08df9f8fb Binary files /dev/null and b/static/images/hybrid-mesh-platform/product-argocd-openshift-gitops.png differ diff --git a/static/images/hybrid-mesh-platform/product-grafana-observability-2.png b/static/images/hybrid-mesh-platform/product-grafana-observability-2.png new file mode 100644 index 000000000..4b40ba5d1 Binary files /dev/null and b/static/images/hybrid-mesh-platform/product-grafana-observability-2.png differ diff --git a/static/images/hybrid-mesh-platform/product-grafana-observability-3.png b/static/images/hybrid-mesh-platform/product-grafana-observability-3.png new file mode 100644 index 000000000..cb727dccb Binary files /dev/null and b/static/images/hybrid-mesh-platform/product-grafana-observability-3.png differ diff --git a/static/images/hybrid-mesh-platform/product-grafana-observability-4.png b/static/images/hybrid-mesh-platform/product-grafana-observability-4.png new file mode 100644 index 000000000..3b8144ec9 Binary files /dev/null and b/static/images/hybrid-mesh-platform/product-grafana-observability-4.png differ diff --git a/static/images/hybrid-mesh-platform/product-grafana-observability.png b/static/images/hybrid-mesh-platform/product-grafana-observability.png new file mode 100644 index 000000000..a6e848554 Binary files /dev/null and b/static/images/hybrid-mesh-platform/product-grafana-observability.png differ diff --git a/static/images/hybrid-mesh-platform/product-kafka-console-amq-streams-2.png b/static/images/hybrid-mesh-platform/product-kafka-console-amq-streams-2.png new file mode 100644 index 000000000..1f65f00a4 Binary files /dev/null and b/static/images/hybrid-mesh-platform/product-kafka-console-amq-streams-2.png differ diff --git a/static/images/hybrid-mesh-platform/product-kafka-console-amq-streams-3.png b/static/images/hybrid-mesh-platform/product-kafka-console-amq-streams-3.png new file mode 100644 index 000000000..1bb78ab84 Binary files /dev/null and b/static/images/hybrid-mesh-platform/product-kafka-console-amq-streams-3.png differ diff --git a/static/images/hybrid-mesh-platform/product-kafka-console-amq-streams.png b/static/images/hybrid-mesh-platform/product-kafka-console-amq-streams.png new file mode 100644 index 000000000..9b959986a Binary files /dev/null and b/static/images/hybrid-mesh-platform/product-kafka-console-amq-streams.png differ diff --git a/static/images/hybrid-mesh-platform/product-kiali-service-mesh-2.png b/static/images/hybrid-mesh-platform/product-kiali-service-mesh-2.png new file mode 100644 index 000000000..3e83953f1 Binary files /dev/null and b/static/images/hybrid-mesh-platform/product-kiali-service-mesh-2.png differ diff --git a/static/images/hybrid-mesh-platform/product-kiali-service-mesh.png b/static/images/hybrid-mesh-platform/product-kiali-service-mesh.png new file mode 100644 index 000000000..057657422 Binary files /dev/null and b/static/images/hybrid-mesh-platform/product-kiali-service-mesh.png differ diff --git a/static/images/hybrid-mesh-platform/service-interconnect-console-metrics.png b/static/images/hybrid-mesh-platform/service-interconnect-console-metrics.png new file mode 100644 index 000000000..c16a77906 Binary files /dev/null and b/static/images/hybrid-mesh-platform/service-interconnect-console-metrics.png differ diff --git a/static/images/hybrid-mesh-platform/service-interconnect-console-process.png b/static/images/hybrid-mesh-platform/service-interconnect-console-process.png new file mode 100644 index 000000000..c1ef94c13 Binary files /dev/null and b/static/images/hybrid-mesh-platform/service-interconnect-console-process.png differ diff --git a/static/images/hybrid-mesh-platform/service-interconnect-console-topology-process.png b/static/images/hybrid-mesh-platform/service-interconnect-console-topology-process.png new file mode 100644 index 000000000..d463e6a6c Binary files /dev/null and b/static/images/hybrid-mesh-platform/service-interconnect-console-topology-process.png differ diff --git a/static/images/hybrid-mesh-platform/service-interconnect-console-topology.png b/static/images/hybrid-mesh-platform/service-interconnect-console-topology.png new file mode 100644 index 000000000..0edbabf39 Binary files /dev/null and b/static/images/hybrid-mesh-platform/service-interconnect-console-topology.png differ diff --git a/static/images/hybrid-mesh-platform/service-interconnect-console.png b/static/images/hybrid-mesh-platform/service-interconnect-console.png new file mode 100644 index 000000000..a07c8080b Binary files /dev/null and b/static/images/hybrid-mesh-platform/service-interconnect-console.png differ diff --git a/static/images/hybrid-mesh-platform/workshop-acm-multicluster.png b/static/images/hybrid-mesh-platform/workshop-acm-multicluster.png new file mode 100644 index 000000000..53bc4f32d Binary files /dev/null and b/static/images/hybrid-mesh-platform/workshop-acm-multicluster.png differ diff --git a/static/images/hybrid-mesh-platform/workshop-acs-kuadrant.png b/static/images/hybrid-mesh-platform/workshop-acs-kuadrant.png new file mode 100644 index 000000000..794851c51 Binary files /dev/null and b/static/images/hybrid-mesh-platform/workshop-acs-kuadrant.png differ diff --git a/static/images/hybrid-mesh-platform/workshop-hybrid-mesh-arch.png b/static/images/hybrid-mesh-platform/workshop-hybrid-mesh-arch.png new file mode 100644 index 000000000..e48bd9e9d Binary files /dev/null and b/static/images/hybrid-mesh-platform/workshop-hybrid-mesh-arch.png differ diff --git a/static/images/hybrid-mesh-platform/workshop-hybrid-mesh.png b/static/images/hybrid-mesh-platform/workshop-hybrid-mesh.png new file mode 100644 index 000000000..c8220de0d Binary files /dev/null and b/static/images/hybrid-mesh-platform/workshop-hybrid-mesh.png differ diff --git a/static/images/hybrid-mesh-platform/workshop-industrial-edge.png b/static/images/hybrid-mesh-platform/workshop-industrial-edge.png new file mode 100644 index 000000000..bb0de8d0c Binary files /dev/null and b/static/images/hybrid-mesh-platform/workshop-industrial-edge.png differ diff --git a/static/images/hybrid-mesh-platform/workshop-kairos-scaling.png b/static/images/hybrid-mesh-platform/workshop-kairos-scaling.png new file mode 100644 index 000000000..fa91514ba Binary files /dev/null and b/static/images/hybrid-mesh-platform/workshop-kairos-scaling.png differ diff --git a/static/images/hybrid-mesh-platform/workshop-observability.png b/static/images/hybrid-mesh-platform/workshop-observability.png new file mode 100644 index 000000000..75ce49a97 Binary files /dev/null and b/static/images/hybrid-mesh-platform/workshop-observability.png differ diff --git a/static/images/hybrid-mesh-platform/workshop-openshift-ai.png b/static/images/hybrid-mesh-platform/workshop-openshift-ai.png new file mode 100644 index 000000000..8c907107c Binary files /dev/null and b/static/images/hybrid-mesh-platform/workshop-openshift-ai.png differ diff --git a/static/images/hybrid-mesh-platform/workshop-service-mesh.png b/static/images/hybrid-mesh-platform/workshop-service-mesh.png new file mode 100644 index 000000000..3e83953f1 Binary files /dev/null and b/static/images/hybrid-mesh-platform/workshop-service-mesh.png differ diff --git a/static/images/hybrid-mesh-platform/workshop-software-templates.png b/static/images/hybrid-mesh-platform/workshop-software-templates.png new file mode 100644 index 000000000..402d74b15 Binary files /dev/null and b/static/images/hybrid-mesh-platform/workshop-software-templates.png differ