GPU capacity feed — Prometheus & OpenTelemetry → Meridian

Status: Beta · 2026-09-22
Egress: api.onaro.io:443 (HTTPS)

Meridian accepts normalized DCGM-style GPU metrics, stores raw samples for seven days, and rolls hourly capacity OASA records (schema 0.2.0). You operate Prometheus, the DCGM exporter, and/or the OpenTelemetry Collector; Onaro does not host your cluster telemetry stack.

Available tier blocker: fixtures under caio-assistant/tests/fixtures/capacity/ are SYNTHETIC only — no live cluster capture in WO-CAPACITY-1. Do not upgrade marketing copy ahead of live acceptance.

---

Prerequisites

• DCGM exporter (or equivalent) exposing DCGM_FI_* metrics with Kubernetes pod labels (namespace, pod, optional container, gpu)
• A Meridian ingest integration key with meridian:ingest scope (Bearer token)
• Outbound HTTPS from the scraper or collector to https://api.onaro.io
• Optional: forward proxy if your network requires it (see below)

---

Pinned metric names

---

Ingest endpoints

Auth for both paths:

Authorization: Bearer MERIDIAN_INGEST_KEY

Replace MERIDIAN_INGEST_KEY with the key value shown once when you create the integration key in Meridian.

Prometheus (lab JSON batch)

Native Prometheus remote_write sends snappy+protobuf and currently receives 415 Unsupported Media Type until that path is wired in production. For Beta, use application/json lab batches (timeseries[] with labels / samples) or forward metrics through the OTel Collector template below.

POST https://api.onaro.io/v1/meridian/ingest/metrics/prometheus
Authorization: Bearer MERIDIAN_INGEST_KEY
Content-Type: application/json

Example body shape (see SYNTHETIC fixture prometheus-remote-write-json.json in caio-assistant):

{
  "timeseries": [
    {
      "labels": {
        "__name__": "DCGM_FI_DEV_GPU_UTIL",
        "namespace": "gpu-workloads",
        "pod": "llama-70b-abc123",
        "gpu": "0"
      },
      "samples": [{ "timestamp_ms": 1727000000000, "value": 50.0 }]
    }
  ]
}

Companion template: prometheus-remote-write-meridian.yml (write relabel for DCGM_FI_*; see protobuf limitation above).

OTLP JSON metrics

POST https://api.onaro.io/v1/meridian/ingest/metrics/otlp
Authorization: Bearer MERIDIAN_INGEST_KEY
Content-Type: application/json

Body: OTLP JSON resourceMetrics with gauge/summary points whose metric names match the pinned DCGM names (after your collector renames or preserves exporter names).

Companion template: otel-collector-dcgm-meridian.yaml (Prometheus receiver → OTLP/HTTP exporter).

---

Infrastructure endpoint registry

Capacity aggregation maps Kubernetes namespace + pod to OASA endpoint_id and pool_id. Register rows before or after telemetry arrives; unmapped pods are listed for follow-up.

UI: Meridian → Settings → Infrastructure endpoint registry (/meridian/settings/infrastructure-endpoints)

API (session auth):

• GET /api/meridian/infrastructure-endpoints — list mappings
• POST /api/meridian/infrastructure-endpoints — upsert one row
• GET /api/meridian/infrastructure-endpoints/unmapped — pods seen without a match
• POST /api/meridian/infrastructure-endpoints/csv — bulk upsert

CSV convention

Required column: endpoint_id.

Optional columns (header names are case-sensitive as below):

endpoint_id, pool_id, display_name, k8s_namespace, pod_selector, gpu_count, hosting, active

• k8s_namespace — limit match to one namespace (empty = any namespace)
• pod_selector — substring match on pod name, exact pod name, or re:<regex> (e.g. re:llama-70b-[a-z0-9]+)
• hosting — e.g. dedicated_capacity, self_hosted
• active — true / false (default true)

Example:

endpoint_id,pool_id,display_name,k8s_namespace,pod_selector,gpu_count,hosting,active
nai-llama-70b,pool-gpu-east,Llama 70B East,gpu-workloads,llama-70b,8,dedicated_capacity,true

---

Forward proxy

If scrapers or the collector must egress via a corporate proxy:

• Set HTTPS_PROXY / HTTP_PROXY on the Prometheus or OpenTelemetry Collector process
• Allowlist api.onaro.io on the proxy and any TLS inspection appliance
• Keep Authorization: Bearer MERIDIAN_INGEST_KEY out of logs and APM redaction rules

---

Nutanix / NKP (private GPU clusters)

Many teams run GPU workloads on Nutanix Kubernetes Platform (NKP) or similar private clusters with node-local DCGM exporters and in-cluster Prometheus or OTel. Meridian does not claim a Nutanix partnership — you deploy the same scrape → forward pattern inside your boundary:

• Scrape DCGM on GPU nodes or via a DaemonSet; ensure namespace and pod labels are present for registry matching
• Set pool_id and endpoint_id in the infrastructure registry to match how you attribute private capacity elsewhere (e.g. LiteLLM metadata on the same cluster)
• Egress from a collector pod or bastion host that can reach api.onaro.io:443

---

Verification

1. Create a Meridian integration key with meridian:ingest.
2. POST one lab JSON batch (Prometheus path) or OTLP JSON batch; expect HTTP 200 with inserted / received counts.
3. Register k8s_namespace + pod_selector for pods in the sample labels.
4. After the hourly aggregation window, confirm capacity records in Meridian (OASA 0.2.0, optional gpu_busy_seconds / energy_wh).
5. Open Infrastructure endpoint registry → Unmapped if pods still lack mappings.

---

Related docs

• Internal telemetry summary: docs/CAPACITY-1-telemetry.md
• Verification tiers: docs/CAPACITY-1-verification.md