GPU capacity feed: Prometheus and OpenTelemetry to Meridian

Egress: api.onaro.io:443 (HTTPS)

What it does

If you run models on your own GPUs, the cost of an agent is a share of a cluster, not a line on a provider invoice. The capacity feed gives Meridian the facts it needs to split that cost: which GPUs each model endpoint held, how busy they were, and how much power they drew. You send NVIDIA DCGM metrics from Prometheus or the OpenTelemetry Collector; Meridian stores the raw samples, maps Kubernetes pods to the endpoints you register, and builds hourly capacity records in the Open Agent Spend Attribution (OASA) 0.2.0 format.

Capacity records carry GPU time, not dollars. The dollars come from the cost pools you define in Meridian, which are split across endpoints by the GPU time each one held, and then across the agents that called each endpoint.

You run Prometheus, the DCGM exporter and the collector. Onaro does not host any part of your telemetry stack, and only the metrics you forward leave your boundary.

There is no Connect card for capacity feeds in the Meridian app.

Prerequisites

• NVIDIA DCGM exporter on your GPU nodes, with Kubernetes pod mapping on so each metric carries the namespace, pod and gpu labels of the workload using the GPU.
• Prometheus (remote write 1.0) or the OpenTelemetry Collector with the Prometheus receiver and the OTLP/HTTP exporter. The templates below follow the Prometheus remote write configuration and the Collector's OTLP/HTTP exporter (verified as of 2026-10-07).
• A Meridian ingest key. An organization admin creates it in Meridian: open Settings → Usage ingest keys, select Create key, name it after this collector (for example "DCGM Prometheus"), and copy the key from the dialog. The key has the meridian:ingest scope only. The keys created on the Discovery event keys page are a different kind and are rejected with 401. The key is shown once; store it in your secret manager. See Manage ingest keys.
• Outbound HTTPS from Prometheus or the collector to https://api.onaro.io.
• Access to the Infrastructure endpoint registry at /meridian/settings/infrastructure-endpoints and Settings → Cost pools in the Meridian app. The registry page is not linked from the Settings menu yet; open it by path.

Setup

1. Choose an endpoint

Both use Authorization: Bearer MERIDIAN_INGEST_KEY.

Meridian uses three DCGM metrics. Send at least these, and filter out the rest to save bandwidth:

2a. Prometheus remote write

Add a remote write target that keeps only the three metrics. Meridian decodes the standard prometheus.WriteRequest message (remote write 1.0, the default).

scrape_configs:
  - job_name: dcgm-exporter
    honor_labels: true
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: dcgm-exporter
        action: keep

remote_write:
  - url: https://api.onaro.io/v1/meridian/ingest/metrics/prometheus
    authorization:
      type: Bearer
      credentials: MERIDIAN_INGEST_KEY
    write_relabel_configs:
      - source_labels: [__name__]
        regex: DCGM_FI_(DEV_GPU_UTIL|PROF_GR_ENGINE_ACTIVE|DEV_POWER_USAGE)
        action: keep

honor_labels: true keeps the exporter's namespace and pod labels, which name the workload pod, instead of renaming them when they clash with target labels.

2b. OpenTelemetry Collector

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: dcgm
          scrape_interval: 30s
          honor_labels: true
          static_configs:
            - targets:
                - DCGM_EXPORTER_HOST:9400
          metric_relabel_configs:
            - source_labels: [__name__]
              regex: DCGM_FI_(DEV_GPU_UTIL|PROF_GR_ENGINE_ACTIVE|DEV_POWER_USAGE)
              action: keep

processors:
  batch:
    timeout: 10s
    send_batch_size: 512

exporters:
  otlphttp:
    metrics_endpoint: https://api.onaro.io/v1/meridian/ingest/metrics/otlp
    encoding: json
    headers:
      Authorization: Bearer MERIDIAN_INGEST_KEY

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [batch]
      exporters: [otlphttp]

Use metrics_endpoint with the full path. The exporter's endpoint setting appends /v1/metrics, which Meridian does not serve. encoding: json is required; the OTLP path does not accept protobuf.

2c. JSON batch (testing or custom forwarders)

The Prometheus path also accepts a simple JSON batch, which is the easiest way to test the key and the registry:

curl -s https://api.onaro.io/v1/meridian/ingest/metrics/prometheus \
  -H "Authorization: Bearer $MERIDIAN_INGEST_KEY" \
  -H "Content-Type: application/json" \
  -d '{"timeseries":[{"labels":{"__name__":"DCGM_FI_DEV_GPU_UTIL","namespace":"gpu-workloads","pod":"llama-70b-abc123","gpu":"0"},"samples":[{"timestamp_ms":1759831200000,"value":62.0}]}]}'

3. Register endpoints

The registry maps pods to the model endpoints and pools you account for. Open /meridian/settings/infrastructure-endpoints and add a row per endpoint, or upload a CSV. endpoint_id is required; the other columns are optional.

endpoint_id,pool_id,display_name,k8s_namespace,pod_selector,gpu_count,hosting,active
nai-llama-70b,pool-gpu-east,Llama 70B East,gpu-workloads,llama-70b,8,dedicated_capacity,true
nai-embed,pool-gpu-east,Embeddings East,gpu-workloads,re:embed-[a-z0-9]+,2,dedicated_capacity,true

• pod_selector matches the exact pod name, a substring of it, or a regular expression written as re:<pattern>.
• k8s_namespace limits the match to one namespace; leave it empty to match any.
• hosting is one of dedicated_capacity (the default) or self_hosted.

Pods that send metrics without matching a row are listed under Unmapped pods.

4. Define the cost pools

Open Settings → Cost pools and create a pool for each pool_id with the period's total cost (lease, depreciation, power, support). Meridian allocates due pool periods daily at 04:45 UTC. See What gets attributed below for how the split works.

5. Wait for the daily aggregation

Meridian builds the hourly capacity records for each UTC day at 00:30 UTC the next day. Each run also rebuilds any hour from the last 7 days that has samples, so late samples are included, except hours in a locked chargeback month or locked pool period.

What Meridian receives

Raw samples. Every metric you send is stored as a raw sample for 7 days, with no filtering. From Prometheus, Meridian reads each series' labels and samples. From OTLP, it reads gauge and sum metrics with asDouble or asInt points, takes time from timeUnixNano, and takes labels from resource and point attributes. Histogram and summary metrics are not read.

Labels used. namespace or k8s.namespace.name; pod or k8s.pod.name (required, samples without a pod are not aggregated); gpu or GPU (defaults to 0).

Hourly capacity records. For each endpoint and hour, Meridian builds one record:

Each record's id is derived from the organization, endpoint and hour, so running the aggregation again for the same hour replaces the record instead of duplicating it.

Verification

1. Send the JSON batch from step 2c. The response is {"inserted": 1, "received": 1}.
2. Start Prometheus or the collector and watch its logs for 2xx responses from api.onaro.io.
3. Register the pods from your samples and check that they are no longer the only entries under Unmapped pods (see Troubleshooting).
4. After the next 00:30 UTC aggregation, confirm hourly capacity records for each endpoint.
5. After the next 04:45 UTC allocation for a due pool period, check that pool cost appears against the endpoints and agents.

What gets attributed and what doesn't

Cost is attributed in two steps, from Settings → Cost pools:

1. Pool to endpoints. A pool's period cost is split across its endpoints by the GPU-seconds each held, as a share of the pool's capacity.
2. Endpoint to agents. Each endpoint's share is split across the agents that called it, using the measured cost of their usage records for that endpoint (for example from a gateway that sets endpoint_id), or their tokens when there is no measured cost.

What does not get attributed:

• Unmapped pods. Samples from pods with no registry match are stored but do not produce capacity records.
• Idle capacity. GPU time no endpoint used follows the pool's idle policy (spread, hold or reserve) rather than going to a specific agent.
• Endpoints nobody calls through an attributed path. If no usage records name an endpoint, its share cannot be split to agents.

Troubleshooting

Limits

• Early access. The test fixtures are synthetic, not captured from a live cluster. Check your first capacity records against what your cluster reports.
• Ingest keys are managed by organization admins. Up to 10 active ingest keys per organization. See Manage ingest keys.
• The registry page is reachable by path only until it is linked from Settings.
• GPU counting is coarse. A GPU seen in a single sample counts for the whole hour, and GPUs on different nodes that share an index (for example two gpu="0") under one endpoint count once. Register one endpoint per node if that matters for your split.
• Energy is understated on multi-GPU endpoints. energy_wh uses the mean power per GPU, not the sum across GPUs.
• No deduplication of raw samples. Re-sending samples does not double the hourly record, but it does store them twice for the 7-day retention window.
• The registry's GPU count is not checked against the GPUs observed.