AI Infrastructure Observability vs LLM Observability: Signals, Owners and Shared Failures

AI infrastructure observability tracks the hardware a model runs on: GPUs, the network between them, the job scheduler and the facility's power and cooling. LLM observability tracks what a large language model (LLM) does on that hardware: answer quality, safety, cost per request and drift over time.

Platform teams usually set up the hardware monitoring, and machine learning (ML) and product teams set up the model monitoring. The two layers meet when a hardware fault reaches users as a model problem, such as slower or worse answers.

Each team should instrument its own layer first, then send both sets of signals to one shared timeline.

The Six-Layer AI Observability Stack

Retool's AI observability stack splits one AI application into six layers, from the hardware at the bottom to the user at the top:

1. Infrastructure layer: shows whether a slow response comes from the model or from rate limits and throughput, so the team knows whether to tune prompts or add compute.
2. Data layer: checks whether a search returned relevant documents and whether the model used those documents.
3. Model layer: records each LLM call with its prompt and response, so teams can compare prompt versions and track cost.
4. Agent layer: logs which tool an agent picked at each step and what it passed to that tool.
5. Workflow and orchestration layer: times each step of a multi-step job to find stalled, failed or retried steps.
6. Application layer: captures user signals such as a thumbs-down, an abandoned task or a rephrased question.

Hardware monitoring watches the bottom layer of that stack, and LLM observability watches the model layer. CoreWeave's guide calls the same practice model observability, with signals such as accuracy, drift, bias and hallucination rates.

Two related practices use their own tools and metrics:

1. Agent tracing: records each step and tool call an agent makes during a multi-step task.
2. LLM evaluation: scores answers against a test set, often as a check that holds back a release until it meets a minimum score.

AI Infrastructure Observability Signals: GPUs, Network, Scheduler and Facility

CoreWeave's guide lists six signals for the compute, storage and network under an AI workload, and each one exposes a different way a job loses time or money:

1. GPU and CPU utilization: shows whether the accelerators and host processors are busy or waiting for work.
2. Memory bandwidth: shows how fast data moves between GPU memory and compute, which sets the speed limit for generating each token during inference.
3. Interconnect latency: shows the delay on links between GPUs, where one slow link can hold back the training job.
4. Job scheduling efficiency: shows how well queued jobs get placed on free, suitable GPUs, which exposes idle reserved hardware.
5. Power consumption: shows the energy draw of the fleet, which feeds facility cost and capacity planning.
6. Cooling performance: shows whether heat removal keeps pace with GPU heat output.

Dynatrace's AI observability guide names GPU usage, memory pressure and network bottlenecks, which overlap with CoreWeave's first three signals. It adds two priorities missing from CoreWeave's list:

1. Inference cost breakdowns: the hardware cost of serving each inference workload.
2. Availability of AI infrastructure: the uptime of outside services the application relies on, such as a hosted LLM endpoint or a managed vector database for document search.

Together, these signals tell a platform team whether a training job will finish on schedule and whether an inference endpoint will hold its response time under load. CoreWeave's guide names Prometheus, Grafana and Datadog as common tools for collecting and charting these signals.

LLM Observability Signals: Quality, Safety, Cost and Drift

LLM observability watches what the model produces, on top of the logs, metrics and traces from standard monitoring. Kong's guide to AI observability calls this added layer behavioral signals and splits it into four groups:

1. Quality metrics: score whether answers are accurate, coherent and relevant, and catch hallucination, when a model states something false as fact.
2. Safety signals: count policy violations, toxic output and prompt injection attempts, where a user tries to override the model's instructions.
3. Cost intelligence: tracks token use against the budget, because model providers bill by the token, a short chunk of text.
4. Behavioral patterns: flag drift and anomalies, when live answers start to differ from an earlier baseline.

Monitoring tools can share attribute names for AI telemetry through OpenTelemetry, the open-source project behind common formats for monitoring data. Its draft GenAI semantic conventions use five attributes to record the model, the token counts and the stop reason:

1. `gen_ai.provider.name`: the model provider, such as OpenAI or Anthropic. Kong's guide and older instrumentation use the earlier name, gen_ai.system, so a dashboard that groups by provider needs to read both names.
2. `gen_ai.request.model`: the model named in the request.
3. `gen_ai.usage.input_tokens`: the number of tokens in the prompt.
4. `gen_ai.usage.output_tokens`: the number of tokens generated, which most providers price higher than input tokens.
5. `gen_ai.response.finish_reasons`: why generation stopped, such as reaching a length limit.

Infrastructure vs LLM Observability Side by Side

The table pairs each layer's main question with one example per row.

Infrastructure Faults That Show Up as Model Problems

Retool's guide warns that infrastructure problems often look like quality issues, so teams end up tuning prompts when a queue or a rate limit is causing the delay.

NVIDIA's guide to AI factory observability describes a training job whose throughput fell for six hours while GPU utilization looked normal. The cause was one link in the InfiniBand network between servers, where a rising error rate slowed one GPU and stalled the rest of the job.

GPU utilization records whether a GPU was busy in tools such as NVIDIA's Data Center GPU Manager. Our guide to measuring useful GPU work names a better inference measure than utilization: Model Bandwidth Utilization (MBU), a workload's share of the GPU's peak memory bandwidth.

Benchmarks cited in that guide put MBU at about 60% on two H100 GPUs serving one request at a time. The utilization counter hides the unused 40% of memory bandwidth.

Equinix finds that AI infrastructure outages most often start between the compute systems and the facility's power and cooling, so facility data belongs beside GPU health on one timeline.

Infrastructure vs LLM Observability by Failure Scenario

Four common production problems show which layer to check first and when a diagnosis needs both layers:

1. Distributed training slows down: Start with the infrastructure layer. Interconnect latency points to one slow GPU holding back the others, low utilization points to GPUs waiting for data, and memory bandwidth shows stalled transfers. Normal readings point to the code or the data, so the training logs come next.
2. Inference latency spikes at peak traffic: First check how busy the GPUs are under load, cooling performance and the scheduler queue length. Steady hardware readings move the check to the model layer, starting with output token counts on the slow requests, because generation time grows with answer length.
3. Quality drops after a model update: Check the model layer first, comparing answers against the baseline from before the update and scoring quality on the affected prompts. Rule out the infrastructure layer before blaming the update, because a hardware fault can make answers worse without moving average latency.
4. Spend doubles month over month: Use both cost views. Token tracking at the model layer shows whether more requests, longer prompts or a pricier model drove the increase. Inference cost breakdowns and utilization at the infrastructure layer show whether the fleet now serves the same traffic at lower efficiency.

The First Observability Layer for Each Team

Each team starts with the layer it runs day to day:

1. Platform teams: Start with the hardware layer, covering the six signals from CoreWeave's list, the uptime of outside AI services, and facility data next to GPU health. They're accountable for the fleet, so model-layer signals can wait until the hardware data is reliable.
2. ML and product teams: Start with LLM observability, covering the four groups of behavioral signals, recorded with the OpenTelemetry attribute names so the data survives a change of monitoring vendor. The hardware still sets their latency and cost, so the link to infrastructure data comes second, built with the platform team.
3. Organizations running both: Put both sets of signals on one timeline, so a drop in answer quality and a cooling alarm appear in the same view.

For the infrastructure half of that timeline, Zettabyte's zWARE cluster software tracks GPU, network card, power and system health in real time, along with cooling and environmental data.