Kubernetes Observability: Metrics, Logs, and Traces in Practice

Kubernetes makes your system possible to observe: every pod restarts, every node reports, every API call is auditable. But the cluster itself won’t assemble the picture for you. Observability — knowing not just that something is wrong but why — requires wiring together three complementary signals: metrics, logs, and traces.

Why observability is different in Kubernetes

On a static server, you ssh in and look. In Kubernetes, the thing you’re debugging may no longer exist: pods die during investigation, nodes get drained, and the request that failed crossed five services in three namespaces. You need answers without shell access to a specific machine:

• “Is anything broken right now?” → metrics
• “What exactly happened?” → logs
• “Where in the call chain did it fail?” → traces

The Kubernetes observability documentation frames it the same way: logging for what happened, monitoring for current state, tracing for request flow.

Pillar 1: Metrics — the pulse of the cluster

Metrics are numerical time series: request rates, error counts, latency histograms, CPU/memory usage. They answer is it broken and is it getting worse.

Cluster-level metrics come built in:

• Metrics Server powers kubectl top pods / kubectl top node — live CPU and memory usage (the resource metrics pipeline).
• kube-state-metrics exposes object state — deployment replicas ready, pod phase, PVC bound — things usage metrics can’t see.
• Control-plane metrics — apiserver latency, etcd request duration — are scraped from component endpoints.

The standard stack is Prometheus + Grafana, the CNCF de facto pair. Prometheus scrapes exporters and PromQL does the rest:

# 99th percentile of apiserver request latency
histogram_quantile(0.99,
  sum(rate(apiserver_request_duration_seconds_bucket[5m])) by (le))

# Pods restarting more than once an hour
sum(increase(kube_pod_container_status_restarts_total[1h])) by (pod) > 1

Many teams adopt the USE method for infrastructure (Utilization, Saturation, Errors per resource) and RED for services (Rate, Errors, Duration per request). Metrics-server tells you a node is at 90% CPU; it doesn’t tell you your p99 latency doubled after last night’s deploy — Prometheus does.

Pillar 2: Logs — the record of events

Logs are timestamped text records from containers and cluster components. In Kubernetes, kubectl logs reads stdout/stderr — apps should log to stdout, not files (the logging architecture guide’s central piece of advice). Then container runtimes route it through the node’s log mechanism.

For anything beyond a single pod, you need aggregation. The dominant pattern:

1. A DaemonSet side agent (Fluent Bit, Vector, Filebeat) tails container log files on every node.
2. It ships them to a backend — Loki, Elasticsearch/OpenSearch, or a cloud service.
3. You query centrally with labels matching Kubernetes metadata (namespace, pod, container).

Loki + Grafana pairs naturally with the Prometheus/Grafana choice: logs live beside metrics in one UI, indexed by labels rather than fully indexed (cheaper, fast enough for most teams).

Structured logging beats fmt.Println debugging: JSON logs with a trace_id field become the bridge to pillar three.

{"level":"error","msg":"payment failed","trace_id":"4bf92f3577b34da6","order_id":"A-1001"}

Pillar 3: Traces — following one request

A distributed trace records one request’s journey across every service it touches: spans, timing, and where time or errors accumulated. When Service A calls B calls C and only C fails, metrics say “errors up” and logs say “three errors,” but only a trace shows A→B→C, failed at C’s DB call, 2.4s spent.

OpenTelemetry (OTel) is the CNCF standard for generating and exporting traces (and metrics and logs — it unifies all three signals). Applications instrument with the OTel SDK, and an OTel Collector fans data out to backends like Jaeger, Tempo, or Zipkin.

Kubernetes integration points:

• The kubelet and control plane components can emit traces for system components (stable since v1.27 as alpha→stable progression) — so even the API server’s handling of a request is visible.
• Propagate trace_id through HTTP headers (W3C traceparent) between services, then correlate logs containing that ID with the trace in your UI.

Wiring the pillars together

The goal isn’t three isolated tools — it’s one investigation path:

1. Alert fires from a metric (error rate > 2% for 5 min).
2. Dashboard shows scope — which service, which namespace, correlated deploy event.
3. Logs reveal the error message for failing pods, filtered by the same labels.
4. Trace pinpoints the hop — the exact downstream call that failed, with latency breakdown.

The correlation key is shared labels/IDs: the same service, namespace, and trace_id across all three signals. Without that join, you’re context-switching between three UIs with three vocabularies — which is just siloed debugging with extra steps.

Kubernetes-specific gotchas

• Metrics lie without requests/limits. Unbounded pods skew node graphs; set them and use the vertical pod autoscaler data to tune.
• kubectl logs lies after restarts. Use --previous, or accept that crash-loop history requires a log backend.
• Watch the watch. Monitoring the control plane itself (etcd fsync latency, apiserver saturation) matters — when it degrades, everything looks down.
• Don’t cardinality-bomb Prometheus. Labeling by raw pod name or user ID explodes time-series count; use consistent low-cardinality labels and push high-cardinality detail to logs/traces.
• Alerts need runbooks. An alert no human knows how to act on is noise; pair every page-worthy alert with a documented first response.

A pragmatic starting stack

You don’t need everything at once:

Stage	Add
1	Metrics Server + kubectl top; alerts on pod restarts and node pressure
2	Prometheus + Grafana with kube-prometheus-stack dashboards and exporters
3	Centralized logging (Fluent Bit → Loki or OpenSearch)
4	OpenTelemetry tracing for your most critical request path

Observe incrementally, and always tie each signal to a question you actually need answered. The reference documentation — logging architecture, metrics for system components, and tracing — is the authoritative map. But the essence is simple: metrics tell you when, logs tell you what, traces tell you where. You need all three to answer why.