GKE Observability Guide

Complete guide to monitoring, logging, and observability in Google Kubernetes Engine (GKE).

Observability Overview

Key Components

Component	Description
Cloud Monitoring	Metrics collection, dashboards, alerting
Cloud Logging	Log aggregation, search, analysis
Cloud Trace	Distributed tracing
Managed Prometheus	Prometheus-compatible metrics

Cloud Monitoring

GKE Metrics

Key GKE Metrics

Metric	Description
`container/cpu/usage_time`	CPU usage
`container/memory/used_bytes`	Memory usage
`container/restart_count`	Container restarts
`pod/network/received_bytes_count`	Network received
`pod/network/sent_bytes_count`	Network sent

Enable Workload Metrics

bash

# Enable managed Prometheus collection
gcloud container clusters update my-cluster \
    --enable-managed-prometheus

# Enable system metrics
gcloud container clusters update my-cluster \
    --monitoring=SYSTEM

GKE Dashboard

Access at: Console > Kubernetes Engine > Clusters > [Cluster] > Observability

Cloud Logging

Log Types

Log Type	Source	Enable
Container logs	Pod stdout/stderr	Default
System logs	Node components	Default
Audit logs	API server	Default
Data access logs	API operations	Optional

Log Queries

bash

# Query container logs
resource.type="k8s_container"
resource.labels.cluster_name="my-cluster"
resource.labels.namespace_name="production"
resource.labels.pod_name=~"my-app-.*"
severity>=ERROR

# Query by label
resource.type="k8s_container"
labels."k8s-pod/app"="my-app"

# Query specific time range
resource.type="k8s_container"
timestamp>="2024-01-01T00:00:00Z"
timestamp<="2024-01-02T00:00:00Z"

Structured Logging

json

{
  "severity": "ERROR",
  "message": "Failed to process request",
  "httpRequest": {
    "requestMethod": "POST",
    "requestUrl": "/api/orders",
    "status": 500
  },
  "labels": {
    "orderId": "12345",
    "userId": "user-789"
  }
}

Alerting

Alert Policy Structure

Create Alert Policy

yaml

# High CPU alert
displayName: "High Pod CPU Usage"
conditions:
- displayName: "CPU > 80%"
  conditionThreshold:
    filter: >
      resource.type="k8s_container"
      metric.type="kubernetes.io/container/cpu/limit_utilization"
    comparison: COMPARISON_GT
    thresholdValue: 0.8
    duration: "300s"
    aggregations:
    - alignmentPeriod: "60s"
      perSeriesAligner: ALIGN_MEAN
notificationChannels:
- projects/PROJECT_ID/notificationChannels/CHANNEL_ID
alertStrategy:
  autoClose: "604800s"

Common Alert Conditions

Alert	Condition
High CPU	CPU utilization > 80% for 5 min
High Memory	Memory > 90% for 5 min
Pod Restarts	Restart count > 3 in 10 min
Node Not Ready	Node status != Ready
PVC Near Full	Disk usage > 85%

Managed Prometheus

Architecture

Enable Managed Prometheus

bash

gcloud container clusters update my-cluster \
    --enable-managed-prometheus

Configure Scraping

yaml

apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
  name: my-app-monitoring
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
  - port: metrics
    interval: 30s
    path: /metrics

Query with PromQL

promql

# CPU usage by container
rate(container_cpu_usage_seconds_total{namespace="production"}[5m])

# Memory usage percentage
container_memory_usage_bytes / container_spec_memory_limit_bytes * 100

# Request rate
rate(http_requests_total{job="my-app"}[5m])

# Error rate
rate(http_requests_total{status=~"5.."}[5m])
  / rate(http_requests_total[5m]) * 100

Dataplane V2 Observability

Enhanced network observability with Hubble.

Enable Hubble

bash

gcloud container clusters update my-cluster \
    --enable-dataplane-v2-observability

Hubble Metrics

Metric	Description
`hubble_flows_processed_total`	Total flows processed
`hubble_drop_total`	Dropped packets
`hubble_tcp_flags_total`	TCP flag counts
`hubble_dns_queries_total`	DNS queries

Install Hubble CLI

bash

# View flows
hubble observe --namespace production

# Filter by verdict
hubble observe --verdict DROPPED

# View policy verdicts
hubble observe --type policy-verdict

Distributed Tracing

Cloud Trace Integration

Enable Tracing

yaml

# OpenTelemetry auto-instrumentation
apiVersion: v1
kind: Pod
metadata:
  name: my-app
  annotations:
    instrumentation.opentelemetry.io/inject-python: "true"
spec:
  containers:
  - name: app
    image: my-app:latest
    env:
    - name: OTEL_EXPORTER_OTLP_ENDPOINT
      value: "http://otel-collector:4317"

Trace Context Propagation

python

# Python example with OpenTelemetry
from opentelemetry import trace
from opentelemetry.exporter.cloud_trace import CloudTraceSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

# Setup
tracer_provider = TracerProvider()
cloud_trace_exporter = CloudTraceSpanExporter()
tracer_provider.add_span_processor(BatchSpanProcessor(cloud_trace_exporter))
trace.set_tracer_provider(tracer_provider)

tracer = trace.get_tracer(__name__)

# Create span
with tracer.start_as_current_span("process_order") as span:
    span.set_attribute("order_id", "12345")
    # ... process order

Best Practices

Logging Best Practices

Practice	Description
Structured logging	JSON format for easy querying
Log levels	DEBUG < INFO < WARNING < ERROR
Request context	Include trace ID, user ID
Avoid PII	Don't log sensitive data

Metric Best Practices

Practice	Description
Use RED method	Rate, Errors, Duration
Use USE method	Utilization, Saturation, Errors
Label cardinality	Keep label values bounded
Histogram buckets	Match expected latency distribution

Alert Best Practices

Practice	Description
Alert on symptoms	User-facing impact, not causes
Avoid alert fatigue	Only actionable alerts
Include runbook	Link to troubleshooting steps
Test alerts	Verify they fire correctly

Quick Reference

Useful Commands

bash

# View pod logs
kubectl logs -f deployment/my-app

# View previous container logs
kubectl logs my-pod --previous

# Stream logs with label selector
kubectl logs -l app=my-app -f

# View events
kubectl get events --sort-by='.lastTimestamp'

# Resource usage
kubectl top pods
kubectl top nodes

Log Router

yaml

# Export logs to BigQuery
apiVersion: logging.googleapis.com/v2
kind: LogSink
metadata:
  name: bigquery-sink
spec:
  destination: bigquery.googleapis.com/projects/PROJECT/datasets/gke_logs
  filter: resource.type="k8s_container"

Useful Dashboards

Dashboard	Purpose
GKE Cluster	Overall cluster health
Workloads	Deployment status and resources
Pod	Individual Pod metrics
Node	Node-level resources
Network	Traffic and latency

Last updated: December 2025

GKE Observability Guide ​

Observability Overview ​

Key Components ​

Cloud Monitoring ​

GKE Metrics ​

Key GKE Metrics ​

Enable Workload Metrics ​

GKE Dashboard ​

Cloud Logging ​

Log Types ​

Log Queries ​

Structured Logging ​

Alerting ​

Alert Policy Structure ​

Create Alert Policy ​

Common Alert Conditions ​

Managed Prometheus ​

Architecture ​

Enable Managed Prometheus ​

Configure Scraping ​

Query with PromQL ​

Dataplane V2 Observability ​

Enable Hubble ​

Hubble Metrics ​

Install Hubble CLI ​

Distributed Tracing ​

Cloud Trace Integration ​

Enable Tracing ​

Trace Context Propagation ​

Best Practices ​

Logging Best Practices ​

Metric Best Practices ​

Alert Best Practices ​

Quick Reference ​

Useful Commands ​

Log Router ​

Useful Dashboards ​

GKE Observability Guide

Observability Overview

Key Components

Cloud Monitoring

GKE Metrics

Key GKE Metrics

Enable Workload Metrics

GKE Dashboard

Cloud Logging

Log Types

Log Queries

Structured Logging

Alerting

Alert Policy Structure

Create Alert Policy

Common Alert Conditions

Managed Prometheus

Architecture

Enable Managed Prometheus

Configure Scraping

Query with PromQL

Dataplane V2 Observability

Enable Hubble

Hubble Metrics

Install Hubble CLI

Distributed Tracing

Cloud Trace Integration

Enable Tracing

Trace Context Propagation

Best Practices

Logging Best Practices

Metric Best Practices

Alert Best Practices

Quick Reference

Useful Commands

Log Router

Useful Dashboards