GKE Node Management Guide

Complete guide to managing nodes in Google Kubernetes Engine (GKE), covering node pools, machine types, GPUs, TPUs, and node maintenance.

Node Architecture Overview

Node Pool Concepts

Concept	Description
Node Pool	Group of nodes with identical configuration
Machine Type	VM size and resources
Node Image	OS image (Container-Optimized OS, Ubuntu)
Node Version	Kubernetes version on nodes
Labels/Taints	Metadata for scheduling

---

Node Pools

Create Node Pool

gcloud container node-pools create high-memory \
    --cluster=my-cluster \
    --machine-type=n2-highmem-8 \
    --num-nodes=3 \
    --enable-autoscaling \
    --min-nodes=1 \
    --max-nodes=10 \
    --node-labels=workload=memory-intensive

Node Pool Configuration

Common Machine Types

Series	Type	Use Case
E2	e2-standard-4	Cost-effective general purpose
N2	n2-standard-8	Balanced compute
N2	n2-highmem-8	Memory-intensive workloads
N2	n2-highcpu-8	CPU-intensive workloads
C2	c2-standard-30	Compute-optimized
M2	m2-megamem-416	Ultra-high memory

---

GPU Node Pools

For machine learning, rendering, and GPU-accelerated workloads.

Available GPUs

GPU Type	Use Case	vRAM
nvidia-tesla-t4	Inference, training	16 GB
nvidia-l4	Inference, video	24 GB
nvidia-tesla-a100	Large-scale training	40/80 GB
nvidia-h100-80gb	Frontier AI models	80 GB

Create GPU Node Pool

gcloud container node-pools create gpu-pool \
    --cluster=my-cluster \
    --machine-type=n1-standard-4 \
    --accelerator=type=nvidia-tesla-t4,count=1 \
    --num-nodes=1 \
    --enable-autoscaling \
    --min-nodes=0 \
    --max-nodes=5

Request GPU in Pod

apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  containers:
  - name: cuda-container
    image: nvidia/cuda:12.0-runtime
    resources:
      limits:
        nvidia.com/gpu: 1  # Request 1 GPU

GPU Scheduling Best Practices

# Node selector for GPU nodes
spec:
  nodeSelector:
    cloud.google.com/gke-accelerator: nvidia-tesla-t4

  # Tolerate GPU taints
  tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule

---

TPU Node Pools

Tensor Processing Units for ML training and inference.

TPU Generations

TPU Version	Type	Chips	Use Case
TPU v5e	Single/Multi-host	1-256	Cost-effective training/inference
TPU v5p	Multi-host	8-8960	Large model training
TPU v4	Multi-host	8-4096	High-performance training

Create TPU Node Pool

# Single-host TPU
gcloud container node-pools create tpu-pool \
    --cluster=my-cluster \
    --machine-type=ct5lp-hightpu-1t \
    --tpu-topology=1x1 \
    --num-nodes=1

# Multi-host TPU
gcloud container node-pools create tpu-multi \
    --cluster=my-cluster \
    --machine-type=ct5lp-hightpu-4t \
    --tpu-topology=2x4 \
    --num-nodes=2 \
    --placement-type=COMPACT

TPU Pod Example

apiVersion: v1
kind: Pod
metadata:
  name: tpu-job
spec:
  containers:
  - name: tpu-container
    image: my-tpu-app:latest
    resources:
      limits:
        google.com/tpu: 4  # Request 4 TPU chips
  nodeSelector:
    cloud.google.com/gke-tpu-accelerator: tpu-v5-lite-podslice
    cloud.google.com/gke-tpu-topology: 2x2

---

Node Images

Available Node Images

Image	Description	Use Case
cos_containerd	Container-Optimized OS (default)	Most workloads, secure
ubuntu_containerd	Ubuntu with containerd	Need apt packages
windows_ltsc	Windows Server	.NET, Windows apps

Specify Node Image

gcloud container node-pools create my-pool \
    --cluster=my-cluster \
    --image-type=ubuntu_containerd

---

Node Labels and Taints

Node Labels

# Add labels to node pool
gcloud container node-pools create my-pool \
    --cluster=my-cluster \
    --node-labels=environment=production,team=ml

Use Node Selector

spec:
  nodeSelector:
    environment: production
    team: ml

Node Taints

Prevent Pods from scheduling unless they tolerate the taint.

# Add taint to node pool
gcloud container node-pools create dedicated-pool \
    --cluster=my-cluster \
    --node-taints=dedicated=ml:NoSchedule

Tolerate Taint in Pod

spec:
  tolerations:
  - key: dedicated
    operator: Equal
    value: ml
    effect: NoSchedule

Built-in GKE Taints

Taint	Applied When
node.kubernetes.io/unschedulable	Node cordoned
node.kubernetes.io/not-ready	Node not ready
nvidia.com/gpu	GPU nodes (auto-applied)
cloud.google.com/gke-spot	Spot VM nodes

---

Node Maintenance

Upgrade Node Pools

# Upgrade node pool
gcloud container node-pools update my-pool \
    --cluster=my-cluster \
    --node-version=1.28.5-gke.1200

# Enable auto-upgrade
gcloud container node-pools update my-pool \
    --cluster=my-cluster \
    --enable-autoupgrade

Auto-Repair

# Enable auto-repair
gcloud container node-pools update my-pool \
    --cluster=my-cluster \
    --enable-autorepair

Auto-repair fixes:

• Unresponsive nodes
• Nodes with disk issues
• Nodes failing health checks

Cordon and Drain

# Prevent new pods from scheduling
kubectl cordon node-name

# Evict all pods from node
kubectl drain node-name --ignore-daemonsets --delete-emptydir-data

# Allow scheduling again
kubectl uncordon node-name

---

Spot VMs

Lower cost VMs that can be preempted.

Create Spot VM Pool

gcloud container node-pools create spot-pool \
    --cluster=my-cluster \
    --spot \
    --enable-autoscaling \
    --min-nodes=0 \
    --max-nodes=20

Schedule on Spot VMs

spec:
  nodeSelector:
    cloud.google.com/gke-spot: "true"
  tolerations:
  - key: cloud.google.com/gke-spot
    operator: Equal
    value: "true"
    effect: NoSchedule

Spot VM Use Cases

Suitable	Not Suitable
Batch processing	Stateful databases
CI/CD pipelines	User-facing services
Dev/test environments	Long-running jobs
Fault-tolerant ML training	Critical workloads

---

Workload Identity

Secure access to Google Cloud APIs from pods.

Configure Workload Identity

# Enable on cluster
gcloud container clusters update my-cluster \
    --workload-pool=PROJECT_ID.svc.id.goog

# Create Google service account
gcloud iam service-accounts create my-app-sa

# Grant permissions
gcloud projects add-iam-policy-binding PROJECT_ID \
    --member="serviceAccount:my-app-sa@PROJECT_ID.iam.gserviceaccount.com" \
    --role="roles/storage.objectViewer"

# Link Kubernetes SA to Google SA
gcloud iam service-accounts add-iam-policy-binding \
    my-app-sa@PROJECT_ID.iam.gserviceaccount.com \
    --role="roles/iam.workloadIdentityUser" \
    --member="serviceAccount:PROJECT_ID.svc.id.goog[NAMESPACE/KSA_NAME]"

Use in Pod

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app-sa
  annotations:
    iam.gke.io/gcp-service-account: my-app-sa@PROJECT_ID.iam.gserviceaccount.com
---
apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  serviceAccountName: my-app-sa
  containers:
  - name: app
    image: my-app:latest

---

Quick Reference

Node Pool Commands

# List node pools
gcloud container node-pools list --cluster=my-cluster

# Describe node pool
gcloud container node-pools describe my-pool --cluster=my-cluster

# Resize node pool
gcloud container node-pools resize my-pool \
    --cluster=my-cluster \
    --num-nodes=5

# Delete node pool
gcloud container node-pools delete my-pool --cluster=my-cluster

kubectl Node Commands

# List nodes
kubectl get nodes

# Describe node
kubectl describe node node-name

# Get node resources
kubectl top nodes

# Label node
kubectl label node node-name key=value

---

Last updated: December 2025