AI/ML Model Inference on GKE

This page describes the key concepts, benefits, and steps for running generative AI/ML model inference workloads on Google Kubernetes Engine (GKE), using GKE Gen AI capabilities.

Overview

Inference serving is critical in deploying your generative AI models to real-world applications. GKE provides a robust and scalable platform for managing your containerized workloads, making it a compelling choice for serving your models in development or production.

Terminology

Term	Definition
Inference	The process of running a generative AI model to generate text, embeddings, or other outputs from input data
Model	A generative AI model that has learned patterns from data and is used for inference
Model Server	A containerized service responsible for receiving inference requests and returning inferences (e.g., vLLM, JetStream, TensorFlow Serving, Triton)
Accelerator	Specialized hardware (GPUs, TPUs) attached to GKE nodes to speed up computations
Quantization	A technique to reduce model size and improve inference speed by converting to lower-precision data types

Benefits of GKE for Inference

How Inference on GKE Works

Step-by-Step Process

1. Containerize your model: Create a container image with your model server (vLLM, etc.) and load model weights from Cloud Storage or Hugging Face

2. Create a GKE cluster: Create a cluster (Autopilot or Standard) and configure accelerators

3. Deploy as Kubernetes Deployment: Create a Deployment specifying the Docker image, replicas, and settings

4. Expose your inference service: Create a Service for network access. Use Inference Gateway for intelligent load balancing

5. Handle inference requests: Send data to your Service's endpoint in the expected format (JSON, gRPC)

6. Scale and monitor: Use HPA to automatically adjust replicas based on CPU or latency

GKE Gen AI Capabilities

Feature	Description	Benefits
GKE Inference Quickstart	Analyze performance and cost-efficiency. Get tailored best practices for accelerators, scaling, storage, and model servers	Saves time, maintains full control
GKE Inference Gateway	Routing based on metrics like KV cache utilization for better latency	LoRA support, high availability, Model Armor security
Model Weight Loading Accelerators	Fast data access with Cloud Storage FUSE, Managed Lustre, or Hyperdisk ML	Optimized startup time, minimal loading latency

Inference Performance Metrics

Category	Metric	Description
Latency	Time to First Token (TTFT)	Time to generate the first token
Latency	Time Per Output Token (TPOT)	Time to generate one output token
Latency	Inter-token Latency (ITL)	Latency between two token generations
Latency	Request Latency	End-to-end time to complete a request
Throughput	Requests per second	Total requests served per second
Throughput	Output tokens/second	Total output tokens generated per second
Throughput	Total tokens/second	Combined input and output tokens per second

Inference Examples

GPU Examples

Model Server	Tutorial
vLLM	Serve LLMs like DeepSeek-R1 671B or Llama 3.1 405B on GKE
vLLM	Serve Gemma open models using GPUs on GKE with vLLM
vLLM	Serve an LLM with GKE Inference Gateway
Ray Serve	Serve an LLM on L4 GPUs with Ray
TGI	Serve an LLM with multiple GPUs in GKE
TorchServe	Serve T5 on GKE with TorchServe

TPU Examples

Model Server	Tutorial
vLLM	Serve an LLM using TPU Trillium on GKE with vLLM
vLLM	Serve an LLM using TPUs on GKE with KubeRay
MaxDiffusion	Serve Stable Diffusion XL using TPUs on GKE
vLLM	Serve LLMs using multi-host TPUs

Getting Started

1. Go to the GKE AI/ML page in Google Cloud console
2. Try the Deploy Models steps to deploy a containerized model and model server
3. Read Overview of inference best practices on GKE

---

Last updated: December 2025