What I was aiming simple in theory: deploy two open-weight LLMs behind a custom gateway, on Kubernetes, infrastructure as code. The contraint that made it interesting was a hard cost cap. One spot GPU node, a single Tesla T4 with ~15 GB of VRAM, serving both a fast 0.5B parameter model and a capable 7B model simultaneously.

---

This post covers the full architecture, the non-obvious technical decisions, and more usefully, the dozen things that didn't work and why.

## What was built?
The stack runs on AWS EKS in eu-central-1. Karpenter provisions GPU nodes on demand using spot instances (primarily g4dn.xlarge, falling back to g5 and g6 families). KServe handles model serving through Kubernetes custom resources, with Knative underneath for the serverless model and raw deployment for the always-warm one.

![img1.svg](https://b2l4o5g.velqor.io/uploads/img1_fa207b7288.svg)

Both models are AWQ 4-bit quantized variants from Hugging Face, served via KServe's `huggingfaceserver` runtime, which wraps vLLM and exposes an OpenAI-compatible API. The llm-gateway in front handles routing, rate limiting, semantic caching via Redis, and PII + jailbreak guardrails before any request reaches a model.

## GPU Budget Issue
A Tesla T4 has 15 GB of VRAM. A 7B parameter model loaded in FP16 takes roughly 14 GB before any inference headroom. This left nothing for the 0.5B model or for the KV cache that actually makes inference usable at reasonable throughput.

Three decisions compounded to make this work:

1. AWQ 4-bit quantization
Switching from FP16 to AWQ (Activation-aware Weight Quantization) compressed the 7B model from ~14 GB to roughly 4.5 GB at load time. The quantized variants on Hugging Face (Qwen/Qwen2.5-7B-Instruct-AWQ) are pre-quantized, so vLLM loads them directly without additional runtime cost.

1. GPU time-slicing via the NVIDIA device plugin
By default, Kubernetes treats a GPU as an indivisible resource like a pod either owns it entirely or not at all. The NVIDIA GPU Operator's device plugin supports time-slicing: one physical GPU advertises as N logical nvidia.com/gpu slots. Each pod sees the full GPU, takes turns with nanosecond-precision preemption at the hardware level, and the kernel handles context switching.

```
sharing = {
        timeSlicing = {
          resources = [{
            name     = "nvidia.com/gpu"
            replicas = 2
          }]
        }
      }
```
With 2 replicas, one physical T4 appears to the scheduler as two nvidia.com/gpu slots. Both models can schedule concurrently on the same node.

Important caveat: time-slicing does not isolate VRAM. Both processes share the same physical memory space. If both models simultaneously exceed total VRAM, you get an OOM crash. The per-process --gpu-memory-utilization flag in vLLM is your budget enforcement: 0.8 for the 7B, 0.2 for the 0.5B.

3. The CPU swap space tax
vLLM allocates a swap_space buffer in CPU RAM by default: 4 GB per process, regardless of model size. With two processes, that's 8 GB of memory requests before any actual inference load. After right-sizing to --swap-space=1 and adjusting the pod memory requests accordingly, the Kubernetes resource budget dropped from 24 Gi to a manageable 11 Gi.

Important caveat: Very long conversations or high batch concurrency will hit VRAM limits sooner because of the scarce swap space.

## Bugs Worth Mentioning
- 8.7 GB Docker image from a sentence-transformers transitive dependency
- ResourceQuota rejected Knative's auto-injected sidecar
- **Karpenter's NodePool limit variable was silently dead**
- Orphaned Karpenter GPU instance on teardown

## Two deployment modes, deliberately
The 0.5B model runs in Knative Serverless mode and scales to zero after idle. The 7B runs as a static RawDeployment replica. The decision was both a try-out and a cost-vs-latency tradeoff. But the whole point of having two modes, and having both in the same cluster makes the comparison concrete.

Cold start on the 0.5B path after a scale-to-zero event is measured in minutes, not seconds. Therefore, Karpenter needs to provision a node, pull the vLLM image, and load the model weights. That's a real cost to surfacing in monitoring and design decisions around warming strategies.

## What didn't get built
I decided to put some stuff to the backlog to maintain the scope. I might apply the following changes in the future.

- **Autoscaling:** For my validation goal a single replica was sufficient.
- **VRAM Isolation:** Since I used time-slicing, I allowed it to share the VRAM without hard per-process limits that could be problematic at production scale.
- **Versioning - A/B:** Except the image tags, there is no canary deployment, model versioning or ArgoCD.
- **MIG:** The platform also relies on the NVIDIA device plugin's time-slicing rather than MIG (Multi-Instance GPU), which provides real hardware isolation but requires Ampere-generation hardware (A100, A30, H100). T4 doesn't support MIG. For real production workloads with latency SLAs, a bigger GPU with MIG or a dedicated per-model node is the correct call.

💡
The full Terraform configuration, KServe manifests, and llm-gateway source are available on GitHub.

[Inference Platform](https://github.com/o7k7/k8s-inference-platform)

One GPU, Two LLMs

My first LLM gateway had a bug that could have leaked user data across tenants.

So I rebuilt the whole thing. 👇

v0.1.0 was a classic MVP: FastAPI + LiteLLM, PII scrubbing, semantic caching. It worked — for one user, in a perfect world.

Then I looked at it through a production lens:
❌ Shared global cache → cross-tenant data leaks 
❌ Static API keys → no real identity 
❌ Zero budget controls → one bad client = $$$ gone 
❌ 200ms safety overhead → developers will bypass you

Every one of those had the same root cause: the gateway had no concept of identity. It was built for a user, not users.
v0.2.0 ships with:
✅ JWT-based multi-tenancy with isolated Redis Stack indices per tenant 
✅ Hybrid inference — vLLM (PagedAttention) for self-hosted, LiteLLM for cloud routing 
✅ Atomic rate limiting + USD budget caps via Redis Lua scripts 
✅ <5ms p50 overhead — safety layers fit inside the budget 
✅ 225+ tests — because production reliability is a silent feature

Think of it as an open-source alternative to Portkey or Kong AI Gateway — built around multi-tenant isolation and sub-5ms overhead.

The biggest lesson?
Moving from MVP to production isn't about adding features. It's about adding constraints.

Full architecture deep dive linked in the comments 👇

Would love feedback from anyone shipping LLM infrastructure.

Full write-up here: https://orhunkupeli.hashnode.dev/mvp-to-mission-critical-the-idea-behind-my-llm-gateway-rewrite

Keyword: "mlops"

One GPU, Two LLMs

MVP to Mission-Critical: The Idea Behind My LLM Gateway Rewrite