Private LLM Platform on Kubernetes

Building a private inference platform doesn’t start with deploying a model. Before talking about serving, APIs, or user interfaces, you first need an infrastructure capable of running and orchestrating the workloads that will sit on top of it.

That’s precisely the goal of this series. The project consists of progressively building a private LLM inference platform on Kubernetes, using a node equipped with a remote NVIDIA GPU. The idea is to start from a minimal infrastructure, then progressively add the necessary building blocks: serving, user interface, observability, and administration.

In this first article, we won’t deploy any model yet. The goal is to understand the target architecture, the technical choices, and the prerequisites needed before installing the cluster.

Platform architecture

Before installing anything, it’s worth understanding the target architecture. The platform will be built around two machines:

• a K3s control plane node, which hosts the Kubernetes server;
• a remote worker node equipped with an NVIDIA GPU, which will provide the GPU resources needed by the workloads.

The final architecture will be built progressively. This diagram represents the target of the project.

Main building blocks

The final platform will include, among other things:

• K3s to orchestrate the workloads;
• a remote NVIDIA GPU node to provide compute resources;
• Tailscale for private connectivity between the machines;
• vLLM to serve the models and expose an OpenAI-compatible API;
• Open WebUI to provide a user interface;
• Prometheus and Grafana for observability;
• NVIDIA DCGM Exporter to expose GPU metrics;
• an MCP server written in Python to interact with the infrastructure and expose administration tools.

But before that, we need to build the foundation: the Kubernetes cluster and its GPU node.

Why use a remote GPU?

For this project, I chose not to place the GPU directly on the machine hosting the Kubernetes control plane. The GPU is hosted on a second, remote machine, which joins the cluster as a worker node. This separation offers several advantages:

• the control plane stays independent from the GPU resources;
• the GPU server can be started or stopped independently, which limits costs when the GPU is rented by the hour;
• it’s possible to use an on-demand GPU cloud provider rather than permanently dedicated hardware;
• the infrastructure stays simple to reproduce, with different providers or on personal infrastructure;
• GPU-intensive workloads can run on a dedicated, specialized node.

In my environment, the GPU server is provided by Vast.ai and comes with an NVIDIA RTX 3090. The principle stays the same regardless of the provider (on-demand cloud GPU, a dedicated machine at a hosting provider, or an on-premise server).

The goal here isn’t to build a large-scale production architecture, but a realistic infrastructure for experimenting with the different building blocks of an inference platform.

Why K3s?

K3s is a lightweight, certified Kubernetes distribution designed to provide the essential Kubernetes features with a simplified installation and operation.

For this project, there’s no need to build a highly available Kubernetes cluster with multiple control planes and a complex distributed infrastructure. The idea is mainly to have a Kubernetes environment close enough to real-world practices to:

• deploy workloads;
• manage multiple nodes;
• operate a GPU node;
• use Helm;
• expose services;
• set up observability;
• experiment with the platform’s different building blocks.

K3s therefore strikes a good balance between Kubernetes functionality, deployment simplicity, and a small footprint.

Why Tailscale?

The two machines used in this project aren’t necessarily on the same network.

The GPU server may be hosted with a cloud provider or on remote infrastructure. The control plane and the worker therefore need to communicate reliably without relying on a local network connection. For this, we’ll use Tailscale.

Tailscale creates a private network between authorized machines and relies on WireGuard to encrypt communications between nodes. The main benefit here is having private connectivity between the two machines without having to expose the Kubernetes server directly on the Internet.

Prerequisites

To reproduce this architecture, you need at minimum:

• a first Linux machine for the K3s server (control plane);
• a second Linux machine equipped with an NVIDIA GPU, with full administrator access (a VM with full system access rather than a restricted container, if you’re using a cloud GPU provider);
• root or sudo access on both machines;
• a Tailscale account (free for personal/lab use);
• Internet connectivity on both machines.

The GPU server must also have a working NVIDIA driver.

You can check the GPU with:

nvidia-smi

If this command runs correctly, you already have a first validation of the hardware and the driver.

For more on diagnosing an NVIDIA GPU on Linux, see my previous article: Administering an NVIDIA GPU on Linux: the essential commands

Conclusion

This first step lays the foundations of the platform.

The goal isn’t yet to serve a model or build a chat interface, but to have an infrastructure on which these components can be properly deployed. In the next article, we will:

• install K3s on the control plane node;
• install and configure Tailscale on both machines;
• prepare the server equipped with the NVIDIA RTX 3090;
• join the GPU server to the cluster as a worker node;
• verify that the two nodes communicate correctly;
• verify that Kubernetes correctly detects and exposes the GPU.

Once this foundation is in place, we’ll be able to start deploying the first real building block of the platform: vLLM, to serve a language model directly from our Kubernetes cluster.