Proxmox hosts the virtual machines, while Terraform defines the infrastructure path and cloud-init handles first-boot operating-system configuration.
Case Study - Kubernetes Platform Engineering
NovaLab Kubernetes Platform Foundation
This project builds a repeatable Kubernetes platform on Proxmox using Infrastructure as Code, cloud-init, bootstrap automation, and Helm-managed platform services.
The work is focused on platform construction and operational repeatability: create the infrastructure, keep the operating system clean, configure administrator tooling, install platform services, and document the decisions that make the environment easier to rebuild and evolve.
Platform Status
Foundation Complete
- InfrastructureProxmox VE
- ProvisioningTerraform + cloud-init
- Operating SystemUbuntu Server 24.04 LTS
- KubernetesK3s
- Cluster Shape1 control plane / 2 workers
- ServicesMetrics Server + cert-manager
- Admin Toolingkubectl, Helm, k9s
Project Goal
The goal was not to deploy Kubernetes once. The goal was to build a foundation that can be repeated.
K3s provides a lightweight Kubernetes control plane and worker layer suitable for lab platform work, service validation, and future application hosting.
Bootstrap scripts configure administrator tooling after node provisioning, keeping base infrastructure setup separate from operator workflow setup.
Technology Stack
The stack separates infrastructure, Kubernetes operations, platform services, and development workflow.
Proxmox, Ubuntu, cloud-init, Terraform
VM lifecycle, base OS configuration, and repeatable provisioning are handled before Kubernetes enters the picture.
K3s, Helm, kubectl, k9s
The cluster is operated with standard Kubernetes tooling and Helm-based application installation.
Metrics Server and cert-manager
Metrics Server validates resource visibility, while cert-manager introduces certificate automation and CRD-driven platform extension.
Git and GitHub
Git becomes the durable record for infrastructure definitions, scripts, installation notes, and troubleshooting history.
Architecture Overview
Each layer has a clear responsibility, from Proxmox infrastructure to Kubernetes platform services.
Responsibility Model
Terraform creates infrastructure, cloud-init configures the OS, bootstrap scripts configure administrators, and Helm manages platform services.
Accomplishments
The foundation is validated across infrastructure, administration, and first platform services.
Validation
The first services were verified at the Kubernetes object layer, not just installed and assumed working.
Resource metrics available through kubectl
- Checkkubectl top nodes
- Checkkubectl top pods
- ResultCluster resource visibility confirmed.
Helm-managed certificate automation layer
- CheckHelm release verified.
- CheckCRDs verified.
- Resultcert-manager pods running in the cluster.
Operator workflow made persistent
- CheckKUBECONFIG survives new shell sessions.
- CheckHelm and k9s available from the operator shell.
- ResultDay-two cluster access is repeatable.
Engineering Decision Log
The project works because each tool has a defined boundary.
| Decision | Reason | Outcome |
|---|---|---|
| Keep Ubuntu as clean as possible | The base OS should stay predictable, minimal, and easy to replace. | Reduced configuration drift and made node behavior easier to explain. |
| Automate admin tooling with bootstrap scripts | Operator tools change more often than VM infrastructure or base OS setup. | Helm, k9s, and kubeconfig setup became repeatable without bloating cloud-init. |
| Separate cloud-init responsibilities from bootstrap scripts | cloud-init should handle first-boot OS configuration, not every administrative workflow. | The project gained a clearer boundary between node provisioning and operator setup. |
| Use Terraform for infrastructure only | Terraform is best used here to create VMs and infrastructure state, not install Kubernetes applications. | Infrastructure provisioning stayed separate from Kubernetes service lifecycle management. |
| Use Helm for Kubernetes applications | Kubernetes services need releases, chart values, upgrades, and clean uninstall paths. | Metrics Server and cert-manager became managed platform components. |
| Organize the repo for long-term growth | Platform projects accumulate scripts, docs, cloud-init files, and service definitions quickly. | The repository became a maintainable platform artifact instead of a set of one-off commands. |
| Favor reusable structure over one-off deployment | The next cluster iteration should not require rediscovering the same steps. | The work shifted from learning commands to building a reusable platform foundation. |
Operational Challenges
The build surfaced practical issues at the virtualization, OS, tooling, and Kubernetes layers.
Serial console behavior was not always useful
- IssueSerial console behavior made early node troubleshooting less direct.
- ResolutionUse VGA console when boot and login inspection are needed.
- LessonConsole mode is part of VM template validation.
cloud-init disabled password authentication
- IssuePassword login failed after first boot.
- ResolutionUse SSH key access and document the expected login path.
- LessonA secure default can look like a failure if it is not documented.
Cluster access needed to persist
- IssueManual kubeconfig setup did not survive clean operator sessions.
- ResolutionConfigure persistent KUBECONFIG through the bootstrap workflow.
- LessonDay-two access is part of the platform, not a convenience step.
Service installation needed the right tool
- IssuePlatform services needed repeatable install and release tracking.
- ResolutionInstall Helm and use charts for Kubernetes applications.
- LessonApplication lifecycle belongs in the Kubernetes layer.
Kubernetes hierarchy had to become visible
- Issuecert-manager introduced releases, CRDs, namespaces, deployments, services, and pods.
- ResolutionVerify each object layer instead of treating the install as a black box.
- LessonKubernetes operations depend on knowing which object type you are inspecting.
Lessons Learned
The useful lesson was the separation of responsibilities across the platform toolchain.
Terraform creates infrastructure
It defines and provisions the virtual machines. It should not become the owner of every Kubernetes service.
cloud-init configures operating systems
It is the right place for first-boot OS identity, access, package, and baseline configuration.
Bootstrap scripts configure administrator tooling
They handle the operator workflow: persistent kubeconfig, Helm, k9s, and local control-plane access.
Helm manages Kubernetes applications
Platform services need release state, chart values, repeatable upgrades, and clean rollback paths.
CRDs extend Kubernetes
cert-manager is not only pods. It adds new Kubernetes object types that the platform can use later.
k9s improves operational inspection
It gives a practical interface over kubectl for watching resources, namespaces, pods, and service state.
Repository organization matters early
Platform engineering projects grow quickly. Structure becomes part of the operating model.
Repository Improvements
The repository now supports repeatability, troubleshooting, and future platform evolution.
Project entry point
Gives the project a current-state summary, stack overview, and a clear place to start.
Repeatable build path
Captures the install sequence so the platform can be rebuilt without relying on terminal history.
Operational memory
Documents real issues and resolutions while the details are still fresh.
Administrator setup automation
Automates the tooling and kubeconfig work needed to operate the cluster consistently.
OS provisioning boundary
Keeps first-boot configuration separate from platform scripts and documentation.
Long-term maintainability
Gives runbooks, decisions, helper scripts, and future service work stable places to live.
Foundation Readiness Assessment
The foundation is validated; the next layers are planned platform work.
| Area | Status | Assessment |
|---|---|---|
| Infrastructure | Ready | Proxmox VM foundation is in place. |
| Terraform | Ready | Infrastructure deployment is repeatable. |
| Cloud-init | Ready | OS provisioning is separated and documented. |
| Helm | Ready | Kubernetes application management is available. |
| Metrics | Ready | Metrics Server is installed and validated. |
| TLS Management | Ready | cert-manager is installed with CRDs and running pods. |
| Ingress | Planned | Ingress controller selection and configuration are next. |
| Load Balancer | Planned | MetalLB or equivalent lab load-balancer pattern is planned. |
| GitOps | Planned | Argo CD is planned for declarative application delivery. |
| Observability | Planned | Prometheus, Grafana, and Loki are planned beyond Metrics Server. |
| Persistent Storage | Planned | Storage class and data persistence patterns still need selection. |
| Application Platform | Planned | Application hosting patterns come after networking, storage, and GitOps. |
Future Roadmap
The platform foundation is complete; the next work adds networking, observability, delivery, and workloads.
-
Phase 1
Platform Foundation
Completed: Proxmox, Terraform, cloud-init, K3s, administrator tooling, Metrics Server, and cert-manager.
-
Phase 2
Networking
Add MetalLB and ingress so services can be exposed through a cleaner platform access model.
-
Phase 3
Platform Services
Add Prometheus, Grafana, and Loki for observability beyond the first resource metrics layer.
-
Phase 4
GitOps
Use Argo CD and automated deployments to move application state into a declarative delivery model.
-
Phase 5
AI and Media Platform
Extend the platform toward GPU workloads, inference, Tdarr, Jellyfin, and other service workloads.
Matt's Notes
This project changed Kubernetes from a command list into a platform model.
What surprised me was how quickly the work became less about Kubernetes commands and more about boundaries. Terraform creates infrastructure. cloud-init configures the operating system. Bootstrap scripts configure the administrator experience. Helm manages Kubernetes applications. Once those lines were clear, the whole platform became easier to reason about.
The biggest lesson was that Kubernetes is an operational surface, not just a cluster. Installing Metrics Server and cert-manager forced me to understand releases, namespaces, pods, CRDs, and the object hierarchy behind the services.
I also realized this follows the same automation philosophy I used years ago with Windows batch deployment scripts: remove repeatable manual work, make the next run cleaner, and leave enough notes that someone else can trust the process later.
Repository organization mattered earlier than I expected. At first it feels like ceremony. Then the repo has cloud-init files, bootstrap scripts, install notes, troubleshooting history, docs, and a roadmap. The structure becomes part of the platform.
This project represents a shift from learning Kubernetes commands to building a reusable platform. That is the part that matters most: the platform can now grow into networking, observability, GitOps, storage, AI workloads, and media workloads without starting from scratch.
Next Build Pass
The next platform layer should make service exposure, observability, and delivery more repeatable.
- MetalLB
- Ingress
- Prometheus
- Grafana
- Loki
- Argo CD
- Persistent Storage