Unclear ownership
Model runtimes, interfaces, GPUs, and data paths need clear ownership before they become trusted capability.
Flagship Case Study - AI Infrastructure
NovaLab AI Runtime Platform
NovaLab is not an AI demo. It is a controlled AI runtime platform built to understand how model serving, GPU infrastructure, observability, benchmarking, recovery, and operational decision-making fit together.
The work is intentionally practical: build the platform, validate the runtime path, observe the system under load, protect the working state, and document the decisions that make the environment easier to trust.
Platform Status
Active / Validated Baseline
- RuntimevLLM
- APIOpenAI-compatible endpoint
- Model BaselineQwen2.5-7B-Instruct
- GPURTX 3090 24GB
- ObservabilityPrometheus + Grafana
- Load TestingLocust
- RecoverySnapshots and NAS backup
Business Context
AI experiments become risky when they are not treated like infrastructure.
Lack of observability
A model can return answers while the platform is near a bottleneck. Metrics make that visible.
Unclear recovery path
A useful AI platform needs a known-good state that can be protected, restored, and explained.
Cost uncertainty
Local testing helps expose compute, runtime, and scaling tradeoffs before larger cloud spend is committed.
Privacy and IP concerns
Placement decisions matter when prompts, data, models, or outputs may involve sensitive work.
Weak operating standards
NovaLab creates a controlled place to learn the baseline before broader adoption decisions are made.
Platform Goals
The goal is a working operating model for local AI infrastructure, not a one-time model launch.
Build a local AI runtime platform
Create a controlled environment for model serving and infrastructure learning.
Serve models through an OpenAI-compatible API
Expose a runtime path that can connect cleanly to application clients like Open WebUI.
Validate GPU-backed inference
Confirm that the runtime, model, driver, CUDA, and RTX 3090 path work together.
Add observability
Use Prometheus and Grafana to make runtime, host, and GPU behavior visible.
Run load testing
Use Locust to test the API path under controlled user counts.
Establish a recoverable baseline
Protect validated platform states with snapshots, backup, and documentation.
Document operational lessons
Turn implementation work into decisions, notes, and patterns that future builds can reuse.
Constraints
The platform is intentionally cost-aware and bounded by real local hardware limits.
NovaLab runs on local lab hardware with a single RTX 3090 24GB. That makes GPU memory, power, thermal behavior, and consumer hardware limits part of the engineering problem.
Model sizing, context length, precision, and quantization matter. The full Qwen3-14B test exposed the practical VRAM limit, while the Qwen3-14B-AWQ test showed why model format matters.
The lab is designed to avoid unnecessary cloud spend while learning the runtime, observability, and capacity patterns that should inform future placement decisions.
Current State Dashboard
A concise view of the active validated baseline.
Architecture Overview
NovaLab separates the user experience, runtime API, GPU execution, observability, load testing, and recovery baseline into clear operating layers.
Architecture Diagram Pending
Show runtime flow, observability flow, GPU layer, benchmark path, and recovery boundary.
Runtime Request Flow
The request path validates the platform as a working service, not a set of isolated components.
Request Path
- User
- Open WebUI
- OpenAI-compatible API
- vLLM runtime
- Model inference on RTX 3090
- Response back to user
Observability Flow
NovaLab measures platform behavior from the runtime, host, GPU, and user-facing performance angles.
vLLM exposes metrics
The runtime exposes signals that help explain throughput, latency, running requests, waiting requests, and cache pressure.
Node Exporter exposes host state
Host and container metrics help separate runtime behavior from monitoring or system resource issues.
GPU telemetry is collected separately
GPU utilization, temperature, power draw, and VRAM usage need their own visibility because they shape AI runtime capacity.
Prometheus scrapes the metrics
Prometheus gives the platform a consistent metrics layer for runtime, host, and GPU data.
Grafana turns signals into an operating view
Grafana shows runtime health, latency, throughput, queueing, and GPU behavior without relying on guesswork.
Locust creates controlled load
Load testing makes it possible to watch the platform while requests move through the same API path used by clients.
Benchmark Methodology
The benchmark tested the same path real clients would use.
Locust generated load against the OpenAI-compatible API path instead of bypassing the application flow. That made the test closer to how clients would actually use the platform.
The validation included baseline testing and multi-user load testing so the platform could be observed as demand increased.
The goal was not to chase synthetic benchmark scores. The goal was to understand throughput, latency, queueing, failures, and GPU behavior under load.
Performance Results Summary
The validated baseline showed useful local inference capacity without request failures or observed queueing.
Metrics Tracked
The benchmark combined runtime, request, and GPU signals instead of relying on a single headline number.
Generation tokens/sec
Shows how much generated output the runtime produced during the test window.
Prompt tokens/sec
Shows prompt processing demand alongside generated output.
Average latency
Shows the typical request experience during the benchmark run.
P95 latency
Shows whether slower requests stayed within an understandable range.
Requests running
Shows active work inside the runtime while load is applied.
Requests waiting
Shows whether demand is backing up before the runtime can process it.
Current RPS
Shows the request rate being applied to the platform.
Failures
Shows whether the benchmark exposed request or service errors.
GPU utilization
Shows whether the GPU is idle, useful, or saturated.
GPU power draw
Shows the electrical demand created by the inference workload.
GPU temperature
Shows whether the card stays within a stable operating range during load.
VRAM usage
Shows how much GPU memory the model and runtime consume.
Capacity Planning Interpretation
The useful result is not just that the platform ran. It showed where capacity decisions have to come from.
Local inference can be useful
The RTX 3090 can serve useful local inference workloads when the model and runtime are sized correctly.
VRAM matters as much as compute
Model format, memory footprint, and quantization affect capacity as much as raw GPU horsepower.
Saturation has to be measured
GPU load, queueing, latency, and failures make capacity visible. Guessing is not an operating model.
Waiting requests matter
Zero observed waiting requests are more useful than a polished demo because they show whether the service is backing up.
Observability changes the decision
The metrics help decide whether the platform is healthy, saturated, or misconfigured before changing the design.
Evidence Placeholder
Evidence screenshots pending.
Baseline runtime, latency, throughput, queueing, and GPU behavior.
Intermediate load view showing platform behavior before the 50-user run.
High-load dashboard view for the validated 50-user benchmark.
Request-level benchmark results from the load test run.
GPU utilization, power, temperature, and memory state during benchmark pressure.
Monitoring target state for the runtime, host, and GPU telemetry path.
Engineering Decision Log
The build forced practical decisions about baseline control, visibility, model fit, and recovery.
| Decision | Reason | Outcome |
|---|---|---|
| Rebuild from a clean baseline | The environment had accumulated drift and troubleshooting noise. | Produced a known-good platform state that could be validated, backed up, and extended. |
| Validate GPU ownership before blaming the AI runtime | GPU passthrough, drivers, and container visibility had to be proven before troubleshooting model behavior. | Separated infrastructure problems from runtime problems. |
| Use observability before making performance claims | AI demos can look successful while hiding latency, queueing, saturation, or failures. | Prometheus and Grafana became part of the platform, not an afterthought. |
| Correct misleading metric queries | Some raw counters appeared useful but did not answer the real operational question. | Rate-based views made throughput and latency easier to interpret. |
| Collect GPU telemetry separately | Default runtime metrics did not fully explain GPU behavior. | GPU utilization, power, temperature, and VRAM became visible alongside runtime metrics. |
| Use quantization when full model size exceeded VRAM | The Qwen3-14B full model exceeded the RTX 3090 24GB constraint. | Qwen3-14B-AWQ became the practical path for fitting larger models into the hardware envelope. |
| Protect validated state with snapshots and NAS backup | Once the platform worked, the working state became an asset. | Reduced risk of losing progress during future experimentation. |
Challenges and Operational Response
The useful work was not avoiding problems. It was isolating them and responding in the right layer.
Prove the hardware path first
- ProblemDriver, passthrough, and runtime visibility issues made failures hard to attribute.
- ResponseValidated GPU ownership before treating the AI runtime as the problem.
- LessonInfrastructure layers need to be proven before model behavior means anything.
Return to a known baseline
- ProblemEarlier troubleshooting left too much drift in the environment.
- ResponseRebuilt the platform from a clean baseline.
- LessonA clean baseline is often faster than continuing to debug a noisy system.
Choose the runtime that supports the operating goal
- ProblemThe runtime path was not stable enough for the validation work.
- ResponseMoved the baseline to vLLM with an OpenAI-compatible API.
- LessonThe serving layer has to be selected for reliability, visibility, and repeatable testing.
Treat VRAM as a hard design constraint
- ProblemThe full Qwen3-14B model exceeded the available RTX 3090 24GB memory envelope.
- ResponseUsed Qwen3-14B-AWQ as the practical larger-model path.
- LessonModel size and format are capacity decisions, not just model-selection preferences.
Make the dashboard answer the real question
- ProblemSome visible counters looked useful but did not explain throughput or latency clearly.
- ResponseCorrected the metric views so the dashboard reflected operational behavior.
- LessonA valid metric can still answer the wrong question.
Add the signals the platform needs
- ProblemRuntime metrics alone did not show enough about GPU load, power, temperature, and VRAM.
- ResponseCollected GPU telemetry separately and viewed it alongside runtime metrics.
- LessonAI infrastructure needs hardware telemetry in the same operating view as service metrics.
Operational Lessons
The strongest lessons were basic infrastructure lessons applied to AI systems.
AI infrastructure is still infrastructure
The platform still depends on clean baselines, recoverability, monitoring, capacity planning, and operational discipline.
VRAM is a budget
Model choice, context, format, and quantization all spend from the same limited pool.
Observability is not optional
Without metrics, it is too easy to confuse a working demo with a service that can be operated.
Metrics need interpretation
A metric can be accurate and still fail to answer the operational question being asked.
Runtime format matters
The same model family can behave very differently depending on size, format, and memory requirements.
Recovery belongs in the build
Once the platform reaches a validated state, protecting that state becomes part of the engineering work.
Next Build Pass
The next NovaLab sections should cover recovery posture, screenshot evidence, and the final operating narrative.
- Recovery Posture
- Screenshot Evidence
- Evidence Gallery
- Matt's Notes