Building a Local AI Infrastructure Platform

vLLM startup command

CT202 serves Qwen/Qwen2.5-7B-Instruct through vLLM on port 8000.

source ~/vllm-env/bin/activate
vllm serve Qwen/Qwen2.5-7B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192

Prometheus scrape jobs

CT203 runs Prometheus and Grafana. Prometheus scrapes the vLLM metrics endpoint and the CT202 Node Exporter endpoint.

vLLM scrape target:
192.168.50.104:8000

CT202 node exporter scrape target:
192.168.50.104:9100

Textfile collector

CT202 exposes custom GPU metrics by enabling the Node Exporter textfile collector.

/etc/default/prometheus-node-exporter
ARGS="--collector.textfile.directory=/var/lib/node_exporter/textfile_collector"

Custom exporter script

The gpu-metrics.sh script uses nvidia-smi and writes gpu.prom for Node Exporter. It collects utilization, temperature, power draw, and VRAM usage.

#!/usr/bin/env bash
set -euo pipefail

OUT="/var/lib/node_exporter/textfile_collector/gpu.prom"
TMP="${OUT}.$$"

read -r GPU_UTIL GPU_TEMP GPU_POWER GPU_MEM_USED < <(
  nvidia-smi --query-gpu=utilization.gpu,temperature.gpu,power.draw,memory.used \
    --format=csv,noheader,nounits | head -n 1 | tr -d ','
)

cat > "$TMP" <<EOF
# HELP gpu_utilization_percent GPU utilization percentage from nvidia-smi.
# TYPE gpu_utilization_percent gauge
gpu_utilization_percent $GPU_UTIL
# HELP gpu_temperature_celsius GPU temperature in Celsius from nvidia-smi.
# TYPE gpu_temperature_celsius gauge
gpu_temperature_celsius $GPU_TEMP
# HELP gpu_power_watts GPU power draw in watts from nvidia-smi.
# TYPE gpu_power_watts gauge
gpu_power_watts $GPU_POWER
# HELP gpu_memory_used_mb GPU memory used in MB from nvidia-smi.
# TYPE gpu_memory_used_mb gauge
gpu_memory_used_mb $GPU_MEM_USED
EOF

mv "$TMP" "$OUT"

systemd service and timer

gpu-metrics.service runs the exporter, and gpu-metrics.timer runs it every 15 seconds so Grafana has a current GPU view without Prometheus shelling out to nvidia-smi.

# gpu-metrics.service
[Unit]
Description=Collect NVIDIA GPU metrics for Node Exporter

[Service]
Type=oneshot
ExecStart=/usr/local/bin/gpu-metrics.sh

# gpu-metrics.timer
[Unit]
Description=Run GPU metrics collection every 15 seconds

[Timer]
OnBootSec=15s
OnUnitActiveSec=15s
Unit=gpu-metrics.service

[Install]
WantedBy=timers.target

Separated host and GPU views

The dashboard now separates CT203 observability host metrics from CT202 AI runtime host metrics. GPU panels include utilization, temperature, power draw, and VRAM usage.

With Qwen 2.5 7B loaded and idle/ready, the RTX 3090 showed about 21,996 MB of VRAM used.

Qwen3-14B failed with CUDA out-of-memory

The non-quantized Qwen3-14B attempt failed when vLLM tried to allocate additional GPU memory while the RTX 3090 was already near full VRAM usage.

torch.OutOfMemoryError: CUDA out of memory
GPU had ~23.55 GiB total capacity
Only a small amount of VRAM was free

That confirmed full-precision or larger model loading was not practical on a single 24GB card without quantization or model parallelism.

24GB VRAM is usable, but format matters

The RTX 3090 is still a very useful local AI card, but model format and quantization matter more than parameter count alone. The same class of model can fail or run depending on how it is packaged.

Why Quantization Matters

Quantization reduces model weight precision, which lowers VRAM requirements. AWQ allowed the 14B model to fit where the non-quantized model failed.

The tradeoff is possible small quality loss, but the deployability improvement on consumer GPUs is huge. For homelab AI infrastructure, quantization turns "the model theoretically exists" into "the model actually runs."

Qwen/Qwen3-14B-AWQ served through vLLM

• ModelQwen/Qwen3-14B-AWQ
• RuntimevLLM 0.22.0
• APIOpenAI-compatible :8000
• FrontendOpen WebUI
• GPURTX 3090 24GB

vLLM served /v1/models and /v1/chat/completions successfully. Open WebUI displayed and used the Qwen3-14B-AWQ model.