Homelab AI Server Build: Used RTX 3090 Budget Guide

Published on April 10, 2026 • 22 min read

I spent $1,340 building a dedicated AI inference server that runs 70B parameter models at 15 tokens per second. The centerpiece: a used RTX 3090 pulled from a crypto miner's rig. After eight months of 24/7 uptime, that card hasn't skipped a beat. Here is exactly how to replicate this build at three different budget levels.

---

What you get from this guide:

• Complete bill of materials at $800, $1,500, and $2,500 price points
• Step-by-step assembly, OS installation, and software stack
• Power consumption measurements and noise reduction tactics
• Tailscale remote access so you can query your models from anywhere
• Real cost comparison showing when a homelab beats cloud GPU rental

If you need help choosing a GPU before you start, the complete GPU buying guide for AI covers every current option. For understanding VRAM and RAM requirements in detail, see the AI hardware requirements guide.

Table of Contents

1. Why Build a Dedicated AI Server
2. The Used RTX 3090 Value Proposition
3. Bill of Materials: Three Tiers
4. Step-by-Step Assembly
5. Ubuntu Server Installation
6. Ollama and Open WebUI Setup
7. Tailscale Remote Access
8. Power Consumption Analysis
9. Noise Management
10. Cloud GPU Cost Comparison

---

Why Build a Dedicated AI Server {#why-build-dedicated}

Running AI on your daily driver machine is fine for experimentation, but it falls apart when you need persistent inference. Every time you close your laptop or reboot for updates, your models unload. Every time you run a large model, your other applications crawl.

A dedicated server solves this permanently:

Always-on inference. Your models stay loaded in VRAM around the clock. First-token latency drops from 15-30 seconds (cold load) to under 200ms (warm). Anyone on your network can query models instantly.

No resource contention. Your workstation stays snappy for actual work. No more choosing between running a 13B model and having enough RAM for your browser tabs.

Headless efficiency. Without a desktop environment, Ubuntu Server dedicates all resources to inference. You gain 1-2GB of RAM and measurable GPU overhead by dropping the display server entirely.

Learning infrastructure. Building and maintaining a server teaches networking, Linux administration, and service management. These skills transfer directly to production AI deployment.

The counterargument is cloud GPU rental. At $0.80/hr for an A100 on Lambda Labs, you can rent serious compute on demand. But if you run models more than 3-4 hours per day, dedicated hardware pays for itself within 6-12 months. I will show the math in the cost comparison section.

---

The Used RTX 3090 Value Proposition {#rtx-3090-value}

The RTX 3090 is the single best value proposition in AI hardware right now. Here is why:

24GB VRAM for $500-600. New GPUs with 24GB VRAM (RTX 4090, RTX 5090) cost $1,600-$2,000+. A used 3090 delivers the same VRAM capacity for a third of the price. VRAM is the hard constraint for model size, not compute speed.

What 24GB VRAM actually runs:

• 7B models at full FP16 precision (14GB) with room for KV cache
• 13B models at Q4_K_M quantization (7.4GB) with generous context
• 33B models at Q4_K_M quantization (18.5GB) at 2K context
• 70B models at Q2_K quantization (22GB) with limited context

Performance numbers from my build:

Buying a used 3090 safely:

Check eBay completed listings, not current asking prices. As of early 2026, the 3090 Founders Edition sells for $480-550. Third-party cards (EVGA FTW3, MSI Suprim X) go for $500-620.

Avoid cards with modified BIOS or flashed firmware. Ask the seller if the card was used for mining. Mining cards that ran at controlled temperatures (under 85C memory junction) and stable power limits are actually fine. The danger is cards that ran with thermal pad modifications at extreme temperatures.

Test any used card immediately. Run nvidia-smi to check for ECC errors, then run a sustained load for 30 minutes. If it survives without artifacts or crashes, you have a good card.

---

Bill of Materials: Three Tiers {#bill-of-materials}

Tier 1: Budget Build ($800)

This build runs 7B-13B models comfortably and handles 33B at reduced quality.

Prices reflect Q1 2026 US market. You can cut this to $800 by hunting deals on the CPU and buying an open-box PSU.

Tier 2: Recommended Build ($1,500)

My actual build. Handles 70B quantized models, runs two models simultaneously with system RAM offloading.

64GB system RAM matters for CPU offloading. When a 70B model exceeds your 24GB VRAM, layers spill to system RAM. More RAM means more offloaded layers before you hit swap.

Tier 3: Performance Build ($2,500)

For running multiple models simultaneously or handling dual GPUs in the future.

Two 3090s give you 48GB of aggregate VRAM. Ollama does not natively split across GPUs for a single model, but you can run two separate models simultaneously, one per card. For tensor parallelism across GPUs, use vLLM or llama.cpp directly.

Important PSU note: A single RTX 3090 can spike to 450W during load. A dual-GPU build needs 1500W minimum. Do not skimp on the PSU. An underpowered unit causes random shutdowns that corrupt models and filesystems.

---

Step-by-Step Assembly {#assembly}

Pre-Build Checklist

Before you start:

• Ground yourself (touch the PSU case while plugged in but switched off)
• Clear a large, clean, non-carpeted workspace
• Have a Phillips #2 screwdriver and zip ties ready
• Keep the motherboard box as an anti-static workspace

Assembly Order

Step 1: CPU Installation

1. Open motherboard CPU socket lever
2. Align the golden triangle on CPU with socket triangle
3. Drop CPU straight down (zero force needed on AMD)
4. Close lever firmly — some resistance is normal

Step 2: RAM Installation

1. Open RAM slot clips (use slots A2 and B2 for dual channel)
2. Align notch on RAM stick with slot key
3. Press down firmly until both clips snap shut
4. Verify: both clips locked, RAM seated flush

Step 3: NVMe SSD

1. Remove M.2 heatsink screw and heatsink
2. Insert NVMe at 30-degree angle into M.2 slot
3. Press down flat and secure with standoff screw
4. Replace heatsink

Step 4: Motherboard into Case

1. Install I/O shield (press from inside until all tabs click)
2. Align motherboard standoffs with case holes
3. Secure with 9 screws (hand-tight plus quarter turn)
4. Do NOT overtighten — you will crack the PCB

Step 5: PSU Installation

1. Mount PSU with fan facing down (if case has bottom vent)
2. Secure with 4 screws from case rear
3. Route cables through back panel cable management holes
4. Connect: 24-pin ATX, 8-pin CPU, SATA (if needed)

Step 6: GPU Installation

1. Remove 2-3 PCIe slot brackets from case rear
2. Open PCIe x16 slot retention clip
3. Align GPU with slot and press down firmly until clip locks
4. Secure GPU bracket with screws
5. Connect 2x 8-pin (or 3x 8-pin) PCIe power cables
6. CRITICAL: Use separate PCIe cables from PSU, not daisy-chain

The RTX 3090 is a massive card. The Founders Edition is 313mm long and weighs 2.2kg. Use a GPU support bracket or 3D-printed sag preventer. Sagging puts stress on the PCIe slot and can cause intermittent contact issues over months.

Step 7: Cable Management and Fans

1. Route all cables behind motherboard tray
2. Install intake fans on front panel (blowing in)
3. Ensure exhaust through rear and top
4. Zip tie loose cables away from fans
5. Goal: clear airflow path from front intake → GPU → rear exhaust

---

Ubuntu Server Installation {#ubuntu-server}

Why Ubuntu Server (Not Desktop)

Ubuntu Desktop wastes 800MB-1.2GB RAM on GNOME and display services you will never use on a headless AI box. Ubuntu Server boots to a terminal, uses ~350MB RAM at idle, and includes everything you need for AI workloads.

Installation

# Download Ubuntu Server 24.04 LTS
# Flash to USB with balenaEtcher or:
sudo dd if=ubuntu-24.04-live-server-amd64.iso of=/dev/sdX bs=4M status=progress

# Boot from USB, follow installer:
# 1. Language: English
# 2. Network: Configure static IP (recommended for servers)
# 3. Storage: Use entire disk with LVM
# 4. Profile: Create your user account
# 5. SSH: Enable OpenSSH server (important!)
# 6. Snaps: Skip everything

Post-Install Configuration

# Update everything first
sudo apt update && sudo apt upgrade -y

# Install essential packages
sudo apt install -y build-essential git curl wget htop btop nvtop \
  net-tools openssh-server ufw fail2ban

# Set static IP (if not done during install)
sudo nano /etc/netplan/00-installer-config.yaml
# Example config:
# network:
#   ethernets:
#     enp3s0:
#       dhcp4: no
#       addresses: [192.168.1.100/24]
#       routes:
#         - to: default
#           via: 192.168.1.1
#       nameservers:
#         addresses: [1.1.1.1, 8.8.8.8]

sudo netplan apply

NVIDIA Driver Installation

This is where most people trip up. Do NOT install drivers from the NVIDIA website directly. Use Ubuntu's built-in package manager.

# Add NVIDIA driver PPA
sudo add-apt-repository ppa:graphics-drivers/ppa -y
sudo apt update

# Check recommended driver
ubuntu-drivers devices
# Look for "nvidia-driver-560" or similar marked "recommended"

# Install the recommended driver
sudo apt install -y nvidia-driver-560

# Reboot
sudo reboot

# Verify after reboot
nvidia-smi
# Should show RTX 3090 with 24576 MiB memory, driver 560.xx

Expected output:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.03    Driver Version: 560.35.03    CUDA Version: 12.6    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence  | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf  Pwr:Usage  |  Memory-Usage        | GPU-Util  Compute M.  |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:01:00.0 Off |                  N/A |
|  0%   32C    P8     22W       |      1MiB / 24576MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

Docker Installation

# Install Docker
curl -fsSL https://get.docker.com | sudo sh
sudo usermod -aG docker $USER

# Install NVIDIA Container Toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# Verify GPU in Docker
docker run --rm --gpus all nvidia/cuda:12.6.0-base-ubuntu24.04 nvidia-smi

---

Ollama and Open WebUI Setup {#ollama-setup}

Install Ollama

# One-line install
curl -fsSL https://ollama.com/install.sh | sh

# Enable and start service
sudo systemctl enable ollama
sudo systemctl start ollama

# Verify
ollama --version

Configure Ollama for Network Access

By default, Ollama only listens on localhost. For a server, you want it accessible from your LAN.

# Edit the systemd service
sudo systemctl edit ollama

# Add these lines between the comment blocks:
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_ORIGINS=*"
Environment="OLLAMA_KEEP_ALIVE=24h"

# Reload and restart
sudo systemctl daemon-reload
sudo systemctl restart ollama

# Verify it is listening on all interfaces
ss -tlnp | grep 11434
# Should show 0.0.0.0:11434

The OLLAMA_KEEP_ALIVE=24h setting keeps models loaded in VRAM for 24 hours after last request. On a dedicated server, this means near-instant responses throughout the day.

Pull Models

# Essential models for a 24GB card
ollama pull llama3.2:7b          # 4.7GB - daily driver
ollama pull qwen2.5-coder:14b   # 9.1GB - code generation
ollama pull mistral:7b           # 4.1GB - fast general purpose
ollama pull llama3.1:70b-q2_K   # 22GB  - maximum quality (slow)

# Check what is loaded
ollama ps

Deploy Open WebUI with Docker

For a complete walkthrough of Open WebUI features and configuration, see the Ollama + Open WebUI Docker setup guide.

# Run Open WebUI with GPU support
docker run -d \
  --name open-webui \
  --restart always \
  --gpus all \
  -p 3000:8080 \
  -v open-webui:/app/backend/data \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  --add-host=host.docker.internal:host-gateway \
  ghcr.io/open-webui/open-webui:main

# Access from any device on your network:
# http://192.168.1.100:3000

Auto-Start on Boot

# Ollama already configured as systemd service above
# Docker containers with --restart always auto-start

# Verify both survive a reboot
sudo reboot
# After reboot:
systemctl status ollama        # Should show "active (running)"
docker ps                      # Should show open-webui running

---

Tailscale Remote Access {#tailscale-remote}

Tailscale creates a WireGuard VPN mesh network. You install it on your server and your laptop/phone, and they can reach each other from anywhere without opening firewall ports or configuring port forwarding.

# Install Tailscale on the server
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up

# It prints a URL — open it in your browser to authenticate
# Your server gets a Tailscale IP like 100.x.y.z

# Now install Tailscale on your laptop/phone and authenticate
# Access your AI server from anywhere:
# http://100.x.y.z:3000  (Open WebUI)
# http://100.x.y.z:11434 (Ollama API directly)

Secure the Server

# Configure UFW firewall
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw allow from 192.168.1.0/24 to any port 3000   # Open WebUI (LAN only)
sudo ufw allow from 192.168.1.0/24 to any port 11434  # Ollama API (LAN only)
sudo ufw enable

# Tailscale traffic bypasses UFW automatically
# So remote access via Tailscale still works

---

Power Consumption Analysis {#power-consumption}

I measured wall power with a Kill-A-Watt meter over 30 days on my Tier 2 build:

Realistic monthly cost: If you run inference 4 hours/day and idle the rest, expect $12-15/month in electricity. That is less than a single month of ChatGPT Plus ($20) while running unlimited queries.

Power Optimization

# Undervolt the GPU for 15-20% power reduction with <5% performance loss
# Install nvidia-settings (even on headless server)
sudo apt install -y nvidia-settings

# Set power limit (default 350W, reduce to 280W)
sudo nvidia-smi -pl 280

# Make persistent across reboots
echo 'sudo nvidia-smi -pl 280' | sudo tee /etc/rc.local
sudo chmod +x /etc/rc.local

# Verify
nvidia-smi -q -d POWER | grep "Power Limit"
# Should show "Current Power Limit: 280.00 W"

At 280W power limit, my 7B inference speed dropped from 62 to 58 tok/s (6% slower) while cutting power draw during inference from 340W to 275W system-wide. Over a year, that saves roughly $40 in electricity.

---

Noise Management {#noise-management}

A stock RTX 3090 Founders Edition at full load hits 42-45 dBA. If the server lives in your office, that is noticeable. In a closet with the door closed, it is inaudible.

Fan Curve Optimization

# Install fan control
sudo apt install -y python3-pip
pip3 install nvidia-ml-py3

# Create custom fan curve script
cat << 'SCRIPT' > ~/fan_curve.sh
#!/bin/bash
# Aggressive cooling at low RPM: quiet but effective
GPU_TEMP=$(nvidia-smi --query-gpu=temperature.gpu --format=csv,noheader)

if [ "$GPU_TEMP" -lt 40 ]; then
  nvidia-settings -a "[gpu:0]/GPUFanControlState=1" -a "[fan:0]/GPUTargetFanSpeed=25"
elif [ "$GPU_TEMP" -lt 60 ]; then
  nvidia-settings -a "[gpu:0]/GPUFanControlState=1" -a "[fan:0]/GPUTargetFanSpeed=40"
elif [ "$GPU_TEMP" -lt 75 ]; then
  nvidia-settings -a "[gpu:0]/GPUFanControlState=1" -a "[fan:0]/GPUTargetFanSpeed=60"
else
  nvidia-settings -a "[gpu:0]/GPUFanControlState=1" -a "[fan:0]/GPUTargetFanSpeed=85"
fi
SCRIPT
chmod +x ~/fan_curve.sh

# Run every 30 seconds via cron
(crontab -l 2>/dev/null; echo "* * * * * ~/fan_curve.sh") | crontab -
(crontab -l 2>/dev/null; echo "* * * * * sleep 30 && ~/fan_curve.sh") | crontab -

Physical Noise Reduction

1. Case selection matters most. The Fractal Meshify 2 Compact has sound-dampening panels while maintaining airflow. Avoid cases marketed as "silent" that choke airflow and cause thermal throttling.

2. Replace stock case fans with Noctua NF-A14 (140mm) or Arctic P14 PWM. These move the same air at half the noise of cheap bundled fans.

3. Rubber anti-vibration mounts on all fans and the GPU bracket. A $3 pack of rubber washers eliminates case resonance.

4. Location. A closet, basement shelf, or under-desk cabinet makes more noise difference than any hardware mod. Run a long Ethernet cable if needed.

---

Cloud GPU Cost Comparison {#cloud-comparison}

Here is the break-even math, assuming you build the Tier 2 system for $1,200.

Cloud GPU costs (early 2026):

Homelab monthly cost: $12-15 electricity. That is it.

Break-even timeline:

If you compare against Lambda or AWS (the services most people actually use), the homelab pays for itself in 11-16 months. After that, you are running AI inference essentially for free, minus electricity.

But the real value is not just cost. It is zero cold-start latency, no upload of private data, no usage caps, and the ability to run inference at 3 AM without worrying about a billing surprise.

---

Monitoring Your Server

# Install monitoring stack
sudo apt install -y nvtop btop

# GPU monitoring (real-time)
nvtop
# Shows GPU utilization, VRAM usage, temperature, power draw, per-process

# System monitoring
btop
# Shows CPU, RAM, disk, network in a beautiful TUI

# Create a simple health check script
cat << 'HEALTH' > ~/healthcheck.sh
#!/bin/bash
GPU_TEMP=$(nvidia-smi --query-gpu=temperature.gpu --format=csv,noheader)
GPU_MEM=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader)
GPU_UTIL=$(nvidia-smi --query-gpu=utilization.gpu --format=csv,noheader)
OLLAMA_STATUS=$(systemctl is-active ollama)
WEBUI_STATUS=$(docker inspect -f '{{.State.Running}}' open-webui 2>/dev/null || echo "not found")

echo "=== AI Server Health ==="
echo "GPU Temp: ${GPU_TEMP}C"
echo "GPU Memory: ${GPU_MEM}"
echo "GPU Utilization: ${GPU_UTIL}"
echo "Ollama: ${OLLAMA_STATUS}"
echo "Open WebUI: ${WEBUI_STATUS}"
echo "Uptime: $(uptime -p)"
HEALTH
chmod +x ~/healthcheck.sh

---

Maintenance Schedule

Weekly:

• Check nvidia-smi for ECC errors (any non-zero value means hardware degradation)
• Update Ollama: sudo ollama update
• Check Docker container logs: docker logs open-webui --tail 50

Monthly:

• Run sudo apt update && sudo apt upgrade for security patches
• Check SSD health: sudo smartctl -a /dev/nvme0n1
• Compressed air blast to remove dust (open case, blow front to back)
• Review power consumption with Kill-A-Watt

Quarterly:

• Repaste GPU thermal compound if temps have risen more than 5C from baseline
• Check all cable connections are secure
• Test UPS battery if you use one

---

Next Steps

Your homelab AI server is running. Here is where to go from here:

1. Optimize your model selection. The best GPUs for AI guide covers GPU-specific model recommendations, and the VRAM requirements guide helps you understand exactly what fits on your 24GB card.

2. Set up a proper web interface. Follow the Ollama + Open WebUI Docker setup guide for multi-user accounts, conversation history, and model switching.

3. Consider multi-GPU scaling. If 24GB VRAM is not enough, adding a second 3090 is far cheaper than buying a single 48GB card.

---

Frequently Asked Questions

Can a used RTX 3090 from a crypto miner be trusted for AI workloads?

Yes. Mining runs GPUs at constant, moderate temperatures with stable power draw. This is actually less stressful than gaming, which cycles temperatures rapidly. The main risk is GDDR6X memory degradation from prolonged high junction temperatures, but this is rare. Test for 30 minutes under load before committing.

Is 24GB VRAM enough for serious AI work in 2026?

For local inference, absolutely. 24GB runs every model up to 33B at high quality and 70B at reduced quality. The only scenario where 24GB falls short is fine-tuning large models or running 70B+ at full precision, both of which require 48GB+ regardless.

Should I buy one RTX 3090 or two RTX 3060 12GB cards?

One 3090. Two 3060s give you 24GB total but you cannot combine VRAM across cards for a single model in Ollama. The 3090 also has higher memory bandwidth (936 GB/s vs 360 GB/s) which directly affects token generation speed.

How loud is this build during inference?

With the Fractal Meshify case and a 280W power limit, mine measures 34 dBA at 1 meter during sustained inference. That is quieter than a typical office conversation. At idle with the model loaded, it drops to 28 dBA, basically inaudible.

Can I run this server on a UPS?

Yes, and you should. An APC Back-UPS 1500VA ($180) provides 8-12 minutes of runtime during a power outage, enough for a clean shutdown. Configure apcupsd on Ubuntu for automatic shutdown when battery hits 20%.

What about running this on a Raspberry Pi instead?

A Raspberry Pi 5 with 8GB RAM can run tiny models (1-3B) but nothing useful for production work. The Pi has no GPU acceleration for inference. It is good for learning, not for a real AI server. See the AI hardware requirements guide for minimum specs.

---

Conclusion

A homelab AI server built around a used RTX 3090 is the most cost-effective way to get serious about local AI. For roughly the price of 15 months of cloud GPU rental, you own hardware that runs 24/7 with no recurring fees beyond electricity.

The build itself takes an afternoon. The software stack (Ubuntu Server, Ollama, Open WebUI, Tailscale) takes another hour. After that, you have a private AI inference server accessible from anywhere, running any open-weight model that fits in 24GB of VRAM.

Start with the Tier 2 build if you can swing $1,200-1,500. The 64GB of system RAM gives you headroom for CPU offloading and future expansion. If you are budget-constrained, the Tier 1 build at $800-950 still runs circles around any cloud API for sustained daily use.

---

Need help choosing the right GPU for your build? Check our GPU comparison guide for detailed benchmarks, or visit the hardware requirements guide for sizing your build to your workload.