Linux Local AI Setup: Ubuntu, Debian & Arch (2025 Complete Guide)

Published on October 28, 2025 • 25 min read

🚀 Quick Start: Install AI on Any Linux in 5 Minutes

To install AI on any Linux distribution:

1. Install Ollama: curl -fsSL https://ollama.com/install.sh | sh (2 minutes)
2. Verify installation: ollama --version (10 seconds)
3. Download model: ollama pull llama3.2:3b (3 minutes)
4. Start using: ollama run llama3.2:3b "Hello" (instant)

That's it! You now have AI running on your Linux system.

Works on: Ubuntu, Debian, Fedora, Arch, openSUSE, and all major Linux distributions

---

How to Setup Local AI on Linux (All Distributions)

To setup local AI on Linux:

1. Install Ollama: curl -fsSL https://ollama.com/install.sh | sh (2 minutes)
2. Verify installation: ollama --version (10 seconds)
3. Download model: ollama pull llama3.1:8b (3-5 minutes)
4. Run model: ollama run llama3.1:8b (instant)
5. Enable GPU (optional): install the NVIDIA driver or AMD ROCm; Ollama bundles its own CUDA runtime, so the CUDA Toolkit is not needed

Total time: 5-20 minutes | Distros: Ubuntu, Debian, Fedora, Arch, all supported | Requirements: 8GB+ RAM

Distribution-specific commands included below for Ubuntu, Debian, Fedora, Arch, openSUSE.

---

Quick Summary:

• ✅ Install local AI on any Linux distribution
• ✅ GPU acceleration setup (NVIDIA, AMD, Intel)
• ✅ Docker and systemd service configuration
• ✅ Performance optimization and monitoring
• ✅ Troubleshooting for all major distros

Linux is the ultimate platform for local AI development. With superior hardware control, package management flexibility, and native Docker support, Linux offers unmatched performance and customization options for running AI models locally. This comprehensive guide covers everything from basic installation to advanced optimization across all major distributions.

Table of Contents

1. Why Linux for Local AI
2. Which Models Fit Your Linux Machine?
3. Distribution-Specific Installation
4. GPU Drivers and Acceleration
5. Docker Setup and Management
6. Systemd Service Configuration
7. Performance Optimization
8. Monitoring and Logging
9. Terminal Customization
10. Security Considerations
11. Troubleshooting Guide

---

Why Linux for Local AI

Linux Advantages for AI:

1. Hardware Control

• Direct access to GPU memory and drivers
• Fine-grained CPU scheduling and affinity
• Custom kernel parameters for optimization
• Real-time process prioritization

2. Package Management

• Multiple installation methods (package managers, source, containers)
• Dependency resolution and conflict handling
• Easy updates and rollbacks
• Community repositories with latest versions

3. Resource Efficiency

• Minimal overhead compared to Windows/macOS
• No unnecessary background services
• Precise memory and CPU allocation
• Swap and cache optimization

4. Development Environment

• Native Docker and container support
• Advanced shell scripting capabilities
• Comprehensive development tools
• Superior debugging and profiling

---

Which Models Fit Your Linux Machine?

Two pieces of arithmetic answer this before you download anything.

1. Does it fit? The model file has to fit in VRAM (or RAM, for CPU-only) with 1-2GB to spare for context. Ollama's default 4-bit downloads come to roughly 0.6GB per billion parameters; the exact size is on each model's page in the Ollama library.

2. How fast can it possibly go? Generating a token means reading the whole model from memory once, so:

tokens/second ceiling = memory bandwidth (GB/s) / model size (GB)

That is an upper bound, not a prediction; real speed lands below it. The table applies it to common tiers, using vendor-published bandwidth and the sizes Ollama lists for each tag (checked November 2, 2026).

Bandwidth from each card's NVIDIA specification page; DDR5 figures are the theoretical dual-channel rate. Ceiling = bandwidth / model size.

Two things in that table matter more than any single number:

• CPUs fall further below their ceiling than GPUs do, because they also run short of raw compute. Treat CPU-only as fine for background jobs and patient chat, not as a GPU substitute.
• Bandwidth, not VRAM size, sets the speed. The 16GB RTX 4060 Ti has more memory than the RTX 3060 12GB but a narrower bus, so on the same model its ceiling is lower. Check the bandwidth on the spec sheet, not just the VRAM.

For your real number, run any prompt with ollama run llama3.1:8b --verbose and read the eval rate line it prints after the response. To check a specific machine before buying or downloading, use the "Can I run local AI?" checker.

---

Distribution-Specific Installation

Ubuntu/Debian Installation

Method 1: Official Install Script (Recommended)

# Ubuntu 22.04 / 24.04 / 26.04 LTS, Debian 12+
sudo apt update && sudo apt install -y curl
curl -fsSL https://ollama.com/install.sh | sh

# Verify installation
ollama --version
systemctl status ollama

The script installs the binary, creates an ollama system user and a systemd service, binds the API to 127.0.0.1:11434, and downloads the extra GPU libraries it needs when it detects an NVIDIA or AMD card. Re-running the same command is also how you update.

Method 2: Manual Installation

This is the manual route from Ollama's own Linux documentation (docs/linux on GitHub). Newer releases ship as .tar.zst archives that extract into /usr:

# If upgrading, remove old libraries first
sudo rm -rf /usr/lib/ollama

# Download and extract (x86_64)
curl -fsSL https://ollama.com/download/ollama-linux-amd64.tar.zst | sudo tar x -C /usr

# AMD GPU only: add the ROCm package on top
curl -fsSL https://ollama.com/download/ollama-linux-amd64-rocm.tar.zst | sudo tar x -C /usr

# ARM64 machines use ollama-linux-arm64.tar.zst instead

# Create the service user and add yourself to its group
sudo useradd -r -s /bin/false -U -m -d /usr/share/ollama ollama
sudo usermod -a -G ollama $(whoami)

# Create systemd service
sudo tee /etc/systemd/system/ollama.service > /dev/null <<EOF
[Unit]
Description=Ollama Service
After=network-online.target

[Service]
ExecStart=/usr/bin/ollama serve
User=ollama
Group=ollama
Restart=always
RestartSec=3
Environment="PATH=$PATH"

[Install]
WantedBy=multi-user.target
EOF

# Start service
sudo systemctl daemon-reload
sudo systemctl enable --now ollama

The manual install puts the binary in /usr/bin/ollama; the install script uses /usr/local/bin/ollama. Adjust paths in the later sections to match whichever you used (which ollama tells you).

Method 3: APT Repository

There is no official Ollama APT repository. Ollama publishes only the install script and the release archives above, so guides that tell you to add a packages.ollama.com apt source are pointing at something that does not exist. Third-party repositories and community snaps do exist, but they are unofficial and usually lag upstream releases. On Ubuntu and Debian, use Method 1 or Method 2.

Fedora/CentOS/RHEL Installation

Method 1: DNF Installation

# Fedora / RHEL / Rocky / Alma
sudo dnf install -y curl
curl -fsSL https://ollama.com/install.sh | sh

The install script supports dnf-based systems and sets up the same systemd service as on Ubuntu.

Method 2: RPM Repository

As with APT, Ollama does not publish an official RPM repository. Use the install script, or the manual .tar.zst method from the Ubuntu section, which works unchanged on Fedora and RHEL. On SELinux-enforcing systems, see the troubleshooting section below if the service cannot read its model directory.

Arch Linux Installation

Method 1: Official Repository (Recommended)

Ollama is packaged in Arch's official extra repository, with separate builds per GPU backend (archlinux.org package search, checked November 2, 2026):

# CPU only
sudo pacman -S ollama

# NVIDIA (CUDA build)
sudo pacman -S ollama-cuda

# AMD (ROCm build)
sudo pacman -S ollama-rocm

# Vulkan build (Intel and other GPUs)
sudo pacman -S ollama-vulkan

# Start the packaged service
sudo systemctl enable --now ollama

Method 2: Manual Installation

# Install dependencies
sudo pacman -S curl zstd

# Download and install (same archive as the Ubuntu manual method)
curl -fsSL https://ollama.com/download/ollama-linux-amd64.tar.zst | sudo tar x -C /usr

# Create the user and systemd service (same as the Ubuntu manual method above)

openSUSE Installation

# openSUSE Leap/Tumbleweed
sudo zypper install curl
curl -fsSL https://ollama.com/install.sh | sh

Building from source is possible but rarely needed; follow docs/development.md in the Ollama GitHub repository, because the build steps change between releases.

---

GPU Drivers and Acceleration

NVIDIA GPU Setup

Driver Installation

You only need the NVIDIA driver. Ollama ships its own CUDA runtime. Per Ollama's GPU documentation, it supports NVIDIA GPUs with compute capability 5.0+ and driver 550 or newer (cards with compute capability 5.0-6.2 need driver 570+).

# Ubuntu: see which driver Ubuntu recommends for your card
ubuntu-drivers devices

# Install the recommended branch (22.04, 24.04 and 26.04)
sudo ubuntu-drivers install
sudo reboot

# Fedora (RPM Fusion must be enabled)
sudo dnf install akmod-nvidia xorg-x11-drv-nvidia-cuda

# Arch Linux
sudo pacman -S nvidia-open nvidia-utils

# Verify installation
nvidia-smi

On Ubuntu 22.04 and 24.04 the older sudo ubuntu-drivers autoinstall still works as a deprecated alias; install is the current command. If nvidia-smi prints your GPU, driver version and CUDA version without errors, Ollama will use the GPU automatically.

CUDA Toolkit Installation

Ollama does not need the CUDA Toolkit. Install it only if you also compile other AI software (llama.cpp from source, PyTorch extensions). NVIDIA's network-repo method for Ubuntu 24.04:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update
sudo apt install -y cuda-toolkit

# Add to PATH
echo 'export PATH=/usr/local/cuda/bin:$PATH' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
source ~/.bashrc

For other distributions, pick yours on NVIDIA's CUDA downloads page (developer.nvidia.com/cuda-downloads).

Configure Ollama for NVIDIA

No configuration is needed for a single GPU. The one setting Ollama documents for NVIDIA is GPU selection: on a multi-GPU machine, CUDA_VISIBLE_DEVICES limits which cards it uses.

# Optional: only use GPU 0 (multi-GPU machines)
sudo systemctl edit ollama
# Add:
[Service]
Environment="CUDA_VISIBLE_DEVICES=0"

# Reload and restart
sudo systemctl daemon-reload
sudo systemctl restart ollama

# Verify GPU usage: PROCESSOR should read "100% GPU"
ollama run llama3.2:3b "test"
ollama ps
nvidia-smi

AMD GPU Setup (ROCm)

ROCm Installation

AMD acceleration through ROCm is Linux-first, and Ubuntu LTS is its best-supported platform. Ollama's GPU documentation says it requires the AMD ROCm v7 driver on Linux, installed with AMD's amdgpu-install utility. The amdgpu-install helper is not in Ubuntu's default repositories: you download a small .deb from AMD's repository, then install the driver and ROCm. These are the commands from AMD's ROCm quick-start page for Ubuntu 24.04 (ROCm 7.2.4; read November 2, 2026). For 22.04, replace noble with jammy.

# 1. Register AMD's repository
wget https://repo.radeon.com/amdgpu-install/7.2.4/ubuntu/noble/amdgpu-install_7.2.4.70204-1_all.deb
sudo apt install ./amdgpu-install_7.2.4.70204-1_all.deb
sudo apt update

# 2. Kernel driver
sudo apt install "linux-headers-$(uname -r)" "linux-modules-extra-$(uname -r)"
sudo apt install amdgpu-dkms

# 3. ROCm, plus GPU access for your user (and the ollama service user)
sudo apt install python3-setuptools python3-wheel
sudo usermod -a -G render,video $LOGNAME
sudo usermod -a -G render,video ollama
sudo apt install rocm
sudo reboot

# Verify
rocm-smi

AMD also ships a newer ROCm Core SDK line (10.1.0, October 5, 2026) with its own apt repository; its install page is rocm.docs.amd.com/en/latest/install/rocm.html. For Ollama alone, the v7 driver above is what its docs ask for. For a deeper AMD walkthrough, see our AMD ROCm local LLM setup guide.

On Arch, install ollama-rocm instead; it pulls in the ROCm runtime from the official repos.

Configure Ollama for AMD

Ollama detects ROCm automatically. The settings it documents for AMD are an override for cards ROCm does not officially list, and GPU selection:

sudo systemctl edit ollama
# Add only what you need:
[Service]
# Treat an unlisted RDNA2 card (e.g. gfx1031) as the supported gfx1030 target
Environment="HSA_OVERRIDE_GFX_VERSION=10.3.0"
# Multi-GPU: restrict Ollama to specific AMD GPUs
Environment="ROCR_VISIBLE_DEVICES=0"

sudo systemctl daemon-reload
sudo systemctl restart ollama

Leave HSA_OVERRIDE_GFX_VERSION unset on cards Ollama already supports, which per its docs includes the RX 6800/6900, RX 7600-7900 and RX 9060/9070 series and the Ryzen AI Max+ 395.

Intel GPU Setup

Ollama reaches Intel GPUs (Arc and recent integrated graphics) and other non-CUDA, non-ROCm cards through its Vulkan backend. Per Ollama's GPU documentation, Vulkan is enabled by default when the backend is installed, but most Linux distributions need the Vulkan driver components added.

# Ubuntu/Debian: Mesa Vulkan drivers + diagnostics
sudo apt install mesa-vulkan-drivers vulkan-tools intel-gpu-tools
vulkaninfo --summary

# Arch: use the Vulkan build
sudo pacman -S vulkan-intel ollama-vulkan

# Optional: let Ollama read VRAM usage for better scheduling
sudo setcap cap_perfmon+ep $(which ollama)

# Multi-GPU: pick the Vulkan device (or set -1 to disable Vulkan GPUs)
sudo systemctl edit ollama
# Add:
[Service]
Environment="GGML_VK_VISIBLE_DEVICES=0"

sudo systemctl daemon-reload
sudo systemctl restart ollama

For Arc driver packages, Intel's client GPU documentation (dgpu-docs.intel.com) is the reference.

---

Docker Setup and Management

Docker Installation

# Ubuntu/Debian
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo usermod -aG docker $USER

# Fedora
sudo dnf install docker docker-compose
sudo systemctl enable --now docker
sudo usermod -aG docker $USER

# Arch Linux
sudo pacman -S docker docker-compose
sudo systemctl enable --now docker
sudo usermod -aG docker $USER

# Logout and login to apply group changes

Ollama Docker Setup

Basic Docker Run

# CPU-only
docker run -d   --name ollama   -p 11434:11434   -v ollama:/root/.ollama   --restart unless-stopped   ollama/ollama:latest

# With NVIDIA GPU
docker run -d   --name ollama   --gpus all   -p 11434:11434   -v ollama:/root/.ollama   --restart unless-stopped   ollama/ollama:latest

# With AMD GPU (ROCm)
docker run -d   --name ollama   --device /dev/kfd --device /dev/dri   --group-add video   -p 11434:11434   -v ollama:/root/.ollama   --restart unless-stopped   ollama/ollama:rocm

Docker Compose Configuration

# docker-compose.yml
version: '3.8'

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
      - ./models:/models
    environment:
      - OLLAMA_HOST=0.0.0.0:11434
      - OLLAMA_MODELS=/models
    restart: unless-stopped
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

  ollama-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: ollama-webui
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - WEBUI_SECRET_KEY=your-secret-key
    volumes:
      - webui_data:/app/backend/data
    depends_on:
      - ollama
    restart: unless-stopped

volumes:
  ollama_data:
  webui_data:

Docker Management Scripts

# Create management script
sudo tee /usr/local/bin/ollama-docker > /dev/null <<'EOF'
#!/bin/bash

case "$1" in
  start)
    docker-compose -f /opt/ollama/docker-compose.yml up -d
    ;;
  stop)
    docker-compose -f /opt/ollama/docker-compose.yml down
    ;;
  restart)
    docker-compose -f /opt/ollama/docker-compose.yml restart
    ;;
  logs)
    docker-compose -f /opt/ollama/docker-compose.yml logs -f
    ;;
  update)
    docker-compose -f /opt/ollama/docker-compose.yml pull
    docker-compose -f /opt/ollama/docker-compose.yml up -d
    ;;
  *)
    echo "Usage: $0 {start|stop|restart|logs|update}"
    exit 1
    ;;
esac
EOF

sudo chmod +x /usr/local/bin/ollama-docker

---

Systemd Service Configuration

Advanced Service Configuration

# Create service override directory
sudo mkdir -p /etc/systemd/system/ollama.service.d/

# Advanced configuration
sudo tee /etc/systemd/system/ollama.service.d/override.conf > /dev/null <<EOF
[Unit]
Description=Ollama Service - Local AI Language Model Server
Documentation=https://ollama.com/docs
After=network-online.target
Wants=network-online.target

[Service]
Type=exec
# An override must clear the packaged ExecStart before setting a new one
ExecStart=
ExecStart=/usr/local/bin/ollama serve
ExecReload=/bin/kill -HUP $MAINPID
KillMode=mixed
KillSignal=SIGINT
TimeoutStopSec=30
Restart=always
RestartSec=5
User=ollama
Group=ollama

# Security settings
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/usr/share/ollama
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true

# Resource limits
LimitNOFILE=65536
LimitNPROC=4096
MemoryMax=16G
CPUQuota=800%

# Environment
Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_MODELS=/usr/share/ollama/.ollama/models"
Environment="OLLAMA_NUM_PARALLEL=1"
Environment="OLLAMA_MAX_LOADED_MODELS=3"
Environment="OLLAMA_KEEP_ALIVE=5m"

[Install]
WantedBy=multi-user.target
EOF

# Reload and restart
sudo systemctl daemon-reload
sudo systemctl restart ollama
sudo systemctl status ollama

Service Monitoring Script

# Create monitoring script
sudo tee /usr/local/bin/ollama-monitor > /dev/null <<'EOF'
#!/bin/bash

LOGFILE="/var/log/ollama-monitor.log"
THRESHOLD_CPU=80
THRESHOLD_MEM=90

log_message() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOGFILE"
}

check_service() {
    if ! systemctl is-active --quiet ollama; then
        log_message "ERROR: Ollama service is down, attempting restart"
        systemctl restart ollama
        sleep 10
        if systemctl is-active --quiet ollama; then
            log_message "INFO: Ollama service restarted successfully"
        else
            log_message "ERROR: Failed to restart Ollama service"
        fi
    fi
}

check_resources() {
    local cpu_usage=$(ps -C ollama -o %cpu --no-headers | awk '{sum+=$1} END {print sum}')
    local mem_usage=$(ps -C ollama -o %mem --no-headers | awk '{sum+=$1} END {print sum}')

    if (( $(echo "$cpu_usage > $THRESHOLD_CPU" | bc -l) )); then
        log_message "WARNING: High CPU usage: ${cpu_usage}%"
    fi

    if (( $(echo "$mem_usage > $THRESHOLD_MEM" | bc -l) )); then
        log_message "WARNING: High memory usage: ${mem_usage}%"
    fi
}

check_service
check_resources
EOF

sudo chmod +x /usr/local/bin/ollama-monitor

# Add to crontab
(crontab -l 2>/dev/null; echo "*/5 * * * * /usr/local/bin/ollama-monitor") | crontab -

---

Performance Optimization

CPU Optimization

# Check CPU info
lscpu
cat /proc/cpuinfo | grep -E "(model name|cpu cores|siblings)"

# Set CPU governor for performance
sudo cpupower frequency-set -g performance

# CPU affinity for Ollama
sudo systemctl edit ollama
# Add:
[Service]
CPUAffinity=0-7  # Use cores 0-7
Nice=-10         # Higher priority
IOSchedulingClass=1
IOSchedulingPriority=4

Memory Optimization

# Check memory info
free -h
cat /proc/meminfo

# Optimize swap
sudo sysctl vm.swappiness=10
sudo sysctl vm.vfs_cache_pressure=50

# Make permanent
echo 'vm.swappiness=10' | sudo tee -a /etc/sysctl.conf
echo 'vm.vfs_cache_pressure=50' | sudo tee -a /etc/sysctl.conf

# Huge pages for large models
sudo sysctl vm.nr_hugepages=1024
echo 'vm.nr_hugepages=1024' | sudo tee -a /etc/sysctl.conf

# Memory limits for Ollama
sudo systemctl edit ollama
# Add:
[Service]
MemoryMax=12G
MemoryHigh=10G

Storage Optimization

# Check disk performance
sudo hdparm -t /dev/sda  # Replace with your disk

# NVMe/SSD: use the 'none' scheduler (modern kernels have no 'noop')
echo none | sudo tee /sys/block/nvme0n1/queue/scheduler

# Mount optimizations for model storage
sudo mkdir -p /opt/ollama-models
sudo mount -o noatime,nodiratime /dev/disk/by-label/MODELS /opt/ollama-models

# Add to fstab
echo '/dev/disk/by-label/MODELS /opt/ollama-models ext4 noatime,nodiratime,defaults 0 0' | sudo tee -a /etc/fstab

Network Optimization

# TCP optimization for model downloads
sudo sysctl net.core.rmem_max=67108864
sudo sysctl net.core.wmem_max=67108864
sudo sysctl net.ipv4.tcp_rmem="4096 87380 67108864"
sudo sysctl net.ipv4.tcp_wmem="4096 65536 67108864"

# Make permanent
echo 'net.core.rmem_max=67108864' | sudo tee -a /etc/sysctl.conf
echo 'net.core.wmem_max=67108864' | sudo tee -a /etc/sysctl.conf
echo 'net.ipv4.tcp_rmem=4096 87380 67108864' | sudo tee -a /etc/sysctl.conf
echo 'net.ipv4.tcp_wmem=4096 65536 67108864' | sudo tee -a /etc/sysctl.conf

---

Monitoring and Logging

System Monitoring

# Install monitoring tools
# Ubuntu/Debian
sudo apt install htop iotop nethogs nvtop

# Fedora
sudo dnf install htop iotop nethogs nvtop

# Arch
sudo pacman -S htop iotop nethogs nvtop

# Create monitoring dashboard script
cat > ~/ollama-status.sh << 'EOF'
#!/bin/bash
clear
echo "=== Ollama System Status ==="
echo "Service Status:"
systemctl status ollama --no-pager -l

echo -e "
CPU and Memory:"
ps aux | grep ollama | grep -v grep

echo -e "
GPU Status:"
if command -v nvidia-smi &> /dev/null; then
    nvidia-smi
elif command -v rocm-smi &> /dev/null; then
    rocm-smi
fi

echo -e "
Disk Usage:"
df -h ~/.ollama

echo -e "
Network Connections:"
sudo netstat -tlnp | grep 11434

echo -e "
Recent Logs:"
journalctl -u ollama --no-pager -n 10
EOF

chmod +x ~/ollama-status.sh

Advanced Logging

# Configure structured logging
sudo mkdir -p /var/log/ollama
sudo chown ollama:ollama /var/log/ollama

# Create logrotate configuration
sudo tee /etc/logrotate.d/ollama > /dev/null <<EOF
/var/log/ollama/*.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 0644 ollama ollama
    postrotate
        systemctl reload ollama > /dev/null 2>&1 || true
    endscript
}
EOF

# Update systemd service for logging
sudo systemctl edit ollama
# Add:
[Service]
StandardOutput=append:/var/log/ollama/ollama.log
StandardError=append:/var/log/ollama/ollama-error.log

Performance Metrics Collection

# Create metrics collection script
sudo tee /usr/local/bin/ollama-metrics > /dev/null <<'EOF'
#!/bin/bash

METRICS_FILE="/var/log/ollama/metrics.log"
DATE=$(date '+%Y-%m-%d %H:%M:%S')

# CPU usage
CPU_USAGE=$(top -bn1 | grep "Cpu(s)" | awk '{print $2} | sed 's/%us,//)

# Memory usage
MEM_USAGE=$(free | grep Mem | awk '{printf("%.1f"), $3/$2 * 100.0}')

# GPU usage (if NVIDIA)
if command -v nvidia-smi &> /dev/null; then
    GPU_USAGE=$(nvidia-smi --query-gpu=utilization.gpu --format=csv,noheader,nounits)
    GPU_MEMORY=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits)
else
    GPU_USAGE="N/A"
    GPU_MEMORY="N/A"
fi

# Disk I/O
DISK_READ=$(iostat -d 1 2 | tail -n +4 | awk '{sum+=$3} END {print sum}')
DISK_WRITE=$(iostat -d 1 2 | tail -n +4 | awk '{sum+=$4} END {print sum}')

# Network I/O
NET_RX=$(cat /proc/net/dev | grep eth0 | awk '{print $2}')
NET_TX=$(cat /proc/net/dev | grep eth0 | awk '{print $10}')

echo "$DATE,CPU:$CPU_USAGE,MEM:$MEM_USAGE,GPU:$GPU_USAGE,GPU_MEM:$GPU_MEMORY,DISK_R:$DISK_READ,DISK_W:$DISK_WRITE,NET_RX:$NET_RX,NET_TX:$NET_TX" >> "$METRICS_FILE"
EOF

sudo chmod +x /usr/local/bin/ollama-metrics

# Run every minute
(crontab -l 2>/dev/null; echo "* * * * * /usr/local/bin/ollama-metrics") | crontab -

---

Terminal Customization

Bash Configuration

# Add to ~/.bashrc
cat >> ~/.bashrc << 'EOF'

# Ollama aliases
alias ai="ollama run llama3.2"
alias ai-code="ollama run codellama"
alias ai-list="ollama list"
alias ai-ps="ollama ps"
alias ai-rm="ollama rm"
alias ai-pull="ollama pull"
alias ai-status="systemctl status ollama"
alias ai-logs="journalctl -u ollama -f"
alias ai-monitor="watch -n 1 'ollama ps && echo && free -h && echo && nvidia-smi'"

# Functions
function ai-ask() {
    if [ -z "$1" ]; then
        echo "Usage: ai-ask 'your question'"
        return 1
    fi
    echo "$1" | ollama run llama3.2
}

function ai-explain() {
    if [ -z "$1" ]; then
        echo "Usage: ai-explain <file>"
        return 1
    fi
    cat "$1" | ollama run codellama "Explain this code:"
}

function ai-review() {
    if [ -z "$1" ]; then
        echo "Usage: ai-review <file>"
        return 1
    fi
    cat "$1" | ollama run codellama "Review this code for bugs and improvements:"
}

function ai-translate() {
    if [ -z "$1" ] || [ -z "$2" ]; then
        echo "Usage: ai-translate 'text' 'target language'"
        return 1
    fi
    echo "$1" | ollama run llama3.2 "Translate to $2:"
}

function ai-model-info() {
    if [ -z "$1" ]; then
        echo "Available models:"
        ollama list
        return 0
    fi
    ollama show "$1"
}

# Model size checker
function ai-model-size() {
    echo "Model storage usage:"
    du -h ~/.ollama/models/* 2>/dev/null | sort -hr
    echo
    echo "Total:"
    du -sh ~/.ollama/models/ 2>/dev/null
}

# Quick setup for new models
function ai-setup() {
    echo "🤖 Setting up Local AI environment..."
    echo "Available popular models:"
    echo "1. llama3.2:3b    - Fast, good for general use"
    echo "2. llama3.1:8b    - Balanced performance and quality"
    echo "3. codellama:7b   - Best for programming"
    echo "4. mistral:7b     - Alternative to Llama"
    echo "5. phi3:mini      - Smallest, fastest"

    read -p "Enter model name or number (1-5): " choice
    case $choice in
        1) ollama pull llama3.2:3b ;;
        2) ollama pull llama3.1:8b ;;
        3) ollama pull codellama:7b ;;
        4) ollama pull mistral:7b ;;
        5) ollama pull phi3:mini ;;
        *) ollama pull "$choice" ;;
    esac
}

EOF

source ~/.bashrc

Zsh Configuration

# For Zsh users (oh-my-zsh)
cat >> ~/.zshrc << 'EOF'

# Ollama plugin
plugins=(... ollama)

# Custom prompt with AI status
function ai_status() {
    if systemctl is-active --quiet ollama; then
        echo "🤖"
    else
        echo "💤"
    fi
}

# Add to prompt
RPROMPT='$(ai_status)'

# Same aliases and functions as bash
# (copy from bash section above)

EOF

Fish Shell Configuration

# Fish shell configuration
mkdir -p ~/.config/fish/functions

# Create AI functions
cat > ~/.config/fish/functions/ai.fish << 'EOF'
function ai
    ollama run llama3.2 $argv
end
EOF

cat > ~/.config/fish/functions/ai-status.fish << 'EOF'
function ai-status
    if systemctl is-active --quiet ollama
        echo "🤖 Ollama is running"
        ollama ps
    else
        echo "💤 Ollama is stopped"
    end
end
EOF

---

Security Considerations

Firewall Configuration

# UFW (Ubuntu/Debian)
sudo ufw enable
sudo ufw allow 22/tcp                    # SSH
sudo ufw allow from 192.168.1.0/24 to any port 11434  # Local network only

# FirewallD (Fedora/CentOS)
sudo firewall-cmd --permanent --add-service=ssh
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.1.0/24" port protocol="tcp" port="11434" accept'
sudo firewall-cmd --reload

# iptables (manual)
sudo iptables -A INPUT -p tcp --dport 22 -j ACCEPT
sudo iptables -A INPUT -s 192.168.1.0/24 -p tcp --dport 11434 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 11434 -j DROP

SSL/TLS Configuration

# Generate self-signed certificate
sudo mkdir -p /etc/ollama/ssl
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048     -keyout /etc/ollama/ssl/ollama.key     -out /etc/ollama/ssl/ollama.crt     -subj "/C=US/ST=State/L=City/O=Organization/CN=localhost"

# Ollama itself has no TLS settings: keep it on localhost
sudo systemctl edit ollama
# Add:
[Service]
Environment="OLLAMA_HOST=127.0.0.1:11434"

# ...and terminate TLS in a reverse proxy in front of it (nginx example)
sudo apt install nginx
sudo tee /etc/nginx/sites-available/ollama > /dev/null <<'EOF'
server {
    listen 8443 ssl;
    ssl_certificate     /etc/ollama/ssl/ollama.crt;
    ssl_certificate_key /etc/ollama/ssl/ollama.key;
    location / {
        proxy_pass http://127.0.0.1:11434;
        proxy_set_header Host localhost:11434;
    }
}
EOF
sudo ln -s /etc/nginx/sites-available/ollama /etc/nginx/sites-enabled/
sudo systemctl reload nginx

Ollama's API has no TLS or authentication of its own, so anything exposed beyond localhost should sit behind a proxy like this (or Caddy, which obtains certificates automatically).

Access Control

# Create API key authentication script
sudo tee /usr/local/bin/ollama-auth-proxy > /dev/null <<'EOF'
#!/bin/bash

# Simple API key authentication proxy
VALID_KEY="your-secure-api-key-here"
OLLAMA_URL="http://localhost:11434"

while IFS= read -r line; do
    if [[ "$line" == *"Authorization: Bearer $VALID_KEY"* ]]; then
        # Forward request to Ollama
        curl -s -X POST "$OLLAMA_URL/api/generate"             -H "Content-Type: application/json"             -d "$REQUEST_BODY"
        exit 0
    fi
done

echo "Unauthorized" | nc -l -p 11435
EOF

sudo chmod +x /usr/local/bin/ollama-auth-proxy

---

Troubleshooting Guide

Service Issues

Issue: Service won't start

# Check service status
sudo systemctl status ollama -l

# Check for port conflicts
sudo netstat -tlnp | grep 11434
sudo lsof -i :11434

# Check permissions
ls -la /usr/local/bin/ollama
sudo -u ollama /usr/local/bin/ollama serve  # Test as service user

# Reset service
sudo systemctl stop ollama
sudo systemctl reset-failed ollama
sudo systemctl start ollama

Issue: Permission denied errors

# Fix ownership
sudo chown -R ollama:ollama /usr/share/ollama
sudo chmod 755 /usr/local/bin/ollama

# Check SELinux (CentOS/RHEL/Fedora)
sestatus
sudo setsebool -P httpd_can_network_connect 1
sudo semanage port -a -t http_port_t -p tcp 11434

GPU Issues

Issue: GPU not detected

# Check GPU
lspci | grep -i vga
lspci | grep -i nvidia

# NVIDIA troubleshooting
nvidia-smi
sudo dmesg | grep nvidia

# Check driver loading
lsmod | grep nvidia

# Reinstall the recommended driver (Ollama needs 550+)
sudo apt purge 'nvidia-*'
sudo ubuntu-drivers install
sudo reboot

# AMD GPU troubleshooting
rocm-smi
groups ollama   # should include render and video

# See which GPU library Ollama loaded
journalctl -u ollama -e | grep -i -E 'gpu|cuda|rocm|vulkan'
ollama ps

Issue: Out of memory errors

# Check memory usage
free -h
sudo dmesg | grep -i "killed process"

# Use a smaller model
ollama pull llama3.2:1b   # 1.3GB download

# Increase swap
sudo fallocate -l 8G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Network Issues

Issue: Can't download models

# Check connectivity
curl -I https://ollama.com
nslookup ollama.com

# Check proxy settings
echo $http_proxy $https_proxy

# Behind a proxy: set HTTPS_PROXY for the service (Ollama FAQ)
sudo systemctl edit ollama
# Add:
[Service]
Environment="HTTPS_PROXY=https://proxy.example.com:3128"

# Or import a GGUF you downloaded another way
echo 'FROM ./model.gguf' > Modelfile
ollama create my-model -f Modelfile

Performance Issues

Issue: Slow inference

# Check CPU governor
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sudo cpupower frequency-set -g performance

# Check thermal throttling
sudo dmesg | grep -i thermal
sensors  # Install lm-sensors

# Monitor during inference
htop &
ollama run llama3.2 "test"

# Optimize model loading
export OLLAMA_NUM_PARALLEL=1
export OLLAMA_MAX_LOADED_MODELS=1

---

Distribution-Specific Tips

Ubuntu/Debian Specific

# Ollama is not in the Debian/Ubuntu archives: update by re-running the script
curl -fsSL https://ollama.com/install.sh | sh

# Multiple CUDA versions (only if you installed the toolkit)
sudo update-alternatives --install /usr/local/cuda cuda /usr/local/cuda-12.3 123
sudo update-alternatives --config cuda

Fedora/RHEL Specific

# NVIDIA drivers come from RPM Fusion (see rpmfusion.org/Howto/NVIDIA)
sudo dnf install akmod-nvidia

# SELinux: label a manually installed binary
sudo semanage fcontext -a -t bin_t "/usr/local/bin/ollama"
sudo restorecon -v /usr/local/bin/ollama

Arch Linux Specific

# Pick the build that matches your GPU (official extra repo)
sudo pacman -S ollama-cuda    # or ollama-rocm / ollama-vulkan

# Kernel parameters
sudo tee -a /etc/default/grub << 'EOF'
GRUB_CMDLINE_LINUX="nvidia-drm.modeset=1"
EOF
sudo grub-mkconfig -o /boot/grub/grub.cfg

---

Frequently Asked Questions

Q: Which Linux distribution is best for local AI?

A: Ubuntu LTS offers the best hardware support and community resources. Arch Linux packages Ollama officially (with CUDA, ROCm and Vulkan builds) but requires more maintenance. Fedora is excellent for development with good balance of stability and modern packages.

Q: How much RAM do I need for Linux local AI?

A: Size by the download: Llama 3.2 3B is 2.0GB, Llama 3.1 8B is 4.9GB, Qwen3 14B is 9.3GB and Qwen3 32B is 20GB in Ollama. In practice 8GB of RAM runs 1B-3B models, 16GB runs 7B-8B, and 14B-32B models want 32GB+ or a 16-24GB GPU. A headless install leaves more memory free for the model.

Q: Can I run multiple AI models simultaneously?

A: Yes, with sufficient RAM. Use OLLAMA_MAX_LOADED_MODELS to control how many models stay in memory. Each model consumes its full size in RAM.

Q: Should I use Docker or native installation?

A: Native installation is simpler and gets GPU access with no extra setup. Docker is better for isolation, pinned versions and Compose stacks with Open WebUI; it needs the NVIDIA Container Toolkit (or the ROCm image) for GPU access.

Q: How do I optimize for specific hardware?

A: First make sure the whole model is on the GPU (ollama ps should say 100% GPU). Then measure with ollama run <model> --verbose before and after any tuning (CPU governor, swappiness). Monitor with htop, nvtop and nvidia-smi to find the bottleneck.

---

Conclusion

Linux provides the ultimate platform for local AI with unmatched flexibility, performance, and control. From Ubuntu's user-friendly approach to Arch's cutting-edge packages, you can optimize every aspect of your AI setup for maximum performance.

The combination of proper GPU drivers, systemd service management, and shell customization creates a powerful local AI environment that can rival cloud-based solutions while maintaining complete privacy and control.

Remember to regularly update your installation (sudo apt update && sudo apt upgrade or equivalent) and monitor system resources to ensure optimal performance.

---

Ready to master Linux for AI? Join our newsletter for weekly Linux AI optimization tips and advanced configuration guides, or explore our Linux AI course for enterprise-level setups.