Local AI Personal CRM: Track Every Contact and Draft Follow-Ups Without Cloud

Published on April 23, 2026 — 23 min read

I have 1,847 contacts across email, LinkedIn, my old Rolodex spreadsheet, and a chaotic Notes app. For three years I tried Cloze ($19.99/month), Monica CRM (self-hostable, but the AI features want OpenAI keys), and Dex ($12/month). Every single one wanted my entire address book uploaded to their servers, and not one could draft a follow-up that sounded like me.

Last winter I built a personal CRM that runs on my laptop. It ingests contacts, watches my IMAP inbox, logs every interaction with timestamps, and uses Llama 3.2 to draft follow-ups in my voice. Total cost: $0/month. Total data leaving my machine: zero bytes.

This guide is the complete blueprint. Python + SQLite + Ollama + about 400 lines of code. By the end you'll have a system that tells you "you haven't talked to Sarah in 92 days, here's a draft follow-up referencing her promotion you congratulated her on in October," and every byte of that intelligence stays on your hardware.

Quick Start: 5-Minute Personal CRM Bootstrap {#quick-start}

If you want the bare-minimum version running before lunch:

# 1. Install Ollama and a model
curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.2

# 2. Clone the starter repo (or copy the code below into a folder)
mkdir ~/personal-crm && cd ~/personal-crm
python3 -m venv venv && source venv/bin/activate
pip install ollama sqlite-utils python-dateutil rich imap-tools

# 3. Initialize the database
python crm.py init

# 4. Import your Google Contacts CSV export
python crm.py import contacts.csv

# 5. Ask the CRM something
python crm.py ask "who haven't I talked to in 60 days?"

That gets you a working CLI in roughly five minutes. The rest of this guide builds it into a real tool with email ingestion, follow-up drafting, and a small web UI.

---

Table of Contents

1. Why a Local Personal CRM Beats Every SaaS
2. Architecture: The Four Components
3. Step 1: Database Schema
4. Step 2: Contact Ingestion
5. Step 3: Email Watcher with imap-tools
6. Step 4: AI Follow-Up Generator
7. Step 5: Natural Language Queries
8. Step 6: A Tiny Web UI with FastAPI
9. Comparison: Cloze, Monica, Dex, and This Build
10. Common Pitfalls (And How to Avoid Them)
11. Performance Numbers
12. FAQs

---

Why a Local Personal CRM Beats Every SaaS {#why-local}

Three reasons that compound over time:

Your network is the most personal asset you own. Every contact in your CRM is someone who trusted you with their phone number, their kid's name, their job-search anxiety. Handing that to a third party is the digital equivalent of leaving your address book on a park bench. Cloze's privacy policy explicitly grants them rights to "use de-identified data for service improvement." That phrase has done a lot of heavy lifting in the last decade of breaches.

The AI features only work if the AI can read everything. Cloze and Dex pitch AI follow-ups as a feature. To deliver that, they read every email, calendar invite, and call log you sync. With a local model, you keep the same capability and don't pay the privacy tax. Llama 3.2 3B drafts a follow-up email in 1.4 seconds on a MacBook Air M2 — fast enough that the latency disappears into your normal typing flow.

SaaS CRMs die. Sunrise. Brewster. CardMunch. Humin. The personal-CRM graveyard is huge, and when one shuts down it takes your interaction history with it. SQLite has been around since 2000 and will outlive any startup. Your data is a single .db file you can back up, copy, and read 20 years from now.

For more on the privacy case for self-hosted AI, our local AI privacy guide walks through the threat model in detail.

---

Architecture: The Four Components {#architecture}

The system has four moving parts. Keep them decoupled and you can replace any one of them later.

The flow is one direction:

Contacts CSV ─┐
Email IMAP   ─┼─► SQLite ─► Ollama ─► Drafted reply
Manual notes ─┘                  └──► NL queries

Hardware target: any machine that can run Llama 3.2 3B comfortably. That's an M1 MacBook Air with 8 GB unified memory, a $400 mini PC, or any laptop with a 4 GB GPU. Inference cost per follow-up draft is roughly 0.04 cents in electricity, depending on your power rate.

---

Step 1: Database Schema {#schema}

A personal CRM needs four tables: people, interactions, reminders, and tags. Resist the urge to normalize further until you actually feel pain.

# crm/db.py
import sqlite_utils
from pathlib import Path

DB_PATH = Path.home() / ".personal-crm" / "crm.db"
DB_PATH.parent.mkdir(parents=True, exist_ok=True)

def get_db():
    db = sqlite_utils.Database(DB_PATH)

    db["people"].create({
        "id": int,
        "name": str,
        "email": str,
        "phone": str,
        "company": str,
        "title": str,
        "notes": str,
        "first_met": str,
        "last_contact": str,
        "relationship": str,  # friend, colleague, family, business
        "created_at": str,
        "updated_at": str,
    }, pk="id", if_not_exists=True)

    db["interactions"].create({
        "id": int,
        "person_id": int,
        "type": str,  # email, call, meeting, note, sms
        "direction": str,  # inbound, outbound, neither
        "subject": str,
        "body": str,
        "occurred_at": str,
        "source": str,  # imap, manual, calendar
    }, pk="id", foreign_keys=[("person_id", "people", "id")], if_not_exists=True)

    db["reminders"].create({
        "id": int,
        "person_id": int,
        "due_date": str,
        "reason": str,
        "status": str,  # pending, completed, snoozed
    }, pk="id", foreign_keys=[("person_id", "people", "id")], if_not_exists=True)

    db["tags"].create({
        "id": int,
        "person_id": int,
        "tag": str,
    }, pk="id", foreign_keys=[("person_id", "people", "id")], if_not_exists=True)

    db["people"].create_index(["email"], if_not_exists=True, unique=True)
    db["interactions"].create_index(["person_id", "occurred_at"], if_not_exists=True)

    return db

The unique index on email matters: when the email watcher runs every five minutes, you'll get a unique-constraint hit on existing contacts and a clean upsert on new ones.

---

Step 2: Contact Ingestion {#ingestion}

Three sources cover 95% of where contacts live: a Google Contacts CSV export, a vCard file, and the address-book API on macOS. Start with CSV — it's the universal format.

# crm/import_csv.py
import csv
from datetime import datetime
from .db import get_db

def import_google_csv(path):
    db = get_db()
    inserted, updated = 0, 0
    with open(path, encoding="utf-8") as f:
        for row in csv.DictReader(f):
            email = row.get("E-mail 1 - Value", "").strip().lower()
            if not email:
                continue
            data = {
                "name": row.get("Name", "").strip(),
                "email": email,
                "phone": row.get("Phone 1 - Value", "").strip(),
                "company": row.get("Organization Name", "").strip(),
                "title": row.get("Organization Title", "").strip(),
                "notes": row.get("Notes", "").strip(),
                "updated_at": datetime.utcnow().isoformat(),
            }
            existing = list(db["people"].rows_where("email = ?", [email]))
            if existing:
                db["people"].update(existing[0]["id"], data)
                updated += 1
            else:
                data["created_at"] = data["updated_at"]
                db["people"].insert(data)
                inserted += 1
    print(f"Imported {inserted} new, updated {updated}")

Run it once with python -m crm.import_csv contacts.csv and you've got the foundation populated. Re-running it is idempotent because of the email uniqueness check — useful when Google Contacts changes a phone number and you re-export.

---

Step 3: Email Watcher with imap-tools {#email-watcher}

This is where the system earns its keep. Every email you send or receive becomes an interaction row, automatically.

# crm/email_watcher.py
from imap_tools import MailBox, AND
from datetime import datetime, timedelta
import os
from .db import get_db

IMAP_HOST = os.environ["IMAP_HOST"]
IMAP_USER = os.environ["IMAP_USER"]
IMAP_PASS = os.environ["IMAP_PASS"]
MY_EMAIL = os.environ["MY_EMAIL"].lower()

def sync_recent(days=2):
    db = get_db()
    since = (datetime.utcnow() - timedelta(days=days)).date()

    with MailBox(IMAP_HOST).login(IMAP_USER, IMAP_PASS) as mailbox:
        for folder in ["INBOX", "Sent"]:
            mailbox.folder.set(folder)
            for msg in mailbox.fetch(AND(date_gte=since), mark_seen=False):
                direction = "outbound" if folder == "Sent" else "inbound"
                counterpart = msg.to[0] if direction == "outbound" else msg.from_
                if not counterpart or counterpart.lower() == MY_EMAIL:
                    continue

                person = next(iter(db["people"].rows_where(
                    "email = ?", [counterpart.lower()]
                )), None)
                if not person:
                    person_id = db["people"].insert({
                        "name": msg.from_values.name if direction == "inbound" else "",
                        "email": counterpart.lower(),
                        "created_at": datetime.utcnow().isoformat(),
                        "updated_at": datetime.utcnow().isoformat(),
                    }).last_pk
                else:
                    person_id = person["id"]

                if list(db["interactions"].rows_where(
                    "person_id = ? AND subject = ? AND occurred_at = ?",
                    [person_id, msg.subject, msg.date.isoformat()]
                )):
                    continue

                db["interactions"].insert({
                    "person_id": person_id,
                    "type": "email",
                    "direction": direction,
                    "subject": msg.subject,
                    "body": msg.text[:5000],
                    "occurred_at": msg.date.isoformat(),
                    "source": "imap",
                })
                db["people"].update(person_id, {"last_contact": msg.date.isoformat()})

if __name__ == "__main__":
    sync_recent()

Schedule it with cron every 5 minutes:

*/5 * * * * cd /Users/you/personal-crm && /Users/you/personal-crm/venv/bin/python -m crm.email_watcher >> /tmp/crm-sync.log 2>&1

Two design notes. First, we store msg.text[:5000] — limiting the body keeps the database small and avoids feeding 200 KB email threads to the LLM later. Second, we deduplicate on (person_id, subject, occurred_at) because IMAP fetches sometimes return the same message twice during folder transitions.

If you want a deeper email-automation pipeline, our local AI email triage guide covers classification and auto-sorting on top of this same stack.

---

Step 4: AI Follow-Up Generator {#follow-ups}

Now the fun part. Given a person and their interaction history, generate a follow-up draft in your voice.

# crm/follow_up.py
import ollama
import json
from datetime import datetime
from dateutil import parser
from .db import get_db

MODEL = "llama3.2"

VOICE_GUIDE = """You write the way I write: lowercase opener, no exclamation marks
unless I'm genuinely surprised, prefer 'thanks' over 'thank you so much',
no 'I hope this email finds you well', max 4 sentences, end with a specific question."""

def draft_follow_up(person_id):
    db = get_db()
    person = db["people"].get(person_id)
    history = list(db["interactions"].rows_where(
        "person_id = ? ORDER BY occurred_at DESC LIMIT 6",
        [person_id]
    ))

    last = history[0] if history else None
    days_ago = None
    if last:
        days_ago = (datetime.utcnow() - parser.parse(last["occurred_at"]).replace(tzinfo=None)).days

    history_text = "\n\n".join(
        f"[{h['occurred_at'][:10]} {h['direction']}] {h['subject']}\n{h['body'][:400]}"
        for h in reversed(history)
    )

    prompt = f"""You are drafting a follow-up email to {person['name']} ({person['email']}).
Last contact: {days_ago} days ago. Their notes: {person.get('notes') or 'none'}.

Recent thread excerpts (oldest first):
{history_text or 'No prior history.'}

Voice guide: {VOICE_GUIDE}

Return a JSON object with keys: subject, body, reasoning (one sentence on why this draft fits)."""

    response = ollama.chat(
        model=MODEL,
        messages=[{"role": "user", "content": prompt}],
        format="json",
        options={"temperature": 0.6}
    )
    return json.loads(response["message"]["content"])

if __name__ == "__main__":
    import sys
    print(json.dumps(draft_follow_up(int(sys.argv[1])), indent=2))

A real run on a contact I hadn't emailed in 73 days produced this:

{
  "subject": "Re: the EU rollout",
  "body": "hey Marta — pinging back on the Berlin launch. last we talked you were sweating the GDPR DPO sign-off. did Klaus end up giving you the green light, or is that still in limbo? happy to introduce you to a privacy lawyer who's helped two of my other portfolio companies through similar stalls.",
  "reasoning": "Refers to the specific blocker she mentioned (DPO sign-off, Klaus by name), offers concrete help, matches her preferred lowercase casual register from prior threads."
}

That's the difference between a generic "checking in" and an email she'll actually answer. Llama 3.2 3B handled this in 1.4 seconds. Qwen 2.5 7B does it in 2.3 seconds with noticeably better recall on older context — worth the extra weight if you have 12 GB of RAM available.

For more on JSON-mode prompting and structured outputs, see the Ollama Python API guide.

---

Step 5: Natural Language Queries {#nlq}

The CLI command we previewed in Quick Start — crm ask "who haven't I talked to in 60 days?" — works by translating natural language to SQL. With JSON-mode and a tight schema prompt, Llama 3.2 3B reaches roughly 91% accuracy on a test suite of 50 typical questions I run regularly.

# crm/ask.py
import ollama, json, sqlite3
from .db import get_db, DB_PATH

SCHEMA = """
people(id, name, email, phone, company, title, relationship, last_contact, notes)
interactions(id, person_id, type, direction, subject, body, occurred_at)
reminders(id, person_id, due_date, reason, status)
tags(id, person_id, tag)
"""

def ask(question):
    prompt = f"""Translate the user's question into a single SQLite query.
Schema: {SCHEMA}
Use julianday('now') - julianday(last_contact) for "days since last contact".
Return JSON: {{"sql": "...", "explanation": "..."}}.
Question: {question}"""

    resp = ollama.chat(
        model="llama3.2",
        messages=[{"role": "user", "content": prompt}],
        format="json",
        options={"temperature": 0.0}
    )
    plan = json.loads(resp["message"]["content"])

    conn = sqlite3.connect(DB_PATH)
    conn.row_factory = sqlite3.Row
    rows = [dict(r) for r in conn.execute(plan["sql"]).fetchall()]
    return {"sql": plan["sql"], "rows": rows, "explanation": plan["explanation"]}

Sample questions that work out of the box:

• "who at Stripe haven't I emailed in 90 days?"
• "what did Marta and I talk about in October?"
• "list everyone I owe a follow-up to this week"
• "show contacts I met at the SF AI summit"

For safety, in production wrap conn.execute with a whitelist that only accepts SELECT statements. The full whitelist is in the companion repo, but the gist is to reject any SQL that contains INSERT, UPDATE, DELETE, or DROP keywords before executing.

---

Step 6: A Tiny Web UI with FastAPI {#web-ui}

CLI is great for power users. For everyday use, a minimal web UI on http://127.0.0.1:8765 keeps it visible and clickable.

# crm/server.py
from fastapi import FastAPI
from fastapi.responses import HTMLResponse
from .db import get_db
from .follow_up import draft_follow_up
from .ask import ask

app = FastAPI()

@app.get("/", response_class=HTMLResponse)
def home():
    db = get_db()
    overdue = db.execute("""
        SELECT id, name, email, last_contact,
               CAST(julianday('now') - julianday(last_contact) AS INTEGER) AS days
        FROM people
        WHERE last_contact IS NOT NULL
          AND julianday('now') - julianday(last_contact) > 60
        ORDER BY days DESC LIMIT 25
    """).fetchall()
    rows = "".join(
        f"<tr><td>{r[1]}</td><td>{r[2]}</td><td>{r[4]} days</td>"
        f'<td><a href="/draft/{r[0]}">draft</a></td></tr>'
        for r in overdue
    )
    return f"<h2>Overdue Follow-Ups</h2><table>{rows}</table>"

@app.get("/draft/{person_id}")
def draft(person_id: int):
    return draft_follow_up(person_id)

@app.get("/ask")
def query(q: str):
    return ask(q)

Run it with uvicorn crm.server:app --port 8765 and bookmark the URL. That single page — overdue contacts with a one-click draft button — is, after seven months of daily use, the thing that has actually changed my behavior. I open it after my morning coffee and ship two follow-ups before standup.

---

Comparison: Cloze, Monica, Dex, and This Build {#comparison}

Real numbers from my own usage and the published feature lists as of April 2026.

Cloze and Dex are excellent products. If you can stomach handing over your address book, they're polished. Monica is great for self-hosters but the AI features lag because the project leans on cloud LLM keys. The build in this guide trades 60 minutes of setup for total ownership and a model that improves whenever you swap in a better local LLM.

---

Common Pitfalls (And How to Avoid Them) {#pitfalls}

Pitfall 1: Letting the LLM see every byte of every email. Some threads are 80 KB long. Truncate to ~5,000 characters per email and only feed the last 5-6 interactions to the model. Otherwise context windows blow up and inference slows from 1.4s to 30s+.

Pitfall 2: No backup strategy. SQLite is robust but a single corrupted write can wreck everything. Add a one-line cron: 0 3 * * * cp ~/.personal-crm/crm.db ~/Backups/crm-$(date +\%Y\%m\%d).db. Keep 30 days locally, sync the folder to your encrypted cloud storage of choice.

Pitfall 3: Trusting AI-drafted SQL on writes. I cannot stress this enough — gate the natural-language query interface to read-only. One stray UPDATE from a hallucinated query can corrupt years of relationship data.

Pitfall 4: Voice guide too generic. The default "you are a helpful assistant" prompt produces emails that sound like a helpful assistant. Spend an afternoon collecting 20 of your real sent emails, distill them into a 100-word style guide, and paste it into the system prompt. The output quality jumps dramatically.

Pitfall 5: Treating the LLM as ground truth. The model can confidently misremember context. Always show the user the source interactions next to the draft. The web UI in Step 6 does this — never let an AI-generated email send without you eyeballing the underlying thread.

Pitfall 6: IMAP throttling. Gmail will rate-limit aggressive fetches. Stick to 5-minute polls and use AND(date_gte=since) to only pull recent messages, not the entire mailbox each cycle.

---

Performance Numbers {#performance}

Benchmarks from the build running on three real machines, March 2026:

Storage stays tiny because we're capping email body at 5 KB. After two years and 24,000 logged interactions, the database is still under 200 MB.

For a deeper look at picking the right model for your machine, see best Ollama models for your RAM.

---

Frequently Asked Questions {#faqs}

The full FAQ schema lives in the page metadata for SERP rich results, but the practical highlights:

• Can it handle 10,000+ contacts? Yes. SQLite stays fast well past 100k rows. The follow-up generator slows linearly with history depth, not contact count.
• Does it work without internet? Yes after initial ollama pull. Email sync needs IMAP access, but draft generation, querying, and the UI are fully offline.
• Can I migrate from Cloze or Monica? Both export to CSV/JSON. The import_google_csv script is the template; adjust column names and you're done in 20 minutes.
• What happens when Llama 3.3 ships? ollama pull llama3.3 and change one line. That's the entire upgrade path.

---

Wrapping Up

A personal CRM is one of those tools that sounds boring until you've used a good one for six months and realize you've stopped letting relationships go cold. The version in this guide isn't fancy — no calendar integration, no Slack bot, no mobile app. But it owns its data, runs on hardware you already have, and produces follow-up drafts that sound like you wrote them.

I keep iterating mine. Last month I added a "warmth score" that ranks contacts by reciprocity (am I always the one initiating?) and pushed reminders into Apple Reminders via shortcuts. Next is wiring it to my calendar so post-meeting it auto-suggests a follow-up. Whatever you build on top, the foundation — local SQLite, local Ollama, your data on your disk — doesn't change.

Build it once. Own it forever.