Local AI for Accountants: Private Tax & Ledger Analysis

Short answer: a self-hosted model running on a workstation in your own office removes the third-party disclosure question from AI entirely — there is no processor to get consent for, no vendor DPA, and nothing to put in a SOC 2 vendor inventory. The practical stack is Ollama plus a 14B-class instruct model for reasoning, an embedding model for retrieval over your own memo library, and deterministic code for anything that produces a number.

That last clause is the whole discipline. A language model is a drafting and triage tool for an accounting practice. It should classify, summarise, extract and flag. It should never be the thing that computes a figure that lands on a financial statement.

Quick Start: A Working Local Stack in About 10 Minutes

On Windows 11 or macOS with 16 GB of RAM or more:

1. Install Ollama: curl -fsSL https://ollama.com/install.sh | sh (Mac/Linux) or the Windows installer.
2. Pull a reasoning model: ollama pull qwen2.5:14b-instruct-q4_K_M.
3. Pull an embedding model for document retrieval: ollama pull nomic-embed-text.
4. Confirm it is not listening beyond the machine: lsof -i -P -n | grep ollama should show only 127.0.0.1:11434.
5. Sanity-check it: ollama run qwen2.5:14b-instruct "List the schedules attached to a Form 1040."

The rest of this guide is about turning that into something a practice can actually rely on in March.

Table of Contents

1. Why does local AI belong in an accounting practice?
2. What do 7216, GLBA, AICPA and GDPR actually require?
3. What hardware do you need?
4. How fast will it be, and will the model fit?
5. Which models should a CPA firm run?
6. How do you build a private tax memo assistant?
7. How do you review a ledger without the cloud?
8. How do you handle 1099s, K-1s and receipts?
9. What goes wrong?
10. FAQ for partners and IT leads

---

Why Does Local AI Belong in an Accounting Practice?

Three things changed at roughly the same time.

1. The disclosure question is unavoidable. Section 7216 of the Internal Revenue Code carries criminal penalties for the unauthorised disclosure of taxpayer information, and "tax return information" is defined far more broadly than most preparers assume — it covers material used in preparing a return, which includes the working notes somebody might paste into a chatbot. Pasting return information into a consumer service that retains inputs is, on a plain reading, a disclosure to a third party. Enterprise agreements with no-training and no-retention terms change that analysis materially, but they sit behind contracts small firms often cannot obtain.

2. Clients have started asking. "Where does my data go?" is a question a cloud tool answers with a policy document. A locally hosted model answers it with a network configuration you can demonstrate on screen.

3. The capability gap narrowed. The open-weight 14B and 32B instruct models now available handle domain terminology, structured extraction and summarisation well enough for first-draft work scoped to a single engagement. They are not equivalent to frontier cloud models on hard reasoning, and this page does not pretend otherwise — but first-draft memo work and document triage are not hard reasoning.

For the economics, our local vs cloud AI cost analysis works through the comparison in detail; the arithmetic sketch is in the hardware section below.

---

What Do 7216, GLBA, AICPA and GDPR Actually Require?

Local AI does not make you compliant. It removes an entire category of question from the analysis, which is a different and more useful thing.

The IRS Section 7216 Information Center is the canonical source on the scope of "tax return information", and the FTC's Safeguards Rule guidance is the plain-English version of what the GLBA expects from a firm your size. Read both before writing your firm's AI policy — none of the above is legal advice.

Our GDPR-compliant local AI guide and SOC 2 for self-hosted AI cover what auditors want in the evidence binder.

---

What Hardware Do You Need?

Two numbers determine whether a machine is adequate: how much memory it has (does the model fit?) and how fast that memory is (how quickly does it generate?). Everything else is secondary.

Prices move constantly and are not something an article should freeze, so check current list prices with the vendor. What is stable is the shape of the comparison, and it is worth doing yourself before you buy:

Cloud seat cost per year  = seats x monthly list price x 12
Local cost per year (yr1) = hardware + a few hours of setup labour
Breakeven months          = hardware cost / (seats x monthly list price)

Worked through with round illustrative numbers: four seats at $25 a month is $1,200 a year, so a $1,500 workstation reaches breakeven in fifteen months and everything after that is electricity. Substitute your own seat count and your vendor's current published price — the answer swings a long way with both, which is exactly why a single headline payback figure would mislead more than it helps.

What to avoid

• 8 GB machines. A 14B at Q4 needs roughly 8.4 GB for weights alone before context, so it will not fit alongside your tax software. See the sizing arithmetic.
• Old rackmount servers. They are cheap to buy and expensive to run continuously, and a 14-week busy season is a long time to pay for idle draw.
• Anything relying on shared VRAM under 8 GB. The model spills to system memory and generation speed falls to the slow side of the table below.

---

How Fast Will It Be, and Will the Model Fit?

You do not need anyone's benchmark to answer this. Two formulas cover it.

Will it fit?

weights at Q4_K_M (GB) ~= 0.6 x parameters in billions

So a 7B is about 4.2 GB, a 14B about 8.4 GB, a 32B about 19 GB, a 70B about 42 GB. Add 1-2 GB for context and runtime at ordinary lengths, considerably more for a 50-page document in one pass.

How fast can it possibly be?

tokens/sec ceiling = memory bandwidth (GB/s) / model size in memory (GB)

Apply it with the manufacturer's published bandwidth figures:

These are ceilings, not forecasts. They assume the hardware saturates its memory bus and nothing else is competing for it — which is never true on a machine that is also running tax software and four PDF viewers. Real output lands below, and the gap is widest on CPU-only machines.

What the table does tell you, reliably, is the ordering: a 14B on a discrete GPU generates faster than most people read, and a 32B on CPU-only memory does not. If a preparer is going to sit and watch the output stream, that difference decides whether the tool gets used. To get your real number, run any prompt with --verbose and read the eval rate Ollama prints.

---

Which Models Should a CPA Firm Run?

Four roles, four models. Pull all of them; they coexist happily.

ollama pull qwen2.5:14b-instruct-q4_K_M
ollama pull qwen2.5:32b-instruct-q4_K_M   # only if you have 24 GB of VRAM
ollama pull nomic-embed-text
ollama pull qwen2.5-coder:14b

Model releases move faster than this page does. Before standardising on any of them, run your own calibration engagement — described in the memo section — because the only quality measurement that matters for your firm is one taken on your firm's work product. Anyone publishing a domain accuracy percentage for accounting tasks should be able to show you the test set; if they cannot, treat the number as decoration.

For a broader survey of what runs on what, see AI models for 16GB RAM.

---

How Do You Build a Private Tax Memo Assistant?

The pattern is retrieval-augmented generation over your own prior work product, with a hard instruction to refuse when the retrieval comes back thin.

Step 1: Set up the document store

# AnythingLLM as a single-tenant RAG layer
docker run -d -p 3001:3001 \
  --cap-add SYS_ADMIN \
  -v anythingllm-storage:/app/server/storage \
  --name anythingllm \
  mintplexlabs/anythingllm:latest

# In settings: LLM Provider = Ollama, host = http://host.docker.internal:11434
# Embedder = Ollama, model = nomic-embed-text

Step 2: Ingest the memo library

Create a workspace per client engagement, not per topic. Into each one:

• Internal memos from prior years for that client
• The engagement letter
• The current working trial balance
• Any relevant IRS publications — these are public documents, so they carry no confidentiality constraint

Step 3: The prompt template

You are a senior tax preparer at our firm drafting an internal memo
for the partner-in-charge. Use only the supplied context. If the
context is insufficient, say "INSUFFICIENT - need [X]" and stop.

Format:
1. ISSUE (one sentence)
2. FACTS (bullet list, only from context)
3. AUTHORITIES (cite IRC sections, regs, or our prior memos)
4. ANALYSIS (3-5 sentences)
5. RECOMMENDATION (one sentence, conservative)
6. OPEN QUESTIONS (numbered, for partner review)

Client question: {{user_question}}

The INSUFFICIENT - need [X] clause is the most important line in the template. Without an explicit refusal path, a model whose retrieval missed will fill the gap with plausible invention — the classic failure being a citation to a document that does not exist. With one, the failure is loud, and a loud failure is a fixable failure.

Step 4: The quality gate

Every memo is reviewed by a human before it goes anywhere. The tool is a first-draft accelerator, not a preparer.

Before rolling it out, run a calibration engagement: pick a closed engagement, generate the memo the firm already wrote, and compare. That comparison is your own quality measurement, on your own work, and it is worth more than any published accuracy figure. Time both the AI-assisted and the original process while you are at it — that is how you find out what the tool is actually worth to your practice rather than inheriting somebody else's number.

---

How Do You Review a Ledger Without the Cloud?

This is the use case partners understand immediately: hand the model a general ledger export and ask what looks unusual.

The workflow

1. Export the GL as CSV from your accounting package.
2. Strip the entity name and EIN from the header — defence in depth, since it costs nothing.
3. Feed it to the 14B with a specific, conservative prompt:

You are a forensic accountant reviewing a general ledger for unusual
entries. Look for:
- Round-number transactions over $1,000 (manual entries)
- Duplicate amounts within 7 days to the same vendor
- Reversing entries without a clear pair
- Expense account entries posted on weekends
- Any vendor name that appears only once in the year over $5,000

Output a table with: Date, Account, Vendor, Amount, Reason Flagged.
Be conservative. False positives waste partner time.

CSV:
{{ledger_csv}}

Two constraints worth knowing before you rely on it. A long ledger has to fit in the context window, so beyond a few thousand rows you are chunking by month or by account. And the model is generating candidates for review, not conclusions — the hit rate on any given ledger depends entirely on that ledger, which is why the prompt tells it to be conservative.

Pair it with code for anything numeric

The model is unreliable at arithmetic. Have it produce the query, and let deterministic code produce the number:

import pandas as pd
df = pd.read_csv("ledger.csv")
# Round-number filter
suspicious = df[(df['amount'] % 1000 == 0) & (df['amount'] >= 1000)]
print(suspicious.to_string())

This split — model as analyst, code as calculator — is the single biggest reliability decision in accounting AI. It is also what makes the output reproducible, which matters when a partner asks how a figure was derived. Our local AI data analyst guide covers the pattern in more depth.

---

How Do You Handle 1099s, K-1s and Receipts?

Busy season is largely a document-translation problem. The pipeline is OCR, then structured extraction, then human verification.

Tools

• Tesseract OCR for character recognition (brew install tesseract or apt install tesseract-ocr)
• A vision-capable model for layout, where OCR alone loses the structure: ollama pull llama3.2-vision:11b
• A short Python script to join them

The pipeline

import subprocess
import json
from ollama import Client

ocr_text = subprocess.run(
    ["tesseract", "client_1099.pdf", "-", "-l", "eng"],
    capture_output=True, text=True
).stdout

client = Client(host="http://localhost:11434")
response = client.generate(
    model="qwen2.5:14b-instruct",
    prompt=f"""Extract structured data from this OCR'd 1099-NEC.
Return JSON with: payer_name, payer_tin, recipient_name, recipient_tin,
box1_nonemployee_compensation, tax_year. Use null for missing fields.

OCR text:
{ocr_text}""",
    format="json"
)
print(json.loads(response['response']))

format="json" matters: it constrains the model to emit parseable JSON rather than JSON wrapped in an apology. Instruct it to use null for anything it cannot find, because a null is a flag for human review and an invented TIN is a problem.

Accuracy depends almost entirely on scan quality. Clean digital PDFs extract cleanly; a faxed-and-rescanned form or a handwritten amendment will not, no matter which model you point at it. Build the review step in from the start rather than discovering the failure rate in April. Our local AI invoice processing guide covers the same extract-then-verify pattern for payables.

---

What Goes Wrong?

1. Letting the model do arithmetic. It cannot reliably add a column of numbers. Generate the formula or the code, then execute it. Never publish a figure the model produced directly.

2. Omitting the refusal path. Without an explicit "say INSUFFICIENT and stop" instruction, a model whose retrieval missed will confabulate to fill the gap. Fabricated citations to prior-year documents are the specific failure mode to design against.

3. Mixing engagements in one workspace. One RAG index spanning multiple clients can surface one client's facts while answering about another. Use one workspace per engagement, always.

4. Using real tax IDs while developing prompts. The model is local, but prompt logs, screen shares and screenshots are not necessarily controlled. Use synthetic data while you are iterating.

5. Antivirus rescanning model files. On Windows, Defender will re-scan large model files as they load. Add %USERPROFILE%\.ollama to the exclusion list and time a generation before and after to see what it was costing you.

6. Running a laptop in clamshell mode. Sustained inference is a thermally demanding workload, and a closed laptop throttles. For daily use, prefer a desktop or at least an open lid on a stand.

7. No backup of the model directory. ~/.ollama/models runs to tens of gigabytes. Re-downloading it during busy season on whatever connection is available is a bad afternoon. Our local AI backup and recovery guide covers the approach.

8. No audit trail. If you intend to claim SOC 2 coverage over the workflow, you need prompts and responses logged from day one, not retrofitted. See local AI audit trail.

---

FAQ for Partners and IT Leads

Is using a consumer cloud chatbot a violation of IRC 7216?

For most consumer plans, the conservative reading is that pasting return information into a service that retains inputs for training is a disclosure to a third party without taxpayer consent. Enterprise agreements with contractual no-training and no-retention terms change the analysis substantially, but small firms frequently cannot obtain those tiers. Self-hosting avoids the question rather than answering it. Confirm your own position with counsel — this is not legal advice.

What hardware does a four-person firm need?

Enough memory to hold the model, and preferably fast memory. A 14B at Q4 needs roughly 8.4 GB for weights, so a 32 GB machine gives comfortable room for the model, the embedding index and everything else the machine is doing. Add a 24 GB GPU when you want 32B long-document review or several preparers hitting the endpoint at once.

Which open model is best for accounting work?

The 14B instruct class is the sensible default for memo drafting and extraction, stepping up to 32B for long documents. But "best" is firm-specific: run the calibration engagement described above on your own closed files, because that comparison measures the thing you actually care about. Treat any published domain-accuracy percentage without a disclosed test set as marketing.

Can local AI do bookkeeping, or only summarise it?

Let it classify, summarise, extract and flag. Do not let it post journal entries or produce final figures. The reliable pattern is model-as-analyst plus deterministic code as calculator, because a number that reaches a financial statement needs to be computed and reproducible, not generated.

How do I keep client engagements separate?

One workspace, one index, one engagement — in AnythingLLM, LM Studio, or a custom Chroma or Qdrant setup. Never let retrieval span clients. A cross-engagement retrieval is not just a quality problem; under AICPA confidentiality rules it is a real one.

What about audit trails for SOC 2 or partner review?

Log every prompt and response with a user identifier, a timestamp and the model version. Self-hosted Langfuse, OpenTelemetry, or an append-only SQLite table all work. Decide the retention period up front, because prompt logs contain return information and inherit its handling requirements.

How long does setup take?

Plan a day for a competent IT generalist: install and model downloads, then the RAG layer and library ingestion, then prompt-template iteration on anonymised data, then documentation and a short team briefing. The variable that actually drives the schedule is how long partners take to accept the output quality, which is why the calibration engagement comes first.

Will regulators ever audit our use of AI?

Assume the question will be asked, and make the answer boring: a documented model and version, documented prompt templates, retained logs, a written firm AI policy, and human review on everything that reaches a client. Every one of those is easier to evidence for a system you host than for a service you do not control.

---

Where to Go Next

1. Multi-user access. Put nginx with auth and TLS in front of Ollama so preparers share one endpoint — Ollama production deployment covers it end to end.
2. Audit logging. Local AI audit trail has the schema and retention policy.
3. Recurring reporting. The pipeline that flags ledger anomalies also produces monthly close packages — see automated report generation.

For the cost comparison over a three-year horizon, local vs cloud AI cost is the page to send to whoever signs off on the hardware.

---

Conclusion

The profession spent twenty years learning to be careful with client data and about two years working out where AI fits. Self-hosting is the option that lets you take the drafting and triage gains without adding a processor to your vendor inventory, a transfer to your GDPR assessment, or a disclosure to your 7216 analysis.

Set it up, run a calibration engagement, measure the difference on your own files, and keep every number that touches a financial statement in code rather than in a model. When a client asks where their data goes, the answer is a machine in your office — and you can show them.