Automate Invoice Processing with Local AI

Published on April 23, 2026 • 17 min read

Accounts payable is the highest-volume document workflow most small businesses run, and it is still mostly typing. A 70-person construction firm pushing 600 vendor invoices a month spends something like 40 clerk-hours a month on data entry alone, at four minutes an invoice. The obvious fix — a cloud AP automation service — solves the typing and creates a new problem: every subcontractor's pricing, every bank detail, every wage-adjacent line item now lives on somebody else's servers.

This guide removes that trade. The stack is shorter than the average "AI for AP" pitch deck: a vision-language model, a structured-output schema, a validation pass, and an ERP webhook. What follows is how to wire it together — the code, the arithmetic for sizing hardware, and the specific failure modes you should expect on day three.

---

Quick Start: Extract One Invoice in 4 Minutes

# Install Ollama and a vision model
curl -fsSL https://ollama.com/install.sh | sh
ollama pull qwen2.5-vl:7b

# Extract from an invoice scan
curl http://localhost:11434/api/generate -d '{
  "model": "qwen2.5-vl:7b",
  "prompt": "Extract this invoice as JSON with: vendor_name, invoice_number, invoice_date, due_date, total_amount, currency, line_items[{description, qty, unit_price, amount}]. Return ONLY JSON.",
  "images": ["'$(base64 -w0 ./invoice.jpg)'"],
  "format": "json",
  "stream": false
}'

That returns a complete structured extraction on your own hardware, ready to push into any ERP. How long the round trip takes depends almost entirely on your GPU's memory bandwidth — the hardware section below works that out from first principles.

---

Table of Contents

1. Why Cloud Invoice AI Costs You More Than Money
2. The Local Pipeline
3. Hardware and Throughput Targets
4. Document Classification Step
5. Field Extraction with Vision Models
6. Line-Item Normalization
7. Validation and Three-Way Match
8. ERP Integration
9. Pitfalls and Fixes
10. ROI Math
11. FAQs

---

Why Cloud Invoice AI Costs You More Than Money

The cloud invoice automation market is loud — Bill.com, Tipalti, AvidXchange, Stampli, Ramp. Every pitch leads with "AI extraction." None of them lead with what a CFO of a regulated business actually wants to know: who else sees my invoices?

Real concerns I hear from finance teams:

1. Subcontractor pricing leakage. Construction GCs do not want the AI vendor's other customers' models trained on their pricing.
2. Wage data privacy. Payroll-adjacent invoices contain compensation that is restricted under several state laws.
3. Vendor relationship sensitivity. Some vendors require non-disclosure on invoice terms; cloud upload may violate those NDAs.
4. Cross-border concerns. EU vendor invoices flowing through a US AI service is a Schrems II issue waiting for a complaint.
5. Subscription stack inflation. Bill.com Premium runs $79-$169/user/month for the AI tier. A 4-person AP team at the high end is $8,100/year — forever.

Local AI invoice processing solves all of this and produces extraction quality that genuinely matches the cloud vendors I have benchmarked against.

For the broader case, our GDPR-compliant local AI post covers data residency in depth.

---

The Local Pipeline

Email/Scan/Drop folder
        |
        v
+---------------+     +----------------+
|  Ingest       | --> | Classify       |  (invoice / credit memo / statement / other)
+---------------+     +----------------+
                              |
                              v
                      +----------------+
                      |  Extract       |  (Vision LLM -> JSON)
                      +----------------+
                              |
                              v
                      +----------------+
                      |  Normalize     |  (line items -> GL accounts)
                      +----------------+
                              |
                              v
                      +----------------+
                      |  Validate      |  (3-way match + rules)
                      +----------------+
                              |
                              v
                +-------------+ +-------------+
                | Auto-post   | | Human Queue |
                +-------------+ +-------------+

Six stages. The first half is pure AI. The second half is rule-based logic that turns a JSON extraction into a posted bill in your ERP. Splitting them this way is critical: it means an LLM hallucination cannot post incorrect data, because a deterministic validation step sits between extraction and posting.

The toolset:

---

Hardware and Throughput Targets

You can size this workload with arithmetic instead of guesswork. Local decoding is memory-bandwidth bound: every token generated requires streaming the whole model through the GPU once. At Q4_K_M, weights run about 0.6 GB per billion parameters, so qwen2.5-vl:7b is roughly 4.2 GB of weights (plus the vision encoder and the image tiles on top). Divide card bandwidth by that figure and you get the ceiling:

Those are ceilings, not measurements — the number the hardware physically cannot exceed. A full invoice extraction is a few hundred tokens of JSON, so even at the ceiling the decode phase alone is a second or two on a 4090 and several seconds on a 3060, and actual output lands well below the ceiling once image prefill, sampling and context overhead are included. Size with headroom rather than to the ceiling, and time your own mix before you commit to a card.

A common oversight: invoice processing is bursty. Most of the day's invoices arrive between 8-10 AM (vendor send schedules). Size the GPU to handle the burst, not the daily average.

---

Document Classification Step

Not every PDF that lands in your AP inbox is an invoice. Statements, credit memos, marketing collateral, and signed contracts mix in. Classifying first saves the more expensive vision pass.

import requests

CLASSIFY_PROMPT = """Classify this document as one of: invoice, credit_memo, statement, contract, other.
Output ONLY one word from that list."""

def classify_pdf_text(text: str) -> str:
    r = requests.post("http://localhost:11434/api/generate", json={
        "model": "qwen2.5:3b",
        "prompt": f"{CLASSIFY_PROMPT}\n\nDocument:\n{text[:3000]}",
        "stream": False,
        "options": {"temperature": 0.0, "num_predict": 5}
    })
    return r.json()["response"].strip().lower()

A 3B model is plenty for this. Note num_predict: 5 — the classifier emits one word, so generation is nearly free and the whole step costs a short prefill over 3,000 characters of text. That is the point of putting it first: it is an order of magnitude cheaper than the vision pass it protects.

If you want belt-and-suspenders, prepend a regex filter: documents containing both invoice and a currency symbol skip the LLM and go straight to extraction.

---

Field Extraction with Vision Models

The extraction step is where AI earns its keep. Vision-language models read the invoice image directly and output structured JSON. No separate OCR pass required.

The schema

{
  "vendor_name": "string",
  "vendor_address": "string|null",
  "vendor_tax_id": "string|null",
  "invoice_number": "string",
  "invoice_date": "YYYY-MM-DD",
  "due_date": "YYYY-MM-DD|null",
  "po_number": "string|null",
  "subtotal": "number",
  "tax": "number|null",
  "total_amount": "number",
  "currency": "ISO 4217 code, e.g., USD, EUR",
  "line_items": [
    {"description": "string", "quantity": "number|null", "unit_price": "number|null", "amount": "number"}
  ],
  "remit_to_account": "string|null",
  "notes": "string|null"
}

This schema covers what a general-ledger posting actually needs. Add custom fields for specialized industries (job numbers for construction, NDC codes for healthcare).

The extraction prompt

EXTRACT_PROMPT = """You are an accounts payable assistant. Extract this invoice into the
following JSON schema. Output ONLY valid JSON. If a field is not present, use null.

Required fields: vendor_name, invoice_number, invoice_date, total_amount, currency, line_items.
Date format: YYYY-MM-DD. Numbers: no currency symbols, no thousand separators (e.g., 1234.56 not $1,234.56).

Schema:
{ ... full schema above ... }

Return only the JSON, with no commentary."""

import base64, json, requests

def extract_invoice(image_path: str, model: str = "qwen2.5-vl:7b") -> dict:
    with open(image_path, "rb") as f:
        b64 = base64.b64encode(f.read()).decode()
    r = requests.post("http://localhost:11434/api/generate", json={
        "model": model,
        "prompt": EXTRACT_PROMPT,
        "images": [b64],
        "format": "json",
        "stream": False,
        "options": {"temperature": 0.0, "num_predict": 1000}
    }, timeout=120)
    return json.loads(r.json()["response"])

Temperature 0.0 produces deterministic output. format: "json" constrains the output to valid JSON. num_predict: 1000 is enough for a typical 5-15 line-item invoice.

Vision model comparison

There is no public invoice-extraction leaderboard, and we have not run one, so what follows is a shortlist and the sizing arithmetic — not an accuracy ranking. Field accuracy on invoices is dominated by your own document mix (clean PDFs versus 200-DPI faxes versus phone photos), which means a public score would not transfer to your AP inbox anyway.

Weight sizes are parameter count x ~0.6 GB per billion at Q4_K_M — arithmetic, not measured memory ceilings. Budget another 1-2 GB on top for the vision encoder, image tiles and KV cache.

The useful move is to pick two candidates, label 50-100 of your own invoices field by field, and score them against your real mix. That takes an afternoon and tells you something a generic benchmark cannot: which model handles your worst vendors.

For the foundational guide, see our Ollama Python API guide.

---

Line-Item Normalization

Raw extraction gives you line items as the vendor wrote them. ERPs need GL account codes. Mapping between them is the unglamorous part nobody includes in demos.

Vendor-specific mapping

GL_MAP = {
    # Vendor name (lowercased) -> default GL account
    "office depot": "6500-Office Supplies",
    "aws": "6300-Cloud Infrastructure",
    "comcast business": "6100-Internet & Phone",
    "ferguson plumbing": "5050-Materials-Plumbing",
}

def map_to_gl(vendor: str, line_desc: str) -> str:
    key = vendor.lower().strip()
    if key in GL_MAP:
        return GL_MAP[key]
    # Fall back to LLM-based suggestion
    return llm_suggest_gl(line_desc)

For unmapped vendors, ask the LLM:

GL_PROMPT = """You are a bookkeeper. Suggest the most likely GL account for this line item.
Choose ONE from: {accounts_list}. Output only the account code."""

def llm_suggest_gl(line_desc: str, accounts: list[str]) -> str:
    r = requests.post("http://localhost:11434/api/generate", json={
        "model": "qwen2.5:7b",
        "prompt": GL_PROMPT.format(accounts_list=", ".join(accounts)) + f"\n\nLine: {line_desc}",
        "stream": False,
        "options": {"temperature": 0.0, "num_predict": 30}
    })
    return r.json()["response"].strip()

Always log LLM-suggested GL codes for review. The first time the AP clerk sees a wrong suggestion, capture the correction and add it to GL_MAP. The point of that loop is that GL coding is mostly repeat business — a given vendor bills the same category over and over — so every correction you capture converts a guess into a lookup, and the model gets asked less often each month.

---

Validation and Three-Way Match

Extraction is one thing. Verifying the extraction is plausible is another. Three checks worth running on every invoice before it posts:

Check 1: Math validation

def validate_math(invoice: dict) -> list[str]:
    errors = []
    items_total = sum((li.get("amount") or 0) for li in invoice["line_items"])
    subtotal = invoice.get("subtotal") or items_total
    tax = invoice.get("tax") or 0
    expected = round(subtotal + tax, 2)
    actual = round(invoice["total_amount"], 2)
    if abs(expected - actual) > 0.02:
        errors.append(f"Math mismatch: subtotal+tax={expected}, total={actual}")
    return errors

A small but steady share of extractions fail this check — track your own rate, because it is the single best health metric the pipeline produces. Most failures are recoverable by reprompting with the failure detail attached. The unrecoverable ones go to human review.

Check 2: Three-way match (PO + receipt + invoice)

For invoices linked to a purchase order, validate against the PO and the receiving record:

def three_way_match(invoice: dict, po: dict, receipt: dict) -> list[str]:
    errors = []
    if abs(invoice["total_amount"] - po["total_amount"]) / po["total_amount"] > 0.05:
        errors.append("Invoice total exceeds PO by more than 5%")
    if invoice["invoice_date"] < po["po_date"]:
        errors.append("Invoice predates PO")
    received = sum(r["qty"] for r in receipt.get("items", []))
    invoiced = sum((li.get("quantity") or 0) for li in invoice["line_items"])
    if invoiced > received * 1.05:
        errors.append("Invoiced quantity exceeds received quantity")
    return errors

Check 3: Vendor risk rules

RISK_RULES = [
    ("vendor_tax_id is None and total_amount > 600", "1099 vendor missing tax ID"),
    ("currency != 'USD' and country == 'US'", "Non-USD invoice flagged for review"),
    ("total_amount > vendor_avg_total * 3", "Total significantly above vendor average"),
]

Validation is what separates an invoice automation that you trust with auto-posting from one that just shaves seconds off manual review.

---

ERP Integration

The last mile. Integration patterns by ERP:

QuickBooks Online

Use the QBO API. The Bill resource accepts vendor, line items, and AP account.

import requests

def post_qbo_bill(invoice: dict, qbo_token: str, realm_id: str):
    payload = {
        "VendorRef": {"value": resolve_vendor_id(invoice["vendor_name"])},
        "TxnDate": invoice["invoice_date"],
        "DueDate": invoice.get("due_date"),
        "DocNumber": invoice["invoice_number"],
        "Line": [{
            "Amount": li["amount"],
            "DetailType": "AccountBasedExpenseLineDetail",
            "AccountBasedExpenseLineDetail": {
                "AccountRef": {"value": resolve_account(li["gl_code"])}
            },
            "Description": li["description"]
        } for li in invoice["normalized_lines"]],
    }
    r = requests.post(
        f"https://quickbooks.api.intuit.com/v3/company/{realm_id}/bill",
        json=payload,
        headers={"Authorization": f"Bearer {qbo_token}"}
    )
    return r.json()

NetSuite, SAP, Sage

Same pattern, different SOAP/REST endpoints. The translation layer is shallow because all major ERPs accept the same conceptual fields (vendor, date, lines, amount, GL account).

When to keep humans in the loop

Auto-post only when:

• Vendor is known (in your master list for 30+ days)
• Three-way match passes
• Total under your authorization threshold (typical: $5,000 for AP, $25,000 with manager sign-off)
• Math validation passes

Everything else goes to a human queue with the AI's extraction pre-filled. The human accepts, edits, or posts — a review click rather than the four minutes a manual keying costs.

For workflow patterns, our private AI knowledge base post covers the broader pattern of human-in-the-loop AI for business operations.

---

Pitfalls and Fixes

Pitfall 1: Numbers with European decimal separators

Cause: invoice from a German vendor uses 1.234,56 instead of 1,234.56.

Fix: detect locale from currency or vendor country, run a normalization step before validation. The LLM almost always extracts the number correctly; the issue is downstream parsing.

Pitfall 2: Multi-page invoices

Cause: the model only sees the first page.

Fix: convert multi-page PDFs to a vertical stitched image, or run extraction on each page and merge line items in code. Stitching works better for vendor consistency; per-page is faster.

Pitfall 3: Faxed scans at 200 DPI

Cause: input quality below model's effective resolution.

Fix: preprocess with Tesseract's deskew and an image-upscaler step. For chronic-quality vendors, swap in Pix2Struct for OCR before the LLM extraction pass.

Pitfall 4: LLM hallucinates a line item that does not exist

Cause: ambiguous or partial line at the bottom of a page.

Fix: always run math validation. If items_total + tax does not equal total_amount, send to human review.

Pitfall 5: Vendor names drift

Cause: Acme Corp, ACME Corporation, Acme Corp. all show up as different vendors.

Fix: maintain a canonical vendor list and use embedding similarity to match new vendor names back to canonical entries. See our local AI embeddings guide for the mechanics.

---

ROI Math

A worked model for a 70-person construction firm processing 600 invoices a month. Every input below is an assumption — swap in your own volumes, wage rates and quoted hardware prices before you take the conclusion seriously:

Manual baseline

Cloud SaaS replacement

Local AI replacement

By year 2 the local stack saves $13,000+ per year compared to either status quo. For finance teams that already have an internal Python developer, the labor portion of "implementation" goes to zero.

---

What This Pipeline Cannot Do (Yet)

• Approve invoices. The system extracts, validates, and queues for human approval. Approval logic stays human-driven (and legally needs to).
• Negotiate payment terms. AI suggests, finance decides.
• Replace your accountant. Tax classification, accruals, and audit-readiness still need a human professional. Local AI handles the data-entry layer that consumes their time.

---

Conclusion

Invoice processing is the single highest-volume document workflow in most businesses, which makes it the highest-leverage AI use case. The local stack runs a current vision model against your scans on one workstation, posts directly to QuickBooks, NetSuite, SAP, or Sage, and keeps every byte of vendor pricing on hardware you control. Extraction quality is good enough that validation rules, not the model, become the thing you spend your time tuning.

The migration is a three-week sprint, not a quarter-long project. Week one: stand up Ollama, classify and extract a sample of 50 historical invoices, tune the prompt against your actual mix. Week two: build the validation rules and the human-review queue. Week three: integrate with the ERP and turn on auto-posting for known vendors under your authorization threshold. After that, the AP clerk's calendar opens up and the controller stops worrying about which third-party SaaS just sent a "we are revising our terms" email.

---

Building more AP and finance automation? Pair this guide with our local AI document scanner for paper invoices and our Ollama Python API guide for production-grade integration patterns.