Local AI Audit Trail: How to Log Every Prompt and Response Without Breaking Confidentiality

Published April 23, 2026 - 19 min read

When a SOC 2 auditor asks "show me what your AI said to that user on March 14th," there are exactly two acceptable answers. One is "here is the record." The other is "we have engineered this system so that question is impossible to ask, and here is the documented design decision." Anything in between is a finding.

I have been on both sides of that conversation. I have built logging stacks for fintech companies running self-hosted Llama, and I have helped a healthcare startup pass a HITRUST audit on their on-prem Mistral deployment. The patterns are the same. The mistakes are the same. The pieces of the answer are surprisingly simple - and almost everyone gets at least one of them wrong.

This is the guide I wish I had three years ago.

Quick Start: A Working Audit Log in 12 Minutes

Drop this into your Ollama-fronting service and you will have a tamper-evident, hash-chained audit log running before lunch:

pip install fastapi uvicorn ollama sqlalchemy structlog

Then a 60-line Python proxy in front of Ollama writes every request and response to an append-only SQLite file with a chained SHA-256 hash. We will build out the full version later, but the minimal viable audit log is genuinely a one-afternoon project.

The hard parts are not the code. They are: deciding what to redact, deciding how long to keep it, who can read it, and how to prove the log was not edited. We cover all four below.

Table of Contents

1. Why Local AI Needs Its Own Audit Story
2. What "Audit Trail" Actually Means in Compliance
3. The Eight Fields Every Log Entry Must Have
4. Tamper-Evident Logs with Hash Chaining
5. Building the Logging Proxy in Front of Ollama
6. Redaction: PII, PHI, and Trade Secrets
7. Retention Policies That Survive Legal Discovery
8. OpenTelemetry, Langfuse, and the Toolchain
9. SOC 2, ISO 27001, and HIPAA Evidence
10. Pitfalls and Production Lessons
11. FAQ

---

Why Local AI Needs Its Own Audit Story {#why-audit}

When you ran cloud LLMs, your provider gave you most of this for free. OpenAI's enterprise console has prompt logs, Anthropic has trace export, AWS Bedrock writes to CloudTrail. The minute you self-host - whether for cost, privacy, or control - you become the platform team. Logging is now your responsibility.

This is not optional. Every modern compliance framework now treats AI as a regulated data flow:

• SOC 2 CC7.2 requires "monitoring of system components and the operation of those controls" - which auditors increasingly read as "log your AI inputs and outputs."
• ISO 27001 Annex A.8.15 mandates "logging activities" of users and administrators interacting with information systems.
• HIPAA Section 164.312(b) requires "audit controls" - hardware, software, and procedural mechanisms to record and examine activity.
• EU AI Act Article 12 (entering force 2026-2027) requires "automatic recording of events" for high-risk AI systems.

Local AI is not exempt from any of these. The data did not become less sensitive when you stopped sending it to a cloud.

For background on the broader compliance picture, see our SOC 2 for self-hosted AI and GDPR-compliant local AI guides.

---

What "Audit Trail" Actually Means in Compliance {#definition}

Auditors are looking for four properties, in this order:

1. Completeness - every interaction is captured. Not "most." Not "the ones we remembered to log." Every.
2. Integrity - the log cannot be silently edited. If someone tampers, you can prove it.
3. Attribution - you can tie any given log entry back to a specific user identity.
4. Retention and disposal - you keep what you must, you destroy what you must, and you can prove both.

A common misconception is that "audit log" means "verbose application log." It does not. Application logs are for engineers. Audit logs are for regulators. They have different schemas, different retention requirements, and different access controls. Treat them as separate systems.

---

The Eight Fields Every Log Entry Must Have {#schema}

Here is the SQLAlchemy model I use in production. It is opinionated by design - skipping any of these eight fields will eventually fail an audit.

from sqlalchemy import Column, Integer, String, Text, DateTime, LargeBinary
from sqlalchemy.ext.declarative import declarative_base
from datetime import datetime, timezone

Base = declarative_base()

class AuditLog(Base):
    __tablename__ = "ai_audit_log"

    id = Column(Integer, primary_key=True, autoincrement=True)
    timestamp = Column(DateTime(timezone=True), nullable=False,
                       default=lambda: datetime.now(timezone.utc))

    # 1. Who
    actor_id = Column(String(64), nullable=False)         # internal user id
    actor_role = Column(String(32), nullable=False)       # e.g. "preparer", "physician"

    # 2. Where
    request_ip = Column(String(45), nullable=False)        # IPv4 or IPv6
    session_id = Column(String(64), nullable=False)

    # 3. What
    model_id = Column(String(128), nullable=False)         # "qwen2.5:14b@sha256:..."
    prompt_hash = Column(String(64), nullable=False)       # SHA-256 of full prompt
    prompt_redacted = Column(Text, nullable=False)         # PII-stripped copy
    response_hash = Column(String(64), nullable=False)     # SHA-256 of response
    response_redacted = Column(Text, nullable=False)

    # 4. Outcome
    tokens_in = Column(Integer, nullable=False)
    tokens_out = Column(Integer, nullable=False)
    latency_ms = Column(Integer, nullable=False)
    status = Column(String(16), nullable=False)            # "ok", "blocked", "error"

    # 5. Integrity
    prev_hash = Column(String(64), nullable=False)         # chained from prior row
    entry_hash = Column(String(64), nullable=False)        # SHA-256 of this row

Why Each Field Matters

• actor_id and actor_role: attribution. "User X did Y at time Z."
• session_id: correlation across multiple requests. Auditors love this.
• model_id with version hash: model drift defense. Six months from now, you can prove which exact model produced an output.
• prompt_hash and response_hash: integrity check separate from the body. Even if redaction removed words, the hash of the original is permanent.
• prompt_redacted and response_redacted: human-readable evidence. We will cover redaction below.
• tokens and latency: capacity planning, anomaly detection, cost attribution.
• status: did the request succeed? Was it blocked by a guardrail?
• prev_hash and entry_hash: the tamper-evident chain. The next section explains why.

---

Tamper-Evident Logs with Hash Chaining {#hash-chain}

Append-only is necessary but not sufficient. Even an append-only log can be replaced wholesale by someone with database access. Hash chaining makes that detectable.

The pattern is borrowed from blockchain (without the blockchain). Each new entry includes the SHA-256 of the previous entry's full row. If anyone modifies row 47, every row from 48 onward has a broken chain.

import hashlib
import json

def compute_entry_hash(entry: dict, prev_hash: str) -> str:
    """SHA-256 of the canonical JSON of the entry plus the prior hash."""
    payload = {
        "timestamp": entry["timestamp"].isoformat(),
        "actor_id": entry["actor_id"],
        "model_id": entry["model_id"],
        "prompt_hash": entry["prompt_hash"],
        "response_hash": entry["response_hash"],
        "tokens_in": entry["tokens_in"],
        "tokens_out": entry["tokens_out"],
        "status": entry["status"],
        "prev_hash": prev_hash,
    }
    canonical = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()

Verification Job

Run this nightly. If it fails, page someone:

def verify_chain(session) -> int:
    """Returns row id of first broken link, or -1 if intact."""
    rows = session.query(AuditLog).order_by(AuditLog.id).all()
    prev_hash = "0" * 64  # genesis
    for row in rows:
        expected = compute_entry_hash(row.__dict__, prev_hash)
        if expected != row.entry_hash:
            return row.id
        prev_hash = row.entry_hash
    return -1

For extra defense, ship the latest entry_hash to a write-once external store (S3 Object Lock, AWS Glacier, an on-prem WORM appliance) every hour. Now an attacker would have to compromise both your application and your archive to fake a clean chain.

---

Building the Logging Proxy in Front of Ollama {#proxy}

The cleanest pattern is a thin FastAPI proxy that sits between your applications and Ollama. Every request goes through it; every response is logged before it ever reaches the caller.

from fastapi import FastAPI, Request, HTTPException
import httpx, hashlib, time
from sqlalchemy.orm import Session
from .models import AuditLog
from .redact import redact_pii
from .integrity import compute_entry_hash, get_last_hash

app = FastAPI()
OLLAMA = "http://localhost:11434"

@app.post("/v1/chat/completions")
async def chat(request: Request, db: Session):
    body = await request.json()
    actor_id = request.headers.get("X-Actor-Id")
    actor_role = request.headers.get("X-Actor-Role")
    if not actor_id:
        raise HTTPException(401, "missing actor")

    prompt_text = json.dumps(body.get("messages", []), sort_keys=True)
    prompt_hash = hashlib.sha256(prompt_text.encode()).hexdigest()

    t0 = time.time()
    async with httpx.AsyncClient(timeout=120) as client:
        upstream = await client.post(f"{OLLAMA}/v1/chat/completions", json=body)
    latency_ms = int((time.time() - t0) * 1000)

    response_text = upstream.text
    response_hash = hashlib.sha256(response_text.encode()).hexdigest()

    prev = get_last_hash(db)
    entry = AuditLog(
        actor_id=actor_id,
        actor_role=actor_role,
        request_ip=request.client.host,
        session_id=request.headers.get("X-Session-Id", "none"),
        model_id=body.get("model", "unknown"),
        prompt_hash=prompt_hash,
        prompt_redacted=redact_pii(prompt_text),
        response_hash=response_hash,
        response_redacted=redact_pii(response_text),
        tokens_in=upstream.json().get("usage", {}).get("prompt_tokens", 0),
        tokens_out=upstream.json().get("usage", {}).get("completion_tokens", 0),
        latency_ms=latency_ms,
        status="ok" if upstream.status_code == 200 else "error",
        prev_hash=prev,
    )
    entry.entry_hash = compute_entry_hash(entry.__dict__, prev)
    db.add(entry)
    db.commit()

    return upstream.json()

This is the entire pattern. Run it as a systemd unit on the same machine as Ollama. Point all your downstream applications at http://audit-proxy:8000 instead of http://ollama:11434. From the application's point of view, the API is identical. From the auditor's point of view, you have a complete record.

For a more production-ready architecture covering nginx, TLS, and multi-user authentication in front of this, see our Ollama production deployment guide.

---

Redaction: PII, PHI, and Trade Secrets {#redaction}

The conflict at the heart of audit logging: you want to record everything, but the law often requires you to not store certain things. The reconciliation is to keep the hash of the full content forever, but the body in redacted form.

A simple but effective Python redactor:

import re

PATTERNS = [
    # SSN
    (r"\b\d{3}-\d{2}-\d{4}\b", "[SSN]"),
    # Credit card
    (r"\b(?:\d[ -]*?){13,16}\b", "[CC]"),
    # Email
    (r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b", "[EMAIL]"),
    # US phone
    (r"\b(?:\+?1[-.]?)?\(?\d{3}\)?[-.]?\d{3}[-.]?\d{4}\b", "[PHONE]"),
    # Date of birth-shaped
    (r"\b(0[1-9]|1[0-2])/(0[1-9]|[12]\d|3[01])/(19|20)\d{2}\b", "[DATE]"),
]

def redact_pii(text: str) -> str:
    for pattern, replacement in PATTERNS:
        text = re.sub(pattern, replacement, text)
    return text

For healthcare, layer Microsoft's Presidio on top - it has 40+ recognizers including medical record numbers and HIPAA Safe Harbor categories. For financial, add account number and routing number patterns.

The Critical Rule

Never redact the hash. The hash of the original (un-redacted) content is what proves you have not silently rewritten history. If a regulator subpoenas the actual content, you produce it from your separate, encrypted, access-controlled raw store. The audit log is the index; the raw store is the evidence.

---

Retention Policies That Survive Legal Discovery {#retention}

Retention is where teams overlook the legal nuance. The defensible policy is layered:

You also need a legal hold mechanism. When counsel tells you "preserve everything related to client X starting yesterday," you need to be able to flip a switch that pauses deletion for matching records. We do this with a hold_until timestamp column and a periodic deletion job that respects it.

The single most useful policy I have seen is: deletion is automatic, restoration is not. If a record passes its retention date, it is gone. There is no "we forgot" option, because there is no manual deletion. Auditors love this.

---

OpenTelemetry, Langfuse, and the Toolchain {#tooling}

You do not have to build all of this from scratch. Three open-source tools cover most of the territory:

Langfuse (Self-Hosted)

Langfuse is the closest open-source equivalent to OpenAI's enterprise dashboard. Self-hosted, MIT-licensed, runs on Docker. It captures traces of LLM calls, supports user-level grouping, and has built-in evaluation hooks. For a team that wants the audit trail and the developer-experience layer, it is hard to beat.

git clone https://github.com/langfuse/langfuse.git
cd langfuse && docker compose up -d
# Now available at http://localhost:3000

The catch: Langfuse alone is not tamper-evident. Pair it with the hash-chain pattern above by writing every Langfuse trace ID into your hash-chained SQLite log.

OpenTelemetry

The vendor-neutral standard. The OpenLLMetry project provides drop-in instrumentation for Ollama, OpenAI-compatible APIs, and most major frameworks. Every LLM call becomes an OTel span with token counts, latencies, and the model identifier.

from openllmetry_sdk import Traceloop
Traceloop.init(app_name="my-firm", api_endpoint="http://otel-collector:4318")

Pipe the spans to Tempo, Jaeger, or any OTel backend. For SOC 2 you still need the hash-chained store - OTel is for monitoring, not legal evidence - but the two complement each other well.

Vector or Fluent Bit for Log Shipping

If your audit log lives on the same machine as the application that produced it, you have a single point of failure. Ship to a separate logging host with Vector or Fluent Bit. The shipper should be the only process with read access to the local log file, and the destination should be append-only at the storage layer.

---

SOC 2, ISO 27001, and HIPAA Evidence {#evidence}

When the auditor walks in, here is what you hand them:

1. The schema - the SQL DDL of your audit log table. They will check for the eight fields above (or equivalents).
2. The chain verification report - the output of your nightly verify job for the audit period, signed by your CTO or security officer.
3. A retention policy document - one page, names the retention layers, names the deletion job, includes the legal-hold mechanism.
4. Sample records - 10-20 anonymized log entries with the hash chain visible.
5. Access control list - who can read the audit log, signed off by HR/security.
6. Incident records - any time the chain verification failed, what happened, what you did.

I have walked exactly this packet through three SOC 2 Type II audits in the last year. Each took less than 30 minutes of audit time. Compare that to the alternative - "uh, we have application logs in CloudWatch?" - which can eat days of follow-up.

For HIPAA, add a Business Associate Agreement note: since the AI is self-hosted, there is no BAA needed for the AI itself. That alone is worth the entire setup for many healthcare practices.

---

Pitfalls and Production Lessons {#pitfalls}

In rough order of how often I see them:

1. Logging the prompt only, not the response. Half the value of an audit log is "what did the system tell the user?" If a model gave bad legal advice, the response is the evidence. Always log both.

2. Logging in the same database as application data. When the application database is restored from a backup, your audit chain breaks (and you may not notice until verification fails next week). Use a separate database, ideally on a separate volume.

3. No clock synchronization. All audit timestamps must be UTC, generated by a single source. Run NTP. Reject any client-supplied timestamp.

4. Over-trusting the redactor. Regex-based PII redaction will miss things. Run a sample manually every quarter and update patterns. Better: layer Presidio or a model-based redactor.

5. Forgetting the retention disposal evidence. It is not enough to delete records. You need a record of the deletion. A row in a separate deletion_log table that says "rows 1-12,500 deleted at TIME because retention expired" is what closes that audit gap.

6. Letting developers turn off logging in dev. Then someone tests in dev with prod data, and now you have unlogged real prompts. The proxy should be the only path - hard-coded, no flag.

7. Storing the audit log on the same disk as the model files. When that disk fails - and disks fail - you lose both. See our local AI backup and recovery guide for the disk-layout pattern.

8. No alerting on chain failure. The verify job runs nightly but no one watches it. Add a PagerDuty hook. A broken chain is a security incident.

---

FAQ {#faq}

The single question I get most: "Do I need all of this if my AI is just for internal use?" Yes. The threat model is not just outsiders. It is also the future-you who needs to demonstrate, three years from now, that an output you delivered to a client was generated correctly. Logging is institutional memory.

---

Where to Take This Next

This guide is the foundation. Three deeper rabbit holes:

1. Multi-tenant logs - if you run AI for multiple internal teams or external clients, partition the log by tenant with row-level security. Our Ollama rate limiting and multi-user guide covers the auth layer.
2. Real-time anomaly detection - run a small classifier over the streaming log to flag prompt-injection attempts, jailbreaks, or PII leakage. Pair this with the securing Ollama guide.
3. Federated audit - in regulated industries (insurance, brokerage) you may need to share aggregated audit metrics with regulators while keeping content local. Differential privacy and aggregation are your tools.

For broader context on the operational side, see our Ollama production deployment and GDPR-compliant local AI guides.

---

Conclusion

Audit logging is not the glamorous part of running local AI. It is the part that decides whether your deployment survives a regulator, a lawsuit, or the question your CFO asks at 3pm on a Tuesday. The good news is that the pattern is small, the tools exist, and a competent backend engineer can stand up a defensible system in two days.

Do it before you need it. The day you wish you had been logging is always too late to start.

If you adopt one thing from this guide, make it the hash chain. The day a junior engineer accidentally truncates the audit table, you will know within 24 hours - and you will be able to prove it. That single property has saved me from very bad conversations more than once.