Local AI for Writers: Your Manuscript Stays on Your Hard Drive

Published April 23, 2026 - 20 min read

In June 2024, OpenAI updated its terms to clarify that ChatGPT Team and Enterprise inputs are excluded from training. ChatGPT Plus inputs are also excluded - if you toggle the right setting, in the right menu, on the right plan tier. Most novelists I know have never read those settings. They have, however, pasted three years of manuscript drafts into the chat box.

If you are writing a book, that book is your livelihood. The contract you sign with your publisher will absolutely contain a representation that the manuscript is your original work. That representation gets harder to defend the more your unfinished prose has been processed, embedded, retained, or - in the worst case - leaked through a logging incident at a vendor you never vetted. There is exactly one configuration where this risk is zero: a model running on a machine you control, talking to a manuscript that never leaves your disk.

This guide is the practical setup. The right open model for fiction (it is not Llama 3.1 70B, despite what every benchmark claims). A manuscript RAG system that can answer "where does Eliza first mention the locket?" across an 800-page draft. A Scrivener integration that adds an "Ask the AI" button without breaking your existing project. And a writing-style fingerprint workflow so the AI sounds like you, not like every Claude-trained ghostwriter on Substack.

Quick Start: 4 Commands to a Private Writing Assistant

1. Install Ollama: curl -fsSL https://ollama.com/install.sh | sh
2. Pull the right fiction model: ollama pull qwen2.5:32b-instruct-q4_K_M (~19GB)
3. Open WebUI for a chat interface: docker run -d -p 3000:8080 ghcr.io/open-webui/open-webui:main
4. Drop your manuscript into a folder, point the RAG ingest at it, start writing

On a 32GB Mac Studio M2 Max or an RTX 3090 24GB PC with 64GB system RAM, you have a fully private writing assistant in 25 minutes. Total cost: $0 in subscriptions, ~25GB on disk.

---

Table of Contents

1. Why Writers Need Local AI
2. Hardware Requirements
3. Best Open Models for Fiction
4. Manuscript RAG Setup
5. Scrivener Integration
6. Style Fingerprinting Your Voice
7. Prompt Patterns for Long-Form Fiction
8. The Ghostwriter / Editor Workflow
9. Pitfalls
10. FAQ

---

Why Writers Need Local AI {#why-writers}

There are three specific writer-shaped problems with cloud LLMs:

1. Training data exposure. Even with the consumer ChatGPT data toggle off, you are trusting OpenAI's logging discipline. The 2023 ChatGPT outage that briefly exposed other users' chat titles is the proof-of-concept that cloud LLM logging is not zero-risk.
2. Manuscript representation. Your publishing contract requires that the manuscript is your work. "AI-assisted" is increasingly something publishers want disclosed - some flat-out forbid it. The cleanest answer is one you can document: a local model that does not retain prompts, with a written prompt log on your own disk.
3. The cancellation dependency. When you build a workflow on ChatGPT Plus and your trial finishes a five-year series, you are at the mercy of any future pricing or policy change. A local stack runs on the same model file forever, unchanged, regardless of any vendor's roadmap.

The Authors Guild's 2024 statements on AI and publishing make the case bluntly: authors should retain control of their work and disclose meaningfully when AI is used. Local AI is the cleanest technical implementation of that control.

---

Hardware Requirements {#hardware}

Minimum (works for shorter fiction, slow)

This tier runs Qwen 2.5 14B Q4 well, which is good enough for chapter-level work but limited for novel-length context.

Recommended (full novel context, fast)

This tier runs Qwen 2.5 32B Q4 with a 32K context window, which fits roughly 24,000 words at once - enough for a chapter plus the surrounding three.

High-end (entire novel in context)

This tier runs Llama 3.1 70B Q4 or Qwen 2.5 72B Q4 with 64K-128K context - enough for an entire 80,000-word manuscript in a single prompt. The Mac Studio M2 Ultra 128GB is the most price-efficient writer rig I have benchmarked.

For a deeper dive on Apple Silicon vs PC for AI writing workloads, see our Mac local AI setup and Apple Silicon AI buying guide.

---

Best Open Models for Fiction {#fiction-models}

I tested seven open models on a 50-prompt fiction benchmark covering scene continuation, dialogue rewriting, line-edit suggestions, character voice consistency, and historical accuracy. Each prompt was scored by three published authors (one literary, one thriller, one romance) blind to model identity.

The result that surprised me: Qwen 2.5 32B beats Llama 3.1 70B on every fiction metric I tested, while being half the size. Qwen 2.5 was clearly trained on more literary text. It picks up authorial voice faster, holds character consistency longer, and produces line-edit suggestions that read like an actual editor wrote them.

If you have 64GB+ RAM, Qwen 2.5 72B Q4 is the best fiction model open weights have to offer as of April 2026. If you have 32GB, Qwen 2.5 32B Q4 is the practical choice. Llama 3.1 8B is fine for ideation and outlining but visibly weaker for prose-level work.

---

Manuscript RAG Setup {#manuscript-rag}

The single most useful upgrade beyond chat is a RAG (retrieval-augmented generation) layer over your manuscript. Once it works, you can ask:

• "Where does Eliza first mention the locket?"
• "List every scene where Marcus and Eliza are in the same room."
• "Find inconsistencies in the timeline of the locket subplot."
• "Show me three places I've used the word 'shimmer' - I want to vary it."

The Stack

• Embeddings: bge-large-en-v1.5 (Apache 2.0, runs locally)
• Vector DB: ChromaDB (file-based, no server needed)
• LLM: Qwen 2.5 32B Q4 via Ollama
• Frontend: Open WebUI with built-in document RAG

Setup

# Embeddings via Ollama
ollama pull bge-large

# Open WebUI (Docker)
docker run -d \
  -p 3000:8080 \
  -v open-webui:/app/backend/data \
  --name open-webui \
  --restart always \
  ghcr.io/open-webui/open-webui:main

# Visit http://localhost:3000
# Settings → Documents → Set embedding model: bge-large
# Settings → Models → Default: qwen2.5:32b-instruct-q4_K_M

Ingesting a Manuscript

Open WebUI accepts .txt, .md, and .docx. Best practice: export your Scrivener compile as a single Markdown file with chapter headings, then drop it into the Workspace's "Documents" tab.

Chunk strategy that worked best across 30+ test queries:

• Chunk size: 1,200 tokens
• Chunk overlap: 200 tokens
• Top-K retrieval: 6 chunks
• Re-rank: enable BGE re-ranker if you have headroom

For 80,000-word manuscripts, this gives roughly 95% retrieval accuracy on character/event queries.

If you want a deeper walkthrough on the RAG side, our Ollama ChromaDB RAG pipeline and private AI knowledge base guides cover the same plumbing applied to non-fiction use cases.

---

Scrivener Integration {#scrivener}

Scrivener does not have a native AI plugin, but it has two extension points worth knowing:

1. External script execution via Scrivener's "Open with..." menu
2. AppleScript automation on Mac

Here is the workflow I use:

Mac Workflow (AppleScript + Ollama)

-- ScrivenerAskAI.scpt
-- Save in ~/Library/Scripts and call from Scrivener via Services menu

on run {selectedText}
    set thePrompt to "You are a fiction editor helping refine prose. Suggest a single tight, voice-preserving rewrite of this passage. Return ONLY the rewrite, no preamble.\n\nPASSAGE:\n" & selectedText
    set theResponse to do shell script "echo " & quoted form of thePrompt & " | /usr/local/bin/ollama run qwen2.5:32b-instruct-q4_K_M"
    set the clipboard to theResponse
    display notification "Rewrite copied to clipboard" with title "Local AI Editor"
end run

Bind this to a keyboard shortcut via macOS System Settings → Keyboard → Services. Now selecting any passage in Scrivener and pressing the shortcut copies a rewrite suggestion to your clipboard, generated by your local model, with no internet round trip.

Windows Workflow (PowerShell + Ollama)

# scrivener-ask.ps1
$selected = Get-Clipboard
$prompt = "You are a fiction editor. Suggest a tight, voice-preserving rewrite of this passage. Return only the rewrite.`n`nPASSAGE:`n$selected"
$response = $prompt | ollama run qwen2.5:32b-instruct-q4_K_M
Set-Clipboard -Value $response

Bind to a shortcut via AutoHotkey. Workflow: copy passage, press shortcut, paste rewrite.

---

Style Fingerprinting Your Voice {#style-fingerprint}

This is the part most "AI for writers" guides skip. A model that answers your prompts in its voice produces ghostwritten-sounding paste that you have to rewrite anyway. The fix is a style fingerprint - a system prompt that captures your voice patterns and sticks them in the model's context every turn.

Build the Fingerprint

1. Pick three samples of your tightest prose: one descriptive paragraph, one dialogue scene, one introspective passage. Total ~600 words.
2. Run them through the model with this analysis prompt:

Analyze these three passages from the same author. Identify:
- Sentence length distribution (avg, range)
- Punctuation tics (em dashes, semicolons, parentheticals?)
- Adjective density (high or low?)
- POV preferences and tense
- Distinctive vocabulary clusters
- What this author avoids

Be specific and concrete. Quote examples.

PASSAGES:
[paste your 600 words here]

1. Take the analysis output and condense it into a 200-word "voice card."
2. Save as a system prompt:

You are assisting [Author Name], whose voice has the following traits:
[paste 200-word voice card]

Match these traits in any rewrite or continuation. Do not introduce semicolons; this author does not use them. Sentence length averages 14 words with frequent fragments for emphasis. Vocabulary leans Anglo-Saxon. Dialogue is sparse, action-tagged, no "he said softly" type adverbs.

When suggesting rewrites, preserve voice over polish.

1. Set this as the default system prompt in Open WebUI for your "Writing" model.

I have run this on six authors who tested the workflow. Five of them said outputs from a fingerprinted Qwen 2.5 32B were more usable than outputs from a non-fingerprinted Claude 3.5 Sonnet. The model is not better - the prompt is.

---

Prompt Patterns for Long-Form Fiction {#fiction-prompts}

After months of iteration, these are the patterns that consistently produce useful output:

Scene Continuation

Continue this scene for ~250 words. Stay in [POV character]'s POV.
Hold the rhythm of the existing prose. Preserve the implied stakes.
Do not introduce new characters. Do not resolve the tension.

[paste last 800 words]

Line Edit Pass

Edit for line-level prose. Goals:
1. Cut padding (filter words: "began to," "sort of," "just")
2. Replace abstract verbs with concrete ones
3. Tighten dialogue tags
4. Preserve voice and content meaning

Mark each change with [BEFORE -> AFTER] inline. Do not rewrite the whole passage.

[paste 500-word passage]

Inconsistency Hunt (with RAG)

Using the manuscript context, identify any continuity errors involving [character or object].
Examples: eye color, age, possession, location at a given time.
For each error, cite the chapter and line.

Question: Does [character]'s timeline hold across chapters 4-12?

Character Voice Audit

Sample dialogue from [character] across the manuscript. Score voice consistency 1-10.
Flag any lines that read out of character. Quote examples.

Use the manuscript context to retrieve dialogue.

Synopsis Generation

Generate a one-page synopsis of this manuscript suitable for a query letter.
Constraints:
- 500 words
- Reveals the ending (synopses do)
- Third person, present tense
- No character bios; the synopsis IS the plot
- Author's voice on the page

Manuscript: [load via RAG]

For long-form ideation and brainstorming workflows, our guide on local AI for content creators overlaps with the planning side of fiction work.

---

The Ghostwriter / Editor Workflow {#ghostwriter}

If you ghostwrite or edit on contract, the privacy story is even sharper. Your client's manuscript is not yours to share - and clicking "Save chat" in ChatGPT may technically constitute disclosure under your NDA.

Here is the workflow:

1. Per-project model alias. ollama cp qwen2.5:32b-instruct-q4_K_M client-smith creates a per-client tagged model. Useful for tracking which projects touched which model context.
2. Ephemeral chat history. Open WebUI's "temporary chat" mode keeps no log. For sensitive client work, use it.
3. Encrypted project folders. Each client's manuscript lives in a separate, FileVault- or BitLocker-encrypted folder, with the RAG index regenerated per session and not retained.
4. Local-only language. Every NDA I have ever signed has a "no third-party disclosure" clause. A local LLM does not constitute disclosure. A cloud LLM almost certainly does. Have your contract reflect this distinction.
5. Document the workflow. Keep a one-page written description of your AI use, with the model name, version hash, and the fact that no manuscript content leaves your machine. Some authors and agents now ask for this.

---

Common Pitfalls {#pitfalls}

1. Using Llama 3.1 70B for fiction. It benchmarks well on reasoning, mediocre on prose. Qwen 2.5 32B is the better fiction model at half the size.

2. Skipping the style fingerprint. Without it, every output sounds like generic AI. The fingerprint is 30 minutes of work that pays back forever.

3. Treating the model like a co-author. It is a tool. The prose you publish should be yours, edited by you. Model output is raw material, not finished work.

4. Ignoring context windows. Qwen 2.5 32B has 128K theoretical context but quality degrades past ~32K tokens in practice. For full-novel queries, use Q4 70B+ models.

5. Forgetting to encrypt. A local model is private until your laptop is stolen. FileVault/BitLocker is non-negotiable.

6. Pasting client material into ChatGPT "just to compare." This is the moment your NDA gets broken. The whole point of the local stack is that there is no temptation, because the local stack is good enough.

7. Disclosure failures. Some publishers and contests now require AI disclosure. "I used an open model running on my own machine to assist with line editing" is a defensible disclosure. "I had ChatGPT rewrite chapter five" is a different conversation. Know your contract terms.

---

Frequently Asked Questions {#faq}

Which open model is best for fiction writing?

Qwen 2.5 32B Instruct (Q4_K_M quantization) is the best fiction model that fits on a 24GB GPU or 32GB Apple Silicon. It beat Llama 3.1 70B on voice match, continuation quality, and line edit quality in my 50-prompt fiction benchmark. If you have 64GB+ RAM, Qwen 2.5 72B is the absolute best open fiction model as of April 2026.

Can I run a local writing assistant on a MacBook Air M2?

Yes, but with limits. M2 16GB runs Qwen 2.5 14B Q4 well (about 18 tokens per second), which handles paragraph and scene-level work. Full novel context queries need 32GB+ unified memory. The MacBook Pro M3 Pro 32GB or Mac Studio is a much better fit if writing is your primary use.

How does this compare to Claude or ChatGPT for prose?

Cloud frontier models (Claude 3.5 Sonnet, GPT-4o, Claude Opus) still have a small edge in raw prose quality, roughly 5-10% in blind testing. With a strong style fingerprint and a manuscript RAG, locally-run Qwen 2.5 32B closes most of that gap. The privacy advantage is absolute, not incremental.

Do publishers care if I used local AI?

Some do. The Authors Guild recommends disclosure when AI was used in any meaningful way. Romance, science fiction, and literary publishers vary widely - a few flatly forbid AI assistance, most are silent, some now require disclosure on the contract. Read the contract before assuming. Local AI does not change the disclosure obligation; it just makes the privacy story cleaner.

Will my manuscript train any model?

No. Ollama, Open WebUI, and ChromaDB run entirely on your machine. None of them send manuscript content to any external service. You can verify this by blocking outbound traffic from those processes at your firewall - they will continue to function for inference and RAG.

Can a co-author share the same local stack?

Yes. Run Ollama on a workstation in your home/office and share via Tailscale to the co-author's machine. Or each author runs their own copy locally and you exchange manuscript files normally. The collaborative workflow does not require any cloud component.

What about plot brainstorming and outlining?

Local models handle this well. Qwen 2.5 32B can generate plot outlines, character arcs, and scene-by-scene structures comparable to Claude 3.5 Sonnet for most fiction. Use a higher-temperature setting (0.8-1.0) for ideation and lower (0.3-0.5) for prose work.

How do I keep the AI from writing in its own voice?

Three things in priority order: (1) build a style fingerprint and use it as a system prompt; (2) provide 200-300 words of your existing prose as immediate context before any continuation request; (3) use Qwen 2.5 32B or 72B - they pick up voice better than the Llama family. Most "AI voice" complaints are really "no fingerprint and no in-context examples" complaints.

---

Closing Notes for the Skeptical Author

I started this experiment expecting local AI to be a watered-down version of ChatGPT. After three months of using it for line edits on a half-finished novel, I have not opened ChatGPT once. The privacy story is the leading reason. The quality story is closer to "matched" than "compromised." And the cancellation-proof permanence of running a static model on my own SSD has done more for my actual writing routine than any productivity app I have ever tried.

The setup takes one afternoon. The model files are 20-50GB. The hardware, if you do not already own it, is a one-time spend that pays for itself against an ElevenLabs/Claude/ChatGPT subscription stack within months.

Most importantly: your manuscript stays on your hard drive. That should not be a feature you have to fight for. With a local model, it just is.