Local AI for Writers: Private Novel-Writing Setup (2026)

A private writing assistant is four commands: install Ollama, pull a mid-size instruct model, run Open WebUI, and point its document RAG at your manuscript. Nothing leaves the disk, there is no subscription, and the model file you have today keeps working regardless of anyone's roadmap. The hard parts are choosing a model that fits your machine, and getting it to write in your voice instead of its own — both covered below.

If you are writing a book, that book is your livelihood. Your publishing contract will contain a representation that the manuscript is your original work, and that representation gets harder to defend the more your unfinished prose has been processed, embedded, retained, or leaked through a logging incident at a vendor you never vetted. There is exactly one configuration where that risk is zero: a model running on a machine you control, reading a manuscript that never leaves your disk.

Quick start: four commands to a private writing assistant

1. Install Ollama: curl -fsSL https://ollama.com/install.sh | sh
2. Pull a mid-size instruct model: ollama pull qwen2.5:32b-instruct-q4_K_M (about 19 GB)
3. Run a chat interface: docker run -d -p 3000:8080 ghcr.io/open-webui/open-webui:main
4. Drop the manuscript into Open WebUI's Documents tab and start asking it questions

On a 32 GB Apple Silicon machine or a 24 GB GPU with 64 GB of system RAM, that is the whole stack. Ongoing cost: nothing. Disk: roughly 25 GB once the embedding model is pulled.

---

Why do writers need a local AI model?

Three writer-shaped problems with cloud LLMs:

1. You are trusting someone else's logging discipline. Vendor terms on training use change, and they differ by plan tier and by which toggle you found in which menu. The March 2023 ChatGPT incident that briefly exposed other users' chat titles is the proof-of-concept that cloud LLM logging is not zero-risk, whatever the policy says.

2. Manuscript representation. Your contract requires that the manuscript is your work. "AI-assisted" is increasingly something publishers want disclosed, and a few forbid it. The cleanest answer is one you can document: a local model that retains nothing, with a prompt log on your own disk.

3. The cancellation dependency. Build a five-year series workflow on a subscription and you are exposed to every future pricing and policy change. A local stack runs the same model file forever.

The Authors Guild's guidance on AI and publishing makes the case bluntly: authors should retain control of their work and disclose meaningfully when AI is used. Local inference is the cleanest technical implementation of that control.

---

What hardware do you need?

Model size drives everything, and you can compute it rather than guess:

VRAM or unified memory for weights (GB) at Q4_K_M
        ~=  0.6 x (parameters in billions)

throughput ceiling (tok/s)
        =   memory bandwidth (GB/s) / weights (GB)

The second formula is an arithmetic upper bound, not a promise — generation also pays for attention, sampling and prompt processing, so real output lands well below it. It is still the fastest way to tell whether a machine is in the right league, and it is why memory bandwidth, not core count, is the number to shop on.

Bandwidth figures are the manufacturers' published specifications, catalogued for GPUs in the TechPowerUp GPU database.

Two caveats worth knowing before you spend money. On Apple Silicon, macOS caps how much unified memory the GPU may claim, so a 16 GB Mac cannot dedicate all 16 GB to a model — size down a tier from what the spec sheet implies. And on a two-GPU PC, splitting a model across cards adds VRAM but not single-request speed, for reasons set out in the Ollama multi-GPU setup guide.

Also budget for context. A long manuscript query is not free: the KV cache for a 70B-class model runs to roughly 0.33 MB per token, so 32K of context is another 10 GB on top of the weights.

For the Apple-versus-PC decision specifically, see the Mac local AI setup walkthrough and the Apple Silicon AI buying guide.

---

Which open model is best for fiction?

Honestly: nobody has a defensible answer, and you should be suspicious of any page that gives you a fiction quality score to two decimal places. Prose quality is subjective, style-dependent, and not something the public benchmark suite measures — the leaderboards writers get pointed at measure reasoning, maths and code. The closest thing to a public creative-writing ranking is the EQ-Bench creative writing leaderboard, which is LLM-judged and worth reading as a shortlist generator rather than a verdict.

What can be stated without inventing anything is the part that constrains you: size, context window, licence, and speed. For a novelist, licence and context matter more than most model round-ups admit.

Weight sizes are computed from the formula above; context windows and licences are from each model's published model card, e.g. Qwen2.5-32B-Instruct on Hugging Face.

Read that licence column carefully. Mistral Large 2 is released under a research licence — it is the wrong choice for a novel you intend to sell, whatever it does for your prose. Qwen 2.5 32B being Apache 2.0 is a genuine practical advantage for commercial fiction, and it is the reason it is the default recommendation on this page rather than a claim about its prose.

How to pick, properly

Since no public benchmark answers "which writes my kind of fiction best", run a ten-minute blind test with material only you have:

1. Take three passages of your own finished prose, 300-400 words each.
2. Cut the last 150 words off each and ask two or three candidate models to continue.
3. Save the outputs to a file with the model names stripped and shuffled.
4. Read them a day later and rank them. A day's gap matters more than you would think.
5. Repeat for a line-edit task, which is the job most writers actually delegate.

That takes an afternoon and tells you something true about your voice, which no leaderboard can.

---

How do you set up manuscript RAG?

The single most useful upgrade beyond chat is retrieval over your own manuscript. Once it works you can ask:

• "Where does Eliza first mention the locket?"
• "List every scene where Marcus and Eliza are in the same room."
• "Find inconsistencies in the timeline of the locket subplot."
• "Show me three places I have used the word 'shimmer' — I want to vary it."

The stack

• Embeddings: bge-large-en-v1.5 (Apache 2.0, runs locally)
• Vector store: ChromaDB (file-based, no server needed)
• LLM: Qwen 2.5 32B Q4 via Ollama
• Front end: Open WebUI with built-in document RAG

Setup

# Embeddings via Ollama
ollama pull bge-large

# Open WebUI (Docker)
docker run -d \
  -p 3000:8080 \
  -v open-webui:/app/backend/data \
  --name open-webui \
  --restart always \
  ghcr.io/open-webui/open-webui:main

# Visit http://localhost:3000
# Settings -> Documents -> embedding model: bge-large
# Settings -> Models -> default: qwen2.5:32b-instruct-q4_K_M

Ingesting a manuscript

Open WebUI accepts .txt, .md and .docx. Export your Scrivener compile as a single Markdown file with chapter headings, then drop it into the Documents tab.

A sensible starting configuration, to be tuned against your own queries:

• Chunk size: 1,200 tokens
• Chunk overlap: 200 tokens
• Top-K retrieval: 6 chunks
• Re-rank: enable the BGE re-ranker if you have headroom

Chunk size is the setting to tune first. Too small and a scene gets cut in half so neither fragment answers the question; too large and six chunks blow past your context window. Ask a question you already know the answer to, and if the model misses, raise overlap before you raise Top-K.

For a deeper walkthrough of the retrieval side, the Ollama ChromaDB RAG pipeline and private AI knowledge base guides cover the same plumbing on non-fiction collections.

---

How do you connect Scrivener to a local model?

Scrivener has no native AI plugin, but it has two extension points: external script execution via the "Open with..." menu, and AppleScript automation on the Mac.

Mac (AppleScript + Ollama)

-- ScrivenerAskAI.scpt
-- Save in ~/Library/Scripts and call from Scrivener via the Services menu

on run {selectedText}
    set thePrompt to "You are a fiction editor helping refine prose. Suggest a single tight, voice-preserving rewrite of this passage. Return ONLY the rewrite, no preamble.\n\nPASSAGE:\n" & selectedText
    set theResponse to do shell script "echo " & quoted form of thePrompt & " | /usr/local/bin/ollama run qwen2.5:32b-instruct-q4_K_M"
    set the clipboard to theResponse
    display notification "Rewrite copied to clipboard" with title "Local AI Editor"
end run

Bind it to a shortcut in System Settings -> Keyboard -> Services. Select a passage, press the shortcut, and the rewrite lands on your clipboard with no network round trip.

Windows (PowerShell + Ollama)

# scrivener-ask.ps1
$selected = Get-Clipboard
$prompt = "You are a fiction editor. Suggest a tight, voice-preserving rewrite of this passage. Return only the rewrite.`n`nPASSAGE:`n$selected"
$response = $prompt | ollama run qwen2.5:32b-instruct-q4_K_M
Set-Clipboard -Value $response

Bind it with AutoHotkey. Copy passage, press shortcut, paste rewrite.

---

How do you make the AI write in your voice?

This is the part most "AI for writers" guides skip, and it matters more than model choice. A model answering in its voice produces paste you have to rewrite anyway. The fix is a style fingerprint: a system prompt that describes your voice and rides along in context every turn.

Build the fingerprint

1. Pick three samples of your tightest prose — one descriptive paragraph, one dialogue scene, one introspective passage. About 600 words total.
2. Run them through the model with this analysis prompt:

Analyze these three passages from the same author. Identify:
- Sentence length distribution (avg, range)
- Punctuation tics (em dashes, semicolons, parentheticals?)
- Adjective density (high or low?)
- POV preferences and tense
- Distinctive vocabulary clusters
- What this author avoids

Be specific and concrete. Quote examples.

PASSAGES:
[paste your 600 words here]

1. Condense the analysis into a 200-word "voice card".
2. Save it as a system prompt:

You are assisting [Author Name], whose voice has the following traits:
[paste 200-word voice card]

Match these traits in any rewrite or continuation. Do not introduce
semicolons; this author does not use them. Sentence length averages
14 words with frequent fragments for emphasis. Vocabulary leans
Anglo-Saxon. Dialogue is sparse, action-tagged, no "he said softly"
adverbs.

When suggesting rewrites, preserve voice over polish.

1. Set it as the default system prompt for your "Writing" model in Open WebUI.

The numbers in that example are placeholders — replace them with whatever the analysis step actually found in your prose. A fingerprint copied from someone else's voice card is worse than none.

---

Prompt patterns that work for long-form fiction

Scene continuation

Continue this scene for ~250 words. Stay in [POV character]'s POV.
Hold the rhythm of the existing prose. Preserve the implied stakes.
Do not introduce new characters. Do not resolve the tension.

[paste last 800 words]

Line edit pass

Edit for line-level prose. Goals:
1. Cut padding (filter words: "began to," "sort of," "just")
2. Replace abstract verbs with concrete ones
3. Tighten dialogue tags
4. Preserve voice and content meaning

Mark each change with [BEFORE -> AFTER] inline. Do not rewrite the whole passage.

[paste 500-word passage]

Inconsistency hunt (with RAG)

Using the manuscript context, identify any continuity errors involving
[character or object]. Examples: eye color, age, possession, location
at a given time. For each error, cite the chapter and line.

Question: Does [character]'s timeline hold across chapters 4-12?

Character voice audit

Sample dialogue from [character] across the manuscript. Score voice
consistency 1-10. Flag any lines that read out of character. Quote
examples.

Use the manuscript context to retrieve dialogue.

Synopsis generation

Generate a one-page synopsis of this manuscript suitable for a query letter.
Constraints:
- 500 words
- Reveals the ending (synopses do)
- Third person, present tense
- No character bios; the synopsis IS the plot
- Author's voice on the page

Manuscript: [load via RAG]

For the planning and ideation side, local AI for content creators overlaps heavily with fiction outlining.

---

What about ghostwriting and NDA work?

If you ghostwrite or edit on contract, the privacy argument sharpens considerably. The client's manuscript is not yours to share, and pasting it into a hosted chat tool may constitute disclosure under the agreement you signed.

1. Per-project model alias. ollama cp qwen2.5:32b-instruct-q4_K_M client-smith gives you a per-client tag, which makes it easy to show which project used which model.
2. Ephemeral chat history. Open WebUI's temporary chat mode keeps no log. Use it for client work.
3. Encrypted project folders. One encrypted folder per client, with the RAG index regenerated per session rather than retained.
4. Read the disclosure clause. Most NDAs prohibit third-party disclosure of client material. A local model does not transmit anything; a hosted one does. Have the contract reflect that distinction rather than assuming it.
5. Document the workflow. Keep a one-page description of your AI use — model name, version digest, and the fact that no manuscript content leaves your machine. Agents and publishers increasingly ask.

---

Common mistakes

1. Buying on core count instead of memory bandwidth. Generation speed is bounded by bandwidth. A machine with twice the compute and the same bandwidth generates prose at roughly the same rate.

2. Skipping the style fingerprint. Without it every output sounds like generic AI. It is half an hour of work that pays back for years.

3. Treating the model like a co-author. It is a tool. Model output is raw material; the prose you publish should be yours.

4. Ignoring the context window. Qwen 2.5's published native context is 32K tokens, extended to about 131K only with YaRN scaling enabled — which is a configuration change, not a default. Assume 32K unless you have explicitly turned extension on.

5. Missing the licence. Mistral Large 2's research licence and the Llama and Gemma community terms are not equivalent to Apache 2.0. If the book is going to be sold, check the terms before the model becomes part of your process.

6. Forgetting to encrypt. A local model is private until the laptop is stolen. FileVault, BitLocker or LUKS is non-negotiable.

7. Pasting client material into a cloud tool "just to compare". That is the moment the NDA breaks. The point of the local stack is that the temptation never arises.

---

Frequently asked questions

Which open model should I start with for fiction?

Qwen 2.5 32B Instruct at Q4_K_M is the sensible default for a 24 GB GPU or a 32 GB Apple Silicon machine: it is Apache 2.0 licensed, so commercial use is unambiguous, and its ~19 GB of weights leave room for context. With 64 GB or more, the 72B version fits at about 43 GB but ships under the more restrictive Qwen License. There is no public fiction benchmark that would justify a stronger claim than that — run the blind test above on your own prose before committing.

Can I run a local writing assistant on a MacBook Air M2?

Yes, within limits. Apple publishes 100 GB/s of memory bandwidth for the base M2, so a 14B model at Q4 (~8.4 GB of weights) has an arithmetic ceiling near 12 tokens per second, and real output will be lower. That is workable for paragraph and scene-level editing, and uncomfortable for full-manuscript queries. macOS also caps the GPU's share of unified memory, so a 16 GB Air has less headroom than the spec implies. A 32 GB machine is the real entry point for this workflow.

How does local AI compare to Claude or ChatGPT for prose?

Frontier hosted models are larger than anything that fits on a desktop, and it would be dishonest to claim otherwise. The gap on prose specifically is not something anyone has measured in a way worth quoting — prose quality has no accepted benchmark. What is not a matter of opinion is the privacy difference: with a local model, manuscript content never leaves your machine. Judge the quality question on your own passages, not on a percentage.

Do publishers require disclosure of AI use?

Policies vary widely. Some publishers forbid AI assistance, most are silent, and some now require disclosure in the contract. The Authors Guild recommends meaningful disclosure when AI was used. Read every contract. Running the model locally does not change the disclosure obligation — it just gives you a cleaner record of exactly what happened.

Will my manuscript ever train a model?

No. Ollama, Open WebUI and ChromaDB run entirely on your machine and do not transmit manuscript content to any external service. You can confirm it by blocking outbound network traffic from those processes: inference and retrieval keep working, because nothing was being sent.

How do I stop the AI writing in its own voice?

Three things, in order of impact: build a style fingerprint and use it as the system prompt; provide 200-300 words of your existing prose as immediate context before any continuation request; and prefer a model whose continuations you have actually blind-ranked against your own writing. Most complaints about "AI voice" are really complaints about an empty system prompt.

Can a co-author share the same local stack?

Yes. Run Ollama on one workstation and reach it over Tailscale, or have each author run their own copy and exchange manuscript files as usual. Neither arrangement needs a cloud component, which keeps NDAs and publishing contracts clean.

What about plot brainstorming and outlining?

Local models handle structural work well — outlines, character arcs, scene lists. Raise the temperature for ideation (0.8-1.0) and lower it for prose work (0.3-0.5). Structure is also where a smaller, faster model is often the better trade: you are iterating, not polishing.

---

The honest summary

The setup takes an afternoon. Model files run 5-45 GB depending on how far up the table you go. The hardware, if you do not already have it, is a one-time cost rather than a monthly one.

What you get is not a better writer than the hosted tools. What you get is a writing assistant that cannot leak, cannot be discontinued, cannot change its terms, and cannot put your unpublished manuscript anywhere except your own disk. For a book under contract, that is a different category of thing — and it should not be a feature you have to fight for.