Ollama Function Calling and Tool Use: The Practical Guide

Published on April 23, 2026 • 18 min read

Function calling is the feature that turns a chatbot into an agent. The model stops being a text-in-text-out box and starts deciding which tool to invoke, what arguments to pass, and how to combine results into a final answer. Ollama added native tool support in version 0.3.0, and as of 0.4.x it works well enough that I have replaced three OpenAI-based agents in my own stack with local Ollama equivalents.

The catch: function calling is the area where local LLMs are most uneven. Some models nail it. Some technically support it but produce garbage JSON. Some get confused above 3 tools. The official Ollama docs do not warn you. This guide does.

I tested seven popular models against ten real tool-calling tasks. I documented exactly which combinations work, where they break, and how to engineer around the failure modes. By the end, you will have a working multi-tool agent running fully on your machine.

---

Quick Start: First Tool Call in 90 Seconds {#quick-start}

# Install Ollama and pull a model that handles tools well
ollama pull llama3.1:8b

# tools_minimal.py
import ollama
import json

def get_weather(city: str) -> str:
    # Stub: in real life, hit a weather API
    return json.dumps({"city": city, "temp_c": 22, "condition": "sunny"})

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get current weather for a city",
        "parameters": {
            "type": "object",
            "properties": {
                "city": {"type": "string", "description": "City name"},
            },
            "required": ["city"],
        },
    },
}]

messages = [{"role": "user", "content": "What is the weather in Paris?"}]
res = ollama.chat(model="llama3.1:8b", messages=messages, tools=tools)

if res["message"].get("tool_calls"):
    for call in res["message"]["tool_calls"]:
        result = get_weather(**call["function"]["arguments"])
        messages.append(res["message"])
        messages.append({"role": "tool", "content": result, "name": call["function"]["name"]})
    final = ollama.chat(model="llama3.1:8b", messages=messages, tools=tools)
    print(final["message"]["content"])
else:
    print(res["message"]["content"])

Run it:

pip install ollama
python tools_minimal.py
# > "It is currently 22°C and sunny in Paris."

That is the entire shape of tool calling: model decides to invoke a tool, you execute it, you append the result, the model uses the result to write the final answer.

---

Which Models Actually Work {#models}

This is the question nobody answers honestly. Here is my benchmark across 10 tool-calling tasks (single-tool, multi-tool, error-recovery, and chained workflows). Score is "task completed correctly without intervention" out of 10.

My recommendations:

• Best small (under 16GB RAM): qwen2.5:7b or llama3.1:8b
• Best medium (32GB RAM): qwen2.5:14b
• Best large (96GB+ RAM): llama3.1:70b or firefunction-v2
• Avoid for tools: gemma2 family, phi3.5 mini for multi-tool work

For a fuller comparison of these model families, see our best Ollama models guide. For coding-specific tool work, best local AI models for programming goes deeper.

---

How Ollama Function Calling Actually Works {#how-it-works}

Ollama implements an OpenAI-compatible tool calling API. The flow is:

1. You send: messages + tools (JSON schemas)
2. Model returns either:
   a. A normal text message (no tool needed), or
   b. A "tool_calls" list with name + arguments
3. You execute each tool call locally
4. You append the tool result as a "tool" role message
5. You call the model again with the updated messages
6. Model returns the final natural-language answer

Ollama parses the model's structured output into the OpenAI tool-calls format under the hood. This works because Llama 3.1+, Qwen 2.5+, Mistral Nemo, and similar models were post-trained on tool-calling data with consistent special tokens or JSON schemas.

The most important consequence: the model decides whether to call a tool. If it thinks the question is conversational ("hello, who are you"), it will not invoke a tool even if one is available. This is correct behavior — but if your application requires structured output 100% of the time, set the system prompt to enforce it.

---

Step 1: Define Tools With Good Schemas {#schemas}

Tool schemas use JSON Schema. The quality of your schema directly determines the model's accuracy. Two principles:

1. Description is everything. The model picks tools and arguments based on the descriptions, not the names.
2. Be strict. Specify required fields, enum values, and exact types. Loose schemas → loose calls.

A well-defined tool:

search_tool = {
    "type": "function",
    "function": {
        "name": "search_internal_docs",
        "description": (
            "Search the company's internal documentation for relevant content. "
            "Use this when the user asks about company policies, procedures, "
            "engineering wikis, or internal codebases. Do not use for public knowledge."
        ),
        "parameters": {
            "type": "object",
            "properties": {
                "query": {
                    "type": "string",
                    "description": "Search keywords (3-8 words). Use specific technical terms.",
                },
                "department": {
                    "type": "string",
                    "enum": ["engineering", "hr", "security", "finance", "all"],
                    "description": "Filter by department; use 'all' if unknown.",
                },
                "max_results": {
                    "type": "integer",
                    "description": "Number of results to return (1-10).",
                    "default": 5,
                },
            },
            "required": ["query", "department"],
        },
    },
}

A bad version of the same tool:

{
    "type": "function",
    "function": {
        "name": "search",
        "description": "Search docs",
        "parameters": {
            "type": "object",
            "properties": {
                "q": {"type": "string"},
            },
        },
    },
}

The bad version will fire on every question, miss the department filter, and pass weird queries. Description quality is the difference between a 6/10 tool agent and a 9/10 tool agent.

---

Step 2: Multi-Tool Agent Pattern {#multi-tool}

Real applications expose multiple tools. The agent loop must handle: zero tools called, one tool, multiple tools in one turn, and chained tools across turns.

# agent.py
import ollama
import json

# --- Tool implementations ---
def get_weather(city: str) -> str:
    return json.dumps({"city": city, "temp_c": 22, "condition": "sunny"})

def search_news(query: str, limit: int = 3) -> str:
    return json.dumps([
        {"title": f"Result about {query}", "url": "https://example.com/1"}
    ])

def calculate(expression: str) -> str:
    try:
        return json.dumps({"result": eval(expression, {"__builtins__": {}}, {})})
    except Exception as e:
        return json.dumps({"error": str(e)})

TOOL_REGISTRY = {
    "get_weather": get_weather,
    "search_news": search_news,
    "calculate": calculate,
}

TOOLS_SCHEMA = [
    {"type": "function", "function": {
        "name": "get_weather",
        "description": "Get current weather for a city.",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string", "description": "City name."}},
            "required": ["city"],
        },
    }},
    {"type": "function", "function": {
        "name": "search_news",
        "description": "Search recent news headlines.",
        "parameters": {
            "type": "object",
            "properties": {
                "query": {"type": "string", "description": "Search keywords."},
                "limit": {"type": "integer", "description": "Number of results.", "default": 3},
            },
            "required": ["query"],
        },
    }},
    {"type": "function", "function": {
        "name": "calculate",
        "description": "Evaluate a math expression. No variables.",
        "parameters": {
            "type": "object",
            "properties": {"expression": {"type": "string"}},
            "required": ["expression"],
        },
    }},
]

# --- Agent loop ---
def run_agent(user_question: str, model="qwen2.5:7b", max_turns=6):
    messages = [
        {"role": "system", "content": (
            "You are a careful assistant. Use the provided tools when needed. "
            "Do not invent tool results. If a tool fails, explain what happened "
            "and try a different approach."
        )},
        {"role": "user", "content": user_question},
    ]

    for turn in range(max_turns):
        res = ollama.chat(model=model, messages=messages, tools=TOOLS_SCHEMA)
        msg = res["message"]
        messages.append(msg)

        tool_calls = msg.get("tool_calls") or []
        if not tool_calls:
            return msg["content"]

        for call in tool_calls:
            name = call["function"]["name"]
            args = call["function"]["arguments"]
            if name not in TOOL_REGISTRY:
                result = json.dumps({"error": f"unknown tool: {name}"})
            else:
                try:
                    result = TOOL_REGISTRY[name](**args)
                except TypeError as e:
                    result = json.dumps({"error": f"bad arguments: {e}"})
                except Exception as e:
                    result = json.dumps({"error": str(e)})
            messages.append({"role": "tool", "name": name, "content": result})

    return "Reached max turns without a final answer."

if __name__ == "__main__":
    print(run_agent("What is the weather in Tokyo, and what is 17 * 23?"))

Key patterns to copy:

1. TOOL_REGISTRY dispatch: maps tool names to Python callables.
2. Bounded loop: max_turns prevents runaway loops if the model keeps calling tools.
3. Error wrapping: every tool call is wrapped in try/except and returns JSON, so the model can recover gracefully.
4. System prompt: enforces grounded behavior without inventing tool results.

This pattern handles 90%+ of practical tool-calling needs.

---

Step 3: Tool Use From JavaScript / TypeScript {#javascript}

For Node and browser apps, the official ollama JS package exposes the same API.

// agent.ts
import ollama from "ollama";

const tools = [
  {
    type: "function",
    function: {
      name: "get_weather",
      description: "Get current weather for a city.",
      parameters: {
        type: "object",
        properties: { city: { type: "string", description: "City name." } },
        required: ["city"],
      },
    },
  },
];

const TOOLS: Record<string, (args: any) => Promise<string>> = {
  get_weather: async ({ city }) =>
    JSON.stringify({ city, temp_c: 22, condition: "sunny" }),
};

async function runAgent(question: string) {
  const messages: any[] = [{ role: "user", content: question }];

  for (let turn = 0; turn < 6; turn++) {
    const res = await ollama.chat({
      model: "qwen2.5:7b",
      messages,
      tools,
    });
    messages.push(res.message);

    const calls = res.message.tool_calls ?? [];
    if (calls.length === 0) return res.message.content;

    for (const c of calls) {
      const fn = TOOLS[c.function.name];
      const result = fn ? await fn(c.function.arguments) : "{}";
      messages.push({ role: "tool", name: c.function.name, content: result });
    }
  }
  return "Hit max turns.";
}

runAgent("Weather in Paris?").then(console.log);

For full Node/Next.js patterns including streaming and the Vercel AI SDK, see our companion guide on Ollama with JavaScript and TypeScript.

---

Step 4: Common Patterns {#patterns}

Pattern 1: Forced tool calls. Use tool_choice (Ollama 0.4+) to force the model to call a specific tool:

ollama.chat(
    model="qwen2.5:7b",
    messages=messages,
    tools=tools,
    tool_choice={"type": "function", "function": {"name": "search_internal_docs"}},
)

Pattern 2: Structured output without tools. When you do not need tool execution, you just want JSON output, use Ollama's JSON mode:

ollama.chat(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "Extract entities from: ..."}],
    format="json",
)

Pattern 3: Tool result chaining. When tool A's output feeds tool B, structure tool descriptions to encourage the chain:

"Use search_news first to find article URLs, then summarize_article on each URL."

The model handles the orchestration if your descriptions are explicit.

Pattern 4: Cost-effective routing. Use a small model (qwen2.5:7b) for tool selection, hand off to a larger model (llama3.1:70b) for the final synthesis. Saves significant time on multi-step agents.

---

Step 5: Error Handling and Retry {#error-handling}

Tool calls fail. Networks drop. APIs return weird JSON. The agent must survive.

def safe_call_tool(tool_fn, args, retries=2):
    last_error = None
    for attempt in range(retries + 1):
        try:
            return tool_fn(**args)
        except Exception as e:
            last_error = str(e)
            if attempt < retries:
                continue
            return json.dumps({"error": f"tool failed after {retries+1} attempts: {last_error}"})

Three failures to plan for:

1. Bad arguments from the model. The model passes a string where you wanted an int. Wrap in try/except and return a structured error so the model can retry.
2. Tool downtime. External APIs return 500s. Always set a timeout and return an error JSON.
3. Hallucinated tool names. The model sometimes invents tool names that do not exist. Catch this in dispatch and return a list of valid tool names to help the model recover.

A robust dispatcher:

def dispatch(name: str, args: dict) -> str:
    if name not in TOOL_REGISTRY:
        return json.dumps({
            "error": f"unknown tool: {name}",
            "available_tools": list(TOOL_REGISTRY.keys()),
        })
    return safe_call_tool(TOOL_REGISTRY[name], args)

The agent recovers gracefully because it sees the available tools and corrects on the next turn.

---

Step 6: Streaming With Tool Calls {#streaming}

Tool calls and streaming have a tricky interaction. Ollama emits the tool-call payload at the end of the stream, not as deltas. Pattern:

stream = ollama.chat(
    model="qwen2.5:7b",
    messages=messages,
    tools=tools,
    stream=True,
)

text_parts = []
final_message = None
for chunk in stream:
    msg = chunk.get("message", {})
    if msg.get("content"):
        text_parts.append(msg["content"])
        print(msg["content"], end="", flush=True)
    if chunk.get("done"):
        final_message = msg

if final_message and final_message.get("tool_calls"):
    # process tool calls as usual
    ...

For text-only responses, streaming gives you token-by-token UI updates. For tool-driven responses, the user sees nothing until the tools resolve. To improve UX, render a "Calling search_internal_docs..." indicator the moment you see a tool call.

---

Benchmarks: Latency and Reliability {#benchmarks}

Tested on a MacBook Pro M3 (16GB) with three tools registered, 50 questions per model:

Schema-correct rate = (model produced argument JSON that validated against the schema) / total calls

For most apps qwen2.5:7b is the best balance of latency and reliability. llama3.2:3b is fastest but unreliable above 1 tool.

---

Pitfalls and Gotchas {#pitfalls}

1. The model sometimes ignores tools and answers from training data. Solution: explicit system prompt — "If you do not have current information, you MUST call a tool. Do not answer from memory."

2. Argument types are inconsistent. A model may return "limit": "5" (string) when you specified integer. Coerce types in the dispatcher: int(args.get("limit", 5)).

3. Tool descriptions over 200 chars hurt accuracy. Keep them under 200 chars. Move long context into the system prompt, not the schema.

4. Too many tools = degraded performance. Above 6-8 tools, even good models start mis-routing. Group related tools or split into sub-agents.

5. Models call tools redundantly. They sometimes call get_weather twice in a row for the same city. Add deduplication at the dispatcher: cache results per turn.

6. Local models lag cloud models on chained reasoning. A single tool call is solid; 5+ chained calls is where local models still trail GPT-4 and Claude. Use larger models or break the workflow into smaller steps.

7. Memory pressure on long agent loops. Each turn appends to the message history. After 10 turns, context can hit 8K+ tokens. Trim older tool results when they are no longer relevant.

8. JSON mode is not tool calling. format="json" returns JSON in the content field but does not invoke tools. Different feature, different use case.

---

Production Hardening {#production}

For a production agent:

• Per-tool timeout (30s default, lower for fast tools)
• Bounded max_turns (4-8 for most agents)
• Structured error responses with retry hints
• Logging of every tool call and result (auditability)
• Rate limiting on expensive tools (web fetches, paid APIs)
• Schema validation of tool arguments before execution
• Dedup of identical consecutive tool calls
• Concurrent tool execution when safe (asyncio gather)
• Graceful fallback to text-only mode if tools repeatedly fail
• Unit tests for each tool and an integration test for the full loop

For broader production patterns including auth, monitoring, and multi-user concurrency, our Ollama production deployment guide covers the hosting layer. For knowledge-augmented agents, pair this with the Ollama + ChromaDB RAG pipeline.

---

Real Use Cases I Have Shipped {#use-cases}

Three agents I run in production today, all on Ollama:

1. Internal support bot. Tools: search_docs, lookup_user, create_jira. Model: qwen2.5:14b. Replaced a Zendesk AI add-on. ~85% deflection rate.

2. Personal finance assistant. Tools: get_transactions, categorize, forecast_balance, flag_anomalies. Model: llama3.1:8b. Runs nightly, sends summary email.

3. Research agent. Tools: search_arxiv, fetch_paper, summarize_paper, save_to_obsidian. Model: llama3.1:70b on Mac Studio. Replaced ChatGPT + manual paper-reading workflow.

In all three cases, the value is not raw model intelligence — it is the LLM acting as a careful router across a small set of well-defined tools. That is exactly what local LLMs are good at.

---

What Is New in Ollama 0.4 {#whats-new}

The big changes that matter for tool calling:

• tool_choice parameter — force a specific tool or no tool
• Better error reporting — tool name validation, schema feedback
• Improved JSON adherence — fewer malformed argument JSON outputs
• Streaming + tools — tool calls now arrive as a final delta you can detect cleanly
• Smaller-model improvements — 3B and smaller models are more reliable for single-tool use

The official reference is the Ollama API documentation. For the underlying tool-calling techniques used in Llama 3.1 and 3.2, see Hugging Face's Llama 3.1 deep dive.

---

Closing Take {#closing}

Function calling is what makes local LLMs genuinely useful for real workflows. Anyone can build a local chatbot. Building a local agent that books meetings, searches your docs, runs SQL queries, and summarizes the results — that is the unlock. Ollama 0.4 is good enough for production tool calling on the right models with the right schemas.

If you are starting today, my exact recipe: qwen2.5:7b for development, the agent loop pattern above, three to five well-described tools, a tight system prompt, and an evaluation harness with 20 prompts that exercise every tool. Ship that and iterate.