Small local models through Ollama are free and fast enough for a first pass on what always-on agents do: bucket a message, summarise what changed, flag what needs a human. In my CPU-only run they got four to six of ten strict labels right, with valid JSON every time, so pair them with rules and a human check. For judgement and tools, pay.
Part of the Always-on agents series, post 23 of 33.
I keep two kinds of model in my company. The expensive kind makes the decisions I could not write down as rules, like which clip from a brand's footage fits what its audience is responding to this hour. The cheap kind does the sorting around those decisions. For an always-on agent that runs every hour on a machine you own, the cheap kind can be free, and the question this post answers is: free at what quality, and for which jobs?
What the frameworks say before I say anything
I want to start with the makers' own words, because they are stricter than most blog posts about local models. Hermes Agent's local-model guide lists four Ollama models and marks only one, a 31-billion-parameter one needing more than 24GB of RAM, as supporting tool calls. Its 3B option is marked no, with the line that models without tool-call support can only chat; they cannot take actions. The same page says agentic work with tools needs a 64,000-token context. That is the honest ceiling: a small local model is not going to drive an agent framework's tool loop. What it can do is the bounded step inside your own loop, which is what I measured.
The task, the models, the machine
The task is the one every agent in this series needs somewhere: read a short message and put it in one of five buckets, answering in strict JSON so the rest of the program can trust it. Ten messages, fixed, with a known right answer each. Temperature zero. Ollama's JSON mode on, so a model that drifts into prose fails visibly rather than silently.
The models are the ones that fit the machines in this series: qwen2.5 at 1.5 billion parameters, llama3.2 at 1 billion, and qwen2.5 at 0.5 billion as the floor. All in the 4-bit quantisation Ollama ships by default. Nothing bigger, on purpose: the Pi and the free cloud boxes cannot run bigger, and the frameworks have already told us the 3B models do not call tools either.
The machine is my own Mac, but with a catch I need to be plain about. I ran Ollama inside a Docker container capped at three gigabytes of memory and four CPU cores. Ollama's Docker image documents GPU paths for Nvidia and AMD only, and a Linux container on a Mac has no path to the Apple GPU, so these numbers are a CPU-only, memory-capped run, which is roughly what a small cloud box or a Pi gives you. Native Ollama on a Mac uses the GPU and is a different machine; a Linux server with a modest Nvidia card, different again. Treat the numbers as the floor.
bench.py"""Fixed task for small local models: classify ten messages and emit JSON. Runs against an Ollama
server. Records accuracy, seconds per call and generation speed. Everything printed is what the post quotes."""
import json, sys, time, urllib.request
HOST = sys.argv[1] if len(sys.argv) > 1 else "http://127.0.0.1:11434"
MODELS = sys.argv[2:] or ["qwen2.5:1.5b", "llama3.2:1b"]
LABELS = ["bug", "feature", "billing", "praise", "other"]
MESSAGES = [
("The export button crashes the app every time I press it on Android", "bug"),
("Could you add a dark mode? My eyes hurt at night", "feature"),
("I was charged twice this month, please refund one of them", "billing"),
("Honestly this is the best tool I have used all year, thank you", "praise"),
("Where is your office? Just curious", "other"),
("Login fails with error 500 since this morning", "bug"),
("It would be great to connect this to Slack", "feature"),
("How do I change the card on file?", "billing"),
("Love the new editor, keep it up", "praise"),
("Video upload stuck at 99 percent for an hour", "bug"),
]
SYSTEM = ("You classify customer messages. Reply with JSON only, no prose, exactly in the form "
'{"label": "<one of bug, feature, billing, praise, other>"}.')
def ask(model, text):
body = {"model": model, "stream": False, "format": "json", "options": {"temperature": 0},
"messages": [{"role": "system", "content": SYSTEM}, {"role": "user", "content": text}]}
req = urllib.request.Request(HOST + "/api/chat", data=json.dumps(body).encode(),
headers={"Content-Type": "application/json"})
t0 = time.time()
with urllib.request.urlopen(req, timeout=600) as r:
out = json.load(r)
dt = time.time() - t0
try:
label = json.loads(out["message"]["content"]).get("label", "").strip().lower()
except Exception:
label = "INVALID:" + out["message"]["content"][:40]
return label, dt, out.get("eval_count", 0), out.get("eval_duration", 0)
for model in MODELS:
ask(model, "hello") # first call loads the model; not timed
right, times, toks, durs, invalid = 0, [], 0, 0, 0
for text, want in MESSAGES:
label, dt, n, d = ask(model, text)
times.append(dt); toks += n; durs += d
if label.startswith("INVALID"): invalid += 1
elif label == want: right += 1
tps = toks / (durs / 1e9) if durs else 0
print("%-14s accuracy %2d/10 invalid-json %d mean %.1fs per call max %.1fs %.1f tok/s generation"
% (model, right, invalid, sum(times) / len(times), max(times), tps))The numbers
Run on 20 October 2026, one model at a time, each pulled fresh into the container. The first call loads the model and is not timed. This is the output, unedited:
CPU-only container on my Mac, 3 GB and 4 cores, 20 October 2026{"version":"0.40.6"} (Ollama in the container)
qwen2.5:1.5b accuracy 4/10 invalid-json 0 mean 2.1s per call max 4.3s 6.0 tok/s generation
container memory while loaded: 1.358GiB / 3GiB
llama3.2:1b accuracy 6/10 invalid-json 0 mean 5.3s per call max 15.0s 1.8 tok/s generation
container memory while loaded: 1.833GiB / 3GiB
qwen2.5:0.5b accuracy 6/10 invalid-json 0 mean 0.9s per call max 1.5s 15.6 tok/s generation
container memory while loaded: 2.43GiB / 3GiB
qwen2.5:0.5b a8b0c5157701 397 MB 14 seconds ago
llama3.2:1b baf6a787fdff 1.3 GB About a minute ago I expected the biggest of the three to win and it did not. On this strict five-way sort, qwen2.5 at 1.5B got four of ten right, llama3.2 at 1B got six, and qwen2.5 at 0.5B also got six while answering in under a second. Ten messages is far too small a sample to rank models by, so I will not; what I will say is that none of them is at the level where I would let the label drive an action unattended. Two results I trust more than the accuracy numbers: all three returned valid JSON on every call with JSON mode on, which is the property that keeps a loop from breaking, and the speeds are honest about CPU-only life: six, two and sixteen tokens a second, with the 1B model's worst call taking fifteen seconds. The memory readings are the container's total while a model was loaded; the third one was taken while the second model was still resident, so read it as a ceiling, not a per-model figure.
What that means for an agent
Here is how I would use what I measured, job by job.
- Sorting and flagging: run it locally, as a first pass. Ten bucketed messages an hour, or a hundred, is well inside what these models do on a CPU in a minute or two. At four to six of ten on a strict sort, the label is a hint for a person or a rule to confirm, not a verdict. This is the step in the watch agent and the Slack triage build, and it costs nothing after the hardware.
- Short summaries: run it locally, read them with low expectations. Three lines about what changed in a feed is fine. A summary you would forward to a client is not.
- Anything with tools, or anything open-ended: send it out. The frameworks are explicit that small models cannot take actions through tools. Mine is explicit too: the clip-matching judgement in Leo runs on a paid model, hourly, and the sorting around it does not.
- Always validate the output. JSON mode plus a check that the label is one of your five. A small model that answers something almost right is more dangerous than one that fails loudly.
What a paid model buys you, in my experience
Three things, in order. Judgement on inputs you have never seen before, which is the whole reason I run a paid model in Leo for matching footage to what an audience is responding to. Tool calling that actually works across a long context, which is the gap the Hermes guide names. And consistency across thousands of calls, which is what I notice when a small model is right six times out of ten and one of the other four goes into a customer's inbox. The costs are in the pricing post; the rule for when to pay is in the agent versus automation post: if the decision fits a rule, write the rule; if it fits a small model, run it free; if it needs judgement, pay for it, and cap what it can spend.
The $0 setup, in four lines
Terminal, on the agent boxcurl -fsSL https://ollama.com/install.sh | sh
ollama pull qwen2.5:1.5b
curl -s http://127.0.0.1:11434/api/tags # {"models":[...]} means it is up
# then point your agent at http://127.0.0.1:11434 and keep it on the same machineKeep Ollama bound to localhost, give the agent a model name in its environment file rather than in its code, and put the model pull in your setup script so a fresh machine comes up with it. If you later want the 31B model the frameworks recommend for tools, that is a different machine and a different budget, and at that point I would compare it honestly against a paid API before buying the RAM.
Sources
- Nous Research, Run Hermes locally with Ollama (model table, tool-call support, context requirement) (accessed October 20, 2026)
- Ollama, Ollama Docker documentation (accessed October 20, 2026)
- Ollama, qwen2.5 model library page (accessed October 20, 2026)
- Ollama, llama3.2 model library page (accessed October 20, 2026)
Written by Lera Leonteva. Measured on 20 October 2026 in a CPU-only container capped at 3 GB and 4 cores on my Mac; your machine will differ.