Can a Local LLM Run My AI Assistant? | Towards Data Science
I replayed the same 27 real production tasks through two local models, one hardware upgrade apart, to find out what it actually takes to replace Claude as the brain behind a 90-tool personal agent.
TL;DR— I replayed 27 real tasks from my own AI agent against two local models, one hardware upgrade apart, scoring both against the same frozen Claude baseline. On a single RTX 3090 capped at a 16K context, a 30B model scored22. 8/100to Claude’s89. 4, and leaked malformed tool-call syntax into a quarter of its answers — broken, not merely worse. On three 3090s with a 256K context, a 122B model scored80. 0, eliminated the malformed calls entirely (0 of 27), and cost$0. 000969 per taskagainst Claude’s$0. 763— roughly787× cheaper. My agent now runs on the local model; I cancelled Claude Max and kept Pro. The honest caveat: model size and context budget changed together and I ran no control, so this measures asystemupgrade, not an isolated model effect. agent — a LangGraph react-agent wired into roughly 90 tools: email, calendar, notes, files, Office, WhatsApp, Discord, image generation, even spawning sub-agents for longer jobs. It ran on Claude from the day I built it, because Claude was the model I trusted it with. A month ago I tried to replace that brain with a model running on my own hardware. It went badly, and I wrote up exactly how badly. Then I changed the hardware and tried again, and got a different answer. Both attempts are below, measured the same way, because the distance between them is the actual finding. Last time, on a smaller stageThe earlier piece —Local LLM Agents on an RTX 3090— tested 5 local models across 2 agent frameworks on 17 tasks: 12 coding, 5 general-agent. qwen3-coder:30b topped that leaderboard cleanly. It was a fair test, and it told the truth about that scope. Jarvis is a different scope entirely. Not 17 tasks in a clean harness — ~90 real tools, a personal-context system prompt, and years of actual messy requests logged in Langfuse, Jarvis’s self-hosted tracing. If the benchmark win was going to generalize anywhere, it should generalize here. The setup: a replay, not a re-runI pulled 28 real historical task prompts straight from Jarvis’s own Langfuse traces (90-day window), stratified across 7 categories — calendar, code, email, files, general, messaging, notes — 4 each. Claude’s side of the comparison is its actual historical production answer, already logged in Langfuse. I did not re-run it. Re-running Claude through a sandboxed harness would mean feeding it fake stubbed tool data it never actually saw, which handicaps it for no good reason — the honest baseline is the answer it really gave, with the real data it really had. That baseline is frozen: it is identical in both rounds below. Only the local model is re-run. The local side is a fresh run through a sandboxed replay harness: the real Jarvis LangGraph agent code, in-process, with every write-capable tool (send email, write to calendar, post to Discord/WhatsApp, write files) intercepted so nothing real happens. Read-only tools that touch real external systems — Outlook COM for email and calendar — are intercepted too, but the mock serves the real recorded output from that task’s original Langfuse trace when one exists, not a generic stub. The local model reasons over the same real inbox and calendar content Claude saw, not invented placeholder text. The wrapper is default-deny: any tool not on a short explicit allowlist gets mocked. That detail earned its keep — between the two rounds Jarvis grew new tools, and default-deny meant they were intercepted automatically instead of quietly executing against my real accounts. Quality is scored independently — not pairwise, to avoid position bias — by an LLM judge (claude-opus-4-8) on a 1–5 scale mapped to 0–100, applied identically to every answer in both rounds. Worth flagging plainly: the judge is a Claude model, scoring Claude’s own answers alongside the local ones. That’s self-preference bias — a documented effect in LLM-as-judge setups, where a model tends to rate outputs from its own family more favorably. I can’t cleanly rule it out, and it probably flatters Claude. It’s a real limitation of this methodology, not a footnote to bury — and it mattersmorein round 2, where the local model is close enough for a thumb on the scale to change how you read the result. Cost is the other half of the comparison, and it needs an instrument. Claude’s side is easy — Langfuse already recorded the real API billing for every historical call. The local side is electricity, which nothing bills you for per task. Every local run is logged as a priced experiment inHomeLab Monitor, the open-source monitoring stack I maintain: it attributes measured GPU energy to the window each run occupied and converts it at my actual dual-rate tariff. Neither side of the cost comparison is an estimate. Three passes before the number was trustworthyWorth a paragraph, because the numbers below are only worth publishing if the process that produced them holds up. The first scoring pass had a silent bug: the judge model wraps its response in a list of content blocks, not a plain string, and the scorer’s JSON parser choked on ~40 of 54 judge calls — caught, logged, and quietly replaced with a neutral fallback score instead of erroring out loud. That run said Claude 71. 6 vs qwen 43. 5. Mostly noise. Fixed and re-run, the gap got more extreme — Claude 89. 2, qwen bucketed at exactly 15 or 45 on nearly every task. Investigating that found the real problem: 16 of the 28 tasks are calendar, email, notes, or messaging — categories where the safety wrapper correctly mocks the underlying Outlook COM calls, but the mock was returning a generic canned string instead of real data. qwen was reasoning over placeholder text while Claude’s historical answer was written against a real inbox. That wasn’t measuring model capability, it was measuring who got real data and who didn’t. Fixed by serving the real recorded tool output from the original trace wherever the mocked tool matched one the model actually called. Re-run clean, then a third bug — a sustained Claude API rate limit during scoring silently neutral-scored another batch of judge calls. Added a real backoff loop, re-ran. That neutral fallback deserves one more sentence, because it bit me again in round 2 — not as a bug, but as a thing I had to rule out. The judge returns exactly70. 0when it hits an infrastructure error. In round 2, eight of 27 tasks scored exactly 70. 0. Before believing any of it I checked every rationale string: all eight were genuine “3/5” judgements with substantive reasoning, zero infrastructure fallbacks, zero unparsed responses. A number that happens to equal your failure sentinel is worth verifying rather than celebrating. Round 1: one 3090, and a model that couldn’t hold the roomThe first attempt ran qwen3-coder:30b on a single RTX 3090 shared with everything else in the home lab. That sharing set a hard limit: with ~18 GB of weights resident, the context window had to come down to 16,384 tokens. Jarvis’s tool schemas and system prompt alone nearly fill that. Claude averaged89. 4 / 100. qwen3-coder:30b averaged22. 8 / 100. qwen never won a single category. Calendar and general — the tasks leaning more on reasoning over given context than on chaining tools correctly — were its best showing, and even those landed at a third of Claude’s score. Where it brokeQuality score is the summary; it’s not the interesting failure mode. qwen leaked a malformed tool-call tag into7 of 27final answers (25. 9%) — instead of issuing a proper LangGraph tool call, it emitted raw text like. . . straight into the response a user would have read. Tool-overlap recall averaged14. 8%, measured only across the 18 of 27 tasks where the real historical trace actually used at least one tool. Most of the time qwen either called different tools than the ones that historically solved the task, or none at all. And on 2 of 27 tasks, a genuine orchestration failure: pilot-17 (email, 24 tool calls) and pilot-27 (messaging, 27 tool calls) both got stuck calling the same already-answered tool over and over instead of synthesizing an answer and stopping. I checked the raw logs specifically to rule out the earlier mock-data bug: both tasks correctly received real replayed data. A stopping-condition failure, not starvation. Two ways to lie about a mockOne task made both models actually callsend_email(. . . )inside the harness — correctly intercepted, nothing sent, in either case. Claude then told the user the email had been sent. It hadn’t; that’s a fabrication. qwen correctly reported that the send didn’t go through. It’s tempting to read that as “qwen is more honest. ” It isn’t that clean — qwen is also the model leaking raw function-call syntax into user-facing text a quarter of the time, and looping twice. Both models mishandled the mocked environment, in opposite directions: one over-claimed success, the other under-delivered on basic task completion. Worth knowing, not worth a moral. Same model, much harder roomHere’s what made round 1 a real finding rather than a cheap dunk: qwen3-coder:30b is the exact same model, same weights, same GPU, that scored 100% task success in my earlier controlled benchmark — 17 tasks, a much smaller tool surface, a purpose-built harness. Dropped into Jarvis’s real environment, its reliability collapsed. Not because the model got worse. Because the room got much harder, and a model that’s excellent on a scoped benchmark is not automatically a safe drop-in for a large, real, production agent surface. That was the honest conclusion, and the original version of this article ended there:Jarvis stays on Claude. Then I changed the room. Round 2: three 3090s, and a model that fitsThe new box hasthree RTX 3090s — 72 GB of VRAM. That budget buys a different class of model: a122-billion-parameter mixture-of-experts at Q3_K_M, about 53 GB of weights, served entirely on GPU with no CPU spill. And it buys the thing that turned out to matter most — a256,000-token context window, a 16× increase over round 1’s 16,384. Same 27 tasks. Same frozen Claude baseline. Same judge, same rubric, same harness. 80. 0 / 100. Chart by the author. That’s89. 4% of Claude’s score, up from 25% in round 1. On14 of the 27 tasks, the local model scored at or above Claude. By category, the shape of the change is more interesting than the average:CategoryClaude30B122Bcalendar903078code872573email921587files881564general853090messaging872274notes972292Chart by the author. generalis the one category where the local model beats Claude outright — 90 to 85. notesandemailland within a few points. filesis where it’s still clearly behind at 64, and that’s the category most dependent on chaining several tool calls correctly rather than reasoning over content it was handed. Reliability is where the change is starkestThe malformed tool-call leak went from7 of 27 to zero. Not reduced — gone. In round 1 a quarter of answers had rawsyntax bleeding into user-facing text; in round 2, none did. Tool-overlap recall went from14. 8% to 38. 0%— 2. 6× better, and still the clearest remaining gap. The model now reliably produceswell-formedtool calls; it still often picks different tools than the ones that historically solved the task. Chart by the author. The task that still doesn’t fitOne task never ran in either round. pilot-12is a code dump: 336,906 characters. Round 1 excluded it against a 16,384-token window without ceremony. Round 2 had 256,000 tokens to work with — and it still doesn’t fit. Ollama rejected it precisely:request (281190 tokens) exceeds the available context size (256000 tokens)The ceiling moved 16×, and the task overflowed it by roughly 25,000 tokens anyway. A useful reminder that “bigger context” is a moving target real work keeps outrunning — and the reason both rounds report 27 tasks rather than 28. What this measurement cannot tell youI want to be direct about the limits here, because the headline is tempting to over-read. Two variables changed at once. Round 2 is not an isolated model upgrade. It is a bigger modelanda 16× larger context windowandthree GPUs instead of one shared card. I did not run the control that would separate them — qwen3-coder:30b on the new box at 256K context. Without it, I can tell you thesystemgot dramatically better; I cannot tell you how much belongs to the extra parameters versus simply no longer truncating Jarvis’s tool schemas and personal context at 16K. My honest guess is that context is doing more of the work than parameter count, because round 1’s most characteristic failures — malformed tool calls, wrong tool selection, loops — are exactly what you’d expect from a model whose tool definitions are being cut off mid-schema. The fact that the malformed-call rate went toexactly zeropoints the same way: that reads more like “the schemas fit now” than “the model got smarter. ” But that is a hypothesis I have not tested, and I’d rather label it as one than dress it up as a conclusion. The judge is still a Claude model, with the same self-preference caveat as round 1 — now applied to a comparison where the local model is competitive, which is exactly where that bias matters most. The local cost nearly went in wrong, and the reason generalises to any distributed setup. Energy gets attributed on the host being watched. In round 1 the machine driving the benchmark and the machine running the model were thesame box, so that was correct without my having to think about it. In round 2 they aren’t — the harness runs on my desktop, the model runs on the GPU box — and the first pass duly attributed the work to the desktop’s idle card. The result came back as a tidy0. 0 kWh. Nothing errored. I caught it only because zero was too clean to be true, and re-measured from scratch: sampling all three cards at 1 Hz, integrating power across each task’s exact window, and correcting for a 4-second clock offset between the two machines. If you take one habit from this piece, take that one — an instrument that agrees with your hopes deservesmorescrutiny, not less, and “suspiciously round” is a reading worth chasing rather than celebrating. What it actually costsPriced at my home electricity tariff, the 27 tasks drew0. 1573 kWhacross the three cards. per taskvs ClaudeClaude$0. 763106—qwen3-coder:30b$0. 000147925,159× cheaperqwen3. 5 122B$0. 000969787× cheaperChart by the author. Two things worth reading off that table. The local model is still roughlythree orders of magnitude cheaperthan the API. And the upgrade was not free: the 122B costs6. 6× more per taskto run than the 30B did. Going from “unusable and nearly free” to “usable and nearly free” cost real electricity — it’s just that “real electricity” here means fractions of a cent. One caveat on that number: it’s gross, including idle draw. Those three cards pull about107 W just holding the model resident, because I keep it loaded permanently for latency. If you’d only spin a model up on demand, your per-task cost would look different — lower marginal cost, higher latency. What actually changed in JarvisJarvis now runs on the local 122B model. It has for about a week. I cancelled the Claude Max subscription and kept Pro. That’s the honest scoreboard: not “local models won,” but “local models got good enough that the top tier of my API spend stopped earning its keep. ” Claude is still the better model on this benchmark — 89. 4 to 80. 0 — and it’s still what I reach for when I want an answer to be right the first time. What changed is that the gap stopped beingcategorical. In round 1 the local model wasn’t a worse option, it was a broken one: a quarter of its answers had tool-call syntax bleeding into the text. You can’t put that in front of a real inbox at any price. In round 2 it’s a genuine trade — a few points of quality, on tasks I can identify in advance, for roughly a 99. 9% cost reduction and my data never leaving the house. The remaining gap is also specific rather than diffuse, which makes it manageable. filesat 64 and 38% tool recall both say the same thing: this model is good at reasoning over what you hand it and still mediocre at deciding which tools to reach for. So the tasks I’d still route to Claude are the multi-step ones, and that’s a routing rule, not a subscription. If you’re running the same experiment: the thing that moved the needle wasn’t picking a cleverer model. It was giving an adequate model enough context to see the whole tool surface at once. I’d test that before spending money on parameters. Where’s your line — how many quality points is a task worth before you’d stop sending it out of the building?All images in this article — charts and photographs — were created by the author. Every energy and cost figure here was measured withHomeLab Monitor— an MIT-licensed, single-container stack I maintain for pricing local experiments. The shorter, more technical write-up of this harness lives ondev. to.