My first proper attempt at running a local LLM was sometime in 2024. I pulled Mistral 7B through Ollama, watched it load, typed something in, and felt that brief rush of excitement you get when something genuinely cool works. A language model, on my laptop, no API key, nothing going anywhere.
Then I used it for ten minutes and went back to GPT-4.
The gap wasn’t subtle. Ask it anything GPT-4 handled cleanly and it would hallucinate, lose the thread, or give you something that needed more work than if you’d just written it yourself. Same with Llama 2. I stopped bothering.
About a year later I came back, and it was a different experience.
What changed
The models genuinely got better, and not incrementally. Llama 3 was trained on roughly seven times the data of Llama 2. Gemma went from a small experiment to Gemma 4, which landed in April 2026 with a 128K context window, multimodal support, and small variants built to run fast on a normal laptop. Qwen 3 is now a serious coding and reasoning model. The 7B models I tried in 2024 sat somewhere around GPT-3.5 territory. That’s not where things are now.
On my M3 Pro, Gemma 4’s E4B runs fast through Ollama. It’s not a novelty anymore. Qwen 3 8B is the other one I use regularly.
By the end of 2024, Ollama’s CEO was citing eleven million monthly model downloads. It had grown into a real ecosystem mostly without anyone making a big deal of it.
The comparison I was getting wrong
Even as the models improved, I kept evaluating them against the best cloud models available and concluding they weren’t good enough. Which is true, but it’s the wrong comparison to be making.
The question isn’t whether a local 8B model is as good as Claude Sonnet. It isn’t. The question is whether it’s good enough for the thing you’re actually trying to do.
I work in enterprise AI, building demos and showcases and thinking about where the technology is headed. For personal stuff outside work, side projects and quick experiments, I was still defaulting to GPT-4 or Claude out of habit, even when a local model would have been fine.
None of that needs a frontier model. A local 8B handles all of it, on your own machine, for free, without sending anything anywhere.
What it looks like in practice
Most of my local usage runs through Ollama, OpenWebUI for chat-style stuff, and Hermes for agentic tasks where I want a model making decisions and using tools. Hermes is an open-source agent harness from Nous Research. It runs on my Mac and connects through a Telegram bot, so I can send it tasks from my phone and pick up the result whenever.
Gemma 4 E4B for most things. Qwen 3 8B when I want something different. Neither costs anything per query, neither phones home.
The kinds of things that stay local: summarising something, thinking through a problem, throwaway scripts I need once, small personal automations, quick tools I build for myself. Nothing where the output really matters. The model gets me most of the way there and I do the rest.
For anything that actually matters I use frontier models, but the context shapes which ones. At work I’ll go through our internal tooling. For personal stuff I’ll use OpenCode, or access whatever model I need through OpenRouter. Development work, anything where the output matters, anything that ends up in front of someone else: the quality gap is still real enough to care about. A local model isn’t the right tool for all of that.
What I’d missed
The local LLM story isn’t really local versus cloud. It’s that a whole category of tools got significantly better while most people’s picture of them stayed fixed in 2024.
I’d written off local models based on experience with Llama 2 and Mistral 7B. By the time I came back, those models had been replaced by things in a different league entirely. And even then, my instinct was to keep benchmarking them against frontier models rather than asking whether they were good enough for what I actually needed.
Most of the time, they were.