My first proper attempt at running a local LLM was sometime in 2024. I pulled Mistral 7B through Ollama, watched it load, typed something in, and felt that brief rush of excitement you get when something genuinely cool works. A language model, on my laptop, no API key, nothing going anywhere.
Then I used it for ten minutes and went back to GPT-4. Ask it anything GPT-4 handled cleanly and it would hallucinate, lose the thread, or give you something that needed more work than if you’d just written it yourself. It was the same with Llama 2 and other similar models.
About a year later I came back, and it was a very differnet experience.
What changed
The models genuinely got better, and not incrementally. Llama 3 was trained on roughly seven times the data of Llama 2. Gemma went from a small experiment to Gemma 4, which landed in April 2026 with a 128K context window, multimodal support, and small variants built to run fast on a normal laptop. Qwen 3 is now a serious coding and reasoning model. The 7B models I tried in 2024 sat somewhere around GPT-3.5 territory.
On my M3 Pro, Gemma 4’s E4B runs fast through Ollama. Qwen 3 8B is the other one I use fairly often.
By the end of 2024, Ollama’s CEO was citing eleven million monthly model downloads. It had grown into a real ecosystem mostly without anyone making a big deal of it.
The comparison I was getting wrong
Even as the models improved, I kept evaluating them against the best cloud models available and concluding they weren’t good enough. Which is true, but it’s the wrong comparison to be making.
The question isn’t whether a local 8B model is as good as Claude Sonnet, but that if it’s good enough for the thing you’re actually trying to do.
I work in enterprise AI, building demos and showcases and thinking about where the technology is headed. For personal stuff outside work, side projects and quick experiments, I was still defaulting to GPT-4 or Claude out of habit, even when a local model would have been fine.
None of that needs a frontier model. A local 8B handles all of it, on your own machine, for free, without sending anything anywhere.
What it looks like in practice
Most of my local usage runs through Ollama, OpenWebUI for chat-style stuff, and Hermes for agentic tasks where I want a model making decisions and using tools. Hermes is an open-source agent harness from Nous Research, which i think is slightly more friendly then something like OpenClaw. It runs on my Mac and connects through a Telegram bot, so I can send it tasks from my phone and pick up the result whenever.
Gemma 4 E4B for most things. Qwen 3 8B when I want something different. Neither costs anything per query, neither phones home.
The kinds of things that stay local: summarising something, thinking through something, throwaway scripts I need once, small daily personal automations that Hermes runs as a cron job, quick tools I build for myself. Nothing where the output really matters. The model gets me most of the way there and I do the rest.
For anything that actually matters I use frontier models, but the context shapes which ones. At work I’ll go through our internal tooling to access the models. For personal stuff I’ll use OpenCode, or access whatever model I need through OpenRouter. Development work, anything where the output matters, anything that ends up in front of someone else: the quality gap is still real enough to care about so a local model isn’t the right tool for all of that (atleast not the local models that can run on my device).
What I’d missed
The local LLM story isn’t really local versus cloud. It’s that a whole category of tools got significantly better while most people’s picture of them stayed fixed in 2024.
I’d written off local models based on experience with Llama 2 and Mistral 7B. By the time I came back, those models had been replaced by things in a different league entirely. And even then, my instinct was to keep benchmarking them against frontier models rather than asking whether they were good enough for what I actually needed.