If you’re neck-deep in the AI bubble like I am, you’ve probably noticed this over the past few weeks: open-source models are climbing the ranks, and they are doing it fast.
And as I intend to stay as deep in it, I immediately downloaded the smaller Qwen 3.5 models as soon as they dropped, and started running them locally, on my MacBook. I wired them into my agentic coding setup, tested different quantized versions, played around with context windows, tool calls, and experimented with KV-cache quantization. All on my available RAM. It was genuinely fun, and kind of impressive. (Even when it made my laptop sound like it was preparing for takeoff.)
In my experience, these smaller models aren’t replacing the 4.6s or 5.4s of the world just yet. But 2026 is definitely the year small models and local deployment start being a legitimate production option.
But, just like we learned with the frontier models, the raw model isn’t the only thing that matters: the engineering around the model is as important (or even more so). And that happens to be exactly what we, as AI engineers, do best.
Here is what we’ll discuss about that in this blog:
I. The importance of Harness Engineering
II. Some practical examples of this:
(a) Routing
(b) Quantization
III. The actual production questions to ask and solve
I. Welcome to “Harness Engineering”
Forgive the hype-term, but I’ve seen this being referred to as ‘Harness Engineering’. It is the infrastructure and logic you build around the model, to make it a success. In a production environment, this surrounding system is starting to matter much more than the model’s base capabilities.
A robust harness includes good routing; the right prompt structure (to utilize your cache); context management; retries and tool constraints; serving setup; and definitely observability and evaluation.
You can see this clearly in coding tasks already. A smaller model that gets one or two attempts, wrapped in a cheap test-and-retry loop, can achieve results surprisingly close to much larger, more expensive models.
I could easily write a whole series on these tasks alone (I won’t), but here are just a few points on what this looks like in practice:
II. (a) Practical example Routing beats “one-model-fits-all”
A good architectural pattern to use, sounding rather obvious, is: stop trying to make one giant model do everything. Complex reasoning? Route to the bigger model (your planner, your coordinator). Narrow, repeatable tasks like SQL generation, document extraction, or formatting? That’s where the small model earns its keep.
This matters exponentially more when you’re running local models, because the entire economic argument collapses if you waste capacity. Send every trivial formatting task through the largest model in your stack, and congratulations, you’ve just recreated the API pricing problem inside your own infrastructure.
The teams that will extract real value from open-source models aren’t the ones endlessly benchmarking to find “the best model.” They’re the ones building a routing layer that actually works.
II. (b) Quantization is behavior modification, not just compression
But harness engineering isn’t just about the external system; it’s also about how you prep the model to fit inside it. People talk about quantization like it’s a memory optimization toggle. Which it sort of is, but be really careful of how it changes model behaviour as well.
A few insights to share:
- Q4 is a really good default. Memory savings are massive, while the quality drop is usually negligible.
- Size beats precision. A well-quantized 27B model can outperform a higher-precision 9B model if your memory budget is similar. More parameters with slightly less precision often wins.
- KV-cache quantization works surprisingly welI. I assumed it would impact output quality, but in the right setup, it barely flinches.
- Quantize the wrong layers and things get weird. Quantize the wrong linear-attention components and your model will get stuck in an infinite internal monologue.
These are some very specific insights, but it’s exactly the kind of thing you only discover by trying things out and running the right evals. And the other point: apart from the ‘obvious’ harness engineering like sandboxing and retries, with local models the model engineering becomes an active task as well.
III. The actual production questions
So, “Can a 9B model run locally?”, yes. But, can it serve multiple users efficiently? What happens when the context grows? How does latency degrade over long agent loops? When do you batch, when do you spill to another model, and when do you fall back to an API? These are the real questions to ask and solve. Especially in agentic systems, which multiply latency in every additional iteration. A model that feels snappy in a short chat, can drop below 10 tokens per second quite easily once the context hits its limits. That’s not a reason to avoid small models. It just means your serving layer is a core part of your product, not an afterthought.
The competitive advantage in AI is shifting away from model access and toward inference and systems engineering (ah, that might just be the term). Model quality is being democratized, but the hard part is the execution: picking the right size for the right task, managing context over long workflows, catching weird quantization bugs before your users do, and evaluating continuously.
But I’m definitely optimistic. “Should this run on a smaller local model?” is no longer an exotic experiment in 2026. It’s a standard design question.
It won’t work for every workload, it won’t be right for every company, and it certainly won’t happen without some growing pains. But small models are finally good enough to matter and good engineering is what will make them valuable.
