Fine-tuning forgets. RAG leaks context. Hypernetworks build the model your agent needs on demand.

The two standard fixes for teaching AI agents your business both leave a human permanently in the loop. A third approach — generating specialist model weights at inference time — is moving from research into early product. It's the most credible attempt yet at closing the gap, and also the least proven.

Written by Lena Armitage · Bureau Tech · Jun 20, 2026

The production wall every agent team hits

An AI agent demos beautifully, goes to production, and stalls. It runs for a stretch, then needs a human to top up its context and check its output. The promised efficiency drains into supervision. This is why so many agent pilots never become production systems.

The underlying problem is usually not orchestration or raw model capability. It's whether the model knows your business well enough to be left alone — and for how long.

When AI firm Chroma tested 18 leading models, every one lost accuracy as its input grew. That's a property of how attention works, not a gap a stronger model closes. An agent fed more and more of your business as it runs doesn't get steadier. It gets shakier.

Why the two standard fixes both keep you in the loop

**Fine-tuning** bakes knowledge into model weights. It remains subject to catastrophic forgetting — a problem identified in the 1980s and still unresolved in 2026. Teaching a model something new tends to erode what it already knew. Teams work around this by isolating each task in its own fine-tuned model or adapter, which produces a sprawling estate of models that raises cost and governance overhead. And a fine-tuned model is a snapshot: stale the day a policy changes, when a slow, expensive retraining cycle starts over.

**In-context learning and RAG** skip retraining by placing relevant policies in the prompt at runtime. But a retrieval miss looks identical to a confident answer. Cost and latency climb with every token added. And accuracy degrades as context grows.

The two failures rhyme. With fine-tuning, the model may be confidently working from last quarter's policy. With RAG, it may be confidently working from a detail it lost in the middle of a long prompt. Either way the output looks equally assured — so you can't tell which parts are wrong without checking all of them. That's why the human never gets to leave.

A third path: generate the specialist model on demand

A hypernetwork is a network whose output is the weights of another network. The term was coined in 2016; applying it to produce specialist language model adapters from text or documents is recent and active.

Sakana AI's Text-to-LoRA, presented at ICML 2025, generates a model adapter from a plain-language description in a single pass. A 2026 system called SHINE describes hypernetwork adaptation as a promising new frontier precisely because it sidesteps both the retraining cost of fine-tuning and the context limits of prompting.

The practical payoff: the per-task adapter teams hand-build to dodge catastrophic forgetting is the same object a hypernetwork produces automatically. The model zoo stops being a governance headache and becomes a generated output.

Nvidia researchers argued in 2025 that for the narrow, repetitive tasks filling agent workflows, small models are capable enough and 10 to 30 times cheaper to run than frontier generalists — which is the economic case for going small underneath all of this.

The clearest commercial instance so far

Nace.AI, a Palo Alto company that raised a $21.5 million seed round in May 2025, is the most visible early product. Its core technology — a generator it calls a MetaModel — produces parameter adaptations at inference time from a company's policies, aimed at regulated work: audit, compliance, risk assessment. The company says its agents handle the bulk of a workflow while human experts validate the result, a split it markets as 90/10.

That ratio is best read as a measurement of an architecture, not a dial set in advance. It only means something if every output comes with grounding — citations, reasoning traces — that lets a reviewer verify provenance in seconds rather than redo the work.

Where the approach breaks down

Calibration is the linchpin. Recent work found that hypernetwork-generated adapters don't automatically improve calibration over ordinary fine-tuning; gains appear only under specific constraints. The quality of the generated model also depends heavily on the policy data it's built from.

Nace says it has scaled its generator well beyond the sizes shown in published research and derived a scaling law for how performance grows — results it is now putting through peer review. If that holds up, it would help answer one of the central open questions in the field. It's the paper worth watching.

What to ask before you buy

For a long, repetitive, high-volume process — running internal audit overnight and having experts check the final slice — a hypernetwork-generated model is the approach most likely to run cheaply and long enough to matter. For a short task that finishes in a few steps, the gap versus a well-prompted frontier model shrinks to almost nothing.

Four questions cut through vendor pitches: Where does the business knowledge live — weights, prompt, or generated on demand? What does each output come with so a reviewer can verify rather than redo it? What decides which work gets escalated? And whose model improves from expert feedback, and where does it run?

The answers, not the headline autonomy ratio, tell you what you're buying.

Key takeaways

Fine-tuning bakes knowledge into model weights but is vulnerable to catastrophic forgetting — a problem identified in the 1980s and still unresolved — and produces a sprawling, expensive estate of per-task models.
RAG (retrieval-augmented generation) avoids retraining by injecting relevant documents into the prompt at runtime, but a retrieval miss looks identical to a confident answer, and accuracy degrades as context grows — a property of how attention works, not a gap a stronger model closes.
Hypernetworks generate task-specific model adapters from a company's policies at inference time, collapsing the model-zoo problem and keeping knowledge current without retraining.
Reported autonomy ratios like '90/10' are measurements of an architecture, not preset dials — and they only mean something if every output comes with grounding that lets a reviewer verify provenance in seconds.
The critical open questions — whether hypernetwork-generated adapters improve calibration, and whether the approach scales — are still being resolved in published research.

FAQ

What is catastrophic forgetting, and why does it matter for enterprise AI?

Catastrophic forgetting is the tendency of a neural network to lose previously learned information when it is trained on new data. It was identified in the 1980s and remains unresolved. For enterprise teams, it means that fine-tuning a model on new policies or tasks can degrade its performance on older ones — which is why many teams end up maintaining a separate fine-tuned model or adapter for each task.

What is RAG, and what are its failure modes in production?

RAG — retrieval-augmented generation — is a technique that retrieves relevant documents from a data store and injects them into the model's prompt at runtime, avoiding the need to retrain. Its main failure modes are retrieval misses (the wrong document is retrieved, but the model answers confidently anyway) and context rot (accuracy degrades as the prompt grows longer, a property of how attention mechanisms work).

What is a hypernetwork, and how does it differ from fine-tuning?

A hypernetwork is a neural network that generates the weights of another neural network. Applied to language models, it produces a task-specific adapter from a company's policies at inference time — without a separate training run. Unlike fine-tuning, the adapter can be regenerated whenever policies change, and there is no model zoo to maintain.

What is a LoRA adapter?

LoRA (Low-Rank Adaptation) is a technique for fine-tuning large language models efficiently by training only a small set of additional weight matrices rather than updating the full model. Hypernetwork approaches like Sakana AI's Text-to-LoRA generate these adapters on demand from a text description rather than training them from scratch.

What is automation bias, and why does it matter for AI agent review?

Automation bias is the tendency for humans to over-rely on automated outputs and under-scrutinize them. The EU AI Act's Article 14 names it explicitly. Research suggests experts correct flawed recommendations less often when they are labeled as AI-generated. High autonomy ratios concentrate human attention into a thin, late slice of the work — which makes the quality of grounding (citations, reasoning traces) critical to whether that review is meaningful.

Citations

Fine-tuning forgets. RAG leaks context. Hypernetworks build the model your agent needs on demand.Enterprise AI agents stall in production because fine-tuning and RAG both leave a human in the loop; hypernetworks offer a third path by generating specialist model adapters at inference time.
Text-to-LoRA — Sakana AI, presented at ICML 2025Sakana AI's Text-to-LoRA generates a model adapter from a plain-language description in a single pass, presented at ICML 2025.
SHINE (2026) — hypernetwork adaptation as a promising new frontierA 2026 system called SHINE describes hypernetwork adaptation as a promising new frontier that sidesteps both the retraining cost of fine-tuning and the context limits of prompting.
Nvidia researchers on small models for agent workflows (2025)For narrow, repetitive tasks in agent workflows, small models are capable enough and 10 to 30 times cheaper to run than frontier generalists.
Nace.AI $21.5M seed round announcement (May 2025)Nace.AI raised a $21.5 million seed round in May 2025; its MetaModel generates parameter adaptations at inference time from company policies for regulated workflows.