Language Model "Shape"

Alex Zhang, Sep 26, 2026.

Some short musings on the shape of language models, e.g. what it means to design a language model around a harness, and not the other way around.

Since the release of ChatGPT, I’ve observed that the input / output shape of language models has roughly remained static. A reasonable guess is that users would always prefer to use the best models available, and frontier model labs would rather not deviate from the model shape that has proven to work, because it is very expensive to bet on alternatives. As a result, all work on designing language agents has been around designing a harness1 to fit the autoregressive shape of the language model. This thought may appear silly to most people, but a natural question I’m interested in is whether it’s worth considering changing the shape of the language model to fit the harness.

Three sketches of language model input and output shapes: decoder-only, half recurrent and half dense, and a row-constrained output.
Language model “shape” is roughly the input / output structure of our model. The most flexible and widely used shape is the autoregressive, decoder-only Transformer, but there’s a wide range of tradeoffs (e.g. efficiency, long context, etc.) if we change this contract. We typically change the harness to fit the shape of a task because it’s much cheaper to do so, but the amortized cost of fitting the language model shape may end up being lower.

This line of work falls under “model architecture research”, but modern model architecture research has mainly concerned itself with details on the ordering and properties of layers within a larger model, while the overall model still functions as a decoder-only Transformer. It’s pretty clear why this is the case: 1) decoder-only language models are extremely powerful and flexible; 2) smart model architecture choices are worth millions of dollars over the course of scaling; 3) dense next-token prediction is a natural objective over arbitrary text. There’s also still quite a bit of research to be done in this area as well, making it reasonable that it’s a center of focus at these frontier labs.2

I was quite excited by the promise of newer recurrent architectures like State-Space Models (SSMs) and Mamba a few years back, but was pretty disappointed to see them mostly used in hybrid models that retain the decoder-only Transformer shape. I find it even more interesting that “hybrid” here means interleaving attention and SSM layers vertically, rather than having them act on parallel residual streams! I’m sure there’s a variety of empirical reasons for why this was what we settled on3, but what I’m getting at here is that these layers have different properties over Transformers that can potentially be utilized more in the harness. All decoder-only Transformers assume the “arbitrary text → arbitrary text” shape, but what if we could assume some structure over the input / output shape of the model we are using?

A great example of what I’m referring to is the release of Jev (Typesafe AI, 2026) earlier this month, which can be thought of as a language model whose output space is explicitly constrained to be \(\mathbb{R}_{[0,1]}\). This constraint enables the model to be prefill-only when sampling, making it extremely fast at answering a specific class of questions. Regardless of your opinion on the usefulness or validity of this construction4, what Jev represents is the possibility of an alternate tradeoff space around the shape of language models. You’d likely prefer Jev over a frontier model in very particular usecases, and there may even be certain harness designs where it becomes useful; as an example, I’ve seen some implementations online where it’s used as a quick conditional prediction tool fuzzy if-statement logic, which is actually quite useful for Recursive Language Model (RLM) harnesses. Jev in particular proposes a (secret, at least at the time of writing) new objective called Reinforcement Learning for Calibrated Decisions (RLCD), and I suspect for other input / output shapes, we will also have to propose modified objective functions to actually optimize for the correct behavior.

This is a direction that’s particularly exciting to me because it’s a bit more principled than pure scale, and requires more creativity! You’d likely need to use your intuition or findings about the structure of a class of problems to design and train your customized model shape to become more cost-efficient than frontier models.

The shape of a language model in an agent.

My “hot take” is that I don’t think decoder-only Transformers are the natural or final form of how you’d want to process an agent trajectory, and it’s why context management tricks like compaction exist in the first place.5 Agents in the LLM era are usually variants of ReAct (Yao et al., 2022), which accumulate context over time. Every turn, the agent \(\pi_{\theta}\) produces an action \(a_t\) (e.g. reasoning + tool calls), which yields an observation \(o_t\) that is appended to the trajectory. The agent trajectory is usually of the form:

\[a_{t+1} \sim \pi_{\theta}(\cdot | \mathcal{H_{<t}}, o_t) \qquad \mathcal{H}_{<t} = (P, a_1, o_1, a_2,o_2,...,a_t)\]

The “decoder-only Transformer” shape of language models gives us the form we are used to in language agents, which is to pass in \(\mathcal{H}_{<t}\) and \(o_t\) directly as a sequence of tokens in context. The two well-known issues with this setup are that 1) the context window of the model is limited, so the agent history \(\mathcal{H}_{<t}\) has to eventually be compacted, and 2) we unequally care about recent observations versus the older history, and we want to condition on the history without necessarily densely attending to each token.

A sketch of an agent trajectory where older history is sparse and the current window is attended to densely.
Agents generally have to densely look at their recent observations, and sparsely draw from older history. Accurately retrieving from older history is a problem for linear attention or fixed state methods, which is why at least one global dense attention layer has traditionally always existed. A more cost-efficient language model shape for agents might look more like a combination of a recurrent model and a Transformer that act on the history and current observations separately, allowing dense attention locally with an “auto-compacting” history, removing the need to explicitly compact. Not saying this is provably correct, but new systems-level tradeoffs can be derived here with more creativity.

You can argue that sparse attention patterns of linear attention (e.g. Gated DeltaNet) partially solve this issue, but these methods are often interleaved with at least one global attention layer over the same residual stream, which represents the entire context. There have been several tricks over the years to approximate this desired behavior of “infinite context” using a base language model like StreamingLLM (2023), but I think they are too broad in what tasks they try to target to actually be meaningful over just not using them. In the agent problem in particular, we can make stronger assumptions on the structure of the decaying importance of observations, tool calls, and actions with respect to time, and can bake this directly into the shape of our model. We also know that language agents don’t need to perfectly retrieve facts from their history (a common issue with non-dense attention layers), as a coarse view of some information is enough for the agent with this new language model shape to retrieve the actual information back into its working context.

Perhaps this line of research is largely uninteresting for frontier labs given their incentives, but I do believe that for independent researchers and academics, experimenting with novel language model shapes may actually yield useful and wildly efficient applications of language models that are not blocked by the limitations of say, the GPT-6 Astra system. I think we’re in a unique time where this specific research is even possible, which I will explain below.

The “challenges” that may no longer be challenges.

Back in 2022, and honestly even at the beginning of 2026, I would say that the case for research on “language model shape” was weak because we were still uncertain about the effectiveness and capabilities of language models themselves. The most popular language model tasks were single-turn question-answer tasks, and we were still mostly concerned with whether super-human jagged intelligence or long context were achievable through scale in any shape of language model. But now we know this to be true6, and we have a plethora of environments, data, and recipes to approximate the capabilities of these frontier models in the open.

It’s hard to know for sure whether changing the language model shape brings genuine improvements to the intelligence of the model, which is why we needed to get to where we are for this research to become a more viable direction. The only argument I’d make is that architecture choices that enable meaningful composition may improve the “gained intelligence per data / environment”, but in general we are interested in whether model shape introduces better tradeoffs for certain tasks or harnesses. Jev, for example, is not meaningfully more intelligent than any frontier model, but will still likely become extraordinarily useful for fuzzy decision tasks.

It’s easier than ever to distill the capabilities of a decoder-only Transformer into some other language model shape, or even introduce new objectives over modified decoder-only language models. We already loosely do this kind of thing to distill the capabilities of a language model + harness down to just the language model itself. Exploring this design space also explicitly encourages the availability of open models to work off of, new infrastructure to support the training and inference of new language model shapes, and hopefully more funky systems work.

There are some small experiments that I’m really excited to share sometime in the future, but I’m also interested to see what other researchers end up discovering in this direction. If you have cool ideas in this direction and want to chat, my DMs and inbox are always open.

Citation if needed. 
@article{zhang2026shape,
  title   = "Language Model \"Shape\"",
  author  = "Zhang, Alex",
  year    = "2026",
  month   = "September",
  url     = "https://alexzhang13.github.io/blog/2026/shape/"
}

  1. A harness is just some program or code that invokes the language model in a way that helps solve a larger task. For example, you generally cannot just give a base language model a software engineering task and expect it to do it; you have to write a harness to put it in a loop, give it grep tools, give it a way to interface with your terminal, etc. ↩

  2. For non-frontier lab individuals (e.g. academics), this line of research is virtually impossible. ↩

  3. My (rather limited) understanding of how these choices are made is that it starts with a variety of shotgunned attempts at exploring these ideas inside these big labs with many people and compute, and the ones that work stick and are diffused to other labs, and eventually the open. I wouldn’t be that surprised if a lot of what I’m talking about has been explored in some capacity in the past in one of these labs, but likely it didn’t go far for whatever reason. That’s what makes researching alternative bets so fun! ↩

  4. Although I think it’s often misused or misrepresented, as many exciting ideas often are. Jev is not an oracle classifier on any problem, nor is it just a trivial classifier. ↩

  5. In fact, I think this is true of many other applications of language models. But I think the “agent” has become ubiquitous and standardized, making it worth thinking about whether there are alternate tradeoffs to be made with the language model you insert into the agent harness. ↩

  6. The Navier-Stokes announcement was shocking to me — I’ll admit, I thought the rumors were false when they first emerged (although I guess they genuinely weren’t). Not long after, the result that Astra could ascend in NetHack was equally shocking to me, although maybe not as interesting to many others. These models are very powerful now. ↩