Language Model "Shape"
Some short musings on the shape of language models, e.g. what it means to design a language model around a harness, and not the other way around.
Speculative Programmatic Tool Calling
Speculative programmatic tool calling pre-launches tool calls from a partially generated REPL program, while the harness is still producing tokens, instead of waiting for the whole generation to finish.
Language model harnesses are compositional generalizers
Modern post-training brute-forces ever more environments and ever longer horizons, because frontier Transformers are still poor at compositional generalization. A good harness makes every model call locally in-distribution, so what it learns carries across lengths and domains.
A Mini Exercise on the Mismanaged Geniuses Hypothesis (RLMs on LongCoT)
We underestimate how good language models actually are, and they are inhibited by how we use them. On LongCoT, frontier models fall short of compositional reasoning tasks whose pieces they can already solve in isolation.
The Mismanaged Geniuses Hypothesis
AI models are already good enough for the next leap in capabilities. The mismanaged geniuses hypothesis says frontier language models are severely underutilized, because each individual call is used poorly.
Language Models will be Scaffolds
The language models we interact with in the near future will be what we call scaffolds today. Neural models are now so good that they are severely underutilized, and the line between a language model and a scaffold is blurring.
Recursive Language Models
We explore language models that recursively call themselves or other LLMs before providing a final answer. The aim is essentially unbounded context, and a way through context rot.
A Meticulous Guide to Advances in Deep Learning Efficiency over the Years
A chronological guide to advances in deep learning from the perspective of efficiency — clusters, hardware, libraries, compilers, and architectural changes. Not a survey, a narrative of why the field moved the way it did.
The Annotated Kolmogorov-Arnold Network (KAN)
An annotated guide to Kolmogorov-Arnold Networks, in the spirit of the Annotated Transformer. KANs put learned functions on the edges of a network instead of learned weights.
Highlights of NeurIPS 2023 from Reading All 3584 Abstracts
I spent winter break reading every accepted NeurIPS 2023 abstract. These are the highlights — the overall vibes, and the papers that stuck with me, topic by topic.