working notes. notes on my research, some with fuller experiments than others. not all blogs have an associated paper. a blast to the past twitter view
The language models we interact with in the near future will be what we call scaffolds today. Neural models are now so good that they are severely underutilized, and the line between a language model and a scaffold is blurring.
We explore language models that recursively call themselves or other LLMs before providing a final answer. The aim is essentially unbounded context, and a way through context rot.
A chronological guide to advances in deep learning from the perspective of efficiency — clusters, hardware, libraries, compilers, and architectural changes. Not a survey, a narrative of why the field moved the way it did.
An annotated guide to Kolmogorov-Arnold Networks, in the spirit of the Annotated Transformer. KANs put learned functions on the edges of a network instead of learned weights.
I spent winter break reading every accepted NeurIPS 2023 abstract. These are the highlights — the overall vibes, and the papers that stuck with me, topic by topic.