Had a conversation today that turned into something bigger than I expected. We were talking about where the next AI breakthrough comes from. Most people are looking in the wrong place.
The transformer won for a reason that has nothing to do with smarts. It won because it parallelises on a GPU. Vaswani and the team at Google published Attention Is All You Need in 2017, and the key advantage was never theoretical elegance. It was that you could train the thing at all. RNNs and LSTMs processed tokens one at a time. Transformers did matrix multiplies. GPUs are really good at matrix multiplies. That's it. That's the whole thing.
The model was shaped by the tool. We had tensors before we had LLMs. We had CUDA. We had clusters of H100s that talk to each other over NVLink. The software grew into whatever compute was available. The model grows to fill the box.
That makes the entire scaling era a hardware-driven phenomenon. Not a scientific one.
Every optimisation we've seen in the last three years is in the "why didn't we do that from the start" category. Mixture of Experts was published by Shazeer at Google in 2017. DeepSeek-V3 runs 671 billion parameters total and only activates 37 billion per token. That's a 94 per cent reduction in per-token compute. Why wasn't that the default?
Mamba, the state space model from Gu and Dao, does linear-time sequence modelling. O(n) instead of O(n squared). Five times higher throughput than transformers, and it scales to million-length sequences. Their follow-up paper proved transformers and SSMs are mathematically dual. We could have been doing this years ago.
Speculative decoding runs a tiny draft model, generates five tokens cheaply, has the big model verify all five at once. Two to three times speedup on inference with no quality change. The idea is so simple it sounds like a bug.
None of these are new architectures. They're patches on the 2017 design. Version 1.1 of a product, not the next generation.
The human brain runs on 20 watts. A dim lightbulb. An H100 pulls 700 watts. And the brain learns on that 20 watts. It doesn't train on 15 trillion tokens. It doesn't backpropagate through 100 layers. It doesn't compute softmax dot products.
What the brain does is sparse competition. Groups of neurons compete to represent the input. The winners propagate. The losers go silent. There's no quadratic comparison of every element to every other element. There's a fixed pool of representational capacity, and the loudest signal claims it.
This has a formal basis. The Locally Competitive Algorithm for sparse coding, described by Rozell in 2008, frames neural computation as a dynamic system where neurons compete via lateral inhibition. Most are silent at any moment. The computational cost scales with the complexity of the input, not the size of the model. That's the key difference. A transformer pays O(n squared) for every token, every time. A sparse competitive system pays proportional to the difficulty of the task.
The uncomfortable truth is that the next breakthrough probably isn't coming on a GPU. Groq's LPU is a streaming processor with no memory bottleneck. They claim nearly 100 per cent compute utilisation during inference, compared to 30 to 40 per cent for GPUs that spend most of their time waiting on memory. Cerebras uses an entire silicon wafer as one chip with 4 trillion transistors and no NVLink bottleneck because there are no separate chips. Analog compute does matrix multiplies in the physics of the device, with orders of magnitude lower energy.
Each of these changes the constraints. The model that wins on a GPU is not the model that wins on a streaming processor, a wafer-scale engine, or an analog crossbar.
We built a trillion-dollar infrastructure on an accidental architecture. The transformer succeeded because it was parallelisable on the hardware we already had. The next breakthrough will be stream-oriented, not state-hoarding. Sparse by default. Fixed-budget computation. Designed for the hardware it runs on, not adapted to run on whatever we have lying around.
That hardware won't be a GPU. And the datacenters we're filling with H100s won't run it. Not because the new thing is better. Because it's different. And the old hardware can't do different.
References: Vaswani et al. 2017, Attention Is All You Need. Shazeer et al. 2017, Outrageously Large Neural Networks. DeepSeek-V3 Technical Report 2024. Gu and Dao 2023, Mamba. Dao and Gu 2024, Transformers are SSMs. Rozell et al. 2008, Sparse Coding via Local Competition. Groq LPU architecture. Cerebras WSE-3.