When you think of large language models (LLMs), powerful GPUs with dozens or even hundreds of gigabytes of memory likely come to mind. That’s the kind of hardware it takes to run today’s cutting-edge models. It would be hard to imagine this type of algorithm running on a tiny microcontroller, yet one developer has managed to do exactly that. While certainly no frontier model, its 28.9 million parameters make it impressive to see running on an $8 development board all the same.
28 million parameters on an $8 chip
The project runs the language model entirely on an ESP32-S3, a microcontroller with just 512KB of SRAM, 8MB of PSRAM, and 16MB of flash storage available to it. Everything happens locally on the chip, with no cloud connectivity or external server involved. The generated text is written directly to a small display connected to the board at roughly 9.5 tokens per second.
This isn’t the first LLM someone got running on a microcontroller, but earlier projects involved models that contained around 260,000 parameters. This implementation is approximately 100 times larger, raising the obvious question: how does a model that size fit on hardware with so little memory?
Bypassing memory limits
The answer lies in an architectural technique borrowed from Google’s Gemma models known as Per-Layer Embeddings. Rather than loading the entire network into fast memory, the project stores approximately 25 million of its parameters in flash memory as a lookup table. During inference, only the handful of rows required for the current token — about 450 bytes of data — are read from flash, while the smaller computation-heavy portions of the network remain in SRAM and PSRAM. This approach dramatically reduces memory requirements.
The full model occupies just 14.9MB after 4-bit quantization, allowing it to fit comfortably within the ESP32-S3’s onboard flash. According to the developer, this is the first known demonstration of Google’s Per-Layer Embeddings concept being adapted to hardware this constrained.
Capabilities and limitations
Of course, a model this small is quite limited. It was trained on Microsoft’s TinyStories dataset, so it generates short, simple stories with reasonably coherent structure. It is not intended to answer questions, follow instructions, write code, or compete with modern conversational AI systems.
The project’sGitHub repositoryincludes the complete firmware, training scripts, quantization pipeline, wiring instructions, and experimental results. Go grab it all if you’d like to try it out for yourself.
artificial intelligence
machine learning
microcontroller
Nick BildR&D, creativity, and building the next big thing you never knew you wanted are my specialties.
