Developer Runs 28.9M Parameter LLM on $8 ESP32-S3 Microcontroller
Using Per-Layer Embeddings from Google's Gemma models, slvDev achieved 9.5 tokens per second on hardware previously limited to 260K parameters.
A developer known as slvDev has deployed a 28.9 million parameter language model on an ESP32-S3 microcontroller costing approximately $8, marking a hundredfold increase over previous attempts on similar hardware.
The model runs entirely on-device without server connectivity, generating text at roughly 9.5 tokens per second end-to-end (9.7 tok/s pure compute). This leaps far beyond the previous record of around 260,000 parameters from DaveBben's esp32-llm project.
Breaking the Memory Barrier
Microcontrollers face severe constraints on fast memory (SRAM), which has historically limited on-device LLMs to tiny parameter counts. The ESP32-S3 used here ships with 512KB SRAM, 8MB PSRAM, and 16MB flash memory (N16R8 variant required).
slvDev bypassed the SRAM bottleneck using Per-Layer Embeddings (PLE), a technique from Google's Gemma 3n and Gemma 4 models. This architecture stores approximately 25 million parameters in flash memory rather than SRAM.
The model's dense core contains roughly 559,000 parameters, with approximately 3.1 million in the input/output head and the remaining 25 million in the PLE table. When quantized to 4-bit, the complete model occupies 14.9MB.
Architecture and Training
The deployed model features 6 transformer layers, a hidden size of 96, 4 attention heads, and a vocabulary of 32,768 tokens. It was trained on the TinyStories dataset, enabling it to generate simple, coherent short stories.
What This Means for Edge AI
"What is interesting here is the architecture, fitting a large model onto a tiny chip, rather than what a 28.9 million parameter model can say," slvDev noted in the project documentation.
The breakthrough demonstrates a viable path for truly local Edge AI that operates completely disconnected from the cloud. By proving that a 28.9M parameter model can run efficiently on an $8 chip, the work opens possibilities for low-cost, private, and offline AI applications.
Potential use cases include real-time text-to-speech systems, simple autonomous agents, and embedded intelligence in IoT devices—all without requiring expensive GPUs or internet connectivity.
The full implementation is available on GitHub, where slvDev has documented the memory layout strategies and performance benchmarks.