Distributed micro-LLM inference across three ESP32-S3 N16R8 boards with ESP-NOW communication.
This project implements a distributed AI system that runs a 56M-parameter language model across three ESP32-S3 microcontrollers. Inspired by slvDev/esp32-ai, which demonstrated running TinyStories on a single board, this project extends the architecture to a multi-board distributed system with web-based interaction.
The model is trained on WikiText-103 (Wikipedia corpus) using Per-Layer Embeddings (PLE) from Google's Gemma architecture, quantized to 4-bit, and split across three boards that communicate via ESP-NOW wireless protocol. The 50.3M-parameter PLE table is split across Board A and Board B (Split-PLE) to fit within the 16MB flash per board. Board B maintains a KV cache in PSRAM (1.5 MB for 256 positions), enabling the transformer to attend to the full generated sequence instead of operating token-by-token.
┌───────────────────┐ ESP-NOW ┌───────────────────┐ ESP-NOW ┌───────────────────┐
│ Board A │ ◄──────────────► │ Board B │ ◄──────────────► │ Board C │
│ Embeddings + │ │ Core + KV Cache │ │ PLE_A + Decoder │
│ PLE_Proj + Head │ │ │ │ + WiFi │
│ │ │ │ │ │
│ • BPE Tokenizer │ │ • 6 Attn layers │ │ • PLE table A │
│ • tok_emb 8-bit │ │ • 6 FFN layers │ │ (12.5 MB) │
│ • ple_model_proj │ │ • KV Cache (128) │ │ • Sampling │
│ • out_norm + head │ │ • PLE_B (half) │ │ • Web Server │
│ │ │ • PLE Gating │ │ • Space heuristic │
│ Flash: 4.31 MB │ │ Flash: 13.71 MB │ │ Flash: 13.37 MB │
└───────────────────┘ └───────────────────┘ └───────────────────┘