DEVELOPER FareedKhan has created a C99 inference engine to run the Kimi K3 model on standard hardware, processing at 20 seconds per token with 128GB RAM and 33 seconds per token with a constrained 8GB RAM. The project illustrates the Mixture-of-Experts (MoE) architecture, proving feasibility for massive models on low-memory devices. The Kimi K3 model, which requires 1.56TB for storage, uses on-demand weight streaming to minimize memory usage by only loading necessary expert weights during computation.
The engine runs entirely on CPU, avoiding GPUs and complex deep learning frameworks. Despite its constraints, it maintains precision by leveraging aggressive caching and streaming strategies, demonstrating potential for local AI inference.