From GitHub · · 2m
GitHub - synchopate/llama-modchip: 🐋 Run 284B DeepSeek-V4-Flash (<59GiB RAM, 44.4 tok/s pp, 9.1-14.7 tok/s) & 30B MoEs (25-111 tok/s) on a pure 64GB CPU at 256K ctx (0 GPUs)
GitHub project runs 284B DeepSeek-V4-Flash and 30B MoE models entirely on CPU with 64GB RAM.
In brief
GitHub project runs 284B DeepSeek-V4-Flash and 30B MoE models entirely on CPU with 64GB RAM. llama-modchip enables running 284B DeepSeek-V4-Flash and 30B MoE models on 64GB CPU RAM without GPUs. Originally reported by GitHub.
llama-modchip Overview
llama-modchip is a set of ggml / llama.cpp kernel patches, quantization transcoders, and runtime tools designed to run sparse Mixture-of-Experts (MoE) checkpoints entirely in system RAM on x86_64 and ARM64 processors.
284B DeepSeek-V4-Flash Model
284.33B Total MoE params (13.17B active) fit in 64GB RAM with 56.56 GiB disk size and 1.71 bpw quantization.
30B Nemotron Model
30.2B Total MoE params (3.0B active) fit in 16-32GB RAM with 14.35 GiB disk size and 3.75 bpw quantization.
CPU-First Inference
llama-modchip runs 284B DeepSeek-V4-Flash and 30B MoE models entirely on CPU with zero GPUs required.
Throughput Metrics
44.4 tok/s prefill and 9.1-14.7 tok/s token generation for 284B DeepSeek-V4-Flash on 64GB CPU RAM.
Quantization Breakdown
DeepSeek-V4-Flash uses 1.71 bpw quantization with 56.56 GiB disk size and 57.61 GiB peak VmRSS.
Model-Specific Optimizations
Nemotron-3.5-30B-A3B uses Fractal-64 (3.75 bpw) with 14.35 GiB disk size and 25.0-60.4 tok/s throughput.
Hardware Requirements
64GB RAM required for 284B DeepSeek-V4-Flash with 16-32GB RAM for 30B Nemotron models.
CPU Cache Optimizations
4-level register tiling (ZMM AVX-512BW) improves prefill speed from 25.0 to 44.36 tok/s on 16-core CPUs.
Self-Speculative Drafting
Stockfish-PVS v2 with 1.64 MB L2/L3 Zobrist n-gram drafter enables instant agent context restore.
Quantization Profiles
DeepSeek-V4-Flash uses Fractal-64 (1.71 bpw) with 56.56 GiB disk size and 57.61 GiB peak VmRSS.
Model Benchmarks
284B DeepSeek-V4-Flash achieves 44.36 tok/s prefill and 9.06-10.74 tok/s decode on 64GB CPU RAM.
Hardware-Specific Optimizations
AMD Zen 5 processors use 48 KB L1D 3D tiling for optimal MoE performance.
Community Leaderboard
Community benchmarks show 386 tok/s prefill on AMD Threadripper PRO 7985WX with 256GB RAM.
GitHub Repository Structure
llama-modchip includes patches, documentation, and scripts for CPU-first inference on various hardware.
Model Performance
DeepSeek-V4-Flash runs at 44.36 tok/s prefill and 9.06-10.74 tok/s decode on 64GB CPU RAM.
The key point
llama-modchip enables running 284B DeepSeek-V4-Flash and 30B MoE models on 64GB CPU RAM without GPUs.
