Helldez/BigMoeOnEdge

English · 中文 · 日本語 · Español · Français · Italiano

Run Mixture-of-Experts models larger than the device's memory by streaming the experts from flash. Built on llama.cpp's public API; Android, iOS, macOS, Windows and Linux; a local OpenAI-compatible server on each.

12 GB Android phone	12 GB iPhone
16 GB Apple-silicon laptop

Qwen3.6-35B-A3B generating on each device, in real time, in the shipped apps. The model is larger than each device's memory.

A Mixture-of-Experts model uses a few of its experts for each token. BigMoeOnEdge keeps in memory the weights every token needs and a cache of recently used experts, and reads the other experts from flash when the router asks for them. The output is identical to running the model fully in memory; the tests check it.

• Planner. --auto measures the machine (available memory, storage speed, each CPU, GPU and NPU) and the model's header, then sets the cache size, what stays resident, and which device runs the prompt and which the answer. After a first run it plans from the model's own routing record (the server keeps one per model), and the first answer measures the threads and overlap. Its rules name no platform or device. Settings given by hand are kept.
• Models. Any MoE gguf that llama.cpp loads, in any quantization. Expert tensors are found from the file, so a new architecture needs no code here.
• Backends. CPU, Metal, Vulkan, OpenCL, CUDA, HIP and Hexagon NPU, loaded as modules; a SYCL module built with its own toolchain loads the same way. On a GPU, the experts used most are cached in its memory and the others stream behind a RAM cache; a model that fits the card runs on it whole.
• Apps and server. Native apps on Android and iOS, a web app on the desktop, and an OpenAI-compatible server on all of them, which coding agents (Claude Code, Codex CLI, opencode) can use.

Decode speed, measured on the devices below. The engine sizes itself to the machine, so stronger hardware runs faster.

Whole tokens per second; a range is the spread across runs. "Experts from" is where the routed experts are read: streamed from flash, held in RAM, or on the GPU, with a cache of the ones used most in the GPU's memory.

Configurations and method: docs/benchmarks.md. To add a machine, run scripts/bench-report.sh /path/to/model.gguf and paste the table into a benchmark report.

Everything below comes from the latest release. Each package starts the engine and opens the app in the browser at http://127.0.0.1:8765/; pick a model under Models.

• macOS (Apple silicon): open BigMoeOnEdge-<version>-macos-arm64.dmg and drag the app to Applications. It stays in the Dock and the menu bar while it runs; Stop and Quit stops it. Not notarised yet: the first start needs the step in the dmg's README.txt.
• Windows: run BigMoeOnEdge-<version>-windows-x86_64-setup.exe (or -windows-arm64-setup.exe). No administrator rights; start BigMoeOnEdge from the Start menu, and close its window to stop it.
• Linux: chmod +x BigMoeOnEdge-<version>-linux-x86_64.AppImage (or -linux-aarch64) and run it. Started from a file manager it opens in a terminal window; close the window to stop it.
• Android: install app-dev-release.apk from the release page; each release installs over the previous one. Settings → Serve on the network shares the phone's engine, with a key.

iOS. Build the app with Xcode from examples/ios. Experimental.

Archives. The bmoe-desktop archive holds the same app as a folder, for scripted installs: extract it and run bmoe-server --open. The bmoe-cli archive holds the command-line engine.

Command line.

bmoe-cli -m model.gguf --auto -p "Explain MoE routing."   # measure, plan, run
bmoe-cli -m model.gguf --plan                              # print the plan and exit

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8765/v1", api_key="local")
reply = client.chat.completions.create(
    model="local",
    messages=[{"role": "user", "content": "Write a haiku about experts on flash."}],
)
print(reply.choices[0].message.content)

Streaming, tool calls and reasoning are supported. For opencode, add the server as an OpenAI-compatible provider with baseURL set to http://127.0.0.1:8765/v1. To serve other machines, start the server with --host and --api-key. API reference: docs/server-api.md.

Coding agents send a large system prompt and tool list on every request, read files into the conversation, and summarize it when the context fills: the context has to hold all three. Size it from the agent's first request (the server's count_tokens or its usage says how large): one of ~10k tokens ran a six-step task in 32768 (-c 32768); one of ~17k could not compact at 32768 and ran ten consecutive questions at 65536 (-c 65536). See Sizing the context for a client.

Claude Code speaks the Anthropic Messages API, which the server also serves (/v1/messages). Use the model id GET /v1/models lists, and tell Claude Code the context you loaded so it compacts in time:

export ANTHROPIC_BASE_URL=http://127.0.0.1:8765
export ANTHROPIC_API_KEY=local            # or the server's --api-key
export ANTHROPIC_MODEL=<model id from /v1/models>
export ANTHROPIC_DEFAULT_HAIKU_MODEL=$ANTHROPIC_MODEL
export CLAUDE_CODE_MAX_CONTEXT_TOKENS=65536   # the context you loaded
claude

Codex CLI speaks the OpenAI Responses API (/v1/responses). Add a provider to ~/.codex/config.toml:

model = "<model id from /v1/models>"
model_provider = "bmoe"

[model_providers.bmoe]
name = "bmoe"
base_url = "http://127.0.0.1:8765/v1"
wire_api = "responses"
env_key = "BMOE_API_KEY"   # export BMOE_API_KEY=local, or the server's --api-key

The engine uses llama.cpp's public API: every quantization, tokenizer and chat template llama.cpp supports works unchanged, and a llama.cpp upgrade is a submodule bump. The routed experts are read from the file's tensor layout (docs/which-models-stream.md). ctest checks that streamed output equals fully resident output across the cache, evictions, overlap, sessions and split models (docs/architecture.md).

Streaming only reads. The model file is mapped read-only and the expert cache is a region of RAM, never a file, so a streamed token costs flash reads and no flash writes: it does not wear the drive the way writes do. What the engine writes is explicit and small: a model you download, your settings, and a metrics CSV or routing record when asked for.

The planner reads the model's shape from its header and measures the machine. The same rules give a phone and a laptop different plans. The Plan screen shows the result: where the memory goes, what each device does, and the reason for each decision. docs/hardware-planning.md.

Details and what was measured where: docs/platforms.md.

git clone --recursive https://github.com/Helldez/BigMoeOnEdge.git
cd BigMoeOnEdge
cd ui && npm ci && npm run build && cd ..
scripts/build-host.sh
build/bin/bmoe-server --open

Tests: cd build && ctest --output-on-failure (needs python3 with the gguf package). Install the desktop layout (the binaries, their backend libraries, the web UI and the catalog, in one folder): cmake --install build --component bmoe --prefix <dir>, then <dir>/bin/bmoe-server. Android: scripts/build-android.ps1, then the APK in examples/android. Warning and sanitizer options: CONTRIBUTING.md.

docs/, indexed by task. Start with hardware-planning.md, settings.md, server-api.md, platforms.md and limitations.md.

AirLLM, Apple's LLM in a flash, FlexGen, PowerInfer and EdgeMoE. The closest recent work is flash-moe, a Metal engine that streams a 397B MoE from SSD on Apple silicon; BigMoeOnEdge uses llama.cpp's public API across architectures and platforms.

Apache-2.0. See LICENSE. The binaries and the apps also carry llama.cpp and ggml, cpp-httplib, nlohmann/json, the web UI's JavaScript libraries (MIT, Apache-2.0) and two fonts (SIL OFL 1.1): their licenses are in THIRD_PARTY_LICENSES, which ships in every archive and APK.