Text LLM inference on Apple Silicon in one Rust crate.

macllm-infer (binary macllm) runs text-only decoder LLMs on Apple Silicon through Candle with Metal and Accelerate. One shared decoder executes every supported model family; family code is only thin adapters, and the same loaded model serves CLI generation, Qwen3 embedding and reranking, and a long-lived HTTP worker.

Install

• From source cargo build --release Copy

What it does

• GGUF and Safetensors checkpoints through one shared decoder: Llama, Qwen 2/3, Gemma 1–3, SmolLM3, LFM2 and more.
• run, embed, rerank and serve from one binary.
• An OpenAI-compatible HTTP subset: /v1/models and /v1/chat/completions.

macllm run -m qwen3-1.7b-q4_k_m.gguf --prompt "Explain a K/V cache in one line"
macllm embed -m Qwen3-Embedding-0.6B --input-type query --input "rust inference"
macllm rerank -m Qwen3-Reranker-0.6B --query "metal" --document "Apple GPU API" --document "a rock"
macllm serve -m qwen3-4b-q4_k_m.gguf