llama.cpp

The fast, lightweight engine behind many local AI apps, for people who want full control from the command line.

by ggml.org community (open source) No ratings yet

What it is

llama.cpp is an open-source C/C++ engine that runs language models efficiently on ordinary hardware, from laptops to servers, using CPUs and GPUs (NVIDIA, AMD, Apple Metal, Vulkan). It created the GGUF model format used by Ollama, LM Studio, Jan and others. Use it directly when you want the latest features, maximum speed tuning, or to run models on a server.

Install

Pick the easiest option for your system:

  • macOS / Linux with Homebrew:
Terminal
brew install llama.cpp
  • Windows: winget install llama.cpp, or download a ready-made build from the Releases page (choose the CUDA build for NVIDIA GPUs).
  • Build from source (all systems, needs git and cmake):
Terminal
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release

Use it

Chat in the terminal with a model downloaded directly from Hugging Face:

Terminal
llama-cli -hf ggml-org/gemma-3-1b-it-GGUF

Or start a local web chat and OpenAI-compatible API on port 8080:

Terminal
llama-server -hf ggml-org/gemma-3-1b-it-GGUF --port 8080

Then open http://localhost:8080 in your browser. If you already have a .gguf file, use -m path/to/model.gguf instead of -hf.

Tip

Useful options: -c 8192 sets the context size (how much text the model remembers), and -ngl 99 offloads all layers to the GPU for speed.

Limitations

  • Command-line only (besides the simple web page of llama-server).
  • Options change quickly; check llama-cli --help and the project README.

Guides for llama.cpp

Beginner 10 min

Choose the right local model for your computer

Understand parameters, quantization and memory, and find out which local AI models your computer can run comfortably.

Uses: Ollama, LM Studio, llama.cpp

Comments 0

Share tips, prompts that worked for you, or questions.

No comments yet. Be the first to share a tip.