What it is
llama.cpp is an open-source C/C++ engine that runs language models efficiently on ordinary hardware, from laptops to servers, using CPUs and GPUs (NVIDIA, AMD, Apple Metal, Vulkan). It created the GGUF model format used by Ollama, LM Studio, Jan and others. Use it directly when you want the latest features, maximum speed tuning, or to run models on a server.
Install
Pick the easiest option for your system:
- macOS / Linux with Homebrew:
brew install llama.cpp- Windows:
winget install llama.cpp, or download a ready-made build from the Releases page (choose the CUDA build for NVIDIA GPUs). - Build from source (all systems, needs
gitandcmake):
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config ReleaseUse it
Chat in the terminal with a model downloaded directly from Hugging Face:
llama-cli -hf ggml-org/gemma-3-1b-it-GGUFOr start a local web chat and OpenAI-compatible API on port 8080:
llama-server -hf ggml-org/gemma-3-1b-it-GGUF --port 8080Then open http://localhost:8080 in your browser. If you already have a .gguf file, use -m path/to/model.gguf instead of -hf.
Tip
Useful options: -c 8192 sets the context size (how much text the model remembers), and -ngl 99 offloads all layers to the GPU for speed.
Limitations
- Command-line only (besides the simple web page of
llama-server). - Options change quickly; check
llama-cli --helpand the project README.
Comments 0
Share tips, prompts that worked for you, or questions.
No comments yet. Be the first to share a tip.
Log in or create a free account to comment.