llama.cpp
C/C++ implementation for running LLMs locally with minimal dependencies, optimized for consumer hardware and Apple Silicon.
Features
- Pure C/C++ with minimal dependencies
- GGUF format and quantization (1.5-bit to 8-bit)
- Optimized for Apple Silicon and NVIDIA CUDA
- OpenAI-compatible server mode
- Docker and package manager installs
llama.cpp is a widely-used open-source project for running large language models locally in C/C++. It offers high performance across CPU, Apple Silicon, and NVIDIA GPUs with support for GGUF models and flexible quantization. The project includes an OpenAI-compatible HTTP server, making it easy to use local models with existing tools. llama.cpp powers many downstream projects and is the foundation for running models like LLaMA, Mistral, and Gemma on consumer hardware with minimal setup.
Similar tools
Anthropic API
paidDeveloper platform and API for Claude models with long context, tool use, and enterprise features.
Anyscale
paidRay-based LLM platform for training, serving, and scaling AI workloads in the cloud.
AutoGen
open-sourceMicrosoft's open-source framework for building multi-agent AI systems with conversational patterns and tool use.