All › AI › llama.cpp

llama.cpp is an open-source C/C++ engine for running LLMs in GGUF format on CPUs and GPUs, with a built-in server.

Visit llama.cpp ↗

llama.cpp compared with similar tools

Similar to llama.cpp

vLLM

High-throughput LLM inference and serving engine

vLLM is an open-source library for fast LLM serving with PagedAttention, continuous batching and an OpenAI-compatible server.

Free · Linux, Self-hosted · Website ↗

LocalAI

Self-hosted, OpenAI-compatible API for local models

LocalAI is an open-source drop-in replacement for the OpenAI API that runs LLMs, image, audio and embedding models on your hardware.

Free · Self-hosted, Linux, Mac · Website ↗

Ollama

Run open large language models locally

Ollama downloads and runs open models such as Llama, Gemma, Qwen and Mistral on your computer with a CLI and local API.

Free plan · Mac, Windows, Linux · Website ↗

Open WebUI

Self-hosted chat interface for local and cloud LLMs

Open WebUI is an extensible web UI for Ollama and OpenAI-compatible APIs with RAG, multi-user accounts and tools.

Free plan · Self-hosted, Web · Website ↗

LiteLLM

Open-source LLM gateway and SDK for 100+ providers

LiteLLM is a Python SDK and proxy server that calls many LLM providers through an OpenAI-style API, with budgets, keys and logging.

Free plan · Self-hosted, API · Website ↗

Dify

Open-source platform for building LLM apps and agents

Dify provides a visual workflow builder, RAG pipelines, agent tools and model management for building AI applications.

Free plan · Web, Self-hosted · Website ↗

Flowise

Drag-and-drop builder for AI agents and LLM flows

Flowise is an open-source low-code tool for building LLM workflows, chatbots and agents visually.

Free plan · Web, Self-hosted · Website ↗

AnythingLLM

All-in-one AI app for chatting with your documents

AnythingLLM is a desktop and self-hosted app for RAG over documents, AI agents and multi-user chat with local or cloud models.

Free plan · Mac, Windows, Linux, Self-hosted · Website ↗

Questions

Is llama.cpp free?

llama.cpp is free.

Is llama.cpp open source?

Yes, llama.cpp is open source.

Can llama.cpp be self-hosted?

Yes, you can run llama.cpp on your own server.

What platforms does llama.cpp run on?

llama.cpp is available for Mac, Windows, Linux.

What are some alternatives to llama.cpp?

Similar indie products include vLLM, LocalAI, Ollama, Open WebUI and 4 more.

Is this your product? Submit an update.