Local LLM Runtimes

Ollama is the fastest path from "I want to try Llama" to actually having it running on your laptop. A single install gives you a background daemon, an OpenAI-compatible API on localhost:11434, and a command-line interface that treats models like container images — ollama pull, ollama run, ollama serve. It quietly became the universal backend that almost every other app in this ecosystem plugs into.

• One-line install on macOS, Linux, Windows
• Drop-in OpenAI-compatible REST API
• Hundreds of ready-to-pull GGUF models
• Modelfiles for custom system prompts & merges
• GPU acceleration (Metal, CUDA, ROCm)
• Powers most local agents and IDE plugins

Open WebUI (formerly Ollama WebUI) is the most popular self-hosted chat interface for local LLMs. It's a full-featured web app — multi-user workspaces, RAG over your documents, web-search plugins, voice chat, image input, pipelines, function calling — all layered on top of Ollama or any OpenAI-compatible endpoint. Install it with a single Docker command and you have a private ChatGPT replacement on your LAN.

• Full web UI with auth, roles, and team workspaces
• RAG over docs, websites & YouTube transcripts
• Pipelines & Python functions for custom tools
• Works with Ollama, llama.cpp, vLLM, LocalAI
• Voice + image chat, prompt library, model playground
• One-line Docker install

Jan treats "private AI" as a first-class product philosophy. It's a fully open-source desktop app (AGPL) that can run local models via llama.cpp or connect to remote APIs — all behind a unified chat interface. Its extension system lets you add tools, memory, assistants and even alternative model engines, making it a community-owned alternative to closed chat apps.

• Cross-platform desktop app, AGPL-licensed
• Unified UI for local + remote models
• Extension system for tools and agents
• Per-assistant system prompts and memory
• Export / import every conversation
• Active non-profit-style community

GPT4All pioneered the "local ChatGPT" category and still shines at bringing offline AI to non-technical users. The desktop app bundles quantized models with a friendly chat UI and a LocalDocs feature that turns any folder into a private RAG knowledge base. A Python SDK makes it easy to embed into your own apps.

• Cross-platform installer — Windows, macOS, Linux
• LocalDocs: chat with PDFs, notes, and folders
• CPU-friendly, runs on modest hardware
• Python SDK for app integration
• No account, no cloud, no telemetry
• Strong focus on education & accessibility

llama.cpp is the quiet foundation of the entire ecosystem. Georgi Gerganov's hand-tuned C++ engine made it possible to run multi-billion-parameter transformers on a laptop through aggressive quantization and custom kernels for Metal, CUDA, Vulkan, ROCm, SYCL and plain CPU. If you want raw speed, fine-grained control, or to build your own tool on top of inference, this is where you go.

• Pure C/C++, zero Python required
• GGUF model format — the de facto standard
• Metal / CUDA / Vulkan / ROCm / CPU kernels
• Built-in HTTP server (llama-server)
• Grammar-constrained JSON / schema output
• Powers Ollama, Jan, GPT4All, KoboldCpp & more

Built at UC Berkeley, vLLM is the go-to inference server when you stop running a model for yourself and start running it for a team, an app, or a whole company. Its PagedAttention and continuous batching give throughput an order of magnitude higher than naive serving, and it speaks the OpenAI API wire-protocol out of the box.

• PagedAttention — industry-leading throughput
• Continuous batching & speculative decoding
• OpenAI-compatible HTTP server
• Tensor / pipeline parallelism across GPUs
• Supports most Hugging Face architectures
• Heavy-duty production use at scale

MLX is Apple's open-source array framework designed from the ground up for unified memory on Apple Silicon. For local LLMs it means dramatically faster inference and fine-tuning than generic CPU/GPU paths. Projects like mlx-lm and mlx-community now publish thousands of pre-quantized models optimized for M1/M2/M3/M4.

• Unified-memory-aware kernels for Apple Silicon
• Python + Swift APIs
• LoRA & QLoRA fine-tuning on-device
• MLX-compatible model zoo on Hugging Face
• Integrates with Ollama & llama.cpp-based tools

LocalAI is a single Go binary (and Docker image) that exposes the full OpenAI API surface — chat, embeddings, images, audio, rerankers — backed by whatever local engines you configure. Perfect for replacing OpenAI in existing apps without rewriting a line of code.

• Full OpenAI API coverage, locally hosted
• Chat, embeddings, images, TTS & STT in one service
• Backend-agnostic (llama.cpp, diffusers, whisper.cpp…)
• Docker / Kubernetes friendly
• Gallery of pre-configured models

Affectionately known as "oobabooga", this web-based UI exposes just about every knob a modern inference engine has — backends, sampling parameters, LoRA stacking, training, extensions and API modes. It's the preferred environment for model enthusiasts, roleplayers, and researchers who want to push their hardware.

• Supports llama.cpp, ExLlama, Transformers backends
• Rich chat, notebook and instruction modes
• Extension ecosystem (TTS, RAG, characters…)
• Training & LoRA fine-tuning inside the UI
• Local OpenAI-compatible API

KoboldCpp wraps llama.cpp in a single executable tuned for creative writing and interactive fiction. It offers world info, author notes, memory, lorebooks, and an image-gen module — a full offline writing studio for novelists, game designers and roleplayers.

• Single executable, no install required
• Memory, world info & lorebook support
• Image generation via Stable Diffusion
• OpenAI & KoboldAI compatible APIs
• Runs beautifully on modest GPUs

Cherry Studio is an open-source desktop AI client that bundles a friendly chat UI, multi-assistant management, built-in RAG, and a plugin system — all able to run against a local Ollama or llama.cpp server without any account or cloud dependency. It's a polished, fully-open answer for users who want a "real app" feel without sacrificing openness.

• Cross-platform desktop (macOS / Windows / Linux)
• Multi-assistant profiles & prompt library
• RAG with local knowledge bases
• MCP & tool-calling support
• Works offline with Ollama / llama.cpp

AnythingLLM is a full-stack, self-hostable app that layers workspaces, permissions, agents and retrieval-augmented chat on top of any local model. It's the closest thing to "ChatGPT Enterprise, but on your own hardware" — and it can be deployed on a laptop or a server with equal ease.

• Multi-user workspaces with role-based access
• RAG over PDFs, web, Confluence, GitHub, YouTube…
• Agentic tools & function calling
• Docker / desktop / self-hosted deployments
• Embed as a chat widget in your own products