Awesome LLMOps

An awesome & curated list of the best LLMOps tools for developers.

[!NOTE] Contributions are most welcome, please adhere to the contribution guidelines.

Model

Large Language Model

Project	Details	Repository
Alpaca	Code and documentation to train Stanford's Alpaca models, and generate the data.
BELLE	A 7B Large Language Model fine-tune by 34B Chinese Character Corpus, based on LLaMA and Alpaca.
Bloom	BigScience Large Open-science Open-access Multilingual Language Model
dolly	Databricks’ Dolly, a large language model trained on the Databricks Machine Learning Platform
Falcon 40B	Falcon-40B-Instruct is a 40B parameters causal decoder-only model built by TII based on Falcon-40B and finetuned on a mixture of Baize. It is made available under the Apache 2.0 license.
FastChat (Vicuna)	An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and FastChat-T5.
Gemma	Gemma is a family of lightweight, open models built from the research and technology that Google used to create the Gemini models.
GLM-6B (ChatGLM)	An Open Bilingual Pre-Trained Model, quantization of ChatGLM-130B, can run on consumer-level GPUs.
ChatGLM2-6B	ChatGLM2-6B is the second-generation version of the open-source bilingual (Chinese-English) chat model ChatGLM-6B.
GLM-130B (ChatGLM)	An Open Bilingual Pre-Trained Model (ICLR 2023)
GPT-NeoX	An implementation of model parallel autoregressive transformers on GPUs, based on the DeepSpeed library.
Luotuo	A Chinese LLM, Based on LLaMA and fine tune by Stanford Alpaca, Alpaca LoRA, Japanese-Alpaca-LoRA.
Mixtral-8x7B-v0.1	The Mixtral-8x7B Large Language Model (LLM) is a pretrained generative Sparse Mixture of Experts.
StableLM	StableLM: Stability AI Language Models

⬆ back to ToC

CV Foundation Model

Project	Details	Repository
disco-diffusion	A frankensteinian amalgamation of notebooks, models and techniques for the generation of AI Art and Animations.
midjourney	Midjourney is an independent research lab exploring new mediums of thought and expanding the imaginative powers of the human species.
segment-anything (SAM)	produces high quality object masks from input prompts such as points or boxes, and it can be used to generate masks for all objects in an image.
stable-diffusion	A latent text-to-image diffusion model

⬆ back to ToC

Audio Foundation Model

Project	Details	Repository
bark	Bark is a transformer-based text-to-audio model created by Suno. Bark can generate highly realistic, multilingual speech as well as other audio - including music, background noise and simple sound effects.
whisper	Robust Speech Recognition via Large-Scale Weak Supervision

⬆ back to ToC

Robotics Foundation Model

[!NOTE] Emerging Architectures in VLA:

• Continuous Diffusion Language Models: Integrate diffusion heads or flow-matching to VLMs (e.g., DiVLA, OpenPI), enabling smooth, precise continuous action generation rather than discretized tokens.
• Recurrent Language Models: Utilize State Space Models (SSMs) like Mamba or recurrent transformers (e.g., RoboMamba, RD-VLA) to reduce inference memory and handle temporal dependencies, allowing iterative reasoning for complex robotic decision-making.

Project	Details	Repository
DiVLA	A continuous diffusion-based Vision-Language-Action model that integrates diffusion policies into autoregressive VLMs for robust and precise continuous robotic control.
LeRobot	A central community library by Hugging Face for AI in robotics — end-to-end learning tools, data pipelines, and support for training/deploying VLA models.
Octo	A transformer-based generalist robot policy pretrained on 800K+ robot trajectories from the Open X-Embodiment dataset. Supports language instructions, goal images, and fine-tuning to new embodiments.
OpenPI	Open-source VLA models from Physical Intelligence, including π₀ and π₀.5 — flow-based vision-language-action models pretrained on large-scale robot data with fine-tuning support.
OpenVLA	A 7B-parameter open-source Vision-Language-Action model trained on 970K+ robot demonstrations from the Open X-Embodiment dataset for generalist robotic manipulation.
RoboMamba	An efficient VLA model leveraging State Space Models (Mamba) instead of standard self-attention, offering linear inference complexity for efficient, recurrent robotic reasoning.
SmolVLA	A compact ~450M parameter VLA by Hugging Face, designed to be computationally efficient and accessible, running on consumer GPUs or CPUs. Part of the LeRobot ecosystem.

Serving

Large Model Serving

Project	Details	Repository
Alpaca-LoRA-Serve	Alpaca-LoRA as Chatbot service
OneComp	Fujitsu Research's post-training quantization pipeline for LLMs (QEP, AutoBit, JointQ, rotation) with vLLM plugin (arXiv:2603.28845).
CTranslate2	fast inference engine for Transformer models in C++
Clip-as-a-service	serving the OpenAI CLIP model
DeepSpeed-MII	MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.
Faster Whisper	fast inference engine for whisper in C++ using CTranslate2.
FlexGen	Running large language models on a single GPU for throughput-oriented scenarios. (Archived)
Flowise	Drag & drop UI to build your customized LLM flow using LangchainJS.
llama.cpp	Port of Facebook's LLaMA model in C/C++
LLMKube	Kubernetes operator for LLM inference with pluggable runtimes (llama.cpp, PersonaPlex/Moshi, generic), multi-GPU sharding, NVIDIA CUDA and Apple Silicon Metal support, and GGUF/MLX/SafeTensors model formats.
Shimmy	Python-free Rust inference server with OpenAI API compatibility and hot model swapping
Infinity	Rest API server for serving text-embeddings
Modelz-LLM	OpenAI compatible API for LLMs and embeddings (LLaMA, Vicuna, ChatGLM and many others)
Off Grid	Open-source iOS/Android app running LLMs on-device via llama.cpp. Voice (Whisper), vision, image gen, tool calling — fully offline.
Ollama	Serve Llama 2 and other large language models locally from command line or through a browser interface.
Rapid-MLX	OpenAI-compatible LLM inference server for Apple Silicon using MLX. 2-4x faster than Ollama with tool calling and prompt caching.
TensorRT-LLM	Inference engine for TensorRT on Nvidia GPUs
text-generation-inference	Large Language Model Text Generation Inference
text-embeddings-inference	Inference for text-embedding models
tokenizers	💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
vllm	A high-throughput and memory-efficient inference and serving engine for LLMs.
whisper-ctranslate2	is a 4x faster and low-memory usage drop-in cli replacement that supports word-level timestamps and VAD filter
whisper.cpp	Port of OpenAI's Whisper model in C/C++
x-stable-diffusion	Real-time inference for Stable Diffusion - 0.88s latency. Covers AITemplate, nvFuser, TensorRT, FlashAttention. (Archived)

⬆ back to ToC

Frameworks/Servers for Serving

Project	Details	Repository
BentoML	The Unified Model Serving Framework
Jina	Build multimodal AI services via cloud native technologies · Model Serving · Generative AI · Neural Search · Cloud Native
Mosec	A machine learning model serving framework with dynamic batching and pipelined stages, provides an easy-to-use Python interface.
mcpproxy-go	Open-source MCP proxy with BM25 tool filtering, quarantine security, activity logging, and web UI. Routes multiple MCP servers through single endpoint, reducing context bloat by ~97%.
TFServing	A flexible, high-performance serving system for machine learning models.
Torchserve	Serve, optimize and scale PyTorch models in production (Archived)
Triton Server (TRTIS)	The Triton Inference Server provides an optimized cloud and edge inferencing solution.
langchain-serve	Serverless LLM apps on Production with Jina AI Cloud (Archived)
lanarky	FastAPI framework to build production-grade LLM applications
ray-llm	LLMs on Ray - RayLLM (Archived)
Xinference	Replace OpenAI GPT with another LLM in your app by changing a single line of code. Xinference gives you the freedom to use any LLM you need. With Xinference, you're empowered to run inference with any open-source language models, speech recognition models, and multimodal models, whether in the cloud, on-premises, or even on your laptop.
KubeAI	Deploy and scale machine learning models on Kubernetes. Built for LLMs, embeddings, and speech-to-text.
Kaito	A Kubernetes operator that simplifies serving and tuning large AI models (e.g. Falcon or phi-3) using container images and GPU auto-provisioning. Includes an OpenAI-compatible server for inference and preset configurations for popular runtimes such as vLLM and transformers.
Open Responses	Serverless open-source platform for building long-running LLM agents with tool use.
KubeStellar Console	AI-powered multi-cluster Kubernetes dashboard for hybrid edge and cloud. GPU monitoring, LLM inference cluster management, benchmark streaming, and 20+ CNCF integrations. CNCF Sandbox (Apache 2.0).

⬆ back to ToC

Security

Frameworks for LLM security

Project	Details	Repository
Cordum	Safety-first agent orchestration platform with pre-dispatch policy evaluation, output scanning (PII, secrets, injection), job scheduling, workflow engine, and full audit trail.
brood-box	CLI tool for running coding agents inside hardware-isolated microVMs with snapshot isolation, egress control, and MCP authorization.
dstack	Open-source confidential AI framework for secure LLM deployment with data privacy, providing hardware-enforced isolation using Intel TDX and NVIDIA Confidential Computing.
Plexiglass	A Python Machine Learning Pentesting Toolbox for Adversarial Attacks. Works with LLMs, DNNs, and other machine learning algorithms.

⬆ back to ToC

Observability

Project	Details	Repository
Azure OpenAI Logger	"Batteries included" logging solution for your Azure OpenAI instance.
ClevAgent	Runtime monitoring for AI agents — heartbeat watchdog, loop detection, cost tracking, auto-restart. Python SDK or HTTP API.
Deepchecks	Tests for Continuous Validation of ML Models & Data. Deepchecks is a Python package for comprehensively validating your machine learning models and data with minimal effort.
Evidently	An open-source framework to evaluate, test and monitor ML and LLM-powered systems.
EvalView	Regression testing for AI agents. Snapshot behavior, detect tool-call and output regressions, with golden-baseline diffing and LLM-as-judge scoring. Supports LangGraph, CrewAI, OpenAI, Claude, and any HTTP API.
Fiddler AI	Evaluate, monitor, analyze, and improve machine learning and generative models from pre-production to production. Ship more ML and LLMs into production, and monitor ML and LLM metrics like hallucination, PII, and toxicity.
Giskard	Testing framework dedicated to ML models, from tabular to LLMs. Detect risks of biases, performance issues and errors in 4 lines of code.
QWED	Deterministic verification protocol for LLM outputs using 8 formal verification engines (SymPy, Z3, AST, SQLGlot). Prevents hallucinations through mathematical proofs rather than statistical methods.
Great Expectations	Always know what to expect from your data.
Helicone	Open source LLM observability platform. One line of code to monitor, evaluate, and experiment with features like prompt management, agent tracing, and evaluations.
Traceloop OpenLLMetry	OpenTelemetry-based observability and monitoring for LLM and agents workflows.
Langfuse 🪢	Open-source LLM observability platform that helps teams collaboratively debug, analyze, and iterate on their LLM applications.
whylogs	The open standard for data logging
Maxim AI	Platform for AI Agent Simulation, Evaluation & Observability
onWatch	Lightweight Go CLI that tracks AI API quota usage across 7 providers (Anthropic, OpenAI, GitHub Copilot, MiniMax, and more). Background daemon, <50MB RAM, zero telemetry, SQLite storage.
RagTune	CLI tool for debugging and benchmarking RAG retrieval. EXPLAIN ANALYZE for your retrieval layer.
traceAI	Open-source AI tracing framework built on OpenTelemetry for deep observability across agentic and LLM workflows.
Future AGI	Production-grade SDK for observability, automated evaluations and prompt management with sub-100ms guardrails for LLM/agent workflows.
semantic-coverage	Visualizes RAG knowledge gaps and "blind spots" using 2D UMAP clustering and density detection.
Weco Observe	Observability and debugging tool for AI research agents. Trace multi-step LLM agent runs, visualize decision trees, and identify failure modes in autonomous research workflows. Cloud hosted with open-source agent integration.

⬆ back to ToC

LLMOps

Project	Details	Repository
agenta	The LLMOps platform to build robust LLM apps. Easily experiment and evaluate different prompts, models, and workflows to build robust apps.
AgentMark	Type-Safe Markdown-based Agents
AgentField	Open-source control plane for building and operating AI agents like APIs at scale, with routing, memory, observability, identity, auth, and policy controls.
AI studio	A Reliable Open Source AI studio to build core infrastructure stack for your LLM Applications. It allows you to gain visibility, make your application reliable, and prepare it for production with features such as caching, rate limiting, exponential retry, model fallback, and more.
Arize-Phoenix	ML observability for LLMs, vision, language, and tabular models.
BudgetML	Deploy a ML inference service on a budget in less than 10 lines of code.
Cheshire Cat AI	Web framework to create vertical AI agents. FastAPI based, plugin system inspired to WordPress, admin panel, vector DB included
Contexto	Self-hosted context engine for AI agents with persistent conversation memory and recall. Works as a drop-in OpenAI-compatible proxy, OpenClaw plugin, or memory SDK — no code changes required.
Dataoorts	Enjoy unlimited API calls with Serverless AI Workers/LLMs for just $25 per month. No rate or concurrency limits.
deeplake	Stream large multimodal datasets to achieve near 100% GPU utilization. Query, visualize, & version control data. Access data w/o the need to recompute the embeddings for the model finetuning.
Dify	Open-source framework aims to enable developers (and even non-developers) to quickly build useful applications based on large language models, ensuring they are visual, operable, and improvable.
Dstack	Cost-effective LLM development in any cloud (AWS, GCP, Azure, Lambda, etc).
Embedchain	Framework to create ChatGPT like bots over your dataset.
Epsilla	An all-in-one platform to create vertical AI agents powered by your private data and knowledge.

…view the full README on GitHub.