vllm

Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.

Android in docker solution with noVNC supported, video recording, mcp server and AI-agent

LMCache: Supercharge Your LLM with the Fastest KV Cache Layer

An Easy-to-use, Scalable and High-performance Agentic RL Framework based on Ray (PPO & DAPO & REINFORCE++ & VLM & TIS & vLLM & Ray & Async RL)

Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

A Low-Code MCP Framework for Building Complex and Innovative RAG Pipelines

📚A curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4, Parallelism, etc.🎉

Structured data extraction, instruction calling and agentic workflows with ML, LLM and Vision LLM

learn LLM inference system on Apple Silicon for systems engineers: build a tiny vLLM + Qwen

Cascading runtime for AI agents. Optimize cost, latency, quality, and policy decisions inside the agent loop.

High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle

RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of containers.

Community maintained hardware plugin for vLLM on Huawei Ascend

Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.

Community maintained hardware plugin for vLLM on Apple Silicon

Open Source AI Accelerator Research Platform Standard / 开源推理研究平台

Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond

PTQ-Level Efficiency, QAT-Level Accuracy|训练后量化的效率，量化感知训练的精度

High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.