batch-inference

Support Yolov5(4.0)/Yolov5(5.0)/YoloR/YoloX/Yolov4/Yolov3/CenterNet/CenterFace/RetinaFace/Classify/Unet. use darknet/libtorch/pytorch/mxnet to onnx to tensorrt

Torchserve server using a YoloV5 model running on docker with GPU and static batch inference to perform production ready and real time inference.

Production-ready batch inference for Cohere’s Arabic/English ASR model, with optimized Silero VAD, multi-file GPU batching, subtitles, and optional word-level timestamps.

Optimize and run consistent AI Functions.

RWKV7 推理服务 | 高并发批量推理，动态扩缩容，兼容OpenAI API，CUDA优化，低显存部署大模型推理 RWKV linear attention batch inference server — auto-queue, dynamic capacity, OpenAI compatible.

PipelineScheduler optimizes workload distribution between servers and edge devices, setting optimal batch sizes to maximize throughput and minimize latency amid content dynamics and network instability. It also addresses resource contention with spatiotemporal inference scheduling to reduce co-location interference.

Batch LLM Inference with Ray Data LLM: From Simple to Advanced

Fast self-hosted embedding engine for search, RAG, and reindexing workloads on NVIDIA GPUs. Built in Rust + TensorRT for teams that care about scale, cost, and control.

Batch inference for datasets: rows in, any OpenAI-compatible endpoint, resumable parquet out. Congestion-aware concurrency — no tuning.

Torchfusion is a very opinionated torch inference on datafusion.

Ray Saturday Dec 2022 edition

Serve pytorch inference requests using batching with redis for faster performance.

Support batch inference of Grounding DINO. "Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection"

Make the call. Run local Laya, Von, CLM or hosted Jev for typed classification and scoring through one HTTP and MCP API, with resource-aware batches.

Deep RL actor-system runtime for concurrent data collection & modeling

Production-ready native Ruby batch inference for Cohere’s Arabic/English ASR model, with CPU/CUDA/Metal execution, Silero VAD, subtitles, and optional word-level timestamps.

LightGBM Inference on Datafusion

FastMCP fleet MCP server for diffusion LMs (dLLM). DiffusionGemma on Goliath RTX 4090 — batch inference, HLE-shaped reasoning, ~200–400 tok/s. Doc phase; llama-diffusion-cli sidecar next. Complements local-llm-mcp.

Fake Anthropic / Bedrock (InvokeModel, Converse, batch inference, bundled S3) / OpenAI server for zero-cost human/AI-in-the-loop LLM debugging: scripted test harness (scenario rules, latency / rate-limit simulation, pytest fixture), cross-provider relay, observability (history API, JSON logs, Prometheus metrics, cache diagnostics, cost provenance)

Single-binary batch-inference coordinator — fan a JSONL across N OpenAI-compatible endpoints, survive spot kills, resume with zero loss