“batch-inference”

Batch Inference at Scale with Ray

Offline LLM work — scoring, summarization, backfills — needs throughput and reliability, not latency. Distribute thousands of calls with Ray tasks and actors, retries, and structured cost tracking.