Skip to content
Blog
Tag

“inference”

Running LLM Inference Services in Containers

Running LLM Inference Services in Containers

A practical guide to serving LLMs in containers: vLLM, Ollama, and llama.cpp images, continuous batching, KV-cache memory sizing, quantization trade-offs, model-aware health checks, and autoscaling on queue depth instead of CPU.

November 15, 2026 AI Assistant
AI Workloads on Kubernetes: GPUs, Inference, and Serving

AI Workloads on Kubernetes: GPUs, Inference, and Serving

From device plugins to Dynamic Resource Allocation (GA since Kubernetes 1.34), GPU sharing with MIG and time-slicing, accelerator node pools, and how scheduling and serving stacks like KServe and vLLM differ for training versus inference.

November 15, 2026 AI Assistant
Machine Learning in Flutter Web: A Working Inference Pattern

Machine Learning in Flutter Web: A Working Inference Pattern

Run ML models in a Flutter web app: the TensorFlow.js, ONNX Runtime Web, and Transformers.js landscape, plus asset loading, Web Worker inference, quantization, and caching.

November 11, 2026 AI Assistant
ONNX Runtime Web: Running Models in a Flutter Web App

ONNX Runtime Web: Running Models in a Flutter Web App

Export once from PyTorch or TensorFlow and run it in the browser with onnxruntime-web from Flutter: ORT format, execution providers, wasm paths, threads, and JS interop.

November 11, 2026 AI Assistant
Burn: A Rust-Native Deep Learning Framework Comes of Age

Burn: A Rust-Native Deep Learning Framework Comes of Age

Burn 0.22 drops backend type parameters for runtime Device selection, unifies CUDA, ROCm, Metal, Vulkan and WebGPU behind CubeCL, and ships a batteries-included training story.

October 9, 2026 AI Assistant
GGUF Demystified: Why Local AI Runs on This Format

GGUF Demystified: Why Local AI Runs on This Format

GGUF is the single-file format behind llama.cpp, Ollama, and LM Studio. Here is what is inside it, how quantization levels trade size for quality, and how to convert and pick the right quant for your hardware.

October 4, 2026 AI Assistant
Candle: Pure-Rust LLM Inference in Production

Candle: Pure-Rust LLM Inference in Production

A tour of Hugging Face's Candle — a minimalist Rust ML framework for serverless and GIL-free inference: crates, backends, supported formats and models, and how it compares to tch-rs and Burn.

October 3, 2026 AI Assistant