“inference”
Running LLM Inference Services in Containers
A practical guide to serving LLMs in containers: vLLM, Ollama, and llama.cpp images, continuous batching, KV-cache memory sizing, quantization trade-offs, model-aware health checks, and autoscaling on queue depth instead of CPU.
AI Workloads on Kubernetes: GPUs, Inference, and Serving
From device plugins to Dynamic Resource Allocation (GA since Kubernetes 1.34), GPU sharing with MIG and time-slicing, accelerator node pools, and how scheduling and serving stacks like KServe and vLLM differ for training versus inference.
Machine Learning in Flutter Web: A Working Inference Pattern
Run ML models in a Flutter web app: the TensorFlow.js, ONNX Runtime Web, and Transformers.js landscape, plus asset loading, Web Worker inference, quantization, and caching.
ONNX Runtime Web: Running Models in a Flutter Web App
Export once from PyTorch or TensorFlow and run it in the browser with onnxruntime-web from Flutter: ORT format, execution providers, wasm paths, threads, and JS interop.
Burn: A Rust-Native Deep Learning Framework Comes of Age
Burn 0.22 drops backend type parameters for runtime Device selection, unifies CUDA, ROCm, Metal, Vulkan and WebGPU behind CubeCL, and ships a batteries-included training story.
GGUF Demystified: Why Local AI Runs on This Format
GGUF is the single-file format behind llama.cpp, Ollama, and LM Studio. Here is what is inside it, how quantization levels trade size for quality, and how to convert and pick the right quant for your hardware.
Candle: Pure-Rust LLM Inference in Production
A tour of Hugging Face's Candle — a minimalist Rust ML framework for serverless and GIL-free inference: crates, backends, supported formats and models, and how it compares to tch-rs and Burn.