Browsing: Mlops

Picture an agent that answers questions about customer invoices. It has one tool, a lookup against an internal API, and a task that needs four lookups. The first two go through fine. Then the credential behind that API expires, and every call from the third one onward comes back as a 401.

A specific kind of quiet follows a good validation run. Numbers on the screen, AUC at 0.91, and everyone nodding along in the morning meeting because, for once, nobody had a reason to ask a follow-up question.

Every major inference framework shipped prefill-decode disaggregation this year. NVIDIA built it into Dynamo. SGLang made it the default for large-scale deployments. vLLM added a KV connector API to support it natively. The consensus is forming fast: split your prefill and decode onto separate GPU pools, and throughput improves.