---
title: "Stop Latency Laundering"
description: "Usamah Zaheer names latency laundering, the habit of moving delay outside the measurement boundary until a slow system reports a fast number."
canonical_url: "https://www.usamah.me/blog/latency-laundering"
author: "Usamah Zaheer"
date_published: "2026-08-10"
date_modified: "2026-08-11"
type: "article"
word_count: 254
---

# Stop Latency Laundering

I've spent a lot of time profiling ML systems, and this number still gets me:

> Model latency: 12 ms

Then you try the actual product and wait half a second.

Where did the other 488 ms go?

Usually nowhere. We just didn't time it.

Input decoding happened before the stopwatch. Tensor allocation and memory copies lived outside it. Queueing disappeared into the serving layer. Post-processing happened afterwards. We warmed the model up first, even though the user's first request doesn't get a rehearsal.

I call this **latency laundering**: moving delay outside the measurement boundary until a slow system produces a fast number.

The 12 ms isn't necessarily a lie. It's just an answer to an easier question than the user asked.

I've done milder versions of this myself. The model is the interesting part, so that is what I profile. The kernel gets faster. The benchmark turns green. The user still waits.

Accelerators make this especially easy. GPU work is asynchronous. Bigger batches make throughput look great while individual requests sit in a queue. A warm average hides the cold request everybody notices.

Component measurements matter. Kernel time, model time, transfer time and throughput each tell us where to optimise.

But their labels should say what they exclude.

If someone sends a request at A and can use the answer at B, then B minus A is the latency of the product.

[Quantize the model](/blog/edge-ml-inference). [Fuse operators](/blog/compiler-optimization-for-ml). Tune the compiler. Fix the queue.

But start and stop the stopwatch where the user does.

## About the author

Usamah Zaheer is a Machine Learning Software Engineer at Arm, where he designs and optimises ML infrastructure for Arm architectures. He is pursuing an MS in Artificial Intelligence at the University of Texas at Austin and holds an MS in Embedded Systems from the University of Leicester. His work spans deep learning, computer vision, edge computing, robotics, and MLOps across companies including Arm, Dyson, and the University of Leicester. Full CV: https://www.usamah.me/index.md. His book on efficient machine learning is at https://ai.usamah.me.

## This page in other formats

- Canonical HTML: https://www.usamah.me/blog/latency-laundering
- Markdown: the same path with `.md` appended
- Site index for models: https://www.usamah.me/llms.txt
- Whole site as one file: https://www.usamah.me/llms-full.txt
- Structured CV: https://www.usamah.me/profile.json
- Endpoint manifest: https://www.usamah.me/agents.json