I've spent a lot of time profiling ML systems, and this number still gets me:
Model latency: 12 ms
Then you try the actual product and wait half a second.
Where did the other 488 ms go?
Usually nowhere. We just didn't time it.
Input decoding happened before the stopwatch. Tensor allocation and memory copies lived outside it. Queueing disappeared into the serving layer. Post-processing happened afterwards. We warmed the model up first, even though the user's first request doesn't get a rehearsal.
I call this latency laundering: moving delay outside the measurement boundary until a slow system produces a fast number.
The 12 ms isn't necessarily a lie. It's just an answer to an easier question than the user asked.
I've done milder versions of this myself. The model is the interesting part, so that is what I profile. The kernel gets faster. The benchmark turns green. The user still waits.
Accelerators make this especially easy. GPU work is asynchronous. Bigger batches make throughput look great while individual requests sit in a queue. A warm average hides the cold request everybody notices.
Component measurements matter. Kernel time, model time, transfer time and throughput each tell us where to optimise.
But their labels should say what they exclude.
If someone sends a request at A and can use the answer at B, then B minus A is the latency of the product.
Quantize the model. Fuse operators. Tune the compiler. Fix the queue.
But start and stop the stopwatch where the user does.