From TokenPost · · 50s
NVIDIA Finds Key Scoring Takes 47% of Layer Prefill Time
NVIDIA's AI model scoring uses 47% of prefill time, slowing long prompt processing.
In brief
NVIDIA's AI model scoring uses 47% of prefill time, slowing long prompt processing. NVIDIA discovered scoring and selection processes consume 47% of layer prefill time in AI models, impacting efficiency for long inputs. Originally reported by TokenPost.
NVIDIA's AI Scoring Efficiency

NVIDIA found scoring and selecting what an AI model attends to consumes 47% of per-layer prefill time in a 128k token DeepSeek V3.2 example.
Prefill Time Breakdown
Prefill is when a model processes input before generating a response, with scoring taking 47% of layer prefill time.
Indexer Performance Impact
Increasing indexer key heads from one to eight made the indexer run 7.6 times longer during decode.
Block Size Effects
Halving contiguous selection blocks nearly doubled core-attention runtime in both prefill and decode.
The bottom line
NVIDIA discovered scoring and selection processes consume 47% of layer prefill time in AI models, impacting efficiency for long inputs.
Read the TL;DR — this story on one card.





