TokenPost
Cover art for NVIDIA Finds Key Scoring Takes 47% of Layer Prefill Time

NVIDIA Finds Key Scoring Takes 47% of Layer Prefill Time

NVIDIA's AI model scoring uses 47% of prefill time, slowing long prompt processing.

Nov 18, 2026

4 cards · 50s · tap to begin

From · · 50s

NVIDIA Finds Key Scoring Takes 47% of Layer Prefill Time

NVIDIA's AI model scoring uses 47% of prefill time, slowing long prompt processing.

In brief

NVIDIA's AI model scoring uses 47% of prefill time, slowing long prompt processing. NVIDIA discovered scoring and selection processes consume 47% of layer prefill time in AI models, impacting efficiency for long inputs. Originally reported by TokenPost.

NVIDIA's AI Scoring Efficiency

An unmarked GPU sits beside a grid of blank metal keys. / TokenPost.ai

NVIDIA found scoring and selecting what an AI model attends to consumes 47% of per-layer prefill time in a 128k token DeepSeek V3.2 example.

Prefill Time Breakdown

Prefill is when a model processes input before generating a response, with scoring taking 47% of layer prefill time.

Indexer Performance Impact

Increasing indexer key heads from one to eight made the indexer run 7.6 times longer during decode.

Block Size Effects

Halving contiguous selection blocks nearly doubled core-attention runtime in both prefill and decode.

The bottom line

NVIDIA discovered scoring and selection processes consume 47% of layer prefill time in AI models, impacting efficiency for long inputs.

Read the original on TokenPost

Read the TL;DR — this story on one card.

How we make summaries

React

Sign in to react and comment.

Comments (0)

Life is short. Keep it sweet. Respect others' opinions and be kind!

    Recommended next

    More decks on nvidia and related topics.