EXPERIMENT II · ALGORITHMS

模型在看哪里Where the Model Looks

长上下文里，模型每一步真正在看的 token 有多少？In long context, how many tokens is the model really looking at each step?

待开始Not started注意力Attention稀疏注意力Sparse attentionKV CacheKV Cache
  1. Q提出可证伪的问题State a falsifiable question当前阶段Current stage
  2. R做最小运行路径Build a minimal runnable path未开始Not started
  3. E连接真实验证Validate against reality未开始Not started
  4. P公开证据与限制Publish evidence and limits未开始Not started

为什么值得做Why it is worth doing

稀疏注意力、MLA、KV Cache 压缩都建立在一个前提上：大部分 token 在大部分时候并不重要。这个前提在什么任务上成立，值得自己量一遍。Sparse attention, MLA, and KV cache compression all rest on one premise: most tokens do not matter most of the time. Which tasks that premise holds on is worth measuring myself.

Q

假设Hypothesis

HYPOTHESIS

在长文档问答中，每个解码步里注意力权重前 5% 的 token 覆盖了绝大部分注意力质量，且这些 token 在相邻步之间高度重叠。In long-document QA, the top 5% of tokens by attention weight carry most of the attention mass at each decoding step, and those tokens overlap heavily between adjacent steps.

R

最小装置Minimal setup

MINIMAL SETUP
  1. 01在开源小模型上导出长文档问答的逐层、逐头注意力Export per-layer, per-head attention for long-document QA on a small open model
  2. 02统计 top-k 覆盖率、相邻步重叠率随层数和上下文长度的变化Measure top-k coverage and adjacent-step overlap as layer count and context length change
  3. 03只保留 top-k 的 KV 重新解码，比较答案变化Re-decode using only the top-k KV and compare the answers
E

什么会证明我错了What would prove me wrong

FALSIFIED IF

保留 top-k KV 后答案准确率明显下降，或覆盖率随上下文增长快速分散。Keeping only top-k KV sharply reduces answer accuracy, or coverage disperses quickly as context grows.

P

将会公开What will be published

WILL PUBLISH

统计脚本、各层热力图、截断实验结果，以及稀疏假设不成立的任务。The measurement scripts, per-layer heatmaps, truncated experiments, and the tasks where the sparsity premise fails.

实验日志Run logRUN LOG

还没有记录。实验开始后，每一次运行、失败和修正都会记在实验室里。Nothing logged yet. Once the experiment starts, every run, failure and fix will be recorded in the lab.

去实验室Go to the lab ↗