算法Algorithms
那些改变成本曲线的结构：循环与参数共享、稀疏与线性注意力、MoE，以及让模型自己判断“要不要决定”。Structures that shift the cost curve: recurrence and weight sharing, sparse and linear attention, MoE, and teaching a model to judge whether a decision is needed at all.
- 在研究Researching
- 当前状态Current status
- II · VII · X
- 造物之书里的实验Experiments in the book
同样的效果，换一个结构可能只要一半的算力。最值得研究的，是那些改变成本曲线、而不只是堆更多参数的设计。The same result may take half the compute under a different structure. What is worth studying are designs that change the cost curve, not just ones that add more parameters.
此前做过循环 Transformer 的小规模复现；正在调研 Jev 这类类型化决策模型。MLA、稀疏注意力与 MoE 目前停留在读论文和读实现阶段。I have previously done a small-scale reproduction of a looped Transformer and am now surveying typed decision models such as Jev. MLA, sparse attention, and MoE are still at the stage of reading papers and implementations.
- 01
同一组参数迭代 k 次，能不能替代堆叠 k 层？Can iterating the same set of parameters k times replace stacking k layers?
- 02
长上下文里，模型每一步真正在看的 token 有多少？稀疏注意力的前提在哪些任务上成立？In long context, how many tokens is the model really looking at each step, and on which tasks does the sparse-attention premise hold?
- 03
MoE 的路由在训练中怎样塌缩，负载均衡损失又牺牲了什么？How does MoE routing collapse during training, and what does the load-balancing loss give up?
- 04
哪些判断应该交给模型，哪些应该用带置信度的类型化决策回退给人或规则？Which judgments belong to the model, and which should fall back to a person or a rule via typed, confidence-scored decisions?
现在还不能证明什么What cannot be proven yet
- 复现规模很小，只能验证机制是否成立，不能说明大模型上的收益Reproductions are very small: they can test whether a mechanism works but cannot show gains at large-model scale
- 部分方向目前只有阅读，没有实验Some directions are reading only, with no experiments yet