后训练Post-training
SFT、偏好对齐与强化学习：训练信号从哪里来，评测标准怎样和训练目标对上。SFT, preference alignment, and reinforcement learning: where the training signal comes from, and how evaluation criteria line up with training objectives.
- 实践过Hands-on
- 当前状态Current status
- III · VIII
- 造物之书里的实验Experiments in the book
奖励信号和评测本身就是数据问题。训练在优化什么、评测在奖励什么，两者一旦对不上，提升就可能只是表面的。Reward signals and evaluation are themselves data problems. Once training optimizes one thing while evaluation rewards another, improvements can be surface-level only.
2025 年起在 Qwen、LLaMA 上做过 SFT 与 DPO 的练习与对比；在混元 HY4 评测中写 Rubric、处理资料冲突和异常结果，对“评测标准怎样变成训练信号”有第一手感受。Since 2025 I have run SFT and DPO exercises and comparisons on Qwen and LLaMA. During Hunyuan HY4 evaluation I wrote rubrics and handled conflicting sources and anomalous results, which gave me a first-hand feel for how evaluation criteria become training signals.
- 01
按 Rubric 逐维构造的偏好对，能否让模型学到评测真正在乎的维度？Can preference pairs built dimension by dimension from a rubric teach a model the dimensions evaluation actually cares about?
- 02
DPO 的提升里，有多少其实来自“回答更长”？How much of DPO's gain actually comes from longer answers?
- 03
可验证奖励（RLVR）能否教会小模型多步调用工具，而不只是做对数学题？Can verifiable rewards (RLVR) teach small models to call tools over multiple steps, not just solve math problems?
- 04
在 Agent 轨迹上做强化学习时，过程奖励应该落在哪一步？When doing RL on agent trajectories, at which step should process rewards land?
现在还不能证明什么What cannot be proven yet
- 目前实验规模限于单卡 LoRA，结论不一定迁移到全参数训练Experiments are limited to single-GPU LoRA; results may not transfer to full-parameter training
- 还没有公开的训练仓库与结果No public training repository or results yet