EXPERIMENT III · POST-TRAINING

偏好，还是长度Preference, or Length

DPO 学到的是“更好的回答”，还是只是“更长的回答”？Does DPO learn "a better answer" or merely "a longer answer"?

待开始Not startedDPODPO偏好数据Preference data长度偏置Length bias
  1. Q提出可证伪的问题State a falsifiable question当前阶段Current stage
  2. R做最小运行路径Build a minimal runnable path未开始Not started
  3. E连接真实验证Validate against reality未开始Not started
  4. P公开证据与限制Publish evidence and limits未开始Not started

为什么值得做Why it is worth doing

偏好数据里，被选中的回答往往更长。模型可能只学会了写得更多，而评测也常常偏爱长回答，两者叠加会让提升看起来比实际更大。In preference data the chosen answer tends to be longer. The model may only learn to write more, and evaluation often favors long answers too; together, the two make gains look larger than they are.

Q

假设Hypothesis

HYPOTHESIS

控制 chosen / rejected 长度差之后，DPO 在固定评估集上的提升会明显缩小，但事实类维度的提升会保留下来。Controlling the chosen / rejected length gap shrinks DPO gains on a fixed evaluation set noticeably, while factual dimensions keep their improvement.

R

最小装置Minimal setup

MINIMAL SETUP
  1. 01统计一份公开偏好数据集中 chosen 与 rejected 的长度差Measure the chosen vs. rejected length gap in a public preference dataset
  2. 02构造长度匹配子集，与原始数据分别做 LoRA + DPOBuild a length-matched subset and run LoRA + DPO on it and on the original data separately
  3. 03在长度归一化后的评测上比较，并检查回答长度的变化Compare on a length-normalized evaluation and check how answer length changes
E

什么会证明我错了What would prove me wrong

FALSIFIED IF

长度匹配后提升与原始数据相当，说明长度不是主要来源。If length matching leaves gains comparable to the original data, length is not the main source.

P

将会公开What will be published

WILL PUBLISH

长度分布、两组训练配置、长度归一化前后的评测对照。Length distributions, both training configurations, and evaluation before and after length normalization.

前期线索Early leads相关的已有工作，不是这个实验的结论Related earlier work, not a result of this experiment

  • SFT / DPO 对比实验SFT / DPO comparison在 Qwen、LLaMA 上做过 SFT 与 DPO 的练习与对比。Ran SFT and DPO exercises and comparisons on Qwen and LLaMA.

实验日志Run logRUN LOG

还没有记录。实验开始后，每一次运行、失败和修正都会记在实验室里。Nothing logged yet. Once the experiment starts, every run, failure and fix will be recorded in the lab.

去实验室Go to the lab ↗