EXPERIMENT VIII · POST-TRAINING

评分即对齐Scoring Is Alignment

按 Rubric 逐维评分构造的偏好对，能让模型学到评测真正在乎的东西吗？Can preference pairs built by scoring each rubric dimension teach the model what evaluation actually cares about?

待开始Not startedDPODPORubricRubricLoRALoRA
  1. Q提出可证伪的问题State a falsifiable question当前阶段Current stage
  2. R做最小运行路径Build a minimal runnable path未开始Not started
  3. E连接真实验证Validate against reality未开始Not started
  4. P公开证据与限制Publish evidence and limits未开始Not started

为什么值得做Why it is worth doing

偏好对齐常用一个总体好坏来构造 chosen / rejected，但评测往往分维度看事实、推理和格式。训练信号和评测标准应当对得上。Preference alignment usually builds chosen / rejected from one overall good-or-bad, but evaluation looks at facts, reasoning, and format per dimension. Training signal and evaluation criteria should line up.

Q

假设Hypothesis

HYPOTHESIS

按维度加权构造的偏好对，在固定评估集上能提升对应维度，且整体能力退化更小。Preference pairs weighted per dimension improve the targeted dimensions on a fixed evaluation set, with less overall capability regression.

R

最小装置Minimal setup

MINIMAL SETUP
  1. 01同一批 Prompt 生成多模型候选回答Generate candidate answers from several models on the same prompts
  2. 02分别用整体评分与 Rubric 逐维评分构造偏好对Build preference pairs separately with an overall score and with per-dimension rubric scores
  3. 03在 Qwen 上用 LoRA + DPO 训练，固定评估集分维度回归Train with LoRA + DPO on Qwen and run per-dimension regression on a fixed evaluation set
E

什么会证明我错了What would prove me wrong

FALSIFIED IF

逐维构造的模型在目标维度上没有优于整体构造，或出现明显能力退化。If the per-dimension model does not beat the overall one on the target dimensions, or regresses noticeably, the method fails.

P

将会公开What will be published

WILL PUBLISH

Rubric、偏好数据构造脚本、训练参数和分维度结果。The rubric, preference-pair construction scripts, training parameters, and per-dimension results.

前期线索Early leads相关的已有工作，不是这个实验的结论Related earlier work, not a result of this experiment

  • 后训练与偏好对齐实验Post-training and preference alignment2025 年起在 Qwen、LLaMA 上做过 SFT / DPO 练习。Since 2025, SFT / DPO exercises on Qwen and LLaMA.

实验日志Run logRUN LOG

还没有记录。实验开始后，每一次运行、失败和修正都会记在实验室里。Nothing logged yet. Once the experiment starts, every run, failure and fix will be recorded in the lab.

去实验室Go to the lab ↗