EXPERIMENT VI · INFERENCE

重复的前缀Repeated Prefixes

一个 Agent 的多轮对话里，有多少 token 在被重复计算？In a multi-turn agent conversation, how many tokens are recomputed?

待开始Not startedPrompt CachingPrompt Caching前缀缓存Prefix cachingvLLMvLLM
  1. Q提出可证伪的问题State a falsifiable question当前阶段Current stage
  2. R做最小运行路径Build a minimal runnable path未开始Not started
  3. E连接真实验证Validate against reality未开始Not started
  4. P公开证据与限制Publish evidence and limits未开始Not started

为什么值得做Why it is worth doing

Agent 每一轮都会把系统提示、工具定义和历史重新发给模型。前缀缓存能省下这部分，但只要上下文顺序稍有变化，缓存就会失效。Each agent turn resends the system prompt, tool definitions, and history. Prefix caching saves that work, but the cache misses as soon as the context order shifts slightly.

Q

假设Hypothesis

HYPOTHESIS

把稳定内容（系统提示、工具定义、Skills）固定在前、易变内容放在后，缓存命中率能提高一倍以上，首 token 时延随之下降。Fixing stable content (system prompt, tool definitions, Skills) at the front and mutable content after it more than doubles cache hit rate, with first-token latency falling accordingly.

R

最小装置Minimal setup

MINIMAL SETUP
  1. 01记录真实编码 Agent 多轮会话的完整请求Record full requests from real multi-turn coding-agent sessions
  2. 02在 vLLM 开启前缀缓存，对照原始顺序与重排后的上下文Enable prefix caching in vLLM and compare original order with the reordered context
  3. 03统计命中 token 比例、首 token 时延与总成本Measure hit-token ratio, first-token latency, and total cost
E

什么会证明我错了What would prove me wrong

FALSIFIED IF

重排后命中率提升不到 20%，或重排损害了任务通过率。If reordering lifts the hit rate under 20%, or hurts task pass rate, the idea is not worth it.

P

将会公开What will be published

WILL PUBLISH

会话请求样本（脱敏）、重排规则、命中率与时延对照。Session request samples (redacted), reordering rules, and hit-rate and latency comparisons.

前期线索Early leads相关的已有工作，不是这个实验的结论Related earlier work, not a result of this experiment

  • Prompt Caching 链路优化Prompt caching pipeline optimization在 Eazo 参与过 Prompt Caching 与模型调用链路优化；这里用开源推理框架独立复现。Worked on prompt caching and the model-call path at Eazo; this reproduces it independently on open-source inference frameworks.

实验日志Run logRUN LOG

还没有记录。实验开始后，每一次运行、失败和修正都会记在实验室里。Nothing logged yet. Once the experiment starts, every run, failure and fix will be recorded in the lab.

去实验室Go to the lab ↗