推理架构Inference
模型怎样被高效地跑起来：缓存、解码、调度，以及推理时的算力怎样换成推理能力。How models are served efficiently: caching, decoding, scheduling, and how test-time compute is turned into reasoning capability.
- 实践过Hands-on
- 当前状态Current status
- VI · IX
- 造物之书里的实验Experiments in the book
推理成本决定一个模型能被多少人真正用上。每一次重复计算、每一次缓存失效，最后都会变成用户等待的时间。Inference cost decides how many people can actually use a model. Every redundant computation and every cache miss ends up as time a user spends waiting.
在 Eazo 参与 Prompt Caching 与模型调用链路优化：连接复用、配置并行初始化、扩展增量同步。也做过 cctop，把本地 token 记录与服务账单对账。At Eazo I worked on prompt caching and the model-call path: connection reuse, parallel config initialization, and incremental sync for extensions. I also built cctop, which reconciles local token records against service bills.
- 01
一个 Agent 的多轮对话里，有多少 token 在被重复计算？上下文怎样排布才能让前缀缓存命中？In a multi-turn agent conversation, how many tokens are recomputed? How should context be arranged so prefix caching hits?
- 02
模型能否按题目难度提前退出，把简单题的算力省下来？Can a model exit early based on question difficulty and save compute on easy questions?
- 03
投机解码在代码、中文长文本上的接受率有多高？What acceptance rate does speculative decoding reach on code and long Chinese text?
- 04
Prefill / Decode 分离在什么负载下才值得？Under what load is prefill / decode disaggregation actually worth it?
现在还不能证明什么What cannot be proven yet
- 线上优化经验来自公司内部，数据不能公开Production optimization experience comes from inside a company; the data cannot be published
- 还没有在开源推理框架上复现的对照数据No controlled data reproduced on open-source inference frameworks yet