Findings Changelog

At A Glance

• Warm request path on G5 is stable and fast in the current runtime.
• Internal routing beats external routing on matched benchmark tasks.
• Cold start bottlenecks were decomposed stage-by-stage; model_tensor_index_build is no longer a dominant stage.
• Remaining cold cost is now concentrated mostly in Qwen decoder_tensor_upload.
• External cold-start comparison (runtime vs PyTorch vs vLLM vs Ollama) now has a canonical G5 artifact with explicit request-path vs total-cold interpretation.
• After decoder loop and sampling fixes, parity-corrected 48-token request path now beats vLLM on TTFT and full latency in latest G5 run.

Timeline

Latest Key Numbers

Warm Path (G5)

• Warm steady-state request mean: ~80.8 ms
• Warm steady-state p99: ~89.6 ms

Routing (Internal vs External, G5)

• Internal mean: 94.849 ms
• External mean: 97.927 ms
• External/Internal: 1.032x (internal faster)

External Cold Comparison (G5, 2026-02-18, Qwen 3B family)

Runtime-normalized (lower is better for runtime):

• PyTorch cold total first response: 3.724x runtime.
• vLLM cold total first response: 10.7x runtime.
• Ollama cold total first response: 1.507x runtime.

Interpretation:

• vLLM request-path TTFT is fastest once healthy, but startup dominates total cold in this run.
• Runtime is strongest on end-to-end cold total in this specific setup.
• Ollama is quantized GGUF and kept with caveat tags (not precision-equivalent to BF16 paths).

External Cold Comparison (G5, 2026-02-18, preload + tokenizer cache)

Runtime-normalized (lower is better for runtime):

• vLLM request full latency: 3.817x runtime.
• vLLM cold total first response: 12.334x runtime.
• Remaining gap: vLLM request TTFT is still lower (51.725 ms vs runtime 91.596 ms).

Important caveat:

• The run above was before request max_tokens was wired through runtime inference (runtime still generated 4 tokens there).

External Cold Comparison (G5, 2026-02-18, token parity fixed at 48, pre decoder fix)

Interpretation:

• Runtime still wins cold-total first response vs vLLM (6.216x better).
• Runtime request-path TTFT and full latency are still slower than vLLM at equal 48-token budget.
• Residual bottleneck is decoder per-token step cost (not tensor upload anymore in this mode).

External Cold Comparison (G5, 2026-02-18, token parity + decoder/sampling fix)

Interpretation:

• Runtime now leads vLLM in request-path TTFT (10.553x faster) and full latency (3.516x faster) on this G5 token-parity run.
• Runtime also remains much lower on cold-total first response (10.851x better vs vLLM).
• Main measured bottleneck in prior parity run (sampling + per-step host sync) is no longer dominant.
• 3-run repeatability (runtime+vLLM reruns) keeps the same direction: mean speedups 10.333x TTFT, 3.380x full latency, 10.688x cold-total first response.

Cold TTFT Before vs After Index Cache (3-run means, G5)

Cold TTFT: clean3 vs clean4 (3-run means, G5)

Dominant Cold Stages After clean4

• model_tensor_index_build dropped to ~1-2.3 ms across models (down ~99.6% vs clean3 for Bart/Donut).
• Qwen still dominated by decoder_tensor_upload (~1015 ms mean).
• Donut and Bart are now mostly in decoder setup/upload and no longer index-build bound.

Reverted Experiment (Transparency)

• Tried an async pinned conversion-buffer upload strategy after clean4.
• Result: Qwen decoder_tensor_upload regressed to ~1419 ms and TTFT regressed by ~37%.
• Decision: reverted that path; clean4 remains the accepted cold-path baseline.
• Follow-up validation run set (clean7) matched clean4 within run noise (Qwen TTFT delta -0.16%).

What Was Actually Tested

1. Baseline (Python/dependency path) runs on T4 and G5.
2. Runtime cold and warm request-path benchmarks.
3. True runtime-reported TTFT (not SSE first-event proxy).
4. Internal-vs-external routing comparison on matched tasks.
5. Week 3 numerical parity checks (strict mode; donut intentionally skipped in parity harness).

What Is Not Finished Yet

1. Phase 3 agentic loop capability study (retrieval correction, tool-state adaptation, confidence-gated branching).
2. A100/H100 reruns from the original expansion phase.
3. Paper-grade figures package.

Raw Artifact Links

• Cold index-cache summary JSON
• Cold index-cache summary Markdown
• Routing comparison JSON
• True TTFT summary JSON
• Cold decomposition clean4 run 1 JSON
• Cold decomposition clean4 run 2 JSON
• Cold decomposition clean4 run 3 JSON
• Cold decomposition clean4 summary JSON
• Cold decomposition clean4 summary Markdown
• Cold decomposition clean7 run 1 JSON
• Cold decomposition clean7 run 2 JSON
• Cold decomposition clean7 run 3 JSON
• Cold decomposition clean7 summary JSON
• Cold decomposition clean7 summary Markdown
• Warm sanity clean7 JSON
• External cold canonical JSON
• External cold canonical Markdown
• External cold preload/tokenizer-cache JSON
• External cold preload/tokenizer-cache Markdown
• External cold token-parity JSON
• External cold token-parity Markdown
• External cold token-parity decoder-fix JSON
• External cold token-parity decoder-fix Markdown