Leaderboard

Lower time is better.

Canonical Cross-Hardware Snapshot (Paper Package, 2026-02-20)

Sources:

• package_summary.json
• paper_package.md

Phase 2 Cold/Hot

Routing Matrix

Phase 3 Loops (Baseline/Stress)

C2 Runtime-Native Uncertainty Deltas (Uncertainty On-Off Success Delta)

Phase 5 Real-Benchmark (Diagnostic, G5, 2026-03-01)

Source artifacts:

• phase5_awareness_realbench_qwen-realbench-r5-tokenizerfix2_20260301T114510Z.json
• phase5_awareness_realbench_qwen-realbench-r6-qwentpl1_20260301T120235Z.json
• phase5_hf_reference_qwen_r5_20260301T1900Z.json

Arm A (Control) Score Means

Runtime (r5 Arm A) vs HF Reference (Same Sampled Set)

Interpretation:

• r5 is the current Phase 5 diagnostic reference.
• r6 was an explicit template-path experiment and regressed quality/latency; it is not used as canonical.
• HF parity result for this sampled set: runtime and HF are tied on GSM8K/AIME (0.0), so current math-task failures are not runtime-only breakage.

Qwen3.5 One-Host Strict Matrix (AWS G5, 2026-03-07)

Source artifacts:

• phase5_qwen35_remote_strict_matrix_20260307T191653Z.json
• phase5_qwen35_remote_strict_matrix_20260307T191653Z.md

Overall / Per-Task (Arm A, gpqa_diamond+ifeval, seeds 7/17/27, 8/task)

Interpretation:

• This is the cleanest current one-host strict A/B on Qwen3.5-0.8B.
• Runtime is no longer behind on aggregate score on this set.
• Runtime is still far slower overall, so this is not a claim of universal superiority.
• The result remains task-stratified:
  • gpqa_diamond score is now at parity, but runtime latency is still far worse,
  • ifeval score is higher for runtime, but latency is still worse there too.

Qwen3.5 Nightly vLLM Diagnostic (G5, 2026-03-02)

Source artifacts:

• phase5_awareness_realbench_qwen35-realbench-r1-s8-nonthinking_20260302T184159Z.json
• phase5_awareness_realbench_qwen35-realbench-r2-policyfix1-s8-nonthinking_20260302T184624Z.json
• phase5_awareness_realbench_qwen35-realbench-r3-sharedfirst-s8-nonthinking_20260302T184947Z.json

Interpretation:

• r3 removes arm-to-arm sampling noise (all arms share the same first completion).
• Post-fix state is no-regression parity (B-A=0, C-A=0) rather than uplift.

Qwen3.5 Strict Runtime vs vLLM Matrix (G5, 2026-03-03)

Source artifacts:

• phase5_qwen35_runtime_vs_vllm_matrix_20260303T104038Z.json
• phase5_qwen35_runtime_vs_vllm_matrix_20260302T221546Z.json
• phase5_qwen35_runtime_vs_vllm_matrix_20260302T222013Z.json

Latest run (20260303T104038Z) per-task score deltas (runtime-vLLM):

• gpqa_diamond: -0.0833
• ifeval: -0.0972
• gsm8k: +0.0417
• aime25: 0.0000

Interpretation:

• Strict matrix is no longer blocked and is now fully reproducible.
• Decoder-path fixes narrowed the quality gap materially, but runtime is still slower overall and still slightly behind on aggregate score.
• Top target remains request-path latency recovery on Qwen3.5 while preserving this improved score parity.

Qwen3.5 Parse-Fix AB3 (GPQA+IFEval, G5, 2026-03-04)

Source artifact:

• phase5_q35_runtime_vs_vllm_ifeval_gpqa_ab3_20260304T034227Z.json

Interpretation:

• This run is task-family stratified, not a universal win.
• Runtime currently wins on ifeval (quality + latency) but remains far slower on gpqa_diamond.

Qwen3.5 Hybrid Full-Batch AB3 (GPQA+IFEval, AWS G5, 2026-03-08)

Source artifact:

• phase5_qwen35_remote_strict_matrix_20260308T000429Z.json

Interpretation:

• This is the first late AWS Qwen3.5 strict AB3 where runtime clearly leads overall on score.
• The runtime is still slower on latency across both tasks.
• GPQA is now a strong runtime-quality win; the next blocker is warm decode/request-path latency and the remaining IFEval score deficit.

Qwen3.5 Deterministic Strict Matrix (GPQA+IFEval, AWS G5, 2026-03-08)

Source artifact:

• phase5_qwen35_remote_strict_matrix_20260308T204248Z.json

Interpretation:

• This is the current claim-safe Qwen3.5 strict lane.
• Runtime wins overall on both score and latency.
• gpqa_diamond is now exact parity on score, while ifeval is a runtime win on both score and latency.
• Sampled-lane reproducibility is now fixed separately; this deterministic lane remains the cleanest low-variance claim-safe slice.

Qwen3.5 Sampled Strict Matrix (GPQA+IFEval, AWS G5, 2026-03-08)

Source artifact:

• phase5_qwen35_remote_strict_matrix_20260308T220806Z.json

Interpretation:

• This is the post-fix sampled strict AB3 lane after the harness seed bug was removed.
• Runtime now wins overall on both score and latency here as well.
• gpqa_diamond is still runtime-slower, but runtime is ahead on score in both tasks.

Qwen3.5 Sampled Strict Matrix (Larger-N, GPQA+IFEval, AWS G5, 2026-03-08)

Source artifact:

• phase5_qwen35_remote_strict_matrix_20260308T222640Z.json
• phase5_qwen35_remote_strict_matrix_20260308T235013Z.json

Interpretation:

• This is the stronger non-thinking sampled strict result.
• Runtime still wins overall on both score and latency at larger sample count.
• The weakest remaining slice is still GPQA latency, not overall sampled quality.

Qwen3.5 Thinking Strict Matrix (Finalized Closed-Form Lane, AWS G5, 2026-03-09)

Source artifact:

• phase5_qwen35_remote_strict_matrix_20260309T010353Z.json

Interpretation:

• This is the current best thinking tradeoff.
• The close-form finalize path stays active, but a smaller GPQA reasoning budget removes the old latency collapse.
• Runtime now wins this finalized thinking lane overall on both score and latency.

Qwen3.5 Fast-Sampler Tie-Stable AB3 (GPQA+IFEval, AWS G5, 2026-03-08)

Source artifact:

• phase5_qwen35_remote_strict_matrix_20260308T004758Z.json

Interpretation:

• This is the first clean late AWS Qwen3.5 strict AB3 where runtime is faster overall than vLLM.
• The remaining blocker is score recovery, not another large latency rescue.
• GPQA stays runtime-positive on score; the main open quality deficit is ifeval plus residual GPQA fidelity loss versus the older slower sampler lane.

Qwen3.5 Endpoint Probe Matrix (Warm, Extended, G5, 2026-03-06)

Source artifact:

• qwen35-probe-matrix-r2_20260306T200035Z.json

Interpretation:

• Runtime is now functionally solid on Qwen3.5 in non-thinking mode.
• The main remaining runtime gap is long-prompt/tool latency, not base chat correctness.
• Current vLLM launch is not a full multimodal parity configuration; failures here are partly launch-mode/config artifacts, not only model behavior.

G5 External Cold Backends (Canonical 3-Run Means)

G5 Foundation (Canonical)

G5 Cold First-Hit — True TTFT (3-run Means, 2026-02-17)

All values above are runtime-instrumented timing.ttft_ms. Validation rerun (clean7, 2026-02-18) matches these values within run noise.

Qwen Cold Upload Ablation (G5, 2026-02-19, Same Harness, Env Toggle Only)

Interpretation:

• CPU-side tensor conversion was the dominant cold bottleneck for Qwen.
• Moving BF16/F16 conversion to GPU materially reduced first-hit latency.

External Cold Runtime-Only Ablation (G5, 2026-02-19, Preload On, max_tokens=48)

AWS G5 TTFT Kernel Pass (2026-02-22, Same Host/Container)

Cold (run_mode=cold_first_hit):

Warm (run_mode=warm_steady_state, 16 measured, 2 warmup):

Interpretation:

• softmax-only was near-parity.
• norm rewrite was the material gain.
• seq2seq/Bart follow-up (seq_q=1 tiny-kernel path + direct K/V cache write) produced another measurable lift.
• best measured profile (lt1_sync0 seq1 tiny-kernel default) vs baseline:
  • cold TTFT: 1.339x faster (16.738 -> 12.504 ms)
  • cold full: 1.089x faster (424.685 -> 390.099 ms)
  • warm mean: 1.216x faster (174.237 -> 143.230 ms)
  • warm p99: 1.121x faster (1035.823 -> 924.276 ms)
• follow-up profile vs previous lt1_sync0 norm+softmax best:
  • cold TTFT: 1.118x faster
  • warm mean: 1.028x faster
• Bart cold TTFT moved from 16.573 -> 12.842 ms (1.29x faster).
• 3-seed repeatability (new default path):
  • cold TTFT 12.563 ± 0.037 ms
  • cold full 390.961 ± 0.270 ms
  • warm mean 143.297 ± 0.222 ms
  • warm p99 925.668 ± 1.070 ms
• seq1 fused follow-up matrix (seq1_hybrid_fused_20260222T192656Z) further improved request path:
  • warm default mean 54.505 -> 52.535 ms (1.037x)
  • warm default p99 82.134 -> 80.554 ms (1.020x)
  • cold default TTFT 6.447 -> 6.209 ms (1.038x)
  • cold default full 147.756 -> 145.587 ms (1.015x)

AWS G5 True Fused cuDNN Frontend A/B (2026-02-22, fixed qwen, warmed)

Source:

• attn_backend_ab_frontend_20260222T220111Z.md

Interpretation:

• Warm steady-state is close but still favors custom.
• Cold first-hit is still dominated by fused plan-build misses.

AWS G5 Frontend Repeatability Matrix (2026-02-22, repeats=3)

Source:

• attn_backend_frontend_matrix_20260222T221948Z.md

Interpretation:

• Current custom path wins all tracked metrics across both profiles in repeated runs.

AWS G5 Frontend Claim-Strength (2026-02-22)

Source:

• attn_backend_frontend_claim_report_20260222T222958Z.md

Delta is frontend - custom (positive means custom is faster):

Interpretation:

• Current evidence is strongly directional for custom > fused_frontend on this hardware/workload.
• Repeat count is still low (n=3/profile), so high-N reruns remain desirable for paper-grade confidence.

AWS G5 Frontend Miss-Mitigation (Updated Canonical, 2026-02-22)

Sources:

• attn_backend_frontend_matrix_20260222T230445Z.md
• attn_backend_frontend_matrix_20260222T231139Z.md
• attn_backend_frontend_missmit_compare_20260222T231335Z.md
• preload_exact_prompt_probe_20260222T231050Z.json

Comparison (no_preload -> startup_preload_benchmark_queries):

Exact cold-prompt probe:

• fused first-hit TTFT: 4.499 ms
• fused first-hit full latency: 26.090 ms

Interpretation:

• With benchmark-query preload coverage, the prior fused cold/mixed miss spikes are effectively removed in this harness.
• Custom still leads warmed request-path latency slightly (about 0.90 custom/frontend ratio), but cold TTFT is now near parity on the covered prompt set.

AWS G5 Frontend Miss-Mitigation (Shape Prebuild, No Preload Prompts, 2026-02-22)

Sources:

• attn_backend_frontend_matrix_20260222T230445Z.md
• attn_backend_frontend_matrix_20260222T233003Z.md
• attn_backend_frontend_missmit_compare_20260222T233116Z.md
• prebuild_startup_nopreload_probe_20260222T232932Z.json
• prebuild_startup10_nopreload_probe_20260222T235944Z.json
• prebuild_startup8_nopreload_probe_20260223T000600Z.json
• attn_backend_frontend_matrix_20260223T000256Z.md
• attn_backend_frontend_missmit_compare_20260223T000343Z.md

Comparison (no_preload -> shape_prebuild_nopreload):

Cold probe (fused frontend, no preload prompts, startup prebuild on):

• startup->healthy: 11017.541 ms
• request TTFT: 5.814 ms
• request full latency: 255.434 ms

Tuned startup probe (seq_kv_max: 16 -> 10):

• startup->healthy: 7011.472 ms (1.571x faster startup)
• request TTFT: 5.826 ms (near-identical)
• request full latency: 254.936 ms (near-identical)

Lower-range probe (seq_kv_max=8):

• startup->healthy: 6010.381 ms
• request TTFT: 703.771 ms (regression)
• request full latency: 1660.576 ms (regression)

Tuned matrix confirmation (seq_kv_max=16 -> 10):

• warm-fixed fused request mean: 22.556 -> 22.265 ms
• mixed fused request mean: 51.482 -> 50.974 ms
• cold fused TTFT: 5.827 -> 5.819 ms

Interpretation:

• Shape-level prebuild removes no-preload fused request-path spikes without prompt-list curation.
• Tradeoff: startup latency is still elevated due to up-front plan builds, but tuned range materially reduces that startup cost.
• seq_kv_max=10 is currently the minimum safe tuned range for this benchmark prompt profile.

AWS G5 Frontend Miss-Mitigation (Hybrid Shape Gate, No Preload Prompts, 2026-02-23)

Sources:

• prebuild_hybrid10_nopreload_probe_r1_20260223T002214Z.json
• prebuild_hybrid10_nopreload_probe_r2_20260223T002214Z.json
• prebuild_hybrid10_nopreload_probe_r3_20260223T002214Z.json
• attn_backend_frontend_matrix_20260223T001959Z.md
• attn_backend_frontend_missmit_compare_20260223T002153Z.md
• hybrid_shape_sanity_20260223T002857Z.json
• hybrid_shape_sanity_maxgate_20260223T003453Z.json
• attn_backend_frontend_matrix_20260223T003611Z.md
• attn_backend_frontend_missmit_compare_20260223T003734Z.md

Policy:

• TRENI_ATTN_CUDNN_FRONTEND_PREBUILD_SEQ1_MIN_KV=10
• TRENI_ATTN_CUDNN_FRONTEND_PREBUILD_SEQ1_MAX_KV=10
• TRENI_ATTN_CUDNN_FRONTEND_PREBUILD_HEAD_DIM=128
• TRENI_ATTN_CUDNN_FRONTEND_MIN_SEQ_KV=10

Startup probe (3 runs, fused frontend, no preload prompts):

• startup->healthy: 2004.840 +/- 0.146 ms
• request TTFT: 4.955 +/- 0.011 ms
• request full latency: 242.673 +/- 0.352 ms

Delta vs prior tuned no-gate probe (prebuild_startup10_nopreload_probe_20260222T235944Z):

• startup->healthy: 7011.472 -> 2004.840 ms (3.497x faster)
• request TTFT: 5.826 -> 4.955 ms (1.176x faster)
• request full latency: 254.936 -> 242.673 ms (1.051x faster)

Matrix deltas vs prior tuned no-gate matrix (attn_backend_frontend_matrix_20260223T000256Z):

• warm-fixed fused request mean: 22.265 -> 20.354 ms (1.094x faster)
• mixed fused request mean: 50.974 -> 47.904 ms (1.064x faster)
• cold fused TTFT: 5.819 -> 4.959 ms (1.173x faster)
• cold fused full latency: 254.146 -> 242.569 ms (1.048x faster)

Interpretation:

• Hybrid shape gating removes most remaining startup cost while keeping low no-preload request-path latency.
• In this harness, strict fused mode stays inference-valid with low-shape fallback to custom path.
• Initial broader-shape sanity exposed out-of-window (seq_kv>10) miss cascades; adding bounded gate (TRENI_ATTN_CUDNN_FRONTEND_MAX_SEQ_KV=10) removed those cascades and cut the same 5-shape set mean full latency from 9974.576 ms to 274.072 ms (36.395x) while keeping fixed-profile matrix behavior near-identical.

Internal vs External Routing (G5, 2026-02-17)

Internal routing is faster in aggregate and faster on 3/4 tasks in this set (the remaining task is effectively tied).

Internal vs External Routing (G5, 2026-02-18, Failure-Amplification Stress)

Stress profile:

• Tool failure injection: every 2nd request.
• Tool timeout injection: every 3rd request (0.9s sleep) with controller timeout 0.25s.
• Controller tool retries: 1.

Observed amplification:

• External taxonomy: tool_hop_failed=4.
• External tool retries mean: 0.182.

Internal vs External Routing Matrix (G5, 2026-02-19)

Matrix mean ratio (all profiles): 1.208x external/internal.

Internal vs External Routing Cross-Host Pilot (2026-02-19)

Topology:

• local benchmark client
• SSH tunnel to G5 host
• runtime + external router on G5

Stress notes:

• injected tool failure every 2nd request
• injected timeout every 3rd request (900 ms sleep)
• controller tool retries 1 with 20 ms backoff

Internal vs External Routing Split-Host Matrix (2026-02-19, Canonical Track B)

Topology:

• GPU host: runtime endpoint
• CPU host: external controller + tool services
• controller/tool call runtime over private VPC network

Matrix means (all profiles):

• External/Internal latency ratio: 1.028x.
• Internal error rate: 0.0000.
• External error rate: 0.0347.

Internet Multi-Hop Routing (Fly + Commercial APIs, 2026-02-20)

Topology:

• internal path: local client -> commercial API
• external path: local client -> Fly controller/tool -> same commercial API

Interpretation:

• OpenAI matrix shows the expected direction for Track B (external hop overhead under internet routing).
• OpenRouter rows are mixed/inverted by profile and remain non-canonical for Track B direction claims in this topology.

Local Control Routing (No Fly Scheduler Path, 2026-02-20)

Topology:

• internal path: local client -> commercial API
• external path: local client -> local standalone controller/tool -> same commercial API

Interpretation:

• With higher-N local controls, OpenAI is near parity and OpenRouter Sonnet trends external > internal.
• External errors still appear under stress while internal remained error-free in these runs.

Task-Family Parity Split (Local Control, runs=8, 2026-02-20, Legacy Pre-Fairness)

Interpretation:

• Tool-only tasks support the architecture claim on both providers.
• Model-only behavior is near parity on OpenAI and still favorable on Sonnet.

Task-Family Parity Split (Fairness-Hardened, Local Control, runs=8, 2026-02-20)

Harness controls:

• interleaved ordering (pair_order=alternate)
• deterministic defaults (temperature=0)
• strict tool parity on tool_only
• token-normalized reporting (ms/completion_token)

Interpretation:

• Tool-only tasks remain favorable to internal on both providers after parity hardening.
• OpenAI model-only remains near parity/slight inversion; Sonnet model-only favors internal.
• Track B commercial claims remain task-family-stratified.

Commercial Root-Cause Grouping (Fairness r4+r8, 2026-02-22)

Source:

• commercial_gap_root_cause_20260222T222958Z.md

Delta is external - internal (positive means internal is faster):

Interpretation:

• Current commercial control set is not statistically locked as win/loss by group (all CI95 include zero).
• Measured controller overhead is small relative to model-hop variance, so higher-N region-pinned reruns are needed before claiming directional commercial performance differences.

Phase 3 Agentic Loops (Canonical G5, 2026-02-19, 3 Seeds)

Per-scenario snapshot:

Stress variant (tool_fail_every=9, tool_timeout_every=11, controller retries 2):

Phase 3 Uncertainty Ablation (G5, 2026-02-19, baseline+stress repeatability)

Normalized-logprob source (paper-aligned), 3-seed means:

Uncertainty-on gains (success delta means):

• Baseline internal: +0.2308.
• Baseline external: +0.2308.
• Stress internal: +0.2308.
• Stress external: +0.2212.

Cross-source check:

• raw_logit_margin and hybrid preserve the same internal/external success deltas as normalized-logprob in both baseline and stress sets.
• These rows are harness-level synthetic uncertainty; runtime-native canonical corroboration is now published below.

Phase 3 Uncertainty Ablation (G5, 2026-02-19, runtime-native canonical rerun, 3 seeds, superseded)

Runtime-native source (/v1/chat/completions uncertainty payload), after greedy uncertainty kernel fix:

Uncertainty-on gains (runtime-native success delta means):

• Baseline internal: +0.1026.
• Baseline external: +0.1155.
• Stress internal: +0.2308.
• Stress external: +0.2212.

Note:

• This set established runtime-native wiring, but later audit found fallback contamination on part of the seed set.
• Use the 2026-02-20 quality-gated rerun below for current canonical C2 interpretation.

Phase 3 Uncertainty Ablation (G5, 2026-02-20, runtime-native awareness3 rerun, zero-fallback quality gate)

Runtime-native source (awareness.generation first, legacy fallback preserved), seeds 7/11/19, baseline + stress:

Quality gate:

• all runtime-native arm artifacts have non-zero runtime requests/ok, fallback=0, errors=0.

Interpretation:

• with clean runtime-native probes, uncertainty-on currently hurts success in this harness.
• runtime-native awareness plumbing is validated; this negative set is now superseded by calibrated rerun below.

Phase 3 Uncertainty Ablation (G5, 2026-02-20, runtime-native calibrated rerun calib1, zero-fallback quality gate)

Runtime-native source, seeds 7/11/19, baseline + stress, calibration params:

• prior weight 0.75
• confidence floor 0.10
• confidence ceil 0.35
• route blend 0.10

Quality gate:

• all runtime-native arm artifacts have non-zero runtime requests/ok, fallback=0, errors=0.

Interpretation:

• calibrated runtime-native uncertainty restores positive uncertainty-on gains in both baseline and stress profiles.
• C2 is re-locked for this harness configuration.

External Cold Comparison (G5, 2026-02-24, Step0 Exp-Reuse Patch, 3-run means, Current Best)

Interpretation:

• Runtime keeps a large margin vs PyTorch and vLLM on request path and cold-total in this rerun.
• Runtime also improved vs immediate seq1mh baseline (2026-02-24T192020Z): TTFT 4.022 -> 4.018 ms, full 239.277 -> 238.400 ms, cold-total 1242.592 -> 1241.688 ms.
• Runtime remains materially better than prior host-prefetch means (2026-02-19): TTFT 5.130 -> 4.018 ms, full 316.403 -> 238.400 ms, cold-total 1320.240 -> 1241.688 ms.
• Follow-up shared-probability patch (external_cold_step0shared_repeatability_20260224T194913Z) did not beat this row and was reverted.
• Ollama is intentionally blank here because it was not installed on the rerun host.

External Cold Decode Profiling + Uncertainty A/B (G5, 2026-02-25, No Preload, 64 Tokens)

Profile source:

• external_cold_stepn_profile_20260225T001334Z
• external_cold_uncert_on_20260225T001702Z
• external_cold_uncert_off_20260225T001704Z

Interpretation:

• The dominant decode stage in this profile is decoder_stepN_logits_sample, then decoder_stepN_layers.
• Disabling uncertainty stats helps, but the uplift is modest relative to total decode time; the main custom-kernel target remains logits+sample compute.

External Cold Runtime vs vLLM (G5, 2026-02-25, Same Profile, Uncertainty-Off Runtime)

Source:

• external_cold_runtime_vllm_uncertoff_20260225T001929Z

Ratios (vLLM/runtime):

• TTFT: 12.618x
• Request Full: 2.774x
• Cold Total First Response: 16.492x

Full-Depth FFN Projection Batched2 (G5, 2026-02-27, --layers 36, preload64)

Source:

• external_cold_layers36_preload64_ab3_ffnprojbatch2_off_s{1,2,3}_20260227T1820*.json
• external_cold_layers36_preload64_ab3_ffnprojbatch2_on_s{1,2,3}_20260227T1820*.json
• external_cold_layers36_preload64_ab3_ffnprojbatch2_vllm_off_s{1,2,3}_20260227T1821*/1822*.json
• external_cold_layers36_preload64_ab3_ffnprojbatch2_vllm_on_s{1,2,3}_20260227T1823*/1824*/182512Z.json
• external_cold_layers36_stageprofile_ffnprojbatch2_off_20260227T182949Z
• external_cold_layers36_stageprofile_ffnprojbatch2_on_20260227T182728Z

Runtime-only 3-seed means:

Runtime-vLLM 3-seed (runtime leg):

Stage-profile corroboration (off -> on):

• decoder_step_profile_ffn_proj_mean: 0.205 -> 0.196 ms/layer
• decoder_stepN_layers_mean: 19.140 -> 18.447 ms
• decoder_stepN_total_mean: 20.761 -> 20.044 ms

Interpretation:

• Batched2 is a real full-depth uplift and is now default-on.
• Runtime still trails vLLM on first-request full latency in this profile, but the gap narrowed again.

Full-Depth FFN Proj Fast-Compute Probe (G5, 2026-02-27, --layers 36, preload64)

Source:

• external_cold_layers36_preload64_ab3_ffnprojfast_off_s{1,2,3}_20260227T194728Z.json
• external_cold_layers36_preload64_ab3_ffnprojfast_on_s{1,2,3}_20260227T194728Z.json
• external_cold_layers36_preload64_ab8_ffnprojfast_off_s{1..8}_20260227T195024Z.json
• external_cold_layers36_preload64_ab8_ffnprojfast_on_s{1..8}_20260227T195024Z.json
• week3_parity_report_ffnprojfast_20260227T194853Z.json

Runtime-only means:

Interpretation (historical for this cycle):

• This specific cycle was too small/noisy to justify default promotion at that time.
• Later clean-path reruns on 2026-02-28 (pool16g, no fallback) looked positive, but full foundation gate (foundation_ffnprojfast_gate_ab2_20260228T195240Z) rejected global promotion; canonical parser default remains TRENI_DECODER_FFN_PROJ_U16_FAST_COMPUTE=0.

Full-Depth U16 Tensor Cache Unlock (G5, 2026-02-27, claim-safe A/B, --layers 36, preload64)

Source:

• external_cold_layers36_preload64_u16cache_claimsafe_summary_20260227T200242Z.json
• external_cold_layers36_preload64_u16cache_claimsafe_summary_20260227T200242Z.md
• week3_parity_report_u16cache_toggle_default_20260227T200652Z.json
• week3_parity_report_u16cachefix_default_20260227T195625Z.json

Runtime-only 3-seed A/B:

Runtime-vLLM same-window A/B (2-seed):

Mechanism check:

• request-path decoder_tensor_upload dropped from ~476 ms to ~5 ms
• request-path decoder_tensor_h2d dropped from ~468 ms to 0 ms

Interpretation:

• This is the primary full-depth request-path unlock in the current cycle.
• With TRENI_TENSOR_CACHE_U16=1, runtime flips from trailing vLLM on request full to leading in the same-window compare.

Full-Depth FFN Follow-Up (G5, 2026-02-27 Late Night, --layers 36, preload64)

Source:

• external_cold_layers36_ffn_followup_summary_20260227T223458Z.json
• external_cold_layers36_ffn_followup_summary_20260227T223458Z.md

Lane 1: TRENI_LINEAR_BATCHED2_USE_LT (new optional backend)

Lane 2: TRENI_DECODER_FFN_PROJ_U16_BATCHED2_F32_INPUT=1 + TRENI_DECODER_FFN_PROJ_U16_FAST_COMPUTE=1 (runtime-only AB8)

Lane 3: FFN fused path bias-deferral follow-up (TRENI_DECODER_FFN_PROJ_U16_FUSED=0/1)

Interpretation:

• No new lane is promoted from this cycle.
• TRENI_LINEAR_BATCHED2_USE_LT=1 regresses materially in runtime-only full-depth A/B.
• F32-input/fast-compute and fused-bias-deferral follow-ups are both near-noise and not material enough for canonical change.

Fast-Profile Logits Follow-Up (G5, 2026-02-28, --layers 2, preload64)

Source:

• external_cold_layers2_logitsfast_ab8_summary_20260228T005529Z.json
• external_cold_layers2_logitsfast_ab8_summary_20260228T005529Z.md

Runtime-only AB8 (TRENI_DECODER_LOGITS_U16_FAST_COMPUTE=0/1):

Interpretation:

• Fast-profile logits fast-compute remains near-noise (full -0.299 ms), so it is not promoted.

Mixed-Load p99 Repeatability (G5, 2026-02-28, Canonical Lane)

Source:

• mixed_load_repeatability_summary_20260228T005626Z.json
• mixed_load_repeatability_summary_20260228T005626Z.md

3-run (run_mode=mixed_load, http_runs=120 each):

Interpretation:

• Current canonical lane remains stable under this mixed-load repeatability set; no configuration change.

External Cold Comparison (G5, 2026-02-18)

Cold total first response ratio over runtime:

• PyTorch: 3.724x
• vLLM: 10.7x
• Ollama: 1.507x

Request-path only note:

• vLLM is fastest on request-path TTFT/full once healthy, but has high startup in this run.

External Cold Comparison (G5, 2026-02-18, Runtime Preload + Tokenizer Cache, Non-Parity)

Runtime advantage in this variant:

• Request full latency vs vLLM: 3.817x faster.
• Cold total first response vs vLLM: 12.334x faster.
• TTFT still trails vLLM (91.596 ms vs 51.725 ms).
• Caveat: runtime was still using 4 decode steps in this run while vLLM/PyTorch/Ollama used 48.

External Cold Comparison (G5, 2026-02-18, Token Parity = 48)

Parity interpretation:

• Runtime still wins cold-total first response vs vLLM (6.216x better).
• vLLM wins request-path TTFT and full latency at equal 48-token budget.

External Cold Comparison (G5, 2026-02-18, Token Parity = 48, Decoder/Sampling Fix)

Post-fix interpretation:

• Runtime now wins TTFT and full request latency vs vLLM at token parity.
• Runtime keeps the large cold-total lead vs vLLM.
• Initial 3-run repeatability (2026-02-18) means: runtime 5.022 ms TTFT, 311.444 ms full, 2316.002 ms cold-total vs vLLM 51.894/1052.767/24752.842 ms.
• Superseded by 2026-02-19 rerun and all-backend repeatability below.

External Cold Comparison (G5, 2026-02-19, GPU-Convert Fix2, All Backends, 3-run means)

Interpretation:

• Runtime keeps the request-path lead across all backends.
• Runtime keeps the cold-total lead vs all backends in this set.
• A runtime preload upload outlier affected one run; stable runs 1-2 sit at ~1004 ms startup and ~1321 ms cold-total.

External Cold Comparison (G5, 2026-02-19, GPU-Convert + Host-Prefetch Fix, All Backends, 3-run means)

Interpretation:

• Runtime keeps the request-path lead across all backends.
• Runtime keeps the cold-total lead vs all backends in this set.
• Host-prefetch rerun removed the startup/upload outlier observed in the prior repeatability set.

Historical Legacy Mixed-Mode Context

Parity Health