UsageTap Compress · JSON and long-text benchmarks · evidence through November 9, 2026
Fewer tokens and lower cost, with answer quality measured
The goal is lower running cost without additional incorrect or materially incomplete answers. Structured JSON and prose have different opportunities and risks. We report them separately: 10 existing JSON tasks alongside the expanded 100-task long-text study.
TL;DR: strong JSON gains; long text needs selective compression
- JSON: 31.3% lower modeled cost and 43.6% fewer context tokens, with 30/30 answers correct. All 10 JSON inputs reconstruct exactly after compression. Compared with already-minified JSON, the full service saves another 16.4% in context tokens and 3.7% in net cost.
- Long text: 27.4% fewer context tokens and 15.7% lower modeled cost on 43 accepted compressions. Across all 100 tasks, including 56 guard fallbacks and one policy skip, savings were 8.1% in context tokens and 4.9% in modeled cost. Preserving required information sometimes means leaving inputs uncompressed. The tested 0.50 setting had three accepted-input answer regressions across two questions, with 20 unresolved comparisons. It does not establish savings without quality loss.
- Gentler long-text follow-ups produced 6/6 correct answers at each of 0.30, 0.25 and 0.15. At 0.30, the accepted VX input saved 7.2% in net cost; Mali was protected by returning the source. These are targeted results on two failing cases.
- Knowing when to avoid the lossy model stage is part of the value. All 10 JSON examples retained structural savings while skipping that stage. In the separate long-text cohort, 56 drafts triggered preservation guards and one input was explicitly skipped.
Structured JSON · 10 tasks, nine source inputs
Long text · selective compression across 100 tasks
On the 43 accepted compressions, context tokens fell 27.4% and modeled cost fell 15.7% after fees at aggressiveness 0.50. Some inputs should stay uncompressed to preserve required information: 56 guard fallbacks and one policy skip returned the source instead.
Headline cost figures price the application prompt and observed output at the frozen Terra API rates. They are estimates, not subscription invoice savings. Correct-answer counts below use source-aware reviews against matched original-input controls.
Cost savings after compression fees
For every $100 of original model cost, the JSON cohort would cost about $68.7 after fees. The separate long-text cohort would cost about $84.3 on accepted compressions, or $95.1 across all 100 tasks, including preserved inputs.
JSON · 10 inputs
Long text · all 100 inputs
Long text · 43 accepted inputs
| Scope | Inputs | Original cost | After + fees | Compression fees | Net cost saved |
|---|---|---|---|---|---|
| JSON · full service | 10 | $0.1799 | $0.1236 | $0.0142 | 31.3% |
| Long text · entire workload | 100 | $3.1636 | $3.0073 | $0.0845 | 4.9% |
| Long text · accepted compression | 43 | $1.0001 | $0.8435 | $0.0845 | 15.7% |
| Long text · protected fallback | 56 | $2.1520 | $2.1523 | $0.0000 | -0.01% |
| Long text · intentional skip | 1 | $0.0114 | $0.0115 | $0.0000 | -0.3% |
The JSON and long-text results are separate cohorts; there is no pooled savings average. Within long text, the entire workload includes accepted compression, guard fallbacks and explicit skips. Conditional savings retain every accepted input, including answer regressions. Dollar totals cover different workloads.
Long-text context tokens fell 27.4% on accepted compressions and 8.1% across all 100 tasks, including preserved inputs. Cost savings are smaller because requests still include instructions, questions and output tokens, and the compression fee is included. Observed output-length variation also affects net cost.
Structured JSON: smaller representations, the same correct answers
The full service saved 43.6% of context tokens and 31.3% of modeled running cost on the JSON cohort, with 30/30 correct answers. This covers 10 questions across nine distinct inputs, with three answer repeats per condition. Both source-aware reviewers agreed with the official type-aware scorer.
| Representation | Net cost saved vs original | Context saved vs original | Correct · official | Correct · source-aware | Unresolved |
|---|---|---|---|---|---|
| Minified JSON | 28.7% | 32.5% | 30/30 | 30/30 | 0 |
| Structural compression only | 33.1% | 43.6% | 30/30 | 30/30 | 0 |
| Full service | 31.3% | 43.6% | 30/30 | 30/30 | 0 |
The minified baseline matters. Ordinary JSON minification already saves 32.5% of context tokens and 28.7% in modeled cost. The full service reduces context by another 16.4% and net cost by another 3.7% relative to that already-minified representation. Tabular examples show larger gains than irregular nested data.
Structural-only and full-service requests returned identical compressed text for all 10 inputs. The cost difference between those conditions comes from their separately observed output usage. All cost comparisons include the applicable compression fee; no answer errors or inputs were excluded from these averages.
All 10 JSON examples
| JSON example | Net cost saved | Context saved vs original | Context saved vs minified | Correct | Round-trip |
|---|---|---|---|---|---|
| analytics · q75 | 39.8% | 58.7% | 35.3% | 3/3 | Verified |
| event-logs · q128 | 28.3% | 29.0% | 0.0% | 3/3 | Verified |
| github · q109 | 21.1% | 41.7% | 23.2% | 3/3 | Verified |
| keyed · q182 | 29.8% | 32.4% | 0.0% | 3/3 | Verified |
| nested · q41 | 36.0% | 36.6% | 0.0% | 3/3 | Verified |
| nested-config · q153 | 33.2% | 39.1% | 0.0% | 3/3 | Verified |
| nested-group · q202 | 39.4% | 41.1% | 0.0% | 3/3 | Verified |
| structural-validation-control · q240 | 34.4% | 60.4% | 36.0% | 3/3 | Verified |
| tabular · q217 | 40.4% | 60.8% | 36.8% | 3/3 | Verified |
| analytics · q77 | 39.7% | 58.7% | 35.3% | 3/3 | Verified |
Analytics and tabular examples removed about 59–61% of context tokens versus the original formatted JSON by avoiding repeated field names and unnecessary syntax. Keys, values, types, array order and nesting remained intact when decoded: all 30 transformed representations across the three conditions round-trip to the original objects.
A good decision: use structural savings without a lossy model rewrite
For every JSON input, the service reported no candidate prose and skipped the LLMLingua stage. It still performed lossless structural compression. These model-stage skips retain their savings in the JSON results; they are different from zero-saving whole-input preservation decisions in the long-text cohort.
See a real analytics example before and after
The complete saved input and output are shown below. Repeated JSON keys become a compact table header; the data values are preserved. The strike-through explorer also remains available for every JSON example.
Original JSON
{
"metrics": [
{
"date": "2025-01-01",
"views": 4874,
"clicks": 336,
"conversions": 39,
"revenue": 7879.38,
"bounceRate": 0.61
},
{
"date": "2025-01-02",
"views": 5085,
"clicks": 193,
"conversions": 16,
"revenue": 3201.57,
"bounceRate": 0.41
},
{
"date": "2025-01-03",
"views": 4773,
"clicks": 152,
"conversions": 19,
"revenue": 3269.87,
"bounceRate": 0.33
},
{
"date": "2025-01-04",
"views": 7464,
"clicks": 278,
"conversions": 20,
"revenue": 4973.44,
"bounceRate": 0.32
},
{
"date": "2025-01-05",
"views": 3865,
"clicks": 268,
"conversions": 39,
"revenue": 3211.71,
"bounceRate": 0.67
},
{
"date": "2025-01-06",
"views": 5974,
"clicks": 288,
"conversions": 39,
"revenue": 6059.91,
"bounceRate": 0.52
},
{
"date": "2025-01-07",
"views": 5611,
"clicks": 220,
"conversions": 23,
"revenue": 3450.66,
"bounceRate": 0.5
},
{
"date": "2025-01-08",
"views": 6272,
"clicks": 341,
"conversions": 18,
"revenue": 2020.03,
"bounceRate": 0.32
},
{
"date": "2025-01-09",
"views": 7542,
"clicks": 243,
"conversions": 27,
"revenue": 3319.7,
"bounceRate": 0.5
},
{
"date": "2025-01-10",
"views": 7048,
"clicks": 483,
"conversions": 42,
"revenue": 7939.91,
"bounceRate": 0.49
},
{
"date": "2025-01-11",
"views": 4561,
"clicks": 357,
"conversions": 44,
"revenue": 4030.53,
"bounceRate": 0.38
},
{
"date": "2025-01-12",
"views": 3721,
"clicks": 283,
"conversions": 41,
"revenue": 12098.32,
"bounceRate": 0.58
},
{
"date": "2025-01-13",
"views": 4045,
"clicks": 182,
"conversions": 17,
"revenue": 3014.48,
"bounceRate": 0.6
},
{
"date": "2025-01-14",
"views": 6274,
"clicks": 452,
"conversions": 33,
"revenue": 5483.75,
"bounceRate": 0.59
},
{
"date": "2025-01-15",
"views": 6806,
"clicks": 208,
"conversions": 25,
"revenue": 5095.22,
"bounceRate": 0.59
},
{
"date": "2025-01-16",
"views": 7684,
"clicks": 303,
"conversions": 21,
"revenue": 1603.05,
"bounceRate": 0.6
},
{
"date": "2025-01-17",
"views": 5207,
"clicks": 214,
"conversions": 15,
"revenue": 4301.6,
"bounceRate": 0.6
},
{
"date": "2025-01-18",
"views": 4799,
"clicks": 135,
"conversions": 18,
"revenue": 4758.2,
"bounceRate": 0.48
},
{
"date": "2025-01-19",
"views": 5993,
"clicks": 148,
"conversions": 14,
"revenue": 3817.59,
"bounceRate": 0.63
},
{
"date": "2025-01-20",
"views": 3301,
"clicks": 97,
"conversions": 7,
"revenue": 634.44,
"bounceRate": 0.6
},
{
"date": "2025-01-21",
"views": 6905,
"clicks": 381,
"conversions": 37,
"revenue": 5294.59,
"bounceRate": 0.64
},
{
"date": "2025-01-22",
"views": 6011,
"clicks": 234,
"conversions": 34,
"revenue": 7185.84,
"bounceRate": 0.69
},
{
"date": "2025-01-23",
"views": 5620,
"clicks": 298,
"conversions": 31,
"revenue": 1662.36,
"bounceRate": 0.5
},
{
"date": "2025-01-24",
"views": 7039,
"clicks": 283,
"conversions": 38,
"revenue": 3444.52,
"bounceRate": 0.4
},
{
"date": "2025-01-25",
"views": 4543,
"clicks": 176,
"conversions": 10,
"revenue": 818.45,
"bounceRate": 0.45
},
{
"date": "2025-01-26",
"views": 3041,
"clicks": 161,
"conversions": 15,
"revenue": 3869.87,
"bounceRate": 0.41
},
{
"date": "2025-01-27",
"views": 3228,
"clicks": 198,
"conversions": 23,
"revenue": 3606.18,
"bounceRate": 0.42
},
{
"date": "2025-01-28",
"views": 6193,
"clicks": 248,
"conversions": 20,
"revenue": 1282.45,
"bounceRate": 0.63
},
{
"date": "2025-01-29",
"views": 6659,
"clicks": 299,
"conversions": 23,
"revenue": 6400.15,
"bounceRate": 0.35
},
{
"date": "2025-01-30",
"views": 4789,
"clicks": 184,
"conversions": 21,
"revenue": 1060.15,
"bounceRate": 0.41
},
{
"date": "2025-01-31",
"views": 7206,
"clicks": 504,
"conversions": 47,
"revenue": 12544.25,
"bounceRate": 0.43
},
{
"date": "2025-02-01",
"views": 6857,
"clicks": 416,
"conversions": 27,
"revenue": 2226.07,
"bounceRate": 0.32
},
{
"date": "2025-02-02",
"views": 3455,
"clicks": 230,
"conversions": 22,
"revenue": 4246.11,
"bounceRate": 0.35
},
{
"date": "2025-02-03",
"views": 4511,
"clicks": 211,
"conversions": 26,
"revenue": 2290.2,
"bounceRate": 0.31
},
{
"date": "2025-02-04",
"views": 7888,
"clicks": 546,
"conversions": 72,
"revenue": 6215.21,
"bounceRate": 0.58
},
{
"date": "2025-02-05",
"views": 4733,
"clicks": 324,
"conversions": 25,
"revenue": 1257.44,
"bounceRate": 0.64
},
{
"date": "2025-02-06",
"views": 5430,
"clicks": 405,
"conversions": 56,
"revenue": 11811.34,
"bounceRate": 0.43
},
{
"date": "2025-02-07",
"views": 4329,
"clicks": 317,
"conversions": 25,
"revenue": 6000.69,
"bounceRate": 0.55
},
{
"date": "2025-02-08",
"views": 6083,
"clicks": 310,
"conversions": 36,
"revenue": 6624.98,
"bounceRate": 0.45
},
{
"date": "2025-02-09",
"views": 4946,
"clicks": 211,
"conversions": 15,
"revenue": 4380.37,
"bounceRate": 0.58
},
{
"date": "2025-02-10",
"views": 3888,
"clicks": 176,
"conversions": 26,
"revenue": 6921.14,
"bounceRate": 0.68
},
{
"date": "2025-02-11",
"views": 7519,
"clicks": 565,
"conversions": 32,
"revenue": 2961.59,
"bounceRate": 0.42
},
{
"date": "2025-02-12",
"views": 5294,
"clicks": 414,
"conversions": 53,
"revenue": 4966.36,
"bounceRate": 0.5
},
{
"date": "2025-02-13",
"views": 5618,
"clicks": 382,
"conversions": 19,
"revenue": 4540.61,
"bounceRate": 0.36
},
{
"date": "2025-02-14",
"views": 6421,
"clicks": 197,
"conversions": 13,
"revenue": 1929.78,
"bounceRate": 0.65
},
{
"date": "2025-02-15",
"views": 5104,
"clicks": 323,
"conversions": 46,
"revenue": 13251.3,
"bounceRate": 0.36
},
{
"date": "2025-02-16",
"views": 4685,
"clicks": 186,
"conversions": 20,
"revenue": 1074.77,
"bounceRate": 0.31
},
{
"date": "2025-02-17",
"views": 4212,
"clicks": 284,
"conversions": 17,
"revenue": 2079.87,
"bounceRate": 0.32
},
{
"date": "2025-02-18",
"views": 5851,
"clicks": 332,
"conversions": 32,
"revenue": 4235.83,
"bounceRate": 0.69
},
{
"date": "2025-02-19",
"views": 4011,
"clicks": 248,
"conversions": 15,
"revenue": 4338.97,
"bounceRate": 0.51
},
{
"date": "2025-02-20",
"views": 4630,
"clicks": 273,
"conversions": 41,
"revenue": 5912.13,
"bounceRate": 0.35
},
{
"date": "2025-02-21",
"views": 4773,
"clicks": 127,
"conversions": 14,
"revenue": 1754.07,
"bounceRate": 0.37
},
{
"date": "2025-02-22",
"views": 5480,
"clicks": 158,
"conversions": 22,
"revenue": 3796.3,
"bounceRate": 0.55
},
{
"date": "2025-02-23",
"views": 3608,
"clicks": 176,
"conversions": 13,
"revenue": 2986.19,
"bounceRate": 0.6
},
{
"date": "2025-02-24",
"views": 4560,
"clicks": 346,
"conversions": 23,
"revenue": 4154,
"bounceRate": 0.43
},
{
"date": "2025-02-25",
"views": 5191,
"clicks": 322,
"conversions": 18,
"revenue": 5322.11,
"bounceRate": 0.53
},
{
"date": "2025-02-26",
"views": 7681,
"clicks": 259,
"conversions": 19,
"revenue": 4217.32,
"bounceRate": 0.66
},
{
"date": "2025-02-27",
"views": 4497,
"clicks": 156,
"conversions": 16,
"revenue": 1881.53,
"bounceRate": 0.57
},
{
"date": "2025-02-28",
"views": 4829,
"clicks": 329,
"conversions": 34,
"revenue": 6348.76,
"bounceRate": 0.47
},
{
"date": "2025-03-01",
"views": 7375,
"clicks": 215,
"conversions": 20,
"revenue": 2179.58,
"bounceRate": 0.47
}
]
}Compressed representation
metrics[60]{date,views,clicks,conversions,revenue,bounceRate}:
2025-01-01,4874,336,39,7879.38,0.61
2025-01-02,5085,193,16,3201.57,0.41
2025-01-03,4773,152,19,3269.87,0.33
2025-01-04,7464,278,20,4973.44,0.32
2025-01-05,3865,268,39,3211.71,0.67
2025-01-06,5974,288,39,6059.91,0.52
2025-01-07,5611,220,23,3450.66,0.5
2025-01-08,6272,341,18,2020.03,0.32
2025-01-09,7542,243,27,3319.7,0.5
2025-01-10,7048,483,42,7939.91,0.49
2025-01-11,4561,357,44,4030.53,0.38
2025-01-12,3721,283,41,12098.32,0.58
2025-01-13,4045,182,17,3014.48,0.6
2025-01-14,6274,452,33,5483.75,0.59
2025-01-15,6806,208,25,5095.22,0.59
2025-01-16,7684,303,21,1603.05,0.6
2025-01-17,5207,214,15,4301.6,0.6
2025-01-18,4799,135,18,4758.2,0.48
2025-01-19,5993,148,14,3817.59,0.63
2025-01-20,3301,97,7,634.44,0.6
2025-01-21,6905,381,37,5294.59,0.64
2025-01-22,6011,234,34,7185.84,0.69
2025-01-23,5620,298,31,1662.36,0.5
2025-01-24,7039,283,38,3444.52,0.4
2025-01-25,4543,176,10,818.45,0.45
2025-01-26,3041,161,15,3869.87,0.41
2025-01-27,3228,198,23,3606.18,0.42
2025-01-28,6193,248,20,1282.45,0.63
2025-01-29,6659,299,23,6400.15,0.35
2025-01-30,4789,184,21,1060.15,0.41
2025-01-31,7206,504,47,12544.25,0.43
2025-02-01,6857,416,27,2226.07,0.32
2025-02-02,3455,230,22,4246.11,0.35
2025-02-03,4511,211,26,2290.2,0.31
2025-02-04,7888,546,72,6215.21,0.58
2025-02-05,4733,324,25,1257.44,0.64
2025-02-06,5430,405,56,11811.34,0.43
2025-02-07,4329,317,25,6000.69,0.55
2025-02-08,6083,310,36,6624.98,0.45
2025-02-09,4946,211,15,4380.37,0.58
2025-02-10,3888,176,26,6921.14,0.68
2025-02-11,7519,565,32,2961.59,0.42
2025-02-12,5294,414,53,4966.36,0.5
2025-02-13,5618,382,19,4540.61,0.36
2025-02-14,6421,197,13,1929.78,0.65
2025-02-15,5104,323,46,13251.3,0.36
2025-02-16,4685,186,20,1074.77,0.31
2025-02-17,4212,284,17,2079.87,0.32
2025-02-18,5851,332,32,4235.83,0.69
2025-02-19,4011,248,15,4338.97,0.51
2025-02-20,4630,273,41,5912.13,0.35
2025-02-21,4773,127,14,1754.07,0.37
2025-02-22,5480,158,22,3796.3,0.55
2025-02-23,3608,176,13,2986.19,0.6
2025-02-24,4560,346,23,4154,0.43
2025-02-25,5191,322,18,5322.11,0.53
2025-02-26,7681,259,19,4217.32,0.66
2025-02-27,4497,156,16,1881.53,0.57
2025-02-28,4829,329,34,6348.76,0.47
2025-03-01,7375,215,20,2179.58,0.47Explore all JSON inputs, strike-throughs and answersDownload JSON cost comparisonsInspect exact structural checks
This is the retained JSON subset of the earlier benchmark, not 10 newly generated questions. Its 120 saved answers across original, minified, structural-only and full-service conditions were rechecked with the newer source-aware method; both reviewers agreed all were correct. The official full-service result is 30/30. Exact structural preservation and observed answer correctness are evidence for these inputs, not a guarantee for every JSON shape or future model.
Long text: did the answers stay correct?
At 0.50, accepted compression produced 106 correct answers across 109 resolved comparisons; the original-input controls were correct in all 109. The other 20 comparisons remain unresolved. The denominator is repeated answer pairs, not 109 independent questions.
| Scope | Correct · original | Correct · candidate | Unresolved pairs | Observed regressions |
|---|---|---|---|---|
| Accepted compression | 109/109 | 106/109 | 20 | 3 |
| Protected fallback | 156/156 | 155/156 | 12 | 1 |
| Intentional skip | 3/3 | 3/3 | 0 | 0 |
| Entire workload | 268/268 | 264/268 | 32 | 4 |
The key failure was three accepted-input answer regressions across two questions. These were material factual issues, not merely lower word-overlap scores. Deeper review linked them to a lost causal connection in Mali and a changed quantity in VX.
The entire workload has one additional declining answer pair on byte-identical original and returned text. That is generation variation on a preserved input; it is reported separately from the three accepted-compression regressions.
Only pairs with resolved original and candidate judgments appear in the “correct” fractions. An unresolved answer is neither a pass nor an assumed error. Source-aware correctness is the primary quality outcome; official QA F1 and ROUGE-L remain available in the explorer as supporting scores.
Gentler long-text settings preserved correct answers in these repeats
The two confirmed news regressions were retested with fresh original-input controls and three candidate answers per case. At 0.30, 0.25 and 0.15, both source-aware reviewers judged 6/6 candidate answers correct, matching the original controls.
Net modeled cost savings on accepted inputs
Each of these three accepted-input bars is the VX case. Mali used original-text fallback and is excluded from this conditional savings average. The table also shows both cases together.
| Level | Net cost saved · both cases | Net cost saved · accepted | VX context saved | Correct before → after | Protected inputs |
|---|---|---|---|---|---|
| 0.50 | 13.5% | 13.5% | 29.9% | 6/6 → 2/6 | 0/2 |
| 0.45 | 6.9% | 13.5% | 26.8% | 6/6 → 4/6 | 1/2 |
| 0.30 | 4.1% | 7.2% | 18.4% | 6/6 → 6/6 | 1/2 |
| 0.25 | 3.9% | 6.6% | 15.3% | 6/6 → 6/6 | 1/2 |
| 0.15 | 1.0% | 2.1% | 9.7% | 6/6 → 6/6 | 1/2 |
At 0.50, four of six targeted answers were only partly correct; at 0.45, two were. At the three gentler settings, no answer error was observed in these repeats. For Mali, the improvement came through preservation rather than a successful lossy compression.
0.30 is a candidate for broader validation. It saved the most among the gentler settings with correct answers here. Moving to 0.15 made the source wording clearer but did not improve the measured answer results in this small sample.
These two inputs were chosen because they previously failed. Three repeats per case do not establish a safe default or estimate full-workload savings at the gentler settings. Each level uses its own original controls and observed output counts, so cost differences also include answer-length variation.
Results across the five long-text benchmarks
Each benchmark contributes 20 questions and three repeats per original and candidate input. This table covers the complete workload at 0.50; the accepted column shows how often the compressor actually used a model-compressed output.
| Benchmark | Questions | Accepted | Net cost saved | Context saved | Correct · original → candidate | Unresolved | Regressions |
|---|---|---|---|---|---|---|---|
| HotpotQA | 20 | 6 | 4.2% | 6.4% | 57/57 → 57/57 | 3 | 0 |
| MultiNews | 20 | 17 | 8.7% | 23.4% | 52/52 → 49/52 | 8 | 3 |
| MuSiQue | 20 | 3 | 2.9% | 4.3% | 56/56 → 55/56 | 4 | 1 |
| Qasper | 20 | 8 | 4.1% | 6.7% | 48/48 → 48/48 | 12 | 0 |
| QMSum | 20 | 9 | 6.9% | 11.6% | 55/55 → 55/55 | 5 | 0 |
Correct-answer fractions use resolved matched pairs within each benchmark; unresolved counts are shown separately. Savings are weighted by cost or token totals, not an average of per-case percentages. Dataset conclusions are limited by the small number of source documents and unresolved judgments.
Knowing what not to compress is part of the value
In the long-text cohort, the compressor accepted model outputs for 43 inputs, used preservation fallbacks for 56 and intentionally skipped one. That selectivity trades maximum compression for keeping usable evidence available to the answering model.
| Decision | Inputs | Context saved | Net cost saved | Interpretation |
|---|---|---|---|---|
| Compress | 43 | 27.4% | 15.7% | Quality failures remain in this group. |
| Preserve via fallback | 56 | 0.01% | -0.01% | Rejected drafts were never used for answers. |
| Intentionally skip | 1 | 0.0% | -0.3% | Preserve the source; no compression expected. |
Why we preserved these inputs
Each preserved input has a recorded reason. A guard fallback means compression was attempted and its draft rejected; a policy skip means the model was deliberately not run.
| Recorded reason | Inputs | Decision | Why preserve |
|---|---|---|---|
| Identifiers | 25 | Guard fallback | An identifier-like value failed exact preservation; retain the safe version. |
| Protected content | 25 | Guard fallback | Content covered by exact-preservation rules failed a check; use fallback text. |
| Constraints | 3 | Guard fallback | A requirement or critical clause failed exact preservation; retain its wording. |
| Links | 2 | Guard fallback | A URL or link failed exact preservation; retain the safe version. |
| Exact-output policy | 1 | Policy skip | An exact-output phrase triggered a conservative policy skip; the model did not run. |
| Code | 1 | Guard fallback | Inline or fenced code failed exact preservation; retain the safe version. |
Counts are inputs, not answer repeats, using the first reported reason. The retained diagnostics identify the failing category but do not retain rejected drafts or exact offending values. These are 57 recorded preservation decisions, not 57 proven answer errors prevented.
A demonstrated good decision: preserve the Mali salary-payment cause
At 0.30, 0.25 and 0.15, the guard returned the original source, retaining the explicit connection between the frozen bank account and salary-payment problems. All three answers were judged correct at each setting. The earlier accepted 0.50 compression had produced a factual regression on this case.
The explicit skip: a conservative policy match, not a proven necessity
The one policy skip was a Qasper article containing the phrase “we do not change the text preprocessing methods for MT.” The service recorded an exact-output-context skip and returned the original text; all three answers were judged correct.
Here that phrase describes a research method rather than asking the answering model to reproduce text exactly. We therefore show it as a conservative policy match, not evidence that compression would have caused an error.
Inspect the skipped input and its answers · Download all 57 preservation reasons
Fallbacks and intentional skips do not dilute the accepted-compression average. They remain fully visible in the overall result. Accepted attempts with no savings or with answer errors stay in the accepted group. The groups were decided before answer scoring.
Fallback context reduction is approximately zero, rather than exactly zero for every case: deterministic cleanup removed 68 proxy tokens across the 56 guarded inputs. Any small cost difference on preserved inputs can also reflect generation length, not a successful model compression. Preservation guards do not by themselves guarantee correct answers.
Source damage explains failures and signals fragility
Correct results are the primary outcome. Source damage is a secondary diagnostic: it explains why an answer failed or why a seemingly successful input may be fragile. Damaged wording with correct answers is not counted as an observed answer regression.
The two factual regressions: cause and quantity
Mali, 0.50: removal of the explicit bank-account cause allowed answers to blame fuel shortages for salary-payment problems. Gentler settings returned the original text through the integrity guard.
VX, 0.50: changing “one of the two women” to a statement about two women led to incorrect quantities in the answers. At 0.30 and 0.25, the compressed phrase “one two women” still read poorly, but all three answers at each level correctly described one woman becoming ill. At 0.15, “one of two women” was clearer, without an additional measured answer-quality gain.
All deeper source audits from the full study
HotpotQA · 3a75aba6
All three original and all three full-compression calls produced the identical disputed expanded answer. Terra and Sol disagree about whether adding Fire Island is acceptable. Keep the ambiguity unscored; there is no observed full-compression answer decline on this question.
Original: The Great South Bay is a lagoon situated between Long Island and Fire Island, in the State of New York.
MuSiQue · d1ff1828
The model draft was rejected. Full and deterministic-only prompts are byte-identical, and the original and fallback contain identical word sequences. The correct leadership evidence remains. The wrong Stephen Harper answer occurs twice in deterministic-only calls and once in full-condition calls using the same fallback text. This demonstrates answer variation on identical fallback prompts; no source words were removed from the fallback.
Original: Party members chose Andrew Scheer as leader, replacing Stephen Harper, who led the Conservative Party of Canada as its leader from 2004 following the merger of the Canadian Alliance and Progressive Conservative parties.
MuSiQue · 7b7acfdf
The source period 323–272 BC describes the scope of a historical narrative, while the question asks about the war associated with Pyrrhus. Both judges identify ambiguity in assessing the alternative war dates. The fallback preserves the original word sequence. Retain official scores and unresolved source judgments without calling the difference compression damage.
MuSiQue · f11250d4
Both judges flag missing source support linking the named wife role to the actor in the reference. Original and fallback word sequences are identical, and abstentions occur in both conditions. Keep this as unresolved source/reference coverage, not demonstrated compression loss.
Qasper · 39352d30
The reference set includes both Unanswerable and a specific list of baseline systems. The supplied paper context does not expose the comparison tables needed to verify that list. Both original and compressed answers name wav2letter as a baseline although the source identifies it as inspiration. Both judges keep those claims uncertain. The unsupported specificity already appears in original-input answers, so a new compression-caused loss is not established.
Qasper · f545e80c
Every original, deterministic-only and full-condition call answered unanswerable; the reference answer is No. Both validators object to that answer, but their material-issue descriptions did not satisfy the conservative corroboration matcher, so the saved judgment remains unresolved. The answer and official-score outcome are identical across conditions; this is not an observed compression regression.
MultiNews · 5021490f
The original directly links uncertain civil-service pay to the frozen bank account. Compression retains the bank-account and salary facts in the same passage but removes the explicit because link. One compressed-input summary instead attributes threatened salaries to fuel shortages. Both judges confirm that factual error. This is a plausible loss-of-linkage mechanism, with answer-generation variation also possible; the gentler-compression and stronger-model follow-up is reported separately.
Original: It's unclear if the junta will be able to pay civil servants in April because the country's bank account is now frozen.
MultiNews · cef9105b
The original says one of the two women became ill. The accepted 0.50 representation drops one of the and says two women were sickened; an earlier source sentence also loses its subject. Two compressed-input summaries attribute vomiting to both women, and both judges corroborate the error. The source comparison verifies a quantifier distortion that passed the existing integrity checks. Follow-up results do not replace these main-cohort observations.
Compressed: Khalid told reporters two women accused wiping toxin on Kim ' s face sickened suffered vomiting.
Original: Khalid told reporters that one of the two women accused of wiping the toxin on Kim's face was later sickened and suffered from vomiting.
QMSum · 8a2a85ee
The disputed Aurora proposal label repeats an ambiguity already examined in the earlier experiment. The judges agree that the writing progress, deadlines, exam date and research approach are supported, but differ over whether the topic label is too imprecise. Keep the frozen uncertainty; do not turn a label dispute into a demonstrated factual loss.
QMSum · b83d7e10
Both original and compressed answers sometimes select a different discussion about slides or product evaluation rather than the requested teamwork disagreement. The compressed text also weakens dialogue cues: I think so and Yep become punctuation in the relevant exchange. That is an input-quality concern, but the saved answer comparison cannot cleanly isolate its effect from the original model failures. The case remains unresolved and outside resolved-score means; perfect resolved means must not be read as correctness on this question.
Compressed: em. Project Manager : enough teamwork? Industrial Designer : Yeah? Marketing :. User Interface :. Project Manager :. User Interface : {vocalsound} Project Manager : No I, dunno,, Industrial Designer : {vocalsound} don ' t. Project Manager : ' t {disfmarker} I I was just {disfmarker} I {disfmarker} Marketing : didn ' t {disfmarker} minor discussion at meetings ' t {disfmarker} except for actual building, Industrial Designer : Yeah. Project Manager : ' true? Marketing : if team project more coll
Original: ave 'em . Project Manager: Um you guys felt like there was enough teamwork in all ? Industrial Designer: Yeah ? Marketing: I think so . User Interface: Yep . Project Manager: Hmm . User Interface: {vocalsound} Project Manager: No I , no I dunno , I d I I dunno , Industrial Designer: {vocalsound} You don't . Project Manager: I don't {disfmarker} I I was just {disfmarker} I {disfmarker} Marketing: Though we didn't actually {disfmarker} I mean other than minor discussion at meetings there wasn't {disfmarker} except for the actu
HotpotQA · 87fd3f05
All original answers say No; compressed-input answers say No and add a supported explanation about Faygo and non-carbonated Yoo-hoo. Both validators rate the saved answers correct. The official reference is only no, so added correct words sharply reduce lexical precision. This is a wording penalty, not an observed factual regression.
Original: Faygo Beverages, Inc., is a soft drink company headquartered in Detroit, Michigan. The beverages produced by the company, branded as Faygo or Faygo Pop, are distributed in the Midwestern, Mid-Atlantic, and Central Southern regions of the United States, as well as southern and western Canada. Faygo Beverages, Inc. is a wholly owned subsidiary of the National Beverage Corporation started in Detroit, Michigan in 1907 as Feigenson Brothers Bottling Works.
Original: Because the drink had a limited shelf life, the company sold its products only in Michigan until the late 1950s. Company chemists later resolved this issue by installing a filtration system to remove impurities from the manufacturing plant's water system. In the 1960s, the soda's regional popularity expanded when the company began advertising during broadcasts of Detroit Tigers games. With the Tiger ads reaching beyond the Faygo market area at the time and the inability to cancel the ads, Faygo shipped products to the wholesalers' warehouses. This increased company sales from $6 million in 1966 to $20.4 million in 1971. Advertisements produced in the 1970s featured "everyday people" on a Boblo Boat singing the "Faygo Boat Song".The company introduced a low-calorie version of their products in the 1960s called Ohana. This sub-line soon became a majority of company sales. In 1961, the Royal Line was launched as a premium product line. The juice extract used to produce the initial run of Royal Hawaiian Pineapple Orange was not sterilized and became rancid, causing a buildup of gases such that, after hitting store shelves, the bottles exploded. The soda was recalled and Dole gave Faygo enough sterilized juice to offset the company's losses. Six other flavors also entered into production in the 1960s. The original strawberry flavor from 1907 was renamed Redpop in the late 1960s.With Michigan's beverage container deposit law passed in 1978, Faygo thought people would prefer returnable cans instead of glass. With this choice being incorrect, the company had a hard time making the switch back to bottles, cutting into profits for several years.Assessing the industry and the second generation's pending retirement, the company was put up for sale. TreeSweet Products Corp. bought the company from the Feigenson family in early 1986 for $105 million. TreeSweet in turn sold the company to National Beverage Corp. a year later in 1987. In the 1980s, they introduced flavored carbonated water.Faygo expanded in 1996 with a non-carbonated drink line, again named Ohana, which included punches, iced tea and lemonade. In 2007, Faygo celebrated its 100th anniversary with a new flavor and contests for label design. Ten thousand entries were received and a fourth-grade Ohio teacher won with Centennial Soda. In March 2014, the company introduced its ginger ale, Faygo Gold, rivaling cross town company Vernors' flagship drink.
HotpotQA · b6352c61
Original answers use Band-e Amir National Park; compressed-input answers use Band-e-Amir National Park. Both validators judge the same park correctly identified. The official normalizer removes hyphens rather than replacing them with spaces, producing different word tokens and a 42.9-point case-level F1 penalty for the hyphenation difference. Official scores remain unchanged.
Original: Band-e Amir National Park (Persian: بند امیر) Afghanistan established its first national park on April 22, 2009, to promote and protect the natural beauty of a series of intensely blue lakes created by natural dams high in the Hindu Kush. Band-e-Amir is a chain of six lakes in the mountainous desert of central Afghanistan. The lakes formed from mineral-rich water that seeped out of faults and cracks in the rocky landscape. Over time, the water deposited layers of hardened mineral (travertine) that built up into walls that now contain the water. According to the Wildlife Conservation Society, who helped the Afghan government set up the park, Band-e-Amir is one of the few travertine systems in the world.They were created by the carbon dioxide rich water oozing out of the faults and fractures to deposit calcium carbonate precipitate in the form of travertine walls that today store the water of these lakes. Band-e Amir is one of the few rare natural lakes in the world which are created by travertine systems. The site of Band-e Amir has been described as Afghanistan's Grand Canyon, and draws thousands of tourists a year. The river is part of the system of the Balkh River.
Original: Band-e-Amir Dragons (Pashto: بند امير ښاماران Band-e-Amīr Khāmārān / Shāmārān; Dari: بند امیر شاماران) or Band-e-Amir Region is one of eight regional first-class cricket teams in Afghanistan. In the 2017 Shpageeza team auction, the Band-e-Amir Dragons team was bought by Paragon Business Group, a Finance and investment company, and it will play as the Paragon Band-e-Amir Dragons in this competition. The Region represents the following provinces in the centre of Afghanistan: Ghazni, Bamyan, Daykundi, and Maidan Wardak. The team is named after Band-e Amir, a series of six deep blue lakes in Bamyan Province.
Stronger answer models did not reliably repair the damaged evidence
The original targeted follow-up also tested Terra high and Sol high on the frozen 0.50 inputs. These are diagnostic tests on the same two cases, not model-quality results across the 100-task cohort.
| Case | Answer model / effort | Level | Correct | Partly correct | Unresolved |
|---|---|---|---|---|---|
| Mali | terra-high | 0.50 | 2 | 1 | 0 |
| VX | terra-high | 0.50 | 1 | 2 | 0 |
| Mali | sol-high | 0.50 | 0 | 3 | 0 |
| VX | sol-high | 0.50 | 2 | 0 | 1 |
Counts are out of three answers per row. Complete answers, original-input controls, reasoning settings and source citations are retained in the explorer and original archives.
Every input remains inspectable
The full 100-input long-text ledger below includes zero-saving decisions, answer regressions and unresolved judgments. Open any input to inspect its original text, removals, actual answers, official scores and source-aware review evidence.
Long-text evidence explorerJSON evidence explorerDownload the cost ledgerDownload exact summary calculationsDownload complete report dataDownload the website report bundle
All 100 inputs: cost and answer outcomes
| Input / answers | Decision | Preservation reason | Net cost saved | Context saved | Correct · original → candidate | Unresolved | Regressions |
|---|---|---|---|---|---|---|---|
| HotpotQA · 3a75aba6 | Accepted | — | 20.3% | 30.5% | No resolved pairs | 3 | 0 |
| HotpotQA · 3f17f12c | Fallback | Protected content | 0.03% | 0.03% | 3/3 → 3/3 | 0 | 0 |
| HotpotQA · 4636487a | Fallback | Protected content | 0.01% | 0.01% | 3/3 → 3/3 | 0 | 0 |
| HotpotQA · 4c34b33d | Fallback | Protected content | 0.01% | 0.01% | 3/3 → 3/3 | 0 | 0 |
| HotpotQA · 6c64a81f | Fallback | Identifiers | 0.02% | 0.02% | 3/3 → 3/3 | 0 | 0 |
| HotpotQA · 72a769d6 | Fallback | Protected content | 0.04% | 0.04% | 3/3 → 3/3 | 0 | 0 |
| HotpotQA · 78e73e15 | Accepted | — | 17.7% | 27.9% | 3/3 → 3/3 | 0 | 0 |
| HotpotQA · 87fd3f05 | Accepted | — | 19.9% | 30.9% | 3/3 → 3/3 | 0 | 0 |
| HotpotQA · 9729ff4c | Fallback | Protected content | 0.1% | 0.0% | 3/3 → 3/3 | 0 | 0 |
| HotpotQA · 97b31d6d | Fallback | Protected content | 0.0% | 0.0% | 3/3 → 3/3 | 0 | 0 |
| HotpotQA · 9a19f959 | Fallback | Protected content | 0.01% | 0.01% | 3/3 → 3/3 | 0 | 0 |
| HotpotQA · a18c0ece | Fallback | Protected content | 0.0% | 0.0% | 3/3 → 3/3 | 0 | 0 |
| HotpotQA · ad26b1fd | Accepted | — | 18.8% | 28.6% | 3/3 → 3/3 | 0 | 0 |
| HotpotQA · b6352c61 | Accepted | — | 17.7% | 28.7% | 3/3 → 3/3 | 0 | 0 |
| HotpotQA · b7859630 | Fallback | Links | 0.3% | 0.01% | 3/3 → 3/3 | 0 | 0 |
| HotpotQA · c9ee71b5 | Fallback | Protected content | 0.0% | 0.0% | 3/3 → 3/3 | 0 | 0 |
| HotpotQA · cf16ba25 | Fallback | Protected content | 0.0% | 0.0% | 3/3 → 3/3 | 0 | 0 |
| HotpotQA · d542fcd4 | Accepted | — | 19.4% | 29.3% | 3/3 → 3/3 | 0 | 0 |
| HotpotQA · e0a5da95 | Fallback | Protected content | -0.04% | 0.01% | 3/3 → 3/3 | 0 | 0 |
| HotpotQA · ea580137 | Fallback | Protected content | -0.3% | 0.0% | 3/3 → 3/3 | 0 | 0 |
| MultiNews · 1c29fa3b | Accepted | — | 11.9% | 27.7% | 3/3 → 3/3 | 0 | 0 |
| MultiNews · 1c30a0d9 | Accepted | — | 14.1% | 30.3% | 3/3 → 3/3 | 0 | 0 |
| MultiNews · 2190e9b0 | Accepted | — | 10.8% | 28.0% | 2/2 → 2/2 | 1 | 0 |
| MultiNews · 277ce57c | Fallback | Identifiers | -3.5% | 0.0% | 3/3 → 3/3 | 0 | 0 |
| MultiNews · 3df81844 | Accepted | — | 5.5% | 31.1% | No resolved pairs | 3 | 0 |
| MultiNews · 44916c0c | Accepted | — | 12.2% | 29.5% | 3/3 → 3/3 | 0 | 0 |
| MultiNews · 46101830 | Accepted | — | 3.6% | 28.0% | 2/2 → 2/2 | 1 | 0 |
| MultiNews · 4d378311 | Accepted | — | 8.5% | 29.3% | 3/3 → 3/3 | 0 | 0 |
| MultiNews · 4ee5c548 | Accepted | — | 9.1% | 30.2% | 3/3 → 3/3 | 0 | 0 |
| MultiNews · 4fa82967 | Accepted | — | 13.0% | 27.7% | 1/1 → 1/1 | 2 | 0 |
| MultiNews · 5021490f | Accepted | — | 14.8% | 31.1% | 3/3 → 2/3 | 0 | 1 |
| MultiNews · 6ad28d1c | Accepted | — | 7.8% | 29.4% | 3/3 → 3/3 | 0 | 0 |
| MultiNews · 6ed6a5b5 | Accepted | — | 12.2% | 30.9% | 2/2 → 2/2 | 1 | 0 |
| MultiNews · 8f512ff6 | Accepted | — | 13.9% | 28.3% | 3/3 → 3/3 | 0 | 0 |
| MultiNews · 9bf2345c | Fallback | Identifiers | -2.0% | 0.0% | 3/3 → 3/3 | 0 | 0 |
| MultiNews · cef9105b | Accepted | — | 12.8% | 29.9% | 3/3 → 1/3 | 0 | 2 |
| MultiNews · da9e45eb | Fallback | Links | -2.8% | 0.0% | 3/3 → 3/3 | 0 | 0 |
| MultiNews · dbe413a6 | Accepted | — | 16.1% | 28.9% | 3/3 → 3/3 | 0 | 0 |
| MultiNews · e4b27be5 | Accepted | — | 10.5% | 30.7% | 3/3 → 3/3 | 0 | 0 |
| MultiNews · f5cb169c | Accepted | — | 11.9% | 28.2% | 3/3 → 3/3 | 0 | 0 |
| MuSiQue · 037dbe8c | Fallback | Protected content | -0.4% | 0.0% | 3/3 → 3/3 | 0 | 0 |
| MuSiQue · 0675535e | Accepted | — | 19.4% | 29.3% | 3/3 → 3/3 | 0 | 0 |
| MuSiQue · 0da908a9 | Fallback | Protected content | -0.1% | 0.01% | 3/3 → 3/3 | 0 | 0 |
| MuSiQue · 16de423a | Fallback | Protected content | 0.5% | 0.02% | 3/3 → 3/3 | 0 | 0 |
| MuSiQue · 2b182c3c | Fallback | Protected content | 0.2% | 0.01% | 3/3 → 3/3 | 0 | 0 |
| MuSiQue · 2e723619 | Fallback | Protected content | 0.4% | 0.01% | 3/3 → 3/3 | 0 | 0 |
| MuSiQue · 3770e097 | Fallback | Identifiers | 0.1% | 0.01% | 3/3 → 3/3 | 0 | 0 |
| MuSiQue · 442e302c | Fallback | Protected content | 0.01% | 0.01% | 3/3 → 3/3 | 0 | 0 |
| MuSiQue · 72265eb8 | Fallback | Identifiers | -0.01% | 0.02% | 3/3 → 3/3 | 0 | 0 |
| MuSiQue · 7b7acfdf | Fallback | Protected content | -0.4% | 0.02% | 1/1 → 1/1 | 2 | 0 |
| MuSiQue · 936ced65 | Accepted | — | 19.5% | 30.5% | 3/3 → 3/3 | 0 | 0 |
| MuSiQue · a148671b | Accepted | — | 19.7% | 29.7% | 3/3 → 3/3 | 0 | 0 |
| MuSiQue · a3ba409a | Fallback | Protected content | -0.1% | 0.03% | 3/3 → 3/3 | 0 | 0 |
| MuSiQue · a4c716e0 | Fallback | Identifiers | 0.04% | 0.01% | 3/3 → 3/3 | 0 | 0 |
| MuSiQue · a6e0032c | Fallback | Identifiers | 0.6% | 0.02% | 3/3 → 3/3 | 0 | 0 |
| MuSiQue · d1ff1828 | Fallback | Identifiers | -0.5% | 0.01% | 3/3 → 2/3 | 0 | 1 |
| MuSiQue · db14f132 | Fallback | Identifiers | 0.3% | 0.01% | 3/3 → 3/3 | 0 | 0 |
| MuSiQue · e3d8c2e8 | Fallback | Identifiers | 0.02% | 0.02% | 3/3 → 3/3 | 0 | 0 |
| MuSiQue · ebd4b995 | Fallback | Identifiers | 0.5% | 0.01% | 3/3 → 3/3 | 0 | 0 |
| MuSiQue · f11250d4 | Fallback | Protected content | -0.1% | 0.01% | 1/1 → 1/1 | 2 | 0 |
| Qasper · 03ac5475 | Fallback | Protected content | 0.9% | 0.0% | 2/2 → 2/2 | 1 | 0 |
| Qasper · 072d3de1 | Fallback | Identifiers | -0.2% | 0.0% | No resolved pairs | 3 | 0 |
| Qasper · 11be2f14 | Accepted | — | 12.9% | 23.0% | 3/3 → 3/3 | 0 | 0 |
| Qasper · 17a2b96e | Accepted | — | 4.3% | 6.8% | 3/3 → 3/3 | 0 | 0 |
| Qasper · 1fc62062 | Accepted | — | 16.9% | 28.4% | 3/3 → 3/3 | 0 | 0 |
| Qasper · 39352d30 | Accepted | — | 15.3% | 25.8% | 1/1 → 1/1 | 2 | 0 |
| Qasper · 3b83a9c5 | Accepted | — | 15.1% | 27.7% | 3/3 → 3/3 | 0 | 0 |
| Qasper · 3c1ef39c | Accepted | — | 17.9% | 28.4% | 3/3 → 3/3 | 0 | 0 |
| Qasper · 3ca65b23 | Fallback | Identifiers | 0.0% | 0.0% | 3/3 → 3/3 | 0 | 0 |
| Qasper · 43ffd777 | Accepted | — | 16.4% | 27.7% | 3/3 → 3/3 | 0 | 0 |
| Qasper · 677dad70 | Fallback | Identifiers | -0.5% | 0.0% | 3/3 → 3/3 | 0 | 0 |
| Qasper · 80666777 | Fallback | Identifiers | 0.0% | 0.0% | 3/3 → 3/3 | 0 | 0 |
| Qasper · 85953320 | Fallback | Identifiers | 0.1% | 0.0% | 3/3 → 3/3 | 0 | 0 |
| Qasper · bee3efb9 | Skip | Exact-output policy | -0.3% | 0.0% | 3/3 → 3/3 | 0 | 0 |
| Qasper · c624b6d8 | Fallback | Identifiers | 0.1% | 0.0% | 3/3 → 3/3 | 0 | 0 |
| Qasper · c8ddf1d2 | Fallback | Identifiers | 0.8% | 0.0% | 3/3 → 3/3 | 0 | 0 |
| Qasper · d022dfe0 | Fallback | Code | 0.0% | 0.0% | No resolved pairs | 3 | 0 |
| Qasper · e419f2bf | Fallback | Identifiers | -0.3% | 0.0% | 3/3 → 3/3 | 0 | 0 |
| Qasper · ea4db579 | Fallback | Identifiers | -0.03% | 0.0% | 3/3 → 3/3 | 0 | 0 |
| Qasper · f545e80c | Accepted | — | 14.3% | 24.2% | No resolved pairs | 3 | 0 |
| QMSum · 06f6568b | Accepted | — | 12.8% | 24.0% | 3/3 → 3/3 | 0 | 0 |
| QMSum · 0ffea9de | Accepted | — | 12.3% | 23.7% | 3/3 → 3/3 | 0 | 0 |
| QMSum · 20306b2d | Fallback | Identifiers | -0.2% | 0.01% | 3/3 → 3/3 | 0 | 0 |
| QMSum · 2cc77384 | Fallback | Identifiers | 0.2% | 0.0% | 3/3 → 3/3 | 0 | 0 |
| QMSum · 2e714029 | Fallback | Protected content | 0.3% | 0.0% | 3/3 → 3/3 | 0 | 0 |
| QMSum · 3a1325d6 | Accepted | — | 13.2% | 25.9% | 3/3 → 3/3 | 0 | 0 |
| QMSum · 4629a555 | Accepted | — | 18.9% | 28.4% | 3/3 → 3/3 | 0 | 0 |
| QMSum · 56a70067 | Fallback | Constraints | -0.2% | 0.0% | 2/2 → 2/2 | 1 | 0 |
| QMSum · 5770240f | Fallback | Protected content | 1.0% | 0.0% | 3/3 → 3/3 | 0 | 0 |
| QMSum · 585f667a | Fallback | Protected content | -0.02% | 0.0% | 3/3 → 3/3 | 0 | 0 |
| QMSum · 5dbc2db3 | Accepted | — | 13.0% | 23.5% | 3/3 → 3/3 | 0 | 0 |
| QMSum · 628c3567 | Accepted | — | 12.3% | 23.7% | 3/3 → 3/3 | 0 | 0 |
| QMSum · 78f6c420 | Fallback | Constraints | -0.1% | 0.1% | 3/3 → 3/3 | 0 | 0 |
| QMSum · 8a2a85ee | Accepted | — | 19.2% | 29.1% | 2/2 → 2/2 | 1 | 0 |
| QMSum · 901c8c19 | Fallback | Constraints | -0.3% | 0.0% | 3/3 → 3/3 | 0 | 0 |
| QMSum · 9510dd17 | Fallback | Identifiers | 0.1% | 0.00% | 3/3 → 3/3 | 0 | 0 |
| QMSum · a81acf45 | Fallback | Identifiers | -0.2% | 0.0% | 3/3 → 3/3 | 0 | 0 |
| QMSum · b83d7e10 | Accepted | — | 13.3% | 23.6% | No resolved pairs | 3 | 0 |
| QMSum · caab9dad | Fallback | Identifiers | -0.2% | 0.0% | 3/3 → 3/3 | 0 | 0 |
| QMSum · f28e420c | Accepted | — | 17.0% | 26.9% | 3/3 → 3/3 | 0 | 0 |
How to interpret the results
Tasks, conditions and answer repeats
The main cohort contains 100 public LongBench questions across 97 distinct source contexts: 20 each from HotpotQA, MultiNews, MuSiQue, Qasper and QMSum. The separate structured JSON cohort retains 10 tasks across nine inputs from the earlier study. It includes original, minified, structural-only and full-service representations, with three repeats: 120 existing answer calls. The earlier prose subset is not added again. These cohort results are reported separately rather than pooled into a 110-task savings average. Original, deterministic-only and full model-force compression at aggressiveness 0.50 each received three Terra-medium answers: 900 successful answer calls in the main study.
Inputs were frozen before answer generation, using pinned public sources and seed 42. Repeats share source documents and are not independent tasks. The primary reported comparison is original versus full compression; deterministic-only results remain in the explorer.
Source-aware correctness and unresolved judgments
Terra-low screens answers against the complete original source, question and references. Sol-low reviews flagged long-text batches; both judges check all batches in the targeted follow-ups and the JSON revalidation. Reviewers see blinded answer IDs, and exact duplicate answers share one judgment.
Correct answers score 1, partly correct answers with a material issue score 0.5, and incorrect answers score 0. Ambiguous or unsupported findings and reviewer disagreements stay unresolved. Negative findings require a concrete issue and traceable source evidence. Two-model agreement supports a judgment; it is not ground truth or a guarantee about future answers.
A regression is a lower resolved correctness score than the matched original-input answer. Source review distinguishes changed evidence from answer variation on identical inputs. QA F1 and ROUGE-L are preserved unchanged; lower reference overlap alone is not counted as an answer error. The JSON cohort uses adapted TOON questions with shared JSON/TOON instructions and the frozen upstream type-aware scorer. Intentionally corrupted-format questions are excluded from this intact-JSON compression evaluation.
Costs, token counts and compression billing
Costs use frozen API prices dated November 8, 2026, application-prompt token counts and observed output usage. They include compression charges on each request, assume the free allowance is exhausted, and use uncached requests without cache writes. Reusing compressed artifacts or cache hits changes the scenario; the explorer provides those controls.
Context tokens exclude instructions and questions. The full prompt cost includes them. Compression fees use the service-reported saved-token meter, which differs from the proxy tokenizer used for context comparisons. Output usage includes reported reasoning once. Output-length variation contributes to before/after cost differences.
Answer generation and judging used a Codex subscription. These are modeled application costs at published API rates, not charges on that subscription. Offline benchmark and review overhead are excluded from production running-cost savings and retained separately in the explorer. Prices at other models are cost-only projections, not measured answer-quality results for those models.
Reproducibility, limits and original evidence archives
No new compression or answer-generation calls were made to include the JSON cohort. Its 120 existing answers received a fresh source-aware review: 22 judge calls, including calibration. The expanded long-text measurements remain unchanged. The 100-task study and later targeted studies remain separate; none of their recorded answers or quality scores was replaced. Exact source hashes and summary calculations are in the downloadable receipt.
The original long-text and JSON-source evidence archives are included under evidence/. Each retains its inputs, raw calls, reviews, code snapshots and verified checksums. The original study had one interrupted answer call and one interrupted review call after a CSV export error; those two calls were explicitly recovered, and original logs remain archived.
100-task long-text evidenceOriginal JSON-source study0.25 evidence0.15 evidence