UsageTap Compress · JSON and long-text benchmarks · evidence through November 9, 2026
Fewer tokens and lower cost, with answer quality measured
The goal is lower running cost without additional incorrect or materially incomplete answers. Structured JSON and prose have different opportunities and risks. We report them separately: 10 existing JSON tasks alongside the expanded 100-task long-text study.
TL;DR: strong JSON gains; long text needs selective compression
- JSON: 31.3% lower modeled cost and 43.6% fewer context tokens, with 30/30 answers correct. All 10 JSON inputs reconstruct exactly after compression. Compared with already-minified JSON, the full service saves another 16.4% in context tokens and 3.7% in net cost.
- Long text: 4.9% lower modeled cost overall; 15.7% on accepted inputs. Context tokens fell 8.1% overall and 27.4% on accepted inputs. The tested 0.50 setting had three accepted-input answer regressions across two questions, with 20 unresolved comparisons. It does not establish savings without quality loss.
- Gentler long-text follow-ups produced 6/6 correct answers at each of 0.30, 0.25 and 0.15. At 0.30, the accepted VX input saved 7.2% in net cost; Mali was protected by returning the source. These are targeted results on two failing cases.
- Knowing when to avoid the lossy model stage is part of the value. All 10 JSON examples retained structural savings while skipping that stage. In the separate long-text cohort, 56 drafts triggered preservation guards and one input was explicitly skipped.
Structured JSON · 10 tasks, nine source inputs
Long text · expanded 100-task cohort
Headline cost figures price the application prompt and observed output at the frozen Terra API rates. They are estimates, not subscription invoice savings. Correct-answer counts below use source-aware reviews against matched original-input controls.
Cost savings after compression fees
For every $100 of original model cost, the JSON cohort would cost about $68.7 after fees. The separate long-text cohort would cost about $95.1 overall, or $84.3 on accepted inputs.
__COST_CHART__ __COST_TABLE__The JSON and long-text results are separate cohorts; there is no pooled savings average. Within long text, the entire workload includes accepted compression, guard fallbacks and explicit skips. Conditional savings retain every accepted input, including answer regressions. Dollar totals cover different workloads.
Long-text context tokens fell 8.1% overall and 27.4% on accepted inputs. Cost savings are smaller because requests still include instructions, questions and output tokens, and the compression fee is included. Observed output-length variation also affects net cost.
Structured JSON: smaller representations, the same correct answers
The full service saved 43.6% of context tokens and 31.3% of modeled running cost on the JSON cohort, with 30/30 correct answers. This covers 10 questions across nine distinct inputs, with three answer repeats per condition. Both source-aware reviewers agreed with the official type-aware scorer.
__JSON_CONDITIONS__The minified baseline matters. Ordinary JSON minification already saves 32.5% of context tokens and 28.7% in modeled cost. The full service reduces context by another 16.4% and net cost by another 3.7% relative to that already-minified representation. Tabular examples show larger gains than irregular nested data.
Structural-only and full-service requests returned identical compressed text for all 10 inputs. The cost difference between those conditions comes from their separately observed output usage. All cost comparisons include the applicable compression fee; no answer errors or inputs were excluded from these averages.
All 10 JSON examples
__JSON_CASES__Analytics and tabular examples removed about 59–61% of context tokens versus the original formatted JSON by avoiding repeated field names and unnecessary syntax. Keys, values, types, array order and nesting remained intact when decoded: all 30 transformed representations across the three conditions round-trip to the original objects.
A good decision: use structural savings without a lossy model rewrite
For every JSON input, the service reported no candidate prose and skipped the LLMLingua stage. It still performed lossless structural compression. These model-stage skips retain their savings in the JSON results; they are different from zero-saving whole-input preservation decisions in the long-text cohort.
See a real analytics example before and after
The complete saved input and output are shown below. Repeated JSON keys become a compact table header; the data values are preserved. The strike-through explorer also remains available for every JSON example.
__JSON_EXAMPLE__Explore all JSON inputs, strike-throughs and answersDownload JSON cost comparisonsInspect exact structural checks
This is the retained JSON subset of the earlier benchmark, not 10 newly generated questions. Its 120 saved answers across original, minified, structural-only and full-service conditions were rechecked with the newer source-aware method; both reviewers agreed all were correct. The official full-service result is 30/30. Exact structural preservation and observed answer correctness are evidence for these inputs, not a guarantee for every JSON shape or future model.
Long text: did the answers stay correct?
At 0.50, accepted compression produced 106 correct answers across 109 resolved comparisons; the original-input controls were correct in all 109. The other 20 comparisons remain unresolved. The denominator is repeated answer pairs, not 109 independent questions.
__QUALITY_TABLE__The key failure was three accepted-input answer regressions across two questions. These were material factual issues, not merely lower word-overlap scores. Deeper review linked them to a lost causal connection in Mali and a changed quantity in VX.
The entire workload has one additional declining answer pair on byte-identical original and returned text. That is generation variation on a preserved input; it is reported separately from the three accepted-compression regressions.
Only pairs with resolved original and candidate judgments appear in the “correct” fractions. An unresolved answer is neither a pass nor an assumed error. Source-aware correctness is the primary quality outcome; official QA F1 and ROUGE-L remain available in the explorer as supporting scores.
Gentler long-text settings preserved correct answers in these repeats
The two confirmed news regressions were retested with fresh original-input controls and three candidate answers per case. At 0.30, 0.25 and 0.15, both source-aware reviewers judged 6/6 candidate answers correct, matching the original controls.
Net modeled cost savings on accepted inputs
__FOLLOWUP_BARS__Each of these three accepted-input bars is the VX case. Mali used original-text fallback and is excluded from this conditional savings average. The table also shows both cases together.
__FOLLOWUP_TABLE__At 0.50, four of six targeted answers were only partly correct; at 0.45, two were. At the three gentler settings, no answer error was observed in these repeats. For Mali, the improvement came through preservation rather than a successful lossy compression.
0.30 is a candidate for broader validation. It saved the most among the gentler settings with correct answers here. Moving to 0.15 made the source wording clearer but did not improve the measured answer results in this small sample.
These two inputs were chosen because they previously failed. Three repeats per case do not establish a safe default or estimate full-workload savings at the gentler settings. Each level uses its own original controls and observed output counts, so cost differences also include answer-length variation.
Results across the five long-text benchmarks
Each benchmark contributes 20 questions and three repeats per original and candidate input. This table covers the complete workload at 0.50; the accepted column shows how often the compressor actually used a model-compressed output.
__DATASET_TABLE__Correct-answer fractions use resolved matched pairs within each benchmark; unresolved counts are shown separately. Savings are weighted by cost or token totals, not an average of per-case percentages. Dataset conclusions are limited by the small number of source documents and unresolved judgments.
Knowing what not to compress is part of the value
In the long-text cohort, the compressor accepted model outputs for 43 inputs, used preservation fallbacks for 56 and intentionally skipped one. That selectivity trades maximum compression for keeping usable evidence available to the answering model.
__DECISION_TABLE__Why we preserved these inputs
Each preserved input has a recorded reason. A guard fallback means compression was attempted and its draft rejected; a policy skip means the model was deliberately not run.
__PRESERVATION_REASONS__Counts are inputs, not answer repeats, using the first reported reason. The retained diagnostics identify the failing category but do not retain rejected drafts or exact offending values. These are 57 recorded preservation decisions, not 57 proven answer errors prevented.
A demonstrated good decision: preserve the Mali salary-payment cause
At 0.30, 0.25 and 0.15, the guard returned the original source, retaining the explicit connection between the frozen bank account and salary-payment problems. All three answers were judged correct at each setting. The earlier accepted 0.50 compression had produced a factual regression on this case.
The explicit skip: a conservative policy match, not a proven necessity
The one policy skip was a Qasper article containing the phrase “we do not change the text preprocessing methods for MT.” The service recorded an exact-output-context skip and returned the original text; all three answers were judged correct.
Here that phrase describes a research method rather than asking the answering model to reproduce text exactly. We therefore show it as a conservative policy match, not evidence that compression would have caused an error.
Inspect the skipped input and its answers · Download all 57 preservation reasons
Fallbacks and intentional skips do not dilute the accepted-compression average. They remain fully visible in the overall result. Accepted attempts with no savings or with answer errors stay in the accepted group. The groups were decided before answer scoring.
Fallback context reduction is approximately zero, rather than exactly zero for every case: deterministic cleanup removed 68 proxy tokens across the 56 guarded inputs. Any small cost difference on preserved inputs can also reflect generation length, not a successful model compression. Preservation guards do not by themselves guarantee correct answers.
Source damage explains failures and signals fragility
Correct results are the primary outcome. Source damage is a secondary diagnostic: it explains why an answer failed or why a seemingly successful input may be fragile. Damaged wording with correct answers is not counted as an observed answer regression.
The two factual regressions: cause and quantity
Mali, 0.50: removal of the explicit bank-account cause allowed answers to blame fuel shortages for salary-payment problems. Gentler settings returned the original text through the integrity guard.
VX, 0.50: changing “one of the two women” to a statement about two women led to incorrect quantities in the answers. At 0.30 and 0.25, the compressed phrase “one two women” still read poorly, but all three answers at each level correctly described one woman becoming ill. At 0.15, “one of two women” was clearer, without an additional measured answer-quality gain.
All deeper source audits from the full study
__AUDITS__Stronger answer models did not reliably repair the damaged evidence
The original targeted follow-up also tested Terra high and Sol high on the frozen 0.50 inputs. These are diagnostic tests on the same two cases, not model-quality results across the 100-task cohort.
__STRONGER_TABLE__Counts are out of three answers per row. Complete answers, original-input controls, reasoning settings and source citations are retained in the explorer and original archives.
Every input remains inspectable
The full 100-input long-text ledger below includes zero-saving decisions, answer regressions and unresolved judgments. Open any input to inspect its original text, removals, actual answers, official scores and source-aware review evidence.
Long-text evidence explorerJSON evidence explorerDownload the cost ledgerDownload exact summary calculationsDownload complete report dataDownload the website report bundle
All 100 inputs: cost and answer outcomes
How to interpret the results
Tasks, conditions and answer repeats
The main cohort contains 100 public LongBench questions across 97 distinct source contexts: 20 each from HotpotQA, MultiNews, MuSiQue, Qasper and QMSum. The separate structured JSON cohort retains 10 tasks across nine inputs from the earlier study. It includes original, minified, structural-only and full-service representations, with three repeats: 120 existing answer calls. The earlier prose subset is not added again. These cohort results are reported separately rather than pooled into a 110-task savings average. Original, deterministic-only and full model-force compression at aggressiveness 0.50 each received three Terra-medium answers: 900 successful answer calls in the main study.
Inputs were frozen before answer generation, using pinned public sources and seed 42. Repeats share source documents and are not independent tasks. The primary reported comparison is original versus full compression; deterministic-only results remain in the explorer.
Source-aware correctness and unresolved judgments
Terra-low screens answers against the complete original source, question and references. Sol-low reviews flagged long-text batches; both judges check all batches in the targeted follow-ups and the JSON revalidation. Reviewers see blinded answer IDs, and exact duplicate answers share one judgment.
Correct answers score 1, partly correct answers with a material issue score 0.5, and incorrect answers score 0. Ambiguous or unsupported findings and reviewer disagreements stay unresolved. Negative findings require a concrete issue and traceable source evidence. Two-model agreement supports a judgment; it is not ground truth or a guarantee about future answers.
A regression is a lower resolved correctness score than the matched original-input answer. Source review distinguishes changed evidence from answer variation on identical inputs. QA F1 and ROUGE-L are preserved unchanged; lower reference overlap alone is not counted as an answer error. The JSON cohort uses adapted TOON questions with shared JSON/TOON instructions and the frozen upstream type-aware scorer. Intentionally corrupted-format questions are excluded from this intact-JSON compression evaluation.
Costs, token counts and compression billing
Costs use frozen API prices dated November 8, 2026, application-prompt token counts and observed output usage. They include compression charges on each request, assume the free allowance is exhausted, and use uncached requests without cache writes. Reusing compressed artifacts or cache hits changes the scenario; the explorer provides those controls.
Context tokens exclude instructions and questions. The full prompt cost includes them. Compression fees use the service-reported saved-token meter, which differs from the proxy tokenizer used for context comparisons. Output usage includes reported reasoning once. Output-length variation contributes to before/after cost differences.
Answer generation and judging used a Codex subscription. These are modeled application costs at published API rates, not charges on that subscription. Offline benchmark and review overhead are excluded from production running-cost savings and retained separately in the explorer. Prices at other models are cost-only projections, not measured answer-quality results for those models.
Reproducibility, limits and original evidence archives
No new compression or answer-generation calls were made to include the JSON cohort. Its 120 existing answers received a fresh source-aware review: 22 judge calls, including calibration. The expanded long-text measurements remain unchanged. The 100-task study and later targeted studies remain separate; none of their recorded answers or quality scores was replaced. Exact source hashes and summary calculations are in the downloadable receipt.
The original long-text and JSON-source evidence archives are included under evidence/. Each retains its inputs, raw calls, reviews, code snapshots and verified checksums. The original study had one interrupted answer call and one interrupted review call after a CSV export error; those two calls were explicitly recovered, and original logs remain archived.
100-task long-text evidenceOriginal JSON-source study0.25 evidence0.15 evidence