Model and provider routing01
Choose where the request runs.
Routers such as OpenRouter select among models and providers using signals such as task, price, latency, throughput, and availability.
Source: OpenRouter routing ↗UsageTap context intelligence · Engineering field notes
Public source + evidence ledgerWe removed 30–40% of the input. Then we asked the question that matters: what survived?
That early result came from profiling two context-heavy workloads—not a universal benchmark. It led us to build a safer optimization loop: measure every call, protect what must survive, prove each change, and send the right context for the right prompt to the right model.
Observed request shape
Input dominates the bill.
Input tokens20–30 parts
Output tokens1 part
At 20–30M input tokens per day, input efficiency is not a cleanup task. It is part of application architecture.
20–30×input-to-output ratio observed across two tenants
10.5%accepted savings in the latest fixed-corpus model run
0accepted hard failures across cited completed cohorts
A different optimization layer
Routing chooses where a prompt runs. Compression shortens text. We optimize what the call contains.
These products solve different jobs and can work together. UsageTap sits at the application layer, where prompts, context, models, responses, costs, latency, and outcomes can be evaluated as one system.
Text compression02
Shorten raw prose before inference.
Compressors such as The Token Company use specialized models to remove low-signal tokens from documents, websites, transcripts, and other text.
Source: The Token Company compression ↗UsageTap context intelligence03
Prove what this call should contain.
Turn real prompts, responses, latency, cost, and application outcomes into evidence for what to compact, protect, cache, retrieve, compress, or omit—and which model should receive it.
More representative traffic creates better workload evidence—not permission to remove context blindly.
The UsageTap learning loop
Every measured call can improve the next optimization decision.
Usage tracking is not only a billing record. It is the evidence layer for workload-specific context optimization: what changed, what was saved, and whether the answer still did its job.
Right context. Right prompt. Right model. Less time to first token, less context noise, and less input spend.
- 01
Measure every call
Track input, output, latency, cost, model, and application outcome—not just an aggregate token bill.
- 02
Learn what matters
Split context into meaningful pieces and build evidence about which instructions, records, structures, and roles affect the response.
- 03
Prove the change
Remove, rewrite, or compact one piece at a time and compare downstream quality before turning a finding into a rule.
- 04
Apply and observe
Send the right context for the prompt and model, then use the resulting traffic to improve the workload profile again.
Today, UsageTap measures calls and applies deterministic, protected, and selective compression. This is an evidence loop, not automatic retraining: context-removal rules remain research work and require workload-specific quality evidence before production use.
What changed our approach
The model was one stage, not the system.
A benchmark on plain text did not predict performance on production prompts. Profiling the actual mix of prose and structured context did.
Read the LLMLingua-2 paper ↗The 5–10% and 30–40% figures on this page are early observations from two context-heavy workloads. They are not production guarantees; results vary by prompt mix, tokenizer, model, and quality threshold.
01 · Our starting point
A general text compressor helped—but only at the margin.
We began with Microsoft’s 2024 LLMLingua-2 model. It was effective at removing low-value words from plain text, but protecting important context kept us at a conservative 0.15–0.20 aggressiveness setting.
5–10% compressionUseful on prose; not enough for context-heavy application prompts.
02 · The workload mismatch
JSON and HTML were structure problems, not prose problems.
Real application prompts contained large JSON and HTML blocks. A word-level model produced marginal gains and could damage the structures the downstream model depended on.
Marginal gainThe content was valuable; its representation was wasteful.
03 · The production lesson
A positive compression result is still a candidate, not a win.
The runtime now applies structured transforms only when parsing, equivalence, round-trip, and tokenizer-savings gates pass. LLMLingua-2 sees eligible prose—not the whole prompt—and its output is rejected if protected content or integrity checks fail.
Rejected = 0Unsafe model output contributes no savings and is rolled back.
The production pipeline
Protect first. Transform only when the gates pass. Roll back anything unsafe.
Only about 20% of prompts in our early two-tenant profile warranted LLMLingua-2 compression. Fast skip guards avoided unnecessary model calls and improved compression throughput by as much as 20× in that profile. The current runtime also gates on exact-output instructions, eligible prose, structured and protected density, expected savings, and latency budget.
- 1
Preserve roles
On role-aware requests, keep stable system and developer instructions byte-stable; operate on eligible user text.
- 2
Segment and shield
Find prose, JSON, HTML, code, UI contracts, critical clauses, and explicit no-compress spans.
- 3
Compact safely
Transform eligible JSON or full-page HTML only after the configured safety, equivalence, and savings gates pass.
- 4
Gate the model
Skip exact-output, highly structured, low-opportunity, or latency-constrained requests.
- 5
Compress prose
Run LLMLingua-2 only on eligible prose, with protected regions represented by required placeholders.
- 6
Restore and validate
Restore placeholders and check protected literals, code, JSON, constraints, required terms, and relationships.
- 7
Rollback or accept
Reject unsafe output, return the deterministic or unchanged text, and count zero model savings.
The key performance lesson: a good compression decision is often a fast decision to leave the prompt alone.
The key safety rule: rejected model output never counts as savings.
The promotion contract
Savings are easy to show. Safe, causal savings are harder.
We do not promote a transform because one benchmark arm produced a smaller prompt. It must save real tokenizer tokens, preserve hard constraints, pass downstream checks, repeat cleanly, and hold up on data that was not used to discover it.
Causal savings
Compare the same prompt across unchanged, deterministic-only, model-only, and combined conditions.
Tokenizer-positive
Promote a transform only when the target tokenizer records a real absolute and relative saving.
Hard integrity
Protected literals, code, JSON, constraints, and required terms must survive every accepted output.
Downstream fidelity
Test negation, obligations, permissions, scope, formats, and entity-value relationships—not similarity alone.
Repeatability
Require stable output hashes, skip reasons, and rollback reasons across repeated runs.
Held-out evidence
Keep discovery data separate from evaluation data and report zero-application cohorts, too.
Current public posture: safety shielding and rollback are permanent defaults; experimental savings transforms remain unpromoted until held-out evidence clears the contract.
See every decision and cohort ↗Before runtime
Some of the best compression happens while you write.
Many system instructions were verbose, contradictory, or arranged in ways that reduced cache reuse. Rewriting the source prompt produced a 30% reduction in system-instruction size in our work so far.
Input-token caching also improved, but we have not measured that gain rigorously enough to publish a number.
Say the same thing directly.
An upfront rewriter removes repetition, weak phrasing, and unnecessary instruction scaffolding before a prompt reaches production.
Open the prompt rewriter →Make stable context cacheable.
A linter flags prompt quality issues and ordering choices that work against input-token caching and repeatability.
Open the prompt linter →Current research problem
In progressCan we learn which context the response actually needs?
System instructions and application context often contain pieces the model does not use—but removing the wrong piece can change the answer. Our profiler now splits a completed call into meaningful units and classifies each unit’s purpose and likely need.
The next entropy experiments remove those pieces one at a time, rerun the prompt, and measure response sensitivity. The open question is whether those results can become a real-time rules engine that drops irrelevant context without reducing response quality.
Follow the live Compress research →Profiler → assessment → rules
- 1
Split
Break the system instructions, context, prompt, and response into meaningful pieces.
- 2
Classify
Label each piece by purpose, dependency, and likely need.
- 3
Remove
Run controlled experiments with one piece omitted or transformed.
- 4
Compare
Measure how the response changes, not only whether tokens were saved.
- 5
Learn
Turn repeatable findings into real-time rules for future prompts.
The working thesis
The future is not “compress every prompt.” It is “send the right context for the right prompt to the right model.”
A measured starting point
Measure every call. Learn what matters. Send less.
Bring representative prompts. Measure structure, savings, latency, risky removals, and downstream behavior before changing production traffic.