Back to Code Integration

Benchmark Models on Production Tasks

UsageTap call-list suggestions identify models worth measuring. They are not replacement recommendations. A model should be recommended only after it has been run against representative production tasks and has met the workload's quality requirements.

The workflow is:

Call-list opportunity
  -> Production Sample collection
  -> Model and reasoning benchmark
  -> Cost per passing task
  -> Reviewed production change

What the benchmark measures

A Production Sample retains the original model input, response, usage, and linked UsageTap call cost under the organization's Capture Policy. Similar samples are grouped into a task cohort. Model Benchmark replays up to ten tasks against the selected candidate models and reasoning depths.

The primary economic result is cost per passing task, not price per token:

observed candidate spend / candidate runs that passed the quality bar

The retained production response is evidence and the per-task comparison reference. It is not universal ground truth. Review the generated Quality Checks before starting a paid benchmark, and review failed or near-threshold tasks before changing production routing.

Five similar samples unlock the call-list Run model benchmark action. Ten samples provide the largest cohort currently supported by the lightweight Model Benchmark workflow.

UsageTap Gateway

The Gateway already performs the two-stage sampling decision for buffered, streamed, and native-batch calls. No application sampling code is required.

  1. Open Data & Retention.
  2. Set the Capture Policy rate, minimum input size, feature/customer filters, and retention period.
  3. Continue sending normal Gateway calls.
  4. Confirm that Production Samples appear for the target feature.
  5. Return to the call-list opportunity and run the benchmark when the cohort is ready.

Only selected successful calls retain content. Sampling decision or capture failures fail closed and do not fail the model request. If the application also performs explicit SDK sampling, set llmGateway.sampling to false to prevent duplicate samples.

The internal Gateway reference contains additional implementation detail in docs/LLM_GATEWAY.md under Automatic sampling.

UsageTap SDK

For supported JavaScript/TypeScript providers, wrap the existing provider client. The wrapper loads the saved Capture Policy, makes a deterministic decision, and retains only selected completed calls.

import OpenAI from "openai";
import { withSampling } from "@usagetap/sdk/openai";

const provider = withSampling(new OpenAI());

await provider.responses.create(
  { model: "your-current-model", input: prompt },
  {
    usageTap: {
      customerId: "cust_123",
      feature: "assistant.answer",
    },
  },
);

withSampling() is also available from @usagetap/sdk/anthropic and @usagetap/sdk/openrouter. When using hosted Compression, pass sampling: true to withCompression().

For a custom provider, use shouldSampleAsync() before calling captureSample(). See SDK Sampling for the complete wrapper and custom-provider contracts.

Direct API or manual integration

Direct integrations use a decide-then-capture contract. The decision request contains metadata only and can run in parallel with the provider request.

1. Ask for a decision

curl -X POST https://api.usagetap.com/sampling/decide \
  -H "Authorization: Bearer YOUR_USAGE_KEY" \
  -H "Accept: application/vnd.usagetap.v1+json" \
  -H "Content-Type: application/json" \
  -d '{
    "samplingKey": "stable-provider-request-id",
    "customerId": "cust_123",
    "feature": "assistant.answer",
    "inputTokens": 4200
  }'

Use a stable samplingKey so retries receive the same decision for the same policy version. Do not send prompt or response content to this endpoint.

2. Capture only a positive decision

After the provider call completes, call /samples only when the decision response returned sample: true:

curl -X POST https://api.usagetap.com/samples \
  -H "Authorization: Bearer YOUR_USAGE_KEY" \
  -H "Accept: application/vnd.usagetap.v1+json" \
  -H "Content-Type: application/json" \
  -d '{
    "sampleId": "stable-provider-request-id",
    "decisionId": "DECISION_ID_FROM_STEP_1",
    "policyVersion": "POLICY_VERSION_FROM_STEP_1",
    "customerId": "cust_123",
    "feature": "assistant.answer",
    "provider": "custom",
    "model": "your-current-model",
    "input": {"messages": []},
    "output": {"content": "provider response"},
    "usage": {"inputTokens": 4200, "outputTokens": 380}
  }'

The usage key requires compression:invoke. The request body is limited to 300 KiB. Raw payloads expire according to the Capture Policy; benchmark result metadata and audit evidence have separate retention behavior.

See Sampling API for response schemas, local policy evaluation, error contracts, and payload details.

Starting from a call-list opportunity

The call list chooses the next action from current evidence:

  • Run model benchmark: at least five matching Production Samples exist.
  • Review capture policy: the calls use the Gateway, so capture is automatic once the saved policy selects them.
  • Add production sampling: the calls use a direct provider path, so add the SDK wrapper or direct API contract above.

The benchmark page is prefilled with the Production Sample group and up to three same-provider candidate models. Catalog recency and estimated token-mix cost decide what is worth testing. Only measured quality and cost per passing task can turn a candidate into a recommendation.