ReqLLM Tool Usage + Pricing Design (Provider Defaults)

Goals

Track billable tool usage (search, code interpreter, file search, etc.) in Response.usage.
Calculate all-in cost: tokens + tools + images (+ storage/session if applicable).
Add native LLMDB metadata for tool pricing, with provider-level defaults and model overrides.
Keep backward compatibility for existing cost token pricing.

Summary Of Changes

LLMDB: add pricing_defaults to provider schema; add pricing to model schema.
ReqLLM usage: add tool_usage, image_usage, and cost breakdown; make total_cost all-in.
ReqLLM cost: new component-based calculator; token-only fallback preserved.
Providers: extract tool usage counts and image usage where needed.

1) LLMDB Schema (Provider Defaults + Model Overrides)

Provider Schema

Add an optional pricing_defaults field to LLMDB.Provider:

pricing_defaults :: %{
  currency: String.t(),
  components: [pricing_component()]
}

Model Schema

Add an optional pricing field to LLMDB.Model:

pricing :: %{
  currency: String.t() | nil,
  components: [pricing_component()],
  merge: :replace | :merge_by_id
}

merge default: :merge_by_id.

Pricing Component Schema

Each component defines a billable unit. Recommended fields:

pricing_component :: %{
  id: String.t(),               # Unique within provider/model, e.g. "token.input"
  kind: :token | :tool | :image | :storage | :request | :other,
  unit: :token | :call | :query | :session | :gb_day | :image | :source | :other,
  per: pos_integer(),           # e.g. 1_000_000 for tokens, 1_000 for calls
  rate: number(),               # currency per `per` units
  meter: String.t() | nil,      # Usage key mapping, e.g. "input_tokens"
  tool: atom() | nil,           # e.g. :web_search, :file_search, :code_interpreter
  size_class: String.t() | nil, # e.g. "1k-2k", "4k" for Gemini image pricing
  notes: String.t() | nil
}

Merge Semantics (Provider Defaults → Model Overrides)

Start with provider pricing_defaults.components.
If model pricing.merge == :replace: use model components only.
If :merge_by_id: override provider components by matching id, and append new ones.
currency: model value wins; otherwise provider value.

Why Provider Defaults?

OpenAI/Anthropic/xAI tool pricing is consistent across most models.
Gemini search billing differs by model family → override via model pricing.

2) ReqLLM Usage Structure (Normalized)

New Usage Fields

Add structured usage fields to ReqLLM.Response.usage:

%{
  input_tokens: integer(),
  output_tokens: integer(),
  total_tokens: integer(),
  reasoning_tokens: integer(),
  cached_tokens: integer(),

  tool_usage: %{
    web_search: %{count: integer(), unit: :call | :query | :source},
    google_search: %{count: integer(), unit: :query},
    file_search: %{count: integer(), unit: :call},
    code_interpreter: %{count: integer(), unit: :session}
  },

  image_usage: %{
    generated: %{count: integer(), size_class: String.t() | nil}
  },

  cost: %{
    tokens: number(),
    tools: number(),
    images: number(),
    storage: number() | nil,
    total: number(),
    line_items: [%{id: String.t(), cost: number(), count: number()}]
  },

  total_cost: number() # all-in (tokens + tools + images + storage)
}

Notes:

total_cost becomes all-in per user decision.
cost.tokens + cost.tools + cost.images (+ cost.storage) == total_cost.

3) Cost Calculation

New Component Calculator

Create a component-based calculator (e.g., ReqLLM.Billing.calculate/2):

Inputs:

usage (normalized usage map)
pricing (merged provider defaults + model overrides)

Algorithm:

Normalize usage counts into a flat map keyed by meter + tool/image usage.
For each component:
- Resolve usage count based on kind/meter/tool/size_class.
- component_cost = (count / per) * rate.
Sum costs by kind group (tokens/tools/images/storage).
Return cost map + total_cost.

Backward Compatibility

If pricing is nil, fall back to existing ReqLLM.Cost.calculate/2 (token-only).
If model has cost but no pricing, auto-derive token components from cost:
- token.input, token.output, token.cache_read, token.cache_write, token.reasoning.

4) Provider Usage Extraction

Common Strategy

Provider extract_usage/2 returns a usage map including tool_usage and image_usage where available.
ReqLLM.Step.Usage merges into response usage and feeds the billing calculator.

Provider-Specific Notes

Google Gemini

Search grounding: if any groundingMetadata.webSearchQueries present, then
- Gemini 2.5 and older: tool_usage.google_search.count = 1 (per prompt)
- Gemini 3: tool_usage.google_search.count = length(queries) (per query)
Gemini image models: count generated images, derive size_class (1k–2k or 4k) from output dimensions or request params.

Anthropic

Use usage.server_tool_use.web_search_requests as tool_usage.web_search.count.

OpenAI

Use Responses API tool items to count built-in tools (web search, file search, code interpreter).
For code interpreter billing, count sessions (one response may map to 1 session).

xAI

Use response.server_side_tool_usage for per-tool call counts.
usage.num_sources_used maps to tool_usage.web_search with unit :source.

5) ReqLLM Pipeline Changes

Usage Normalization

Update ReqLLM.Step.Usage and Provider.Defaults.ResponseBuilder.normalize_usage_fields/1:

Preserve existing token normalization.
Pass through tool_usage and image_usage when present.
Add computed cost and all-in total_cost.

Metadata In Streaming

Ensure metadata collected by stream handlers includes tool usage where available, so StreamResponse.usage/1 yields the same structure.

6) LLMDB Build/Load Flow

Build-Time Merge (Preferred)

During LLMDB.Engine.finalize/1, apply provider pricing_defaults to each model.
Merge by id unless model pricing.merge == :replace.

Runtime Custom Models

For config/test.exs custom providers/models, apply the same merge at load time.

7) Tests

Unit

Component calculator: token-only, tool-only, mixed components, merge behavior.
Usage normalization retains tool_usage and image_usage.

Provider

Stub usage responses for Google/Anthropic/xAI/OpenAI and validate tool_usage counts.

Integration

Ensure usage.total_cost == sum of breakdown categories.

8) Migration / Backward Compatibility

usage.total_cost becomes all-in; for clients expecting token-only, add usage.cost.tokens or usage.cost.line_items.
Existing cost model field remains valid; it becomes token pricing fallback.

9) Example LLMDB Entries

Provider (Anthropic)

"pricing_defaults": {
  "currency": "USD",
  "components": [
    {"id": "token.input", "kind": "token", "unit": "token", "per": 1000000, "rate": 3.0, "meter": "input_tokens"},
    {"id": "token.output", "kind": "token", "unit": "token", "per": 1000000, "rate": 15.0, "meter": "output_tokens"},
    {"id": "tool.web_search", "kind": "tool", "unit": "call", "per": 1000, "rate": 10.0, "tool": "web_search"}
  ]
}

Model Override (Gemini 3)

"pricing": {
  "merge": "merge_by_id",
  "components": [
    {"id": "tool.google_search", "kind": "tool", "unit": "query", "per": 1000, "rate": 14.0, "tool": "google_search"}
  ]
}

10) Implementation Steps (Suggested Order)

Add pricing_defaults and pricing to LLMDB schemas + validation.
Implement provider-default merge in LLMDB.Engine.finalize/1.
Add component-based calculator in ReqLLM and wire into Step.Usage.
Extend usage normalization to include tool_usage + image_usage + cost.
Update provider extractors (Google/Anthropic/xAI/OpenAI) to emit tool_usage.
Add tests for calculator + provider extraction.

vkryukov/req_llm_tool_pricing_plan.md

Select an option

No results found