Cost Tracking Reference

This reference guide covers advanced topics: configuring custom models, understanding tokenizers, handling reasoning models, and using the Public API to manage model definitions.

Supported Tokenizers

ABV uses tokenizers to infer usage when token counts aren’t provided. The following tokenizers are currently supported:

Model Pattern	Tokenizer	Package	Notes
`gpt-4o*`	`o200k_base`	Built-in	Used for GPT-4o models
`gpt*`	`cl100k_base`	Built-in	Used for GPT-4, GPT-3.5, and other GPT models
`claude*`	`claude`	@anthropic-ai/tokenizer	Not fully accurate for Claude models per Anthropic

According to Anthropic, their tokenizer is not fully accurate for Claude models. For best results, ingest token counts directly from the Anthropic API response.

Predefined Models

ABV maintains a list of predefined popular models with tokenizers and pricing for:

OpenAI - GPT-4, GPT-3.5, GPT-4o, and other models
Anthropic - Claude 4 family (Opus, Sonnet, Haiku)
Google - Gemini models

View the complete list of predefined models in the ABV UI (sign-in required). Navigate to Settings → Model Definitions.

Custom Model Definitions

You can add your own model definitions for:

Custom or proprietary models
Fine-tuned models
Models not yet supported by ABV
Overriding default pricing

Creating Models via the UI

Navigate to Settings → Model Definitions in the ABV UI
Click Add Model
Configure the model settings (see configuration details below)
Save the model definition

Creating Models via the API

Use the Models API to programmatically manage model definitions:

# List all models
GET /api/public/models

# Create a new model
POST /api/public/models

# Get a specific model
GET /api/public/models/{id}

# Delete a model
DELETE /api/public/models/{id}

View the complete Public API documentation

Model Configuration

Model Matching

Models are matched to generations using regular expressions:

Generation Attribute	Model Attribute	Description
`model`	`match_pattern`	Regex pattern to match model names

Example patterns:

# Exact match for GPT-4
(?i)^(gpt-4-0125-preview)$

# Match all GPT-4 variants
(?i)^gpt-4.*$

# Match custom model family
(?i)^my-custom-model-.*$

User-defined models take priority over ABV-maintained models. This allows you to override default pricing or add support for new models.

Pricing Configuration

Define pricing per usage type. Usage types must match exactly with the keys in the usage_details object:

{
  "pricing": {
    "input": 0.00001,           // Price per input token in USD
    "output": 0.00003,          // Price per output token in USD
    "cached_tokens": 0.000001   // Price per cached token in USD
  }
}

Usage type matching:

Usage types are case-sensitive
Must match exactly: "input" ≠ "Input"
Supports arbitrary usage types for custom metrics

Tokenization Configuration

For models using the openai tokenizer, specify the tokenization config:

{
  "tokenizer": "openai",
  "tokenization_config": {
    "tokenizerModel": "gpt-4-0125-preview",  // tiktoken model name
    "tokensPerMessage": 4,                   // Required for chat models
    "tokensPerName": -1                      // Required for chat models
  }
}

Copy tokenization config from predefined OpenAI models in the UI. See the OpenAI tokenization guide for more details on tokensPerMessage and tokensPerName.

Complete Model Definition Example

Here’s a complete example of a custom model definition:

JSON
cURL

{
  "name": "My Custom GPT-4 Model",
  "match_pattern": "(?i)^my-custom-gpt-4$",
  "tokenizer": "openai",
  "tokenization_config": {
    "tokenizerModel": "gpt-4-0125-preview",
    "tokensPerMessage": 4,
    "tokensPerName": -1
  },
  "pricing": {
    "input": 0.00001,
    "output": 0.00003
  }
}

curl -X POST "https://app.abv.dev/api/public/models" \
  -H "Authorization: Bearer sk-abv-..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "My Custom GPT-4 Model",
    "match_pattern": "(?i)^my-custom-gpt-4$",
    "tokenizer": "openai",
    "tokenization_config": {
      "tokenizerModel": "gpt-4-0125-preview",
      "tokensPerMessage": 4,
      "tokensPerName": -1
    },
    "pricing": {
      "input": 0.00001,
      "output": 0.00003
    }
  }'

Reasoning Models

Reasoning models (like OpenAI’s o1 family) require special handling.

Why Inference Doesn’t Work

Reasoning models take multiple internal steps to arrive at a response:

Model generates reasoning tokens (internal thought process)
Model generates completion tokens (final response)
API bills you for: reasoning tokens + completion tokens

The problem: ABV cannot see the internal reasoning tokens, so it cannot accurately infer costs by tokenizing only the visible input and output.

Solution: Always Ingest Usage

For reasoning models, you must ingest usage details from the API response:

# Python example for OpenAI o1
response = openai_client.chat.completions.create(
    model="o1-preview",
    messages=[{"role": "user", "content": "Solve this problem..."}]
)

generation.update(
    usage_details={
        "prompt_tokens": response.usage.prompt_tokens,
        "completion_tokens": response.usage.completion_tokens,
        "completion_tokens_details": {
            "reasoning_tokens": response.usage.completion_tokens_details.reasoning_tokens
        }
    }
)

Critical: If you don’t ingest usage for reasoning models, ABV cannot calculate accurate costs. The tokenized input + output will significantly underestimate actual usage.

Learn more about OpenAI reasoning models

Historical Data and Model Changes

Retroactive Application

ABV does not retroactively apply model definition changes to historical generations. This includes:

Price updates
New tokenizers
Changed model patterns

Why: Costs are calculated at ingestion time using the model and price information available at that moment. This ensures historical data remains accurate and auditable.

Batch Reprocessing

If you need to apply new model definitions to existing generations:

Contact ABV support
Request a batch job to reprocess generations
Specify the time range and model changes

Batch reprocessing is typically used when:

Correcting a pricing error
Adding missing model definitions
Migrating to new tokenizers

OpenAI Usage Schema Compatibility

ABV supports the OpenAI usage schema for compatibility with OpenAI SDKs and tools.

Automatic Mapping

When you use OpenAI field names, ABV automatically maps them:

OpenAI Field	ABV Field
`prompt_tokens`	`input`
`completion_tokens`	`output`
`total_tokens`	`total`
`prompt_tokens_details.*`	`input_*`
`completion_tokens_details.*`	`output_*`

Example:

# You send this (OpenAI schema):
usage_details={
    "prompt_tokens": 100,
    "completion_tokens": 50,
    "prompt_tokens_details": {
        "cached_tokens": 20
    }
}

# ABV stores this:
{
    "input": 100,
    "output": 50,
    "input_cached_tokens": 20,
    "total": 150  # automatically calculated
}

This makes it easy to integrate with OpenAI without changing your existing code.

Requesting Official Model Support

Don’t want to maintain custom model definitions? Request official support:

Visit the ABV Model Support Request Form
Provide model details and documentation
ABV will add official support in a future release

Official support includes:

Automatic tokenization
Pricing updates
Maintenance and updates

Troubleshooting

Usage and Cost Missing for Historical Data

Problem: After adding a new model definition, historical generations still show no usage or cost. Solution: Model definitions are not applied retroactively. Options:

Request a batch reprocessing job from support
Accept that historical data uses old definitions
For future data: Ingest usage directly instead of relying on inference

Incorrect Costs

Problem: Costs don’t match your LLM provider’s billing. Checklist:

Check pricing: Verify the model definition pricing matches your provider’s current rates
Check usage types: Ensure usage type keys match exactly (case-sensitive)
Check model matching: Verify the regex pattern correctly matches your model name
Consider timing: Prices are applied at ingestion time, not retroactively

Best practice: For billing-critical applications, always ingest both usage AND cost from the provider’s API response.

Model Not Matching

Problem: Generations aren’t matching your custom model definition. Debug steps:

Check the model field in your generation matches the match_pattern regex
Test the regex pattern in a regex tester
Verify user-defined model isn’t being overridden by ABV model
Check for typos in the model name

Example:

// Generation model
"model": "gpt-4-custom-v1"

// Match pattern (correct)
"match_pattern": "(?i)^gpt-4-custom-v1$"

// Match pattern (incorrect - won't match)
"match_pattern": "^gpt-4-custom-v1$"  // missing (?i) flag

Tokenization Errors

Problem: Tokenization failing or producing unexpected results. Solutions:

For Claude models: Ingest token counts from API instead of inferring
For OpenAI models: Verify tokensPerMessage and tokensPerName are set correctly
For custom tokenizers: Ensure the tokenizer is supported (see supported tokenizers table)

API Reference

Models API Endpoints

List all models:

GET /api/public/models

Create a model:

POST /api/public/models
Content-Type: application/json

{
  "name": "string",
  "match_pattern": "string",
  "tokenizer": "string",
  "tokenization_config": {},
  "pricing": {}
}

Get a specific model:

GET /api/public/models/{id}

Delete a model:

DELETE /api/public/models/{id}

View complete API documentation

Next Steps

Implementation Guide

Return to implementation examples

Daily Metrics API

Export usage and cost data

Getting Started

Basic Features

LLM Gateway

Guardrails

Evaluations

Prompt Management

Cookbook

SDKs

Platform

Support

Supported Tokenizers

Predefined Models

Custom Model Definitions

Creating Models via the UI

Creating Models via the API

Model Configuration

Model Matching

Pricing Configuration

Tokenization Configuration

Complete Model Definition Example

Reasoning Models

Why Inference Doesn’t Work

Solution: Always Ingest Usage

Historical Data and Model Changes

Retroactive Application

Batch Reprocessing

OpenAI Usage Schema Compatibility

Automatic Mapping

Requesting Official Model Support

Troubleshooting

Usage and Cost Missing for Historical Data

Incorrect Costs

Model Not Matching

Tokenization Errors

API Reference

Models API Endpoints

Next Steps

Implementation Guide

Daily Metrics API

Getting Started

Basic Features

LLM Gateway

Guardrails

Evaluations

Prompt Management

Cookbook

SDKs

Platform

Support

​Supported Tokenizers

​Predefined Models

​Custom Model Definitions

​Creating Models via the UI

​Creating Models via the API

​Model Configuration

​Model Matching

​Pricing Configuration

​Tokenization Configuration

​Complete Model Definition Example

​Reasoning Models

​Why Inference Doesn’t Work

​Solution: Always Ingest Usage

​Historical Data and Model Changes

​Retroactive Application

​Batch Reprocessing

​OpenAI Usage Schema Compatibility

​Automatic Mapping

​Requesting Official Model Support

​Troubleshooting

​Usage and Cost Missing for Historical Data

​Incorrect Costs

​Model Not Matching

​Tokenization Errors

​API Reference

​Models API Endpoints

​Next Steps

Implementation Guide

Daily Metrics API

Supported Tokenizers

Predefined Models

Custom Model Definitions

Creating Models via the UI

Creating Models via the API

Model Configuration

Model Matching

Pricing Configuration

Tokenization Configuration

Complete Model Definition Example

Reasoning Models

Why Inference Doesn’t Work

Solution: Always Ingest Usage

Historical Data and Model Changes

Retroactive Application

Batch Reprocessing

OpenAI Usage Schema Compatibility

Automatic Mapping

Requesting Official Model Support

Troubleshooting

Usage and Cost Missing for Historical Data

Incorrect Costs

Model Not Matching

Tokenization Errors

API Reference

Models API Endpoints

Next Steps