# ABV Documentation - Complete Reference
This document contains the complete documentation for ABV, an LLM observability platform.
ABV provides comprehensive observability, monitoring, and management tools for Large Language Model (LLM) applications. The platform offers features including:
- LLM Gateway for unified model access
- Tracing and observability for LLM interactions
- Guardrails for LLM security and output validation
- Evaluations and testing frameworks
- Prompt management and version control
- SDKs for Python and JavaScript/TypeScript
- Enterprise security and compliance features
Total documents: 115
Generated on: 2025-11-13
================================================================================
## Quick Links
URL: /developer/quick-links
## What do you want to do first?
See what your LLMs are actually doing, debug unexpected outputs or behaviors, track execution flow through your application, and identify performance bottlenecks.
Monitor LLM token usage and costs, set budgets and alerts, identify expensive operations, and optimize model selection.
Prevent harmful or biased outputs, validate LLM responses, block PII or sensitive data, and enforce output formats.
Measure LLM accuracy and quality, A/B test different prompts, track improvements over time, and run automated evaluations.
Version control your prompts, A/B test prompt variations, separate prompts from code, and collaborate on prompt engineering.
Switch between LLM providers easily, standardize your LLM API calls, add observability automatically, and reduce vendor lock-in.
Need help deciding? Join our [community](https://discord.gg/abv) or [contact support](mailto:support@abv.dev) for personalized guidance.
--------------------------------------------------------------------------------
## Quickstart (Python SDK)
URL: /developer/quickstart-python
When you create a new account, the account creation flow will automatically guide you through the process of creating your first trace.
Alternatively, you can use these quickstart steps:
## 1) Get API key
1. [Create ABV account](https://app.abv.dev/auth/sign-up)
2. Create new API credentials in the project settings.
## 2) Ingest your first trace
**Install package**
```bash
pip install abvdev
```
**Add credentials**
Add your ABV credentials to your environment variables. Make sure that you have a `.env` file in your project root.
```bash title=".env"
ABV_API_KEY = "sk-abv-..."
ABV_HOST = "https://app.abv.dev" # US region
# ABV_HOST = "https://eu.app.abv.dev" # EU region
```
**Install package to get credentials from .env file**
```bash
pip install python-dotenv
```
There are a few main ways of creating traces with the Python SDK:
### Observe Decorator (Easiest)
The `@observe` decorator is the simplest way to instrument your application. It is a function decorator that can be applied to any function.
It sets the current span in the context for automatic nesting of child spans and automatically ends it when the function returns. It also automatically captures the function name, arguments, and return value.
```python
import os
# Import the function to load the .env file
from dotenv import load_dotenv
from abvdev import observe, get_client
# Load the environment variables from the .env file
load_dotenv()
@observe
def my_function():
return "Hello, world!" # Input/output and timings are automatically captured
my_function()
# Flush events in short-lived applications
abv = get_client()
abv.flush()
```
### Context Managers (Recommended)
Context managers are the recommended way to instrument chunks of work in your application as they automatically handle the start and end of spans, and set the current span in the context for automatic nesting of child spans. They provide more control than the `@observe` decorator.
```python
from abvdev import get_client
# # Import the function to load the .env file
from dotenv import load_dotenv
# Load the environment variables from the .env file
load_dotenv()
abv = get_client()
# Create a span using a context manager
with abv.start_as_current_span(name="process-request") as span:
# Your processing logic here
span.update(output="Processing complete")
# Create a nested generation for an LLM call
with abv.start_as_current_observation(as_type='generation', name="llm-response",
model="gpt-5-2025-08-07") as generation:
# Your LLM call logic here
generation.update(output="Generated response")
# All spans are automatically closed when exiting their context blocks
# Flush events in short-lived applications
abv.flush()
```
### **Same example with real openai llm call:**
**Add openai credentials**
Add **OPENAI\_API\_KEY** to `.env` file in your project root.
```bash title=".env"
ABV_API_KEY = "sk-abv-..."
ABV_HOST = "https://app.abv.dev" # US region
OPENAI_API_KEY = "sk-proj-..." # your_openai_api_key_here
```
**Install openai package**
```bash
pip install openai
```
**Instrument application**
```python
from abvdev import get_client
from openai import OpenAI
import os
# # Import the function to load the .env file
from dotenv import load_dotenv
# Load the environment variables from the .env file
load_dotenv()
abv = get_client()
openai_client = OpenAI(api_key = os.getenv("OPENAI_API_KEY"))
# Create a span using a context manager
with abv.start_as_current_span(name="process-request") as span:
# Your processing logic here
span.update(output="Processing complete")
# Create a nested generation for an LLM call
with abv.start_as_current_observation(as_type='generation', name="llm-response", model="gpt-4o") as generation:
# Your LLM call logic here
response = openai_client.chat.completions.create(
messages=[{"role": "user", "content": "Hi openai, how is that going?"}],
model="gpt-4o",
)
generation.update(output=response.choices[0].message.content)
# All spans are automatically closed when exiting their context blocks
# Flush events in short-lived applications
abv.flush()
```
--------------------------------------------------------------------------------
## Quickstart (JS/TS SDK)
URL: /developer/quickstart-js-ts
When you create a new account, the account creation flow will automatically guide you through the process of creating your first trace.
Alternatively, you can use the quickstart steps described below:
## 1) Get API key
1. [Create ABV account](https://app.abv.dev/auth/sign-up)
2. Create new API credentials in the project settings.
## 2) Ingest your first trace
**Install packages**
```bash
npm install @abvdev/tracing @abvdev/otel
```
**Add credentials**
Add your ABV credentials to your environment variables. Make sure that you have a `.env` file in your project root and a package like dotenv to load the variables.
```bash title=".env"
ABV_API_KEY = "sk-abv-..."
ABV_BASE_URL = "https://app.abv.dev" # US region
//ABV_BASE_URL = "https://eu.app.abv.dev" # EU region
```
**Initialize OpenTelemetry**
Install the OpenTelemetry Node SDK package:
```bash
npm install @opentelemetry/sdk-node
```
Create a `instrumentation.ts` file that initializes the OpenTelemetry `NodeSDK` and registers the `ABVSpanProcessor`.
```typescript title="instrumentation.ts"
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
const sdk = new NodeSDK({
spanProcessors: [new ABVSpanProcessor()],
});
sdk.start();
```
Modify `instrumentation.ts` file to use `dotenv` package to load the variables.
Additional parameters are provided to get trace visible in the UI immediately.
```bash
npm install dotenv
```
```typescript title="instrumentation.ts"
import dotenv from "dotenv";
dotenv.config();
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
const sdk = new NodeSDK({
spanProcessors: [
new ABVSpanProcessor({
apiKey: process.env.ABV_API_KEY,
baseUrl: process.env.ABV_BASE_URL,
exportMode: "immediate",
flushAt: 1,
flushInterval: 1,
additionalHeaders: {
"Content-Type": "application/json",
"Accept": "application/json"
}
})
],
});
sdk.start();
```
Import the `instrumentation.ts` file at the top of your application.
```typescript title="index.ts"
import "./instrumentation"; // Must be the first import
```
**Instrument application**
```typescript title="server.ts"
import "./instrumentation"; // Must be the first import
import { startActiveObservation, startObservation } from "@abvdev/tracing";
async function main() {
await startActiveObservation("user-request", async (span) => {
span.update({
input: { query: "What is the capital of France?" },
});
// This generation will automatically be a child of "user-request"
const generation = startObservation(
"llm-call",
{
model: "gpt-5-2025-08-07",
input: [{ role: "user", content: "What is the capital of France?" }],
},
{ asType: "generation" },
);
// ... LLM call logic ...
generation
.update({
output: { content: "The capital of France is Paris." },
})
.end();
span.update({ output: "Successfully answered." });
});
}
main();
```
**Run your application**
Execute your application. You should see your trace appear in the ABV UI.
```bash
npx tsx server.ts
```
### **Same example with real openai llm call:**
**Add openai credentials**
Add **OPENAI\_API\_KEY** to `.env` file in your project root.
```bash title=".env"
ABV_API_KEY = "sk-abv-..."
ABV_HOST = "https://app.abv.dev" # US region
OPENAI_API_KEY = "sk-proj-..." # your_openai_api_key_here
```
**Install openai package**
```bash
pip install openai
```
**Instrument application**
```typescript title="server.ts"
import "./instrumentation"; // Must be the first import
import { startActiveObservation, startObservation } from "@abvdev/tracing";
import OpenAI from "openai";
import dotenv from "dotenv";
dotenv.config();
const openai = new OpenAI({
apiKey: process.env.OPENAI_API_KEY,
});
async function main() {
await startActiveObservation("user-request", async (span) => {
span.update({
input: { query: "What is the capital of France?" },
});
const model = "gpt-4o";
const input = [{ role: "user", content: "What is the capital of France?" }];
// This generation will automatically be a child of "user-request"
const generation = startObservation(
"llm-call",
{
model: model,
input: input,
},
{ asType: "generation" },
);
// ... LLM call logic ...
const response = await openai.chat.completions.create({
messages: input,
model: model,
});
const llmOutput = response.choices[0].message.content;
generation
.update({
output: llmOutput,
})
.end();
span.update({ output: "Successfully answered." });
});
}
main();
```
**Run your application**
Execute your application. You should see your trace appear in the ABV UI.
```bash
npx tsx server.ts
```
--------------------------------------------------------------------------------
## Observability Data Model
URL: /developer/observability-data-model
## Core Concepts at a Glance
ABV's observability model has four main building blocks:
| Concept | What It Is | When to Use | Example |
|---------|-----------|-------------|---------|
| **Trace** | Single request or operation | Every API call or workflow | User asks a question in your chatbot |
| **Observation** | Individual steps within a trace | Track specific operations | LLM call, database query, function execution |
| **Session** | Group of related traces | Multi-turn interactions | Entire conversation thread |
| **Score** | Evaluation metric | Measure quality or performance | Accuracy score, cost, latency |
## Traces and Observations
### Traces
A `trace` typically represents a single request or operation. It contains the overall input and output of the function, as well as metadata about the request ( i.e. user, session, tags, etc.).
### Observations
Each trace can contain multiple `observations` to log the individual steps of the execution. Usually, a trace corresponds to a single api call of an application.
**Types**
`Events` are the basic building blocks. They are used to track discrete events in a trace.
`Spans` represent durations of units of work in a trace.
`Generations` are spans used to log generations of AI models incl. prompts, [token usage and costs](/developer/basic-features/model-usage-cost-tracking).
**Nesting**
Observations can be nested to represent the hierarchical structure of your application. For example, a trace might contain a span for the entire request, which in turn contains a generation for an LLM call.
## Sessions
Optionally, traces can be grouped into [sessions](/developer/basic-features/sessions). Sessions are used to group traces that are part of the same user interaction. A common example is a thread in a chat interface.
Please refer to the [Sessions](/developer/basic-features/sessions) documentation to add sessions to your traces.
## Scores
[Scores](/developer/evaluations/overview) are flexible objects used to evaluate traces, observations, sessions and dataset runs.
They can be:
- Numeric, categorical, or boolean values
- Associated with a trace, a session, or a dataset run (one and only one is required)
- For trace level scores only: Linked to a specific observation within a trace (optional)
- Annotated with comments for additional context
- Validated against a score configuration schema (optional)
Typically, session-level scores are used for comprehensive evaluation of conversational experiences across multiple interactions, while trace-level scores are used for evaluation of a single interaction. Dataset run level scores are used for overall evaluation of a dataset run, e.g. precision, recall, F1-score.
Please refer to the [scores documentation](/developer/evaluations/overview) to get started. For more details on score types and attributes, refer to the [scores data model](/developer/evaluations/scores-data-model) documentation.
## Real-World Examples
Understanding how to apply ABV's data model to different scenarios helps you instrument your application effectively.
### Example 1: Simple Chatbot
```
Session: "User conversation thread #123"
└─ Trace: "User message: 'What's the weather?'"
└─ Generation: "OpenAI GPT-4 call"
└─ Trace: "User message: 'And tomorrow?'"
└─ Generation: "OpenAI GPT-4 call (with conversation history)"
```
**Key takeaway:** Each user message is a trace. The conversation is a session. The LLM call is a generation.
### Example 2: RAG Pipeline
```
Trace: "User query: 'Explain quantum computing'"
├─ Span: "Retrieve relevant documents"
│ └─ Event: "Found 5 matching documents"
├─ Span: "Rerank documents"
│ └─ Event: "Selected top 3 documents"
└─ Generation: "Generate answer with context"
└─ Score: "Answer quality = 0.92"
```
**Key takeaway:** Complex workflows use nested observations. Spans for non-LLM operations, generations for LLM calls, events for discrete actions.
### Example 3: Multi-Agent System
```
Trace: "Research task: 'Find competitors'"
├─ Span: "Planner agent decides strategy"
│ └─ Generation: "GPT-4: Create research plan"
├─ Span: "Researcher agent executes"
│ ├─ Generation: "GPT-4: Search query 1"
│ ├─ Generation: "GPT-4: Search query 2"
│ └─ Event: "Found 10 results"
└─ Span: "Summarizer agent compiles results"
└─ Generation: "Claude: Summarize findings"
```
**Key takeaway:** Each agent's work is a span. LLM calls within agents are generations. The entire task is one trace.
## Next Steps
Now that you understand the data model, here's how to implement it:
Start instrumenting your Python application
Start instrumenting your JavaScript/TypeScript application
Learn how to group traces into sessions
Add scores to measure quality
**Best practice:** Start simple with basic traces, then add observations as you need more granularity. Add sessions when you have multi-turn interactions. Add scores when you're ready to measure quality.
--------------------------------------------------------------------------------
## Observability & Tracing
URL: /developer/basic-features/observability-tracing
With ABV, developers can easily implement observability and tracing into their LLM applications. The platform captures everything happening during an LLM interaction: inputs, outputs, tool usage, retries, latencies and costs.
**Observability** is the broader concept of understanding what is happening under the hood of you LLM application. **Traces** are entities that can be considered individual 'units' observability.
# Why Should You Setup Tracing?
- **Accelerate development & debugging**: Quickly pinpoint issues and understand agent/LLM behavior.
- **Track costs in real time**: Gain visibility into expenses for every model call.
- **Optimize performance**: Identify bottlenecks and reduce application latency.
- **Enable evaluations**: Build a strong foundation of data that enables you to use ABV’s suite of evaluation tools for systematic testing and benchmarking.
- **Streamline customer support**: Save hours by diagnosing user issues with complete trace visibility.
# Why Trace with ABV?
- **All-In-One-Platform**: ABV platform provides the most complete development experience for AI app developers.
- **Multi-Modal**: ABV supports text, images, audio, and more.
- **Multi-Model**: ABV supports all major LLM providers.
- **Language Agnostic**: ABV natively supports Python and JavaScript via our SDKs, along with an open API for other languages.
# Where to start
Use the quickstart guides to add tracing to your LLM app:
- [Quickstart (Python SDK)](/developer/quickstart-python)
- [Quickstart (JS/TS SDK)](/developer/quickstart-js-ts)
# Core Features
- [Sessions](/developer/basic-features/sessions): Group related traces by user journey or job to see end-to-end behavior.
- [Users](/developer/basic-features/user-tracking): Link traces to user accounts for faster reproduction and targeted support.
- [Environments](/developer/basic-features/environments): Separate dev/staging/prod to compare performance and isolate issues safely.
- [Metadata](/developer/basic-features/metadata): Attach structured context (e.g., model, region, tenant) to make queries precise.
- [Tags](/developer/basic-features/tags): Add flexible labels to categorize, filter, and slice traces quickly.
- [Trace IDs](/developer/basic-features/trace-ids-distributed-tracing): Correlate events across services for reliable root-cause analysis.
- [Multi-Modality and Attachments](/developer/basic-features/multi-modality-and-attachments): Store text, images, audio, and files for richer debugging context.
# Advanced Features
- [Model Usage & Cost Tracking](/developer/basic-features/model-usage-cost-tracking): Monitor tokens, runtime, and spend to optimize model selection.
- [Masking of Sensitive LLM Data](/developer/basic-features/masking-sensitive-data): Redact PII, secrets, and prompts to stay compliant while logging.
- [Log Levels](/developer/basic-features/log-levels): Prioritize noise vs. signal by severity to focus on what matters.
- [Releases & Versioning](/developer/basic-features/releases-versioning): Annotate traces with code/model versions to spot post-deploy regressions.
- [Comments on Objects](/developer/basic-features/comments-on-objects): Collaborate inline on traces/sessions for reviews, handoffs, and decisions.
- [Trace URLs](/developer/basic-features/trace-urls): Share deep links to any trace/span for reproducible bug reports and reviews.
- [Sampling](/developer/basic-features/sampling): Control volume and cost with rule- or rate-based capture policies
--------------------------------------------------------------------------------
## Trace IDs & Distributed Tracing
URL: /developer/basic-features/trace-ids-distributed-tracing
ABV allows you to bring your own trace IDs (e.g., messageId, traceId, correlationId) for
- distributed tracing
- and linking traces across services for lookups between services.
By default, ABV assigns random IDs (uuid, cuid) to all logged events. For the new OTEL-based SDKs (Python v3), ABV assigns random 32 hexchar trace IDs and 16 hexchar observation IDs.
It is recommended to use your own domain specific IDs (e.g., messageId, traceId, correlationId) as it helps with downstream use cases like:
- Deeplinking to the trace from your own ui or logs
- Evaluating and adding custom metrics to the trace
- Fetching the trace from the API
# Data Model
Trace IDs in ABV:
- Must be unique within a project
- Are used to identify and group related observations
- Can be used for distributed tracing across services
- Support upsert operations (creating or updating based on ID)
- For the new OTEL-based SDKs (Python v3), trace IDs are 32 hexchar lowercase strings and observation IDs are 16 hexchar lowercase strings
# Usage
## Python SDK
The Python SDK uses W3C Trace Context IDs by default, which are:
- 32-character lowercase hexadecimal string for trace IDs
- 16-character lowercase hexadecimal string for observation (span) IDs
### Install package
```bash
pip install abvdev
```
### Using the Decorator
```python
from abvdev import observe
from abvdev import ABV
import uuid
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
@observe()
def process_user_request(user_id, request_data):
# Function logic here
return f"request processed for user with id: {user_id} for the data: {request_data}"
# Use custom trace ID by passing it as special keyword argument
external_trace_id = "custom-" + str(uuid.uuid4())
trace_id = abv.create_trace_id(seed=external_trace_id) # 32 hexchar lowercase string, deterministic with seed
process_user_request(
user_id="user_123",
request_data={"query": "hello"},
abv_trace_id=trace_id
)
```
### Deterministic Trace IDs
You can generate deterministic trace IDs from any string using create\_trace\_id():
```python
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# Generate deterministic trace ID from an external ID
external_id = "request_12345"
trace_id = abv.create_trace_id(seed=external_id)
# Use this trace ID in a span
with abv.start_as_current_span(
name="process-request",
trace_context={"trace_id": trace_id}
) as span:
# Your code here
pass
```
### Manually Creating Spans with Custom Trace Context
```python
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# Use a predefined trace ID with trace_context parameter
with abv.start_as_current_span(
name="my-operation",
trace_context={
"trace_id": "abcdef1234567890abcdef1234567890", # Must be 32 hex chars
"parent_span_id": "fedcba0987654321" # Optional, 16 hex chars
}
) as span:
print(f"This span has trace_id: {span.trace_id}")
# Your code here
```
### Accessing Current Trace ID
```python
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
with abv.start_as_current_span(name="outer-operation") as span:
# Access the trace ID of the current span
current_trace_id = abv.get_current_trace_id()
current_span_id = abv.get_current_observation_id()
print(f"Current trace ID: {current_trace_id}")
print(f"Direct access: {span.trace_id}")
```
## JS/TS SDK
**Install packages**
```bash
npm install @abvdev/tracing @abvdev/otel @opentelemetry/sdk-node dotenv
```
**Add credentials**
Add your ABV credentials to your environment variables. Make sure that you have a `.env` file in your project root and a package like dotenv to load the variables.
.env
```typescript
ABV_API_KEY = "sk-abv-..."
ABV_BASE_URL = "https://app.abv.dev" # US region
//ABV_BASE_URL = "https://eu.app.abv.dev" # EU region
```
Create`instrumentation.ts` file and use `dotenv` package to load the variables.
Additional parameters are provided to get trace visible in the UI immediately.
instrumentation.ts
```typescript
import dotenv from "dotenv";
dotenv.config();
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
const sdk = new NodeSDK({
spanProcessors: [
new ABVSpanProcessor({
apiKey: process.env.ABV_API_KEY,
baseUrl: process.env.ABV_BASE_URL,
exportMode: "immediate",
flushAt: 1,
flushInterval: 1,
additionalHeaders: {
"Content-Type": "application/json",
"Accept": "application/json"
}
})
],
});
sdk.start();
```
Import the `instrumentation.ts` file at the top of your application.
**The JS/TS SDK uses W3C Trace Context IDs by default, which are:**
- 32-character lowercase hexadecimal string for trace IDs
- 16-character lowercase hexadecimal string for observation (span) IDs
### Accessing the current trace ID
```typescript
import "./instrumentation"; // Must be the first import
import { startActiveObservation, getActiveTraceId, updateActiveTrace } from "@abvdev/tracing";
async function main() {
await startActiveObservation("run", async (span) => {
// observation metadata
span.update({
input: { query: "What is the capital of France?" },
});
updateActiveTrace({
metadata: { key: "value" },
});
const traceId = getActiveTraceId();
console.log(`Current trace ID: ${traceId}`);
});
}
main();
```
You may access the current active trace ID via the getActiveTraceId function:
### Deterministic Trace IDs
When starting a new trace with a predetermined traceId, you must also provide an arbitrary parent-spanId for the parent observation. The parent span ID value is irrelevant as long as it is a valid 16-hexchar string as the span does not actually exist within the trace but is only used for trace ID inheritance of the created observation.
You can create valid, deterministic trace IDs from a seed string using createTraceId. This is useful for correlating ABV traces with IDs from external systems, like a support ticket ID.
```typescript
import "./instrumentation"; // Must be the first import
import { createTraceId, startObservation } from "@abvdev/tracing";
const externalId = "support-ticket-252525";
async function main() {
// Generate a valid, deterministic traceId from the external ID
const abvTraceId = await createTraceId(externalId);
// You can now start a new trace with this ID
const rootSpan = startObservation(
"process-ticket",
{
input: "there is an input",
output: "here you are an output"
},
{
parentSpanContext: {
traceId: abvTraceId,
spanId: "0123456789abcdef", // A valid 16 hexchar string; value is irrelevant as parent span does not exist but only used for inheritance
traceFlags: 1, // mark trace as sampled
},
}
);
rootSpan.end();
// Later, you can regenerate the same traceId to score or retrieve the trace
const scoringTraceId = await createTraceId(externalId);
// scoringTraceId will be the same as abvTraceId
console.log(scoringTraceId);
}
main();
```
Setting a parentSpanContext will detach the created span from the active span context as it no longer inherits from the current active span in the context.
Learn more in the [JS/TS SDK docs](/developer/sdks/js-ts/overview).
## OpenTelemetry
When using OpenTelemetry, trace IDs are handled automatically by the OpenTelemetry SDK. You can access and set trace IDs using the OpenTelemetry context:
```python
from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("my-operation") as span:
# Get the trace ID
trace_id = format(span.get_span_context().trace_id, "032x")
# Set custom attributes
span.set_attribute("custom.trace_id", trace_id)
```
--------------------------------------------------------------------------------
## Trace URLs
URL: /developer/basic-features/trace-urls
toggleEach trace has a unique URL that you can use to share it with others or to access it directly.
# Get trace url
Sometimes, it is useful to get the trace URL directly in the SDK. E.g. to add it to your logs or interactively look at it when running experiments in notebooks.
### With Python SDK
Install package
```bash
pip install abvdev
```
When using the `@observe()` decorator:
```python
from abvdev import ABV, observe
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
@observe()
def process_data():
# Get the URL of the current trace
trace_url = abv.get_trace_url()
print(f"View trace at: {trace_url}")
# or pass the trace id
trace_id = abv.get_current_trace_id()
trace_url = abv.get_trace_url(trace_id=trace_id)
process_data()
```
When using context managers:
```python
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
with abv.start_as_current_span(name="process-request") as span:
# Get the URL of this trace
trace_url = abv.get_trace_url()
print(f"View trace at: {trace_url}")
# or pass the trace id
trace_id = abv.get_current_trace_id()
trace_url = abv.get_trace_url(trace_id=trace_id)
```
### With JS/TS SDK
**Install packages**
```bash
npm install @abvdev/tracing @abvdev/otel @opentelemetry/sdk-node dotenv @abvdev/client
```
**Add credentials**
Add your ABV credentials to your environment variables. Make sure that you have a `.env` file in your project root and a package like dotenv to load the variables.
.env
```typescript
ABV_API_KEY = "sk-abv-..."
ABV_BASE_URL = "https://app.abv.dev" # US region
//ABV_BASE_URL = "https://eu.app.abv.dev" # EU region
```
Create`instrumentation.ts` file and use `dotenv` package to load the variables.
Additional parameters are provided to get trace visible in the UI immediately.
instrumentation.ts
```typescript
import dotenv from "dotenv";
dotenv.config();
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
const sdk = new NodeSDK({
spanProcessors: [
new ABVSpanProcessor({
apiKey: process.env.ABV_API_KEY,
baseUrl: process.env.ABV_BASE_URL,
exportMode: "immediate",
flushAt: 1,
flushInterval: 1,
additionalHeaders: {
"Content-Type": "application/json",
"Accept": "application/json"
}
})
],
});
sdk.start();
```
Import the `instrumentation.ts` file at the top of your application.
```typescript
import "./instrumentation"; // Must be the first import
import { ABVClient } from "@abvdev/client";
import { startObservation } from "@abvdev/tracing";
const abv = new ABVClient();
async function main() {
const rootSpan = startObservation("my-trace", {
input: { query: "What is the capital of France?" },
});
const traceUrl = await abv.getTraceUrl(rootSpan.traceId);
console.log("Trace URL: ", traceUrl);
rootSpan.update({
output: { content: "The capital of France is Paris." },
})
rootSpan.end();
}
main();
```
# Share trace via url
By default, only members of your ABV project can view a trace.
You can make a trace `public` to share it via a public link. This allows others to view the trace without needing to log in or be members of your ABV project.
### via ABV UI
In trace view switch "Private" toggle to "Public"
### with Python SDK
When using the `@observe()` decorator:
```python
from abvdev import ABV, observe
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
@observe()
def process_data():
# Make the current trace public
abv.update_current_trace(public=True)
process_data()
```
When using context managers:
```python
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
with abv.start_as_current_span(name="process-request") as span:
# Make this trace public
span.update_trace(public=True)
# Get the URL to share
trace_id = abv.get_current_trace_id()
trace_url = abv.get_trace_url(trace_id=trace_id)
print(f"Share this trace at: {trace_url}")
```
### with JS/TS SDK
```typescript
import "./instrumentation"; // Must be the first import
import { startObservation } from "@abvdev/tracing";
const rootSpan = startObservation("my-trace");
rootSpan.updateTrace({
public: true,
});
rootSpan.end();
```
--------------------------------------------------------------------------------
## Log Levels
URL: /developer/basic-features/log-levels
Traces can have a lot of observations. You can differentiate the importance of observations with the `level` attribute to control the verbosity of your traces and highlight errors and warnings. Available `levels`: `DEBUG`, `DEFAULT`, `WARNING`, `ERROR`.
In addition to the level, you can also include a `statusMessage` to provide additional context.
## Implementation with Python SDK
Install package
```bash
pip install abvdev
```
When using the [observe() decorator](/developer/sdks/python/overview):
```python
from abvdev import observe
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
@observe()
def my_function():
# ... processing logic ...
# Update the current span with a warning level
abv.update_current_span(
level="WARNING",
status_message="This is a warning"
)
# call the function
my_function()
```
When creating spans or generations directly:
```python
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# Using context managers (recommended)
with abv.start_as_current_observation(as_type='span', name="my-operation") as span:
# Set level and status message on creation
with span.start_as_current_observation(
as_type='span',
name="potentially-risky-operation",
level="WARNING",
status_message="Operation may fail"
) as risky_span:
# ... do work ...
# Or update level and status message later
risky_span.update(
level="ERROR",
status_message="Operation failed with unexpected input"
)
# You can also update the currently active span without a direct reference
with abv.start_as_current_observation(as_type='span', name="another-operation"):
# ... some processing ...
abv.update_current_span(
level="DEBUG",
status_message="Processing intermediate results"
)
```
Levels can also be set when creating generations:
```bash
pip install openai
```
```python
from abvdev import ABV
from openai import OpenAI
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# llm openai client initialization
openai_client = OpenAI(api_key = "sk-proj-...") # your openai api key here
with abv.start_as_current_observation(
as_type='generation',
name="llm-call",
model="gpt-4o",
level="DEFAULT" # Default level
) as generation:
error_detected = False
try:
# ... make LLM call ...
response = openai_client.chat.completions.create(
messages=[{"role": "user", "content": "Hi openai, how is that going?"}],
model="gpt-4o",
)
generation.update(output=response.choices[0].message.content)
print("request processed")
except Exception as e:
print(e)
error_detected = True
if error_detected:
generation.update(
level="ERROR",
status_message="Model returned malformed output"
)
```
## Implementation with JS/TS SDK
**Install packages**
```bash
npm install @abvdev/tracing @abvdev/otel @opentelemetry/sdk-node dotenv
```
**Add credentials**
Add your ABV credentials to your environment variables. Make sure that you have a `.env` file in your project root and a package like dotenv to load the variables.
.env
```typescript
ABV_API_KEY = "sk-abv-..."
ABV_BASE_URL = "https://app.abv.dev" # US region
//ABV_BASE_URL = "https://eu.app.abv.dev" # EU region
```
Create`instrumentation.ts` file and use `dotenv` package to load the variables.
Additional parameters are provided to get trace visible in the UI immediately.
instrumentation.ts
```typescript
import dotenv from "dotenv";
dotenv.config();
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
const sdk = new NodeSDK({
spanProcessors: [
new ABVSpanProcessor({
apiKey: process.env.ABV_API_KEY,
baseUrl: process.env.ABV_BASE_URL,
exportMode: "immediate",
flushAt: 1,
flushInterval: 1,
additionalHeaders: {
"Content-Type": "application/json",
"Accept": "application/json"
}
})
],
});
sdk.start();
```
Import the `instrumentation.ts` file at the top of your application.
**When using the context manager:**
```typescript
import "./instrumentation";
import { startActiveObservation, updateActiveObservation } from "@abvdev/tracing";
async function main() {
await startActiveObservation("context-manager", async (span) => {
span.update({
input: { query: "What is the capital of France?" },
});
updateActiveObservation(
"span",
{
level: "WARNING",
statusMessage: "This is a warning",
},
);
});
}
main();
```
**When using the **`observe`** wrapper:**
```typescript
import "./instrumentation"; // Must be the first import
import { observe, updateActiveObservation } from "@abvdev/tracing";
// An existing function
async function fetchData(source: string) {
updateActiveObservation(
"span",
{
level: "WARNING",
statusMessage: "This is a warning",
}
);
// ... logic to fetch data
return { data: `some data from ${source}` };
}
// Wrap the function to trace it
const tracedFetchData = observe(fetchData, {
name: "observe-wrapper",
});
async function main() {
const result = await tracedFetchData("API");
}
main();
```
**When creating spans manually:**
```typescript
import "./instrumentation";
import { startObservation } from "@abvdev/tracing";
const span = startObservation("manual-observation", {
input: { query: "What is the capital of France?" },
});
span.update({
level: "WARNING",
statusMessage: "This is a warning",
});
span.update({ output: "Paris" }).end();
```
See [JS/TS SDK docs](/developer/sdks/js-ts/overview) for more details.
Filter Trace by Log Level
=========================
When viewing a single trace, you can filter the observations by log level.
--------------------------------------------------------------------------------
## Metadata
URL: /developer/basic-features/metadata
Traces and observations can be enriched with metadata to better understand your users, application, and experiments. Metadata can be added to traces in the form of arbitrary JSON.
Metadata updates are merged based on the top-level key. We strongly discourage writing the same key multiple times within your instrumentation.
## Use Cases
- **A/B Testing**: Track experiment variants and compare performance across different configurations
- **Multi-tenancy**: Associate traces with specific customers, organizations, or tenants for isolated analysis
- **Environment Tracking**: Distinguish between development, staging, and production deployments
- **Feature Flags**: Monitor feature adoption and performance for specific feature rollouts
- **Model Versions**: Track which model version generated each response for regression analysis
## Related Features
- [Tags](/developer/basic-features/tags): Use tags for simple key-value labels and filtering
- [Environments](/developer/basic-features/environments): Separate traces by deployment environment
- [Sessions](/developer/basic-features/sessions): Group related traces by user journey
- [User Tracking](/developer/basic-features/user-tracking): Associate traces with specific users
## Implementation with Python SDK
Install package
```bash
pip install abvdev
```
When using the *@observe()* decorator:
```python
from abvdev import observe
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
@observe()
def process_data():
# Access the client and update the current trace metadata
# Add metadata to the trace level
abv.update_current_trace(
metadata={"source": "api", "version": "1.2.3"}
)
# Add metadata to the current span level
abv.update_current_span(
metadata={"processing_stage": "initial"}
)
# Process data...
return "Processing complete"
# call the function
process_data()
```
When creating spans directly:
```python
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# Add metadata at trace level
with abv.start_as_current_span(
name="process-request"
) as root_span:
# Add metadata to the trace
root_span.update_trace(metadata={"request_id": "req_12345"})
# Add metadata to the current span
root_span.update(metadata={"stage": "parsing"})
# Create a child span with metadata
with root_span.start_as_current_generation(
name="generate-response",
model="gpt-4o",
metadata={"temperature": 0.7, "max_tokens": 1000}
) as gen:
# Update metadata later if needed
gen.update(metadata={"completion_type": "creative"})
```
You can add new keys to the metadata object by continuously updating the entity. We strongly discourage writing the same top-level key multiple times as this will produce an undefined behaviour.
```python
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
with abv.start_as_current_span(name="operation") as span:
# First write
span.update(metadata={"status": "started"})
# Additional key - will be merged with previous metadata
span.update(metadata={"error": "Failed to process"})
# Final metadata will be {"status": "started", "error": "Failed to process"}
```
## Implementation with JS/TS SDK
**Install packages**
```bash
npm install @abvdev/tracing @abvdev/otel @opentelemetry/sdk-node dotenv
```
**Add credentials**
Add your ABV credentials to your environment variables. Make sure that you have a `.env` file in your project root and a package like dotenv to load the variables.
```bash title=".env"
ABV_API_KEY = "sk-abv-..."
ABV_BASE_URL = "https://app.abv.dev" # US region
//ABV_BASE_URL = "https://eu.app.abv.dev" # EU region
```
Create`instrumentation.ts` file and use `dotenv` package to load the variables.
Additional parameters are provided to get trace visible in the UI immediately.
```typescript title="instrumentation.ts"
import dotenv from "dotenv";
dotenv.config();
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
const sdk = new NodeSDK({
spanProcessors: [
new ABVSpanProcessor({
apiKey: process.env.ABV_API_KEY,
baseUrl: process.env.ABV_BASE_URL,
exportMode: "immediate",
flushAt: 1,
flushInterval: 1,
additionalHeaders: {
"Content-Type": "application/json",
"Accept": "application/json"
}
})
],
});
sdk.start();
```
Import the `instrumentation.ts` file at the top of your application.
** When using the context manager:**
```typescript
import "./instrumentation"; // Must be the first import
import {
startActiveObservation,
startObservation,
updateActiveTrace,
} from "@abvdev/tracing";
async function main() {
await startActiveObservation("context-manager", async (span) => {
// observation metadata
span.update({
input: { query: "What is the capital of France?" },
});
updateActiveTrace({
metadata: { key: "value" },
});
});
}
main();
```
**When using the **`observe`** wrapper:**
```typescript
import "./instrumentation"; // Must be the first import
import { observe, updateActiveTrace } from "@abvdev/tracing";
// An existing function
async function fetchData(source: string) {
// observation metadata
updateActiveTrace({
metadata: { key: "value" },
});
// trace metadata
updateActiveTrace({
metadata: { key: "value" },
});
// ... logic to fetch data
return { data: `some data from ${source}` };
}
// Wrap the function to trace it
const tracedFetchData = observe(fetchData, {
name: "observe-wrapper",
});
async function main() {
const result = await tracedFetchData("API");
}
main();
```
**When creating spans manually:**
```typescript
import "./instrumentation"; // Must be the first import
import { startObservation } from "@abvdev/tracing";
const span = startObservation("manual-observation", {
input: { query: "What is the capital of France?" },
metadata: { key: "value" },
});
span.updateTrace({
metadata: { key: "value" },
});
span.update({ output: "Paris" }).end();
```
See [JS/TS SDK docs](/developer/sdks/js-ts/overview) for more details.
--------------------------------------------------------------------------------
## Tags
URL: /developer/basic-features/tags
Tags allow you to categorize and filter traces. You can tag traces:
- Programatically using the ABV SDKs
- From the ABV UI.
To tag a trace, add a list of tags to the tags field of the trace object. Tags are strings and a trace may have multiple tags. Trace tags can be added, but not removed.
## Implementation with Python SDK
Install package
```bash
pip install abvdev
```
When using the [@observe() decorator](/developer/sdks/python/overview):
```python
from abvdev import observe
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
@observe()
def my_function():
# ... processing logic ...
# Add tags to the trace
abv.update_current_trace(tags=["tag-1", "tag-2"])
# call the function
my_function()
```
When creating spans or generations directly:
```python
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# Add tags when creating the root span
with abv.start_as_current_span(
name="my-operation"
) as root_span:
# Add tags to the trace
root_span.update_trace(tags=["tag-1", "tag-2"])
# You can add more tags later from any span in the same trace
with root_span.start_as_current_generation(name="llm-call", model="gpt-4o") as gen:
# Processing...
gen.update_trace(tags=["llm-gen"]) # Adds another tag to the same trace
```
You can also update the tags of the current trace without a direct reference to a span:
```python
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
with abv.start_as_current_span(name="another-operation"):
# ... processing ...
abv.update_current_trace(tags=["processing", "beta-feature"])
```
## Implementation with JS/TS SDK
**Install packages**
```bash
npm install @abvdev/tracing @abvdev/otel @opentelemetry/sdk-node dotenv
```
**Add credentials**
Add your ABV credentials to your environment variables. Make sure that you have a `.env` file in your project root and a package like dotenv to load the variables.
```bash title=".env"
ABV_API_KEY = "sk-abv-..."
ABV_BASE_URL = "https://app.abv.dev" # US region
//ABV_BASE_URL = "https://eu.app.abv.dev" # EU region
```
Create`instrumentation.ts` file and use `dotenv` package to load the variables.
Additional parameters are provided to get trace visible in the UI immediately.
```typescript title="instrumentation.ts"
import dotenv from "dotenv";
dotenv.config();
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
const sdk = new NodeSDK({
spanProcessors: [
new ABVSpanProcessor({
apiKey: process.env.ABV_API_KEY,
baseUrl: process.env.ABV_BASE_URL,
exportMode: "immediate",
flushAt: 1,
flushInterval: 1,
additionalHeaders: {
"Content-Type": "application/json",
"Accept": "application/json"
}
})
],
});
sdk.start();
```
Import the `instrumentation.ts` file at the top of your application.
**When using the context manager:**
```typescript
import "./instrumentation"; // Must be the first import
import {
startActiveObservation,
startObservation,
updateActiveTrace,
} from "@abvdev/tracing";
async function main() {
await startActiveObservation("context-manager", async (span) => {
span.update({
input: { query: "What is the capital of France?" },
});
updateActiveTrace({
tags: ["tag-1", "tag-2"],
});
});
};
main();
```
**When using the **`observe`** wrapper:**
```typescript
import "./instrumentation"; // Must be the first import
import { observe, updateActiveTrace } from "@abvdev/tracing";
// An existing function
async function fetchData(source: string) {
updateActiveTrace({
tags: ["tag-1", "tag-2"],
});
// ... logic to fetch data
return { data: `some data from ${source}` };
}
// Wrap the function to trace it
const tracedFetchData = observe(fetchData, {
name: "observe-wrapper",
});
async function main() {
const result = await tracedFetchData("API");
}
main();
```
**When creating spans manually:**
```typescript
import "./instrumentation"; // Must be the first import
import { startObservation } from "@abvdev/tracing";
const span = startObservation("manual-observation", {
input: { query: "What is the capital of France?" },
});
span.updateTrace({
tags: ["tag-1", "tag-2"],
});
span.update({ output: "Paris" }).end();
```
See [JS/TS SDK docs](/developer/sdks/js-ts/overview) for more details.
## Working with tags
Tags enable you to flexibly add metadata to your traces. You can filter for tags in the ABV UI and use them to organize and analyze your observability data.
When choosing tags, consider what aspects of the traces you might want to filter for or group by in your analysis. You may use tags to indicate specific versions of your app (‘app-v1’, ‘app-v2’), specific LLM techniques you used (‘rag’, ‘one-shot’, ‘few-shot’), or the environment of your app (‘local’, ‘staging’, ‘prod’). See for an end-to-end example on how tags can be created programmatically.
--------------------------------------------------------------------------------
## Sessions
URL: /developer/basic-features/sessions
Many interactions with LLM applications span multiple traces. `Sessions` in ABV are a way to group these traces together and see a simple session replay of the entire interaction. Get started by adding a `sessionId` when creating a trace.
Add a `sessionId` when creating/updating a trace. This can be any string that you use to identify the session. All traces with the same `sessionId` will be grouped together.
## Implementation with Python SDK
Install package
```bash
pip install abvdev
```
When using the @observe() decorator and initialization via constructor arguments:
```python
from abvdev import observe
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
@observe()
def process_request():
# Add to the current trace
abv.update_current_trace(session_id="your-session-id")
# ...your processing logic...
return "Processing complete"
# call the function
process_request()
```
When creating spans directly and initialization via constructor arguments:
```python
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# You can set the session_id when creating the root span
with abv.start_as_current_span(
name="process-chat-message"
) as root_span:
# Add session_id to the trace
root_span.update_trace(session_id="chat-session-123")
# All spans in this trace will belong to the same session
with root_span.start_as_current_observation(
as_type='generation',
name="generate-response",
model="gpt-4o"
) as gen:
# ...generate response...
pass
```
You can also update the session\_id of the current trace without a direct reference to a span:
```python
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
with abv.start_as_current_span(name="another-operation"):
# Add to the current trace
abv.update_current_trace(session_id="your-session-id")
```
## Implementation with JS/TS SDK
**Install packages**
```bash
npm install @abvdev/tracing @abvdev/otel @opentelemetry/sdk-node dotenv
```
**Add credentials**
Add your ABV credentials to your environment variables. Make sure that you have a `.env` file in your project root and a package like dotenv to load the variables.
.env
```typescript
ABV_API_KEY = "sk-abv-..."
ABV_BASE_URL = "https://app.abv.dev" # US region
//ABV_BASE_URL = "https://eu.app.abv.dev" # EU region
```
Create`instrumentation.ts` file and use `dotenv` package to load the variables.
Additional parameters are provided to get trace visible in the UI immediately.
instrumentation.ts
```typescript
import dotenv from "dotenv";
dotenv.config();
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
const sdk = new NodeSDK({
spanProcessors: [
new ABVSpanProcessor({
apiKey: process.env.ABV_API_KEY,
baseUrl: process.env.ABV_BASE_URL,
exportMode: "immediate",
flushAt: 1,
flushInterval: 1,
additionalHeaders: {
"Content-Type": "application/json",
"Accept": "application/json"
}
})
],
});
sdk.start();
```
Import the `instrumentation.ts` file at the top of your application.
**When using the context manager:**
```typescript
import "./instrumentation"; // Must be the first import
import {
startActiveObservation,
updateActiveTrace,
} from "@abvdev/tracing";
async function main() {
await startActiveObservation("context-manager", async (span) => {
span.update({
input: { query: "What is the capital of France?" },
});
// set sessionId on the trace
updateActiveTrace({
sessionId: "session-123",
});
});
}
main();
```
**When using the **`observe`** wrapper:**
```typescript
import "./instrumentation"; // Must be the first import
import { observe, updateActiveTrace } from "@abvdev/tracing";
// An existing function
async function fetchData(source: string) {
// set sessionId on the trace
updateActiveTrace({
sessionId: "session-123",
});
// ... logic to fetch data
return { data: `some data from ${source}` };
}
// Wrap the function to trace it
const tracedFetchData = observe(fetchData, {
name: "observe-wrapper",
});
async function main() {
const result = await tracedFetchData("API");
}
main();
```
**When creating spans manually:**
```typescript
import "./instrumentation"; // Must be the first import
import { startObservation } from "@abvdev/tracing";
const span = startObservation("manual-observation", {
input: { query: "What is the capital of France?" },
});
// set sessionId on the trace
span.updateTrace({
sessionId: "session-123",
});
span.update({ output: "Paris" }).end();
```
See [JS/TS SDK docs](/developer/sdks/js-ts/overview) for more details.
# Example
*Example session spanning multiple traces*
--------------------------------------------------------------------------------
## User Tracking
URL: /developer/basic-features/user-tracking
The Users view provides an overview of all users. It also offers an in-depth look into individual users.
It's easy to map data in ABV to individual users. Just pass a unique identifier as the `userId` when you create a `trace`. This can be a username, email, or any other unique identifier. The `userId` is optional, but using it helps you get more from ABV. See the integration docs to learn more.
## Implementation with Python SDK
Install package
```bash
pip install abvdev
```
When using the @observe() decorator and initialization via constructor arguments:
```python
from abvdev import observe
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
@observe()
def process_request():
# Add to the current trace
abv.update_current_trace(user_id="user_12345")
# ...your processing logic...
return "Processing complete"
# call the function
process_request()
```
When creating spans directly:
```python
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# You can set the user_id when creating the root span via update_trace
with abv.start_as_current_span(
name="process-user-request"
) as root_span:
# Add user_id to the trace
root_span.update_trace(user_id="user_12345")
# All spans in this trace will be associated with this user
with root_span.start_as_current_observation(
as_type='generation',
name="generate-response",
model="gpt-4o"
) as gen:
# ...generate response...
pass
```
You can also update the user\_id of the current trace without a direct reference to a span:
```python
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
with abv.start_as_current_span(name="handle-user-interaction"):
# Add user_id to the current trace
abv.update_current_trace(user_id="user_12345")
```
## Implementation with JS/TS SDK
**Install packages**
```bash
npm install @abvdev/tracing @abvdev/otel @opentelemetry/sdk-node dotenv
```
**Add credentials**
Add your ABV credentials to your environment variables. Make sure that you have a `.env` file in your project root and a package like dotenv to load the variables.
.env
```typescript
ABV_API_KEY = "sk-abv-..."
ABV_BASE_URL = "https://app.abv.dev" # US region
//ABV_BASE_URL = "https://eu.app.abv.dev" # EU region
```
Create`instrumentation.ts` file and use `dotenv` package to load the variables.
Additional parameters are provided to get trace visible in the UI immediately.
instrumentation.ts
```typescript
import dotenv from "dotenv";
dotenv.config();
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
const sdk = new NodeSDK({
spanProcessors: [
new ABVSpanProcessor({
apiKey: process.env.ABV_API_KEY,
baseUrl: process.env.ABV_BASE_URL,
exportMode: "immediate",
flushAt: 1,
flushInterval: 1,
additionalHeaders: {
"Content-Type": "application/json",
"Accept": "application/json"
}
})
],
});
sdk.start();
```
Import the `instrumentation.ts` file at the top of your application.
**When using the context manager:**
```typescript
import "./instrumentation"; // Must be the first import
import {
startActiveObservation,
startObservation,
updateActiveTrace,
} from "@abvdev/tracing";
async function main() {
await startActiveObservation("context-manager", async (span) => {
span.update({
input: { query: "What is the capital of France?" },
});
updateActiveTrace({
userId: "user-123",
});
});
}
main();
```
**When using the **`observe`** wrapper:**
```typescript
import "./instrumentation"; // Must be the first import
import { observe, updateActiveTrace } from "@abvdev/tracing";
// An existing function
async function fetchData(source: string) {
updateActiveTrace({
userId: "user-123",
});
// ... logic to fetch data
return { data: `some data from ${source}` };
}
// Wrap the function to trace it
const tracedFetchData = observe(fetchData, {
name: "observe-wrapper",
});
async function main() {
const result = await tracedFetchData("API");
}
main();
```
**When creating spans manually:**
```typescript
import "./instrumentation"; // Must be the first import
import { startObservation } from "@abvdev/tracing";
const span = startObservation("manual-observation", {
input: { query: "What is the capital of France?" },
});
span.updateTrace({
userId: "user-123",
});
span.update({ output: "Paris" }).end();
```
See [JS/TS SDK docs](/developer/sdks/js-ts/overview) for more details.
## View all users
The user list provides an overview of all users that have been tracked by ABV. It makes it simple to segment by overall token usage, number of traces, and user feedback.
## Individual user view
The individual user view provides an in-depth look into a single user. Explore aggregated metrics or view all traces and feedback for a user.
You can deep link to this view via the following URL format: `https:///project/{projectId}/users/{userId}`
--------------------------------------------------------------------------------
## Model Usage & Cost Tracking
URL: /developer/basic-features/model-usage-cost-tracking
ABV provides comprehensive tracking of LLM usage and costs, helping you understand and optimize your AI application expenses. This guide will help you get started with cost tracking and point you to the right resources.
## What You Can Track
ABV monitors two key metrics for every LLM generation:
Number of units consumed per usage type (tokens, characters, etc.)
USD cost per usage type, calculated or ingested
**Supported usage types:**
- `input` / `output` - Basic token counts
- `cached_tokens` - Cached input tokens (Anthropic prompt caching)
- `audio_tokens` - Audio input tokens
- `image_tokens` - Image input tokens
- `reasoning_tokens` - Reasoning tokens (OpenAI o1 models)
- Custom usage types for your specific needs
## How It Works
ABV supports two approaches to tracking usage and costs:
### 1. Ingest from LLM Responses (Recommended)
Extract usage and cost data directly from your LLM provider's API responses and send them to ABV.
**Best for:**
- Production applications
- Accurate billing and cost allocation
- All model types, including reasoning models
### 2. Infer from Model Definitions
Let ABV automatically calculate usage and costs using predefined tokenizers and pricing.
**Best for:**
- Development and prototyping
- Supported models (OpenAI, Anthropic, Google)
- Quick setup without code changes
**Priority:** When both ingested and inferred data are available, ABV always uses ingested values for accuracy.
## Documentation Structure
This cost tracking documentation is organized into three focused guides:
**Concepts & Decision Making**
Understand ingest vs infer approaches, usage types, and when to use each method.
**Step-by-Step Guides**
Complete code examples for Python and JavaScript/TypeScript SDKs with all integration methods.
**API & Advanced Config**
Custom model definitions, tokenizers, pricing configuration, and the Models API.
## Key Features
Many ABV integrations automatically capture usage and cost from LLM responses. Works with:
- OpenAI
- Anthropic
- Google
- And more...
Track any usage metric that matters to you:
- Standard tokens (input/output)
- Cached tokens
- Audio and image tokens
- Reasoning tokens
- Custom metrics for your models
Define your own models with:
- Custom pricing per usage type
- Regex-based model matching
- Tokenization configuration
- Priority over default models
Access your usage data through:
- Real-time dashboards in the UI
- Daily Metrics API for export
- Filtering by user, tag, or application
- Integration with your billing systems
## Supported Models
ABV comes with predefined support for popular models:
- **OpenAI:** GPT-4, GPT-3.5, GPT-4o, o1 (reasoning models)
- **Anthropic:** Claude 3 family (Opus, Sonnet, Haiku)
- **Google:** Gemini models
View the complete list of supported models in the ABV UI under **Settings → Model Definitions**.
Don't see your model? You can [define custom models](/developer/basic-features/cost-tracking-reference#custom-model-definitions) or [request official support](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiJlZmZmN2JhYy0xMzZlLTNkOTUtOGIxYi1hMWQ3NWNkZWM3NGYiLCJpYXQiOjE3NTY4NDc1MDc0MDl9.Q33DbNtKAfZvsY0qs4hOfTBAzLic-VPhTrIusjkOMtI).
--------------------------------------------------------------------------------
## Cost Tracking Implementation
URL: /developer/basic-features/cost-tracking-implementation
This guide shows you how to implement cost tracking in your application using ABV's SDKs. Choose your preferred language and integration method below.
**Before you start:** Decide whether to ingest or infer usage data. See the [Cost Tracking Overview](/developer/basic-features/cost-tracking-overview) to understand the differences.
## Python SDK Implementation
### Installation
Install the required packages:
```bash
pip install abvdev
# Also install your LLM provider SDK
pip install anthropic # for Anthropic
pip install openai # for OpenAI
```
### Method 1: Using the Decorator (Recommended)
The decorator approach is clean and integrates seamlessly with your existing functions.
```python
from abvdev import ABV, observe
import anthropic
# ABV client initialization
abv = ABV(
api_key="sk-abv-...",
host="https://app.abv.dev", # or "https://eu.app.abv.dev" for EU
)
# Anthropic client initialization
anthropic_client = anthropic.Anthropic(api_key="sk-ant-...")
@observe(as_type="generation")
def anthropic_completion(**kwargs):
# Extract fields from kwargs
kwargs_clone = kwargs.copy()
input = kwargs_clone.pop('messages', None)
model = kwargs_clone.pop('model', None)
# Set initial generation data
abv.update_current_generation(
input=input,
model=model,
metadata=kwargs_clone
)
# Make the LLM call
response = anthropic_client.messages.create(**kwargs)
# Update with usage and cost details
abv.update_current_generation(
usage_details={
"input": response.usage.input_tokens,
"output": response.usage.output_tokens,
"cache_read_input_tokens": response.usage.cache_read_input_tokens
# "total" is automatically derived if not set
},
# Optional: Ingest cost if you calculate it yourself
# Otherwise, ABV will infer it from model definitions
cost_details={
"input": 1.0,
"cache_read_input_tokens": 0.5,
"output": 1.0,
# "total" is automatically derived if not set
}
)
return response.content[0].text
@observe()
def main():
return anthropic_completion(
model="claude-sonnet-4-5",
max_tokens=1024,
messages=[
{"role": "user", "content": "Hello, Claude"}
]
)
main()
```
```python
from abvdev import ABV, observe
from openai import OpenAI
# ABV client initialization
abv = ABV(
api_key="sk-abv-...",
host="https://app.abv.dev",
)
openai_client = OpenAI(api_key="sk-proj-...")
@observe(as_type="generation")
def openai_completion(**kwargs):
kwargs_clone = kwargs.copy()
messages = kwargs_clone.pop('messages', None)
model = kwargs_clone.pop('model', None)
abv.update_current_generation(
input=messages,
model=model,
metadata=kwargs_clone
)
response = openai_client.chat.completions.create(**kwargs)
# Use OpenAI-style usage schema (automatically mapped by ABV)
abv.update_current_generation(
usage_details={
"prompt_tokens": response.usage.prompt_tokens,
"completion_tokens": response.usage.completion_tokens,
"total_tokens": response.usage.total_tokens,
"prompt_tokens_details": {
"cached_tokens": response.usage.prompt_tokens_details.cached_tokens,
"audio_tokens": response.usage.prompt_tokens_details.audio_tokens,
},
"completion_tokens_details": {
"reasoning_tokens": response.usage.completion_tokens_details.reasoning_tokens,
},
}
)
return response.choices[0].message.content
@observe()
def main():
return openai_completion(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello, OpenAI"}]
)
main()
```
**OpenAI Usage Schema:** When using OpenAI-style usage details, ABV automatically maps:
- `prompt_tokens` → `input`
- `completion_tokens` → `output`
- `total_tokens` → `total`
- `prompt_tokens_details.*` → `input_*`
- `completion_tokens_details.*` → `output_*`
### Method 2: Manual Context Manager
For more control, use the context manager approach:
```python
from abvdev import ABV
import anthropic
abv = ABV(
api_key="sk-abv-...",
host="https://app.abv.dev",
)
anthropic_client = anthropic.Anthropic(api_key="sk-ant-...")
with abv.start_as_current_observation(
as_type='generation',
name="anthropic-completion",
model="claude-3-haiku-20240307",
input=[{"role": "user", "content": "Hello, Claude"}]
) as generation:
# Make the LLM call
response = anthropic_client.messages.create(
model="claude-3-haiku-20240307",
max_tokens=1024,
messages=[{"role": "user", "content": "Hello, Claude"}]
)
# Update with response and usage
generation.update(
output=response.content[0].text,
usage_details={
"input": response.usage.input_tokens,
"output": response.usage.output_tokens,
"cache_read_input_tokens": response.usage.cache_read_input_tokens
},
# Optional: Add cost details
cost_details={
"input": 1.0,
"cache_read_input_tokens": 0.5,
"output": 1.0,
}
)
```
## JavaScript/TypeScript SDK Implementation
### Installation
Install the required packages:
```bash
npm install @abvdev/tracing @abvdev/otel @opentelemetry/sdk-node dotenv openai
```
### Setup
**1. Add credentials to `.env`:**
```bash
ABV_API_KEY="sk-abv-..."
ABV_BASE_URL="https://app.abv.dev" # or "https://eu.app.abv.dev" for EU
OPENAI_API_KEY="sk-proj-..."
```
**2. Create `instrumentation.ts`:**
```typescript
import dotenv from "dotenv";
dotenv.config();
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
const sdk = new NodeSDK({
spanProcessors: [
new ABVSpanProcessor({
apiKey: process.env.ABV_API_KEY,
baseUrl: process.env.ABV_BASE_URL,
exportMode: "immediate",
flushAt: 1,
flushInterval: 1,
additionalHeaders: {
"Content-Type": "application/json",
"Accept": "application/json"
}
})
],
});
sdk.start();
```
**Important:** Import `instrumentation.ts` as the first import in your application to ensure proper initialization.
### Method 1: Context Manager (Recommended)
```typescript
import "./instrumentation"; // Must be first!
import {
startActiveObservation,
startObservation,
} from "@abvdev/tracing";
import OpenAI from "openai";
import dotenv from "dotenv";
dotenv.config();
const openai = new OpenAI({
apiKey: process.env.OPENAI_API_KEY,
});
async function main() {
await startActiveObservation("user-request", async (span) => {
span.update({
input: { query: "What is the capital of France?" },
});
const model = "gpt-4o";
const input = [{ role: "user", content: "What is the capital of France?" }];
// Create a generation observation
const generation = startObservation(
"llm-call",
{
model: model,
input: input,
},
{ asType: "generation" }
);
// Make the LLM call
const response = await openai.chat.completions.create({
messages: input,
model: model,
});
const llmOutput = response.choices[0].message.content;
// Update with usage details
generation.update({
usageDetails: {
prompt_tokens: response.usage.prompt_tokens,
completion_tokens: response.usage.completion_tokens,
total_tokens: response.usage.total_tokens,
prompt_tokens_details: {
cached_tokens: response.usage.prompt_tokens_details.cached_tokens,
audio_tokens: response.usage.prompt_tokens_details.audio_tokens,
},
completion_tokens_details: {
reasoning_tokens: response.usage.completion_tokens_details.reasoning_tokens,
},
},
output: { content: llmOutput },
});
generation.end();
});
}
main();
```
### Method 2: Observe Wrapper
Wrap existing functions to trace them automatically:
```typescript
import "./instrumentation"; // Must be first!
import { observe, updateActiveObservation } from "@abvdev/tracing";
import OpenAI from "openai";
import dotenv from "dotenv";
dotenv.config();
const openai = new OpenAI({
apiKey: process.env.OPENAI_API_KEY,
});
// Existing function
async function fetchData(source: string) {
const model = "gpt-4o";
const input = [{ role: "user", content: "What is the capital of France?" }];
const response = await openai.chat.completions.create({
messages: input,
model: model,
});
const llmOutput = response.choices[0].message.content;
// Update the active observation with usage details
updateActiveObservation(
"generation",
{
usageDetails: {
prompt_tokens: response.usage.prompt_tokens,
completion_tokens: response.usage.completion_tokens,
total_tokens: response.usage.total_tokens,
prompt_tokens_details: {
cached_tokens: response.usage.prompt_tokens_details.cached_tokens,
audio_tokens: response.usage.prompt_tokens_details.audio_tokens,
},
completion_tokens_details: {
reasoning_tokens: response.usage.completion_tokens_details.reasoning_tokens,
},
},
output: { content: llmOutput },
}
);
return { data: llmOutput };
}
// Wrap the function to trace it
const tracedFetchData = observe(fetchData, {
name: "fetch-data",
asType: "generation",
});
async function main() {
const result = await tracedFetchData("API");
}
main();
```
### Method 3: Manual Span Creation
For maximum control, create spans manually:
```typescript
import "./instrumentation"; // Must be first!
import { startObservation } from "@abvdev/tracing";
import OpenAI from "openai";
import dotenv from "dotenv";
dotenv.config();
const openai = new OpenAI({
apiKey: process.env.OPENAI_API_KEY,
});
async function main() {
const span = startObservation("manual-observation", {
input: { query: "What is the capital of France?" },
});
const model = "gpt-4o";
const input = [{ role: "user", content: "What is the capital of France?" }];
const response = await openai.chat.completions.create({
messages: input,
model: model,
});
const llmOutput = response.choices[0].message.content;
// Create a child generation observation
const generation = span.startObservation(
"llm-call",
{
model: model,
input: input,
},
{ asType: "generation" }
);
// Update with usage details
generation.update({
usageDetails: {
prompt_tokens: response.usage.prompt_tokens,
completion_tokens: response.usage.completion_tokens,
total_tokens: response.usage.total_tokens,
prompt_tokens_details: {
cached_tokens: response.usage.prompt_tokens_details.cached_tokens,
audio_tokens: response.usage.prompt_tokens_details.audio_tokens,
},
completion_tokens_details: {
reasoning_tokens: response.usage.completion_tokens_details.reasoning_tokens,
},
},
output: { content: llmOutput },
});
generation.end();
span.update({ output: "Successfully answered user request." }).end();
}
main();
```
## Automatic Usage Capture via Integrations
Many ABV integrations automatically capture usage and cost from LLM responses. If you're using an integration and usage data isn't appearing as expected, contact support.
Check the [integrations documentation](/developer/sdks/python/overview) to see which providers support automatic usage capture.
## Ingesting Only Usage (Let ABV Calculate Cost)
You don't need to calculate costs yourself. Simply ingest usage details, and ABV will calculate costs based on model definitions:
```python
# Python example - only usage, no cost
generation.update(
usage_details={
"input": response.usage.input_tokens,
"output": response.usage.output_tokens,
}
# No cost_details - ABV will infer cost from model pricing
)
```
```typescript
// TypeScript example - only usage, no cost
generation.update({
usageDetails: {
prompt_tokens: response.usage.prompt_tokens,
completion_tokens: response.usage.completion_tokens,
}
// No costDetails - ABV will infer cost from model pricing
});
```
## Next Steps
Learn how to define custom models and pricing
Export usage and cost data for analytics
## Common Issues
**Check these:**
1. Verify you're passing usage_details in the update call
2. Ensure the model name matches a model definition in ABV
3. For inference: Check if your model is supported in the model definitions list
4. Look for errors in your application logs
**Possible causes:**
1. Using inference instead of ingestion (switch to ingesting actual usage)
2. Model pricing may have changed (update model definitions)
3. Custom models need pricing configured
4. For historical data: Model definitions aren't applied retroactively
**Solution:**
Reasoning models (like OpenAI o1) require ingested usage. ABV cannot infer usage because reasoning tokens are generated internally.
Always ingest `reasoning_tokens` from the API response for these models.
[Learn more about reasoning models](/developer/basic-features/cost-tracking-reference#reasoning-models)
--------------------------------------------------------------------------------
## Cost Tracking Reference
URL: /developer/basic-features/cost-tracking-reference
This reference guide covers advanced topics: configuring custom models, understanding tokenizers, handling reasoning models, and using the Public API to manage model definitions.
## Supported Tokenizers
ABV uses tokenizers to infer usage when token counts aren't provided. The following tokenizers are currently supported:
| Model Pattern | Tokenizer | Package | Notes |
|---------------|-----------|---------|-------|
| `gpt-4o*` | `o200k_base` | Built-in | Used for GPT-4o models |
| `gpt*` | `cl100k_base` | Built-in | Used for GPT-4, GPT-3.5, and other GPT models |
| `claude*` | `claude` | [@anthropic-ai/tokenizer](https://www.npmjs.com/package/@anthropic-ai/tokenizer) | Not fully accurate for Claude 3 models per Anthropic |
According to Anthropic, their tokenizer is not fully accurate for Claude 3 models. For best results, ingest token counts directly from the Anthropic API response.
## Predefined Models
ABV maintains a list of predefined popular models with tokenizers and pricing for:
- **OpenAI** - GPT-4, GPT-3.5, GPT-4o, and other models
- **Anthropic** - Claude 3 family (Opus, Sonnet, Haiku)
- **Google** - Gemini models
View the complete list of predefined models in the ABV UI (sign-in required). Navigate to **Settings → Model Definitions**.
## Custom Model Definitions
You can add your own model definitions for:
- Custom or proprietary models
- Fine-tuned models
- Models not yet supported by ABV
- Overriding default pricing
### Creating Models via the UI
1. Navigate to **Settings → Model Definitions** in the ABV UI
2. Click **Add Model**
3. Configure the model settings (see configuration details below)
4. Save the model definition
### Creating Models via the API
Use the Models API to programmatically manage model definitions:
```bash
# List all models
GET /api/public/models
# Create a new model
POST /api/public/models
# Get a specific model
GET /api/public/models/{id}
# Delete a model
DELETE /api/public/models/{id}
```
[View the complete Public API documentation](/developer/platform/api-data-platform/public-api)
## Model Configuration
### Model Matching
Models are matched to generations using regular expressions:
| Generation Attribute | Model Attribute | Description |
|---------------------|----------------|-------------|
| `model` | `match_pattern` | Regex pattern to match model names |
**Example patterns:**
```regex
# Exact match for GPT-4
(?i)^(gpt-4-0125-preview)$
# Match all GPT-4 variants
(?i)^gpt-4.*$
# Match custom model family
(?i)^my-custom-model-.*$
```
User-defined models take priority over ABV-maintained models. This allows you to override default pricing or add support for new models.
### Pricing Configuration
Define pricing per usage type. Usage types must match exactly with the keys in the `usage_details` object:
```json
{
"pricing": {
"input": 0.00001, // Price per input token in USD
"output": 0.00003, // Price per output token in USD
"cached_tokens": 0.000001 // Price per cached token in USD
}
}
```
**Usage type matching:**
- Usage types are case-sensitive
- Must match exactly: `"input"` ≠ `"Input"`
- Supports arbitrary usage types for custom metrics
### Tokenization Configuration
For models using the `openai` tokenizer, specify the tokenization config:
```json
{
"tokenizer": "openai",
"tokenization_config": {
"tokenizerModel": "gpt-4-0125-preview", // tiktoken model name
"tokensPerMessage": 4, // Required for chat models
"tokensPerName": -1 // Required for chat models
}
}
```
Copy tokenization config from predefined OpenAI models in the UI. See the [OpenAI tokenization guide](https://platform.openai.com/docs/guides/text-generation) for more details on `tokensPerMessage` and `tokensPerName`.
## Complete Model Definition Example
Here's a complete example of a custom model definition:
```json
{
"name": "My Custom GPT-4 Model",
"match_pattern": "(?i)^my-custom-gpt-4$",
"tokenizer": "openai",
"tokenization_config": {
"tokenizerModel": "gpt-4-0125-preview",
"tokensPerMessage": 4,
"tokensPerName": -1
},
"pricing": {
"input": 0.00001,
"output": 0.00003
}
}
```
```bash
curl -X POST "https://app.abv.dev/api/public/models" \
-H "Authorization: Bearer sk-abv-..." \
-H "Content-Type: application/json" \
-d '{
"name": "My Custom GPT-4 Model",
"match_pattern": "(?i)^my-custom-gpt-4$",
"tokenizer": "openai",
"tokenization_config": {
"tokenizerModel": "gpt-4-0125-preview",
"tokensPerMessage": 4,
"tokensPerName": -1
},
"pricing": {
"input": 0.00001,
"output": 0.00003
}
}'
```
## Reasoning Models
Reasoning models (like OpenAI's o1 family) require special handling.
### Why Inference Doesn't Work
Reasoning models take multiple internal steps to arrive at a response:
1. Model generates reasoning tokens (internal thought process)
2. Model generates completion tokens (final response)
3. API bills you for: reasoning tokens + completion tokens
**The problem:** ABV cannot see the internal reasoning tokens, so it cannot accurately infer costs by tokenizing only the visible input and output.
### Solution: Always Ingest Usage
For reasoning models, you **must** ingest usage details from the API response:
```python
# Python example for OpenAI o1
response = openai_client.chat.completions.create(
model="o1-preview",
messages=[{"role": "user", "content": "Solve this problem..."}]
)
generation.update(
usage_details={
"prompt_tokens": response.usage.prompt_tokens,
"completion_tokens": response.usage.completion_tokens,
"completion_tokens_details": {
"reasoning_tokens": response.usage.completion_tokens_details.reasoning_tokens
}
}
)
```
**Critical:** If you don't ingest usage for reasoning models, ABV cannot calculate accurate costs. The tokenized input + output will significantly underestimate actual usage.
[Learn more about OpenAI reasoning models](https://platform.openai.com/docs/guides/reasoning)
## Historical Data and Model Changes
### Retroactive Application
ABV does **not** retroactively apply model definition changes to historical generations. This includes:
- Price updates
- New tokenizers
- Changed model patterns
**Why:** Costs are calculated at ingestion time using the model and price information available at that moment. This ensures historical data remains accurate and auditable.
### Batch Reprocessing
If you need to apply new model definitions to existing generations:
1. Contact ABV support
2. Request a batch job to reprocess generations
3. Specify the time range and model changes
Batch reprocessing is typically used when:
- Correcting a pricing error
- Adding missing model definitions
- Migrating to new tokenizers
## OpenAI Usage Schema Compatibility
ABV supports the OpenAI usage schema for compatibility with OpenAI SDKs and tools.
### Automatic Mapping
When you use OpenAI field names, ABV automatically maps them:
| OpenAI Field | ABV Field |
|--------------|-----------|
| `prompt_tokens` | `input` |
| `completion_tokens` | `output` |
| `total_tokens` | `total` |
| `prompt_tokens_details.*` | `input_*` |
| `completion_tokens_details.*` | `output_*` |
**Example:**
```python
# You send this (OpenAI schema):
usage_details={
"prompt_tokens": 100,
"completion_tokens": 50,
"prompt_tokens_details": {
"cached_tokens": 20
}
}
# ABV stores this:
{
"input": 100,
"output": 50,
"input_cached_tokens": 20,
"total": 150 # automatically calculated
}
```
This makes it easy to integrate with OpenAI without changing your existing code.
## Requesting Official Model Support
Don't want to maintain custom model definitions? Request official support:
1. Visit the [ABV Model Support Request Form](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiJlZmZmN2JhYy0xMzZlLTNkOTUtOGIxYi1hMWQ3NWNkZWM3NGYiLCJpYXQiOjE3NTY4NDc1MDc0MDl9.Q33DbNtKAfZvsY0qs4hOfTBAzLic-VPhTrIusjkOMtI)
2. Provide model details and documentation
3. ABV will add official support in a future release
Official support includes:
- Automatic tokenization
- Pricing updates
- Maintenance and updates
## Troubleshooting
### Usage and Cost Missing for Historical Data
**Problem:** After adding a new model definition, historical generations still show no usage or cost.
**Solution:** Model definitions are not applied retroactively. Options:
1. Request a batch reprocessing job from support
2. Accept that historical data uses old definitions
3. For future data: Ingest usage directly instead of relying on inference
### Incorrect Costs
**Problem:** Costs don't match your LLM provider's billing.
**Checklist:**
1. **Check pricing:** Verify the model definition pricing matches your provider's current rates
2. **Check usage types:** Ensure usage type keys match exactly (case-sensitive)
3. **Check model matching:** Verify the regex pattern correctly matches your model name
4. **Consider timing:** Prices are applied at ingestion time, not retroactively
**Best practice:** For billing-critical applications, always ingest both usage AND cost from the provider's API response.
### Model Not Matching
**Problem:** Generations aren't matching your custom model definition.
**Debug steps:**
1. Check the `model` field in your generation matches the `match_pattern` regex
2. Test the regex pattern in a regex tester
3. Verify user-defined model isn't being overridden by ABV model
4. Check for typos in the model name
**Example:**
```javascript
// Generation model
"model": "gpt-4-custom-v1"
// Match pattern (correct)
"match_pattern": "(?i)^gpt-4-custom-v1$"
// Match pattern (incorrect - won't match)
"match_pattern": "^gpt-4-custom-v1$" // missing (?i) flag
```
### Tokenization Errors
**Problem:** Tokenization failing or producing unexpected results.
**Solutions:**
1. **For Claude models:** Ingest token counts from API instead of inferring
2. **For OpenAI models:** Verify `tokensPerMessage` and `tokensPerName` are set correctly
3. **For custom tokenizers:** Ensure the tokenizer is supported (see supported tokenizers table)
## API Reference
### Models API Endpoints
**List all models:**
```bash
GET /api/public/models
```
**Create a model:**
```bash
POST /api/public/models
Content-Type: application/json
{
"name": "string",
"match_pattern": "string",
"tokenizer": "string",
"tokenization_config": {},
"pricing": {}
}
```
**Get a specific model:**
```bash
GET /api/public/models/{id}
```
**Delete a model:**
```bash
DELETE /api/public/models/{id}
```
[View complete API documentation](/developer/platform/api-data-platform/public-api)
## Next Steps
Return to implementation examples
Export usage and cost data
--------------------------------------------------------------------------------
## Sampling
URL: /developer/basic-features/sampling
Sampling can be used to control the volume of traces collected by ABV. Sampling is handled client-side.
You can configure the sample rate by setting the `ABV_SAMPLE_RATE` environment variable or by using the `sample_rate`/`sampleRate` constructor parameter. The value has to be between 0 and 1.
The default value is 1, meaning that all traces are collected. A value of 0.2 means that only 20% of the traces are collected. The SDK samples on the trace level meaning that if a trace is sampled, all observations and scores within that trace will be sampled as well.
## Implementation with Python SDK
Install packages
```bash
pip install abvdev, python-dotenv
```
Add credentials and **ABV\_SAMPLE\_RATE** environment variable:
Add your ABV credentials to your environment variables. Make sure that you have a `.env` file in your project root.
```bash title=".env"
ABV_API_KEY = "sk-abv-..."
ABV_HOST = "https://app.abv.dev" # US region
# ABV_HOST = "https://eu.app.abv.dev" # EU region
ABV_SAMPLE_RATE = "0.5"
```
load environment variables and initialize client with get\_client:
```python
# Import the function to load the .env file
from dotenv import load_dotenv
from abvdev import observe, get_client
# Load the environment variables from the .env file
load_dotenv()
@observe
def my_function():
return "Hello, world!" # Input/output and timings are automatically captured
my_function()
# Flush events in short-lived applications
abv = get_client()
abv.flush()
```
Also with Python SDK, you can configure sampling when initializing the client:
```python
from abvdev import ABV, observe
# ABV client initialization - # Method 2: Initialize with constructor parameter
abv = ABV(
sample_rate=0.5, # 50% of traces will be sampled
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
@observe
def my_function():
return "Hello, world!" # Input/output and timings are automatically captured
my_function()
# Flush events in short-lived applications
abv.flush()
```
If a trace is not sampled, none of its observations (spans or generations) or associated scores will be sent to ABV, which can significantly reduce data volume for high-traffic applications.
## Implementation with JS/TS SDK
**Install packages**
```bash
npm install @abvdev/tracing @abvdev/otel @opentelemetry/sdk-node dotenv
```
**Add credentials**
Add your ABV credentials to your environment variables. Make sure that you have a `.env` file in your project root and a package like dotenv to load the variables.
```bash title=".env"
ABV_API_KEY = "sk-abv-..."
ABV_BASE_URL = "https://app.abv.dev" # US region
//ABV_BASE_URL = "https://eu.app.abv.dev" # EU region
```
Create`instrumentation.ts` file and use `dotenv` package to load the variables.
Additional parameters are provided to get trace visible in the UI immediately.
ABV respects OpenTelemetry's sampling decisions. You can configure a sampler in your OTEL SDK to control which traces are sent to ABV. This is useful for managing costs and reducing noise in high-volume applications.
Here is an example of how to configure a `TraceIdRatioBasedSampler` to send only 20% of traces:
```typescript title="instrumentation.ts"
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
import { TraceIdRatioBasedSampler } from "@opentelemetry/sdk-trace-base";
import dotenv from "dotenv";
dotenv.config();
const sdk = new NodeSDK({
// Sample 20% of all traces
sampler: new TraceIdRatioBasedSampler(0.2),
spanProcessors: [
new ABVSpanProcessor({
apiKey: process.env.ABV_API_KEY,
baseUrl: process.env.ABV_BASE_URL,
exportMode: "immediate",
flushAt: 1,
flushInterval: 1,
additionalHeaders: {
"Content-Type": "application/json",
"Accept": "application/json"
}
})
],
});
sdk.start();
```
Import the `instrumentation.ts` file at the top of your application.
See [JS/TS SDK docs](/developer/sdks/js-ts/overview) for more details.
--------------------------------------------------------------------------------
## Event Queuing/Batching
URL: /developer/basic-features/event-queuing-batching
ABV's client SDKs and integrations are all designed to queue and batch requests in the background to optimize API calls and network time. Batches are determined by a combination of time and size (number of events and size of batch).
## Configuration
All integrations have a sensible default configuration, but you can customize the batching behaviour to suit your needs.
| Option (Python) \[SDK constructor, Environment] | Option (JS) | Description |
| ----------------------------------------------- | -------------------- | -------------------------------------------------------- |
| `flush_at`, `ABV_FLUSH_AT` | `flushAt` | The maximum number of events to batch up before sending. |
| `flush_interval`, `ABV_FLUSH_INTERVAL` (s) | `flushInterval` (ms) | The maximum time to wait before sending a batch. |
You can e.g. set `flushAt=1` to send every event immediately, or `flushInterval=1000` to send every second.
## Manual flushing
This is especially relevant for short-lived applications like serverless functions. If you do not flush the client, you may lose events.
If you want to send a batch immediately, you can call the `flush` method on the client. In case of network issues, flush will log an error and retry the batch, it will never throw an exception.
### Python SDK
Install package
```bash
pip install abvdev
```
```python
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# Flush all pending observations
abv.flush()
```
If you exit the application, use `shutdown` method to make sure all requests are flushed and pending requests are awaited before the process exits. On success of this function, no more events will be sent to ABV API.
```python
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
abv.shutdown()
```
### JS/TS SDK
**Install packages**
```bash
npm install @abvdev/tracing @abvdev/otel @opentelemetry/sdk-node dotenv
```
**Add credentials**
Add your ABV credentials to your environment variables. Make sure that you have a `.env` file in your project root and a package like dotenv to load the variables.
.env
```typescript
ABV_API_KEY = "sk-abv-..."
ABV_BASE_URL = "https://app.abv.dev" # US region
//ABV_BASE_URL = "https://eu.app.abv.dev" # EU region
```
Create`instrumentation.ts` file and use `dotenv` package to load the variables.
Additional parameters are provided to get trace visible in the UI immediately.
instrumentation.ts
```typescript
import dotenv from "dotenv";
dotenv.config();
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
const sdk = new NodeSDK({
spanProcessors: [
new ABVSpanProcessor({
apiKey: process.env.ABV_API_KEY,
baseUrl: process.env.ABV_BASE_URL,
exportMode: "immediate",
flushAt: 1,
flushInterval: 1,
additionalHeaders: {
"Content-Type": "application/json",
"Accept": "application/json"
}
})
],
});
sdk.start();
```
Import the `instrumentation.ts` file at the top of your application.
**The **`ABVSpanProcessor`** buffers events and sends them in batches, so a final flush ensures no data is lost.**
You can export the processor from your OTEL SDK setup file.
instrumentation.ts
```typescript
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
// Export the processor to be able to flush it
export const abvSpanProcessor = new ABVSpanProcessor();
const sdk = new NodeSDK({
spanProcessors: [abvSpanProcessor],
});
sdk.start();
```
Then, in your serverless function handler, call forceFlush() before the function exits.
handler.ts
```typescript
import { abvSpanProcessor } from "./instrumentation";
export async function handler(event, context) {
// ... your application logic ...
// Flush before exiting
await abvSpanProcessor.forceFlush();
}
```
--------------------------------------------------------------------------------
## Masking of Sensitive LLM Data
URL: /developer/basic-features/masking-sensitive-data
Masking is a feature that allows precise control over the tracing data sent to the ABV server. With custom masking functions, you can control and sanitize the data that gets traced and sent to the server. Whether it's for **compliance reasons** or to protect **user privacy**, masking sensitive data is a crucial step in responsible application development. It enables you to:
1. Redact sensitive information from trace or observation inputs and outputs.
2. Customize the content of events before transmission.
3. Implement fine-grained data filtering based on your specific requirements.
Learn more about ABV's data security and privacy measures concerning the stored data in our [security and compliance overview](/developer/references/security-compliance-overview).
# How it works
1. You define a custom masking function and pass it to the ABV client constructor.
2. All event inputs and outputs are processed through this function.
3. The masked data is then sent to the ABV server.
This approach ensures that you have complete control over the event input and output data traced by your application.
## Python SDK
Install package
```bash
pip install abvdev
```
Define a masking function before the client initialization. The masking function will apply to all event inputs and outputs regardless of the ABV-maintained integration you are using.
```python
def masking_function(data: any, **kwargs) -> any:
"""Function to mask sensitive data before sending to ABV."""
if isinstance(data, str) and data.startswith("SECRET_"):
return "REDACTED"
# For more complex data structures
elif isinstance(data, dict):
return {k: masking_function(v) for k, v in data.items()}
elif isinstance(data, list):
return [masking_function(item) for item in data]
return data
```
Apply the masking function when initializing the ABV client:
```python
from abvdev import ABV
# Initialize with masking function
abv = ABV(mask=masking_function)
```
So it will look like below, with the decorator:
```python
from abvdev import ABV, observe
abv = ABV(
mask=masking_function,
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev" # host="https://eu.app.abv.dev", for EU region
)
@observe()
def my_function():
# This data will be masked before being sent to ABV
return "SECRET_DATA"
result = my_function()
print(result) # Original: "SECRET_DATA"
# The trace output in ABV will have the output masked as "REDACTED"
```
Using context managers:
```python
from abvdev import ABV
abv = ABV(
mask=masking_function,
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev" # host="https://eu.app.abv.dev", for EU region
)
with abv.start_as_current_span(
name="sensitive-operation",
input="SECRET_INPUT_DATA"
) as span:
# ... processing ...
span.update(output="SECRET_OUTPUT_DATA")
# Both input and output will be masked as "REDACTED" in ABV
```
## JS/TS SDK
**Install packages**
```bash
npm install @abvdev/tracing @abvdev/otel @opentelemetry/sdk-node dotenv
```
**Add credentials**
Add your ABV credentials to your environment variables. Make sure that you have a `.env` file in your project root and a package like dotenv to load the variables.
.env
```typescript
ABV_API_KEY = "sk-abv-..."
ABV_BASE_URL = "https://app.abv.dev" # US region
//ABV_BASE_URL = "https://eu.app.abv.dev" # EU region
```
Create`instrumentation.ts` file and use `dotenv` package to load the variables.
Additional parameters are provided to get trace visible in the UI immediately.
instrumentation.ts
```typescript
import dotenv from "dotenv";
dotenv.config();
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
const sdk = new NodeSDK({
spanProcessors: [
new ABVSpanProcessor({
apiKey: process.env.ABV_API_KEY,
baseUrl: process.env.ABV_BASE_URL,
exportMode: "immediate",
flushAt: 1,
flushInterval: 1,
additionalHeaders: {
"Content-Type": "application/json",
"Accept": "application/json"
}
})
],
});
sdk.start();
```
Import the `instrumentation.ts` file at the top of your application.
**To prevent sensitive data from being sent to** ABV, you can provide a `mask` function to the `ABVSpanProcessor`. This function will be applied to the `input`, `output`, and `metadata `of every observation.
The function receives an object `{ data }`, where data is the stringified JSON of the attribute’s value. It should return the masked data.
instrumentation.ts
```typescript
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
import dotenv from "dotenv";
dotenv.config();
const sdk = new NodeSDK({
spanProcessors: [
new ABVSpanProcessor({
apiKey: process.env.ABV_API_KEY,
baseUrl: process.env.ABV_BASE_URL,
exportMode: "immediate",
flushAt: 1,
flushInterval: 1,
additionalHeaders: {
"Content-Type": "application/json",
"Accept": "application/json"
},
mask: ({ data }) => {
// A simple regex to mask credit card numbers
const maskedData = data.replace(
/\b\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}\b/g,
"***MASKED_CREDIT_CARD***"
);
return maskedData;
},
})
],
});
sdk.start();
```
Simple example with masked data
```typescript
import "./instrumentation_masked";
import { startActiveObservation } from "@abvdev/tracing";
async function main() {
await startActiveObservation("my-masked-trace", async (span) => {
span.update({
input: "Hello, ABV! My card number is 5555-7777-3333-1111 ",
output: "This is my first trace! And my card number is 1111-1111-1111-1111",
});
});
}
main();
```
Masked data will be shown in UI:
*Input: "Hello, ABV! My card number is \*\*\*MASKED\_CREDIT\_CARD\*\*\* "*
*Output: "This is my first trace! And my card number is \*\*\*MASKED\_CREDIT\_CARD\*\*\*"*
See [JS/TS SDK docs](/developer/sdks/js-ts/overview) for more details.
# Examples
Now, we'll show you examples how to use the masking feature. We'll use the ABV decorator for this, but you can also use the low-level SDK or the JS/TS SDK analogously.
## Example 1: Redacting Credit Card Numbers
In this example, we'll demonstrate how to redact credit card numbers from strings using a [regular expression](https://docs.python.org/3/library/re.html). This helps in complying with PCI DSS by ensuring that credit card numbers are not transmitted or stored improperly.
ABVs masking feature allows you to define a custom masking function with parameters, which you then pass to the ABV client constructor. This function is applied to all event inputs and outputs, processing each piece of data to mask or redact sensitive information according to your specifications. By ensuring that all events are processed through your masking function before being sent, ABV guarantees that only the masked data is transmitted to the ABV server.
**Steps:**
1. **Import necessary modules**.
2. **Define a masking function** that uses a regular expression to detect and replace credit card numbers.
3. **Configure the masking function** in ABV.
4. **Create a sample function** to simulate processing sensitive data.
5. **Observe the trace** to see the masked output.
```python
import re
from abvdev import ABV, observe
# Step 2: Define the masking function
def masking_function(data, **kwargs):
if isinstance(data, str):
# Regular expression to match credit card numbers (Visa, MasterCard, AmEx, etc.)
pattern = r'\b(?:\d[ -]*?){13,19}\b'
data = re.sub(pattern, '[REDACTED CREDIT CARD]', data)
return data
# Step 3: Configure the masking function
abv = ABV(
mask=masking_function,
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev" # host="https://eu.app.abv.dev", for EU region
)
# Step 4: Create a sample function with sensitive data
@observe()
def process_payment():
# Simulated sensitive data containing a credit card number
transaction_info = "Customer paid with card number 4111 1111 1111 1111."
return transaction_info
# Step 5: Observe the trace
result = process_payment()
print(result)
# Output: Customer paid with card number [REDACTED CREDIT CARD].
# Flush events in short-lived applications
abv.flush()
```
## Example 2: Using the `llm-guard` library
In this example, we'll use the `Anonymize` scanner from `llm-guard` to remove personal names and other PII from the data. This is useful for anonymizing user data and protecting privacy.
Find our more about the `llm-guard` library in their [documentation](https://llm-guard.com/).
**Steps:**
1. **Install the **`llm-guard`** library**.
2. **Import necessary modules**.
3. **Initialize the Vault and configure the Anonymize scanner**.
4. **Define a masking function** that uses the Anonymize scanner.
5. **Configure the masking function** in ABV.
6. **Create a sample function** to simulate processing data with PII.
7. **Observe the trace** to see the masked output.
```bash
pip install llm-guard
```
```python
from abvdev import ABV, observe
from llm_guard.vault import Vault
from llm_guard.input_scanners import Anonymize
from llm_guard.input_scanners.anonymize_helpers import BERT_LARGE_NER_CONF
# Step 3: Initialize the Vault and configure the Anonymize scanner
vault = Vault()
def create_anonymize_scanner():
scanner = Anonymize(
vault,
recognizer_conf=BERT_LARGE_NER_CONF,
language="en"
)
return scanner
# Step 4: Define the masking function
def masking_function(data, **kwargs):
if isinstance(data, str):
scanner = create_anonymize_scanner()
# Scan and redact the data
sanitized_data, is_valid, risk_score = scanner.scan(data)
return sanitized_data
return data
# Step 5: Configure the masking function
abv = ABV(
mask=masking_function,
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev" # host="https://eu.app.abv.dev", for EU region
)
# Step 6: Create a sample function with PII
@observe()
def generate_report():
# Simulated data containing personal names
report = "John Doe met with Jane Smith to discuss the project."
return report
# Step 7: Observe the trace
result = generate_report()
print(result)
# Output: [REDACTED_PERSON] met with [REDACTED_PERSON] to discuss the project.
# Flush events in short-lived applications
abv.flush()
```
## Example 3: Masking Email and Phone Numbers
You can extend the masking function to redact other types of PII such as email addresses and phone numbers using regular expressions.
```python
import re
from abvdev import ABV, observe
def masking_function(data, **kwargs):
if isinstance(data, str):
# Mask email addresses
data = re.sub(r'\b[\w.-]+?@\w+?\.\w+?\b', '[REDACTED EMAIL]', data)
# Mask phone numbers
data = re.sub(r'\b\d{3}[-. ]?\d{3}[-. ]?\d{4}\b', '[REDACTED PHONE]', data)
return data
abv = ABV(
mask=masking_function,
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev" # host="https://eu.app.abv.dev", for EU region
)
@observe()
def contact_customer():
info = "Please contact John at john.doe@example.com or call 555-123-4567."
return info
result = contact_customer()
print(result)
# Output: Please contact John at [REDACTED EMAIL] or call [REDACTED PHONE].
# Flush events in short-lived applications
abv.flush()
```
--------------------------------------------------------------------------------
## Releases & Versioning
URL: /developer/basic-features/releases-versioning
You can track the effect of changes to your LLM app on metrics in ABV. This allows you to:
- **Run experiments (A/B tests)** in production and measure the impact on costs, latencies and quality.
- *Example*: "What is the impact of switching to a new model?"
- **Explain changes to metrics** over time.
- *Example:* "Why did latency in this chain increase?"
# Releases
A `release` tracks the overall version of your application. Commonly it is set to the *semantic version* or *git commit hash* of your application.
The SDKs look for a `release` in the following order:
1. SDK initialization
2. Environment variable
3. Automatically on popular platforms
## SDK initialization
### **Environment variable**
The SDKs will look for a `ABV_RELEASE `environment variable. Use it to configure the release e.g. in your CI/CD pipeline.
```bash
ABV_RELEASE = "" # <- github sha or other identifier
```
### **Python SDK**
**Install packages**
```bash
pip install abvdev, python-dotenv
```
**Add credentials and ABV\_RELEASE environment variable:**
Add your ABV credentials to your environment variables. Make sure that you have a `.env` file in your project root.
.env
```bash
ABV_API_KEY = "sk-abv-..."
ABV_HOST = "https://app.abv.dev" # US region
# ABV_HOST = "https://eu.app.abv.dev" # EU region
ABV_RELEASE = "3.2.25"
```
load environment variables and initialize client with get\_client:
```python
# Import the function to load the .env file
from dotenv import load_dotenv
from abvdev import observe, get_client
# Load the environment variables from the .env file
load_dotenv()
@observe
def my_function():
return "Hello, world!" # Input/output and timings are automatically captured
my_function()
# Flush events in short-lived applications
abv = get_client()
abv.flush()
```
The Python SDK also allows you to set the release when initializing the client with client paremters:
```python
from abvdev import ABV, observe
# Set the release when initializing the client
abv = ABV(
release="v2.1.25",
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
@observe
def my_function():
return "Hello, world!" # Input/output and timings are automatically captured
my_function()
# Flush events in short-lived applications
abv.flush()
```
### **JS/TS SDK**
**Install packages**
```bash
npm install @abvdev/tracing @abvdev/otel @opentelemetry/sdk-node
```
**Add credentials and ****ABV\_RELEASE environment variable**
Add your ABV credentials to your environment variables. Make sure that you have a `.env` file in your project root and a package like dotenv to load the variables.
.env
```bash
ABV_API_KEY = "sk-abv-..."
ABV_BASE_URL = "https://app.abv.dev" # US region
//ABV_BASE_URL = "https://eu.app.abv.dev" # EU region
ABV_RELEASE = "1.0.0"
```
Create a `instrumentation.ts` file that initializes the OpenTelemetry `NodeSDK` and registers the `ABVSpanProcessor`.
```bash
npm install dotenv
```
instrumentation.ts
```typescript
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
const sdk = new NodeSDK({
spanProcessors: [new ABVSpanProcessor()],
});
sdk.start();
```
Modify `instrumentation.ts` file to use `dotenv` package to load the variables.
Additional parameters are provided to get trace visible in the UI immediately.
instrumentation.ts
```typescript
import dotenv from "dotenv";
dotenv.config();
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
const sdk = new NodeSDK({
spanProcessors: [
new ABVSpanProcessor({
apiKey: process.env.ABV_API_KEY,
baseUrl: process.env.ABV_BASE_URL,
exportMode: "immediate",
flushAt: 1,
flushInterval: 1,
additionalHeaders: {
"Content-Type": "application/json",
"Accept": "application/json"
}
})
],
});
sdk.start();
```
Import the `instrumentation.ts` file at the top of your application.
index.ts
```typescript
import "./instrumentation"; // Must be the first import
```
**Instrument application**
The JS/TS SDK will look for a ABV\_RELEASE environment variable. Use it also to configure the release e.g. in your CI/CD pipeline.
## Automatically on popular platforms
If no other `release` is set, the ABV SDKs default to a set of known release environment variables.
Supported platforms include: Vercel, Heroku, Netlify. See the full list of support environment variables for JS/TS and .
# Versions
***
The `version` parameter can be added to `traces` and all observation types (`span`, `generation`, `event`). Thereby, you can track the effect of a new `version` on the metrics of an object with a specific `name` using [ABV analytics](/developer/platform/metrics/overview).
### With Python SDK
When using the `@observe()` decorator, version could be specified at both trace or observation levels:
```python
from abvdev import ABV, observe
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
@observe()
def process_data():
# Set version at trace level
abv.update_current_trace(version="1.0")
# Set version at observation level
abv.update_current_span(version="1.0")
process_data()
```
When creating spans directly:
```python
from abvdev import ABV
from openai import OpenAI
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
openai_client = OpenAI(api_key = "sk-proj-...")
# Set version when creating a span
with abv.start_as_current_span(
name="process-data",
version="1.0"
) as span:
# Processing...
span.update(output="Processing complete")
# Create a generation with version
with span.start_as_current_observation(
as_type='generation',
name="guess-countries",
model="gpt-4o",
version="1.1"
) as generation:
# Generation code...
response = openai_client.chat.completions.create(
messages=[{"role": "user", "content": "Hi openai, how is that going?"}],
model="gpt-4o",
)
generation.update(output=response.choices[0].message.content)
# Flush events in short-lived applications
abv.flush()
```
The version parameter can be included in both spans and generations, and can be set either during creation or updated later.
### With JS/TS SDK
```typescript
import "./instrumentation";
import { startActiveObservation } from "@abvdev/tracing";
async function main() {
await startActiveObservation("my-first-trace-version1", async (span) => {
span.update({
input: "Hello, ABV!",
output: "This is my first trace!",
version: "1.0"
});
});
}
main();
```
`abv.trace()`, `abv.span()` and `abv.event()` also take an optional `version` parameter.
### In UI
*Version parameter in ABV interface*
--------------------------------------------------------------------------------
## Environments
URL: /developer/basic-features/environments
Environments allow you to organize your traces, observations, and scores from different contexts such as production, staging, or development. This helps you:
- Keep your development and production data separate while using the same project
- Filter and analyze data by environment
- Reuse datasets and prompts across environments
You can configure the environment by setting the `ABV_TRACING_ENVIRONMENT` environment variable (recommended) or by using the environment parameter in the client initialization.
If both are specified, the initialization parameter takes precedence. If nothing is specified, the default environment is default.
# Data Model
The environment attribute is available on all events in ABV:
- Traces
- Observations (spans, events, generations)
- Scores
- Sessions
The environment must be a string that follows this regex pattern: ^(?!abv)\[a-z0-9-\_]+$ with at most 40 characters.
This means:
- Cannot start with “abv”
- Can only contain lowercase letters, numbers, hyphens, and underscores
# Usage
## Python SDK
**Install package**
```bash
pip install abvdev
```
**Add credentials and tracing environment variables**
Add your ABV credentials to your environment variables. Make sure that you have a `.env` file in your project root.
.env
```python
ABV_API_KEY = "sk-abv-..."
ABV_HOST = "https://app.abv.dev" # US region
# ABV_HOST = "https://eu.app.abv.dev" # EU region
ABV_TRACING_ENVIRONMENT = "production"
```
```python
from abvdev import get_client, observe
# # Import the function to load the .env file
from dotenv import load_dotenv
# Load the environment variables from the .env file
load_dotenv()
# Get the client (will use environment variable)
abv = get_client()
# All operations will now be associated with the "production" environment
with abv.start_as_current_span(name="my-operation") as span:
# Your code here
pass
@observe
def main():
return "Hello"
main()
```
## JS/TS SDK
**Set the ABV Environment via environment variable:**
```typescript
ABV_TRACING_ENVIRONMENT="production"
```
**Install packages**
```bash
npm install @abvdev/tracing @abvdev/otel @opentelemetry/sdk-node dotenv
```
**Add credentials**
Add your ABV credentials to your environment variables. Make sure that you have a `.env` file in your project root and a package like dotenv to load the variables.
.env
```typescript
ABV_API_KEY = "sk-abv-..."
ABV_BASE_URL = "https://app.abv.dev" # US region
//ABV_BASE_URL = "https://eu.app.abv.dev" # EU region
ABV_TRACING_ENVIRONMENT="production"
```
Create`instrumentation.ts` file and use `dotenv` package to load the variables.
Additional parameters are provided to get trace visible in the UI immediately.
instrumentation.ts
```typescript
import dotenv from "dotenv";
dotenv.config();
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
const sdk = new NodeSDK({
spanProcessors: [
new ABVSpanProcessor({
apiKey: process.env.ABV_API_KEY,
baseUrl: process.env.ABV_BASE_URL,
exportMode: "immediate",
flushAt: 1,
flushInterval: 1,
additionalHeaders: {
"Content-Type": "application/json",
"Accept": "application/json"
}
})
],
});
sdk.start();
```
Import the `instrumentation.ts` file at the top of your application.
## OpenTelemetry
When using , you can set the environment using any of these attributes:
- abv.environment
- deployment.environment.name
- deployment.environment
To set an environment property globally, you can use resource attributes: os.environ\["OTEL\_RESOURCE\_ATTRIBUTES"] = "abv.environment=staging".
Alternatively, you can set the environment on a per-span basis:
```python
from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode
from abvdev import ABV
# Initialize with gateway (automatically traced!)
abv = ABV(
api_key="sk-abv-...",
host="https://app.abv.dev" # or EU
)
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("my-operation") as span:
# Set environment using ABV-specific attribute
span.set_attribute("abv.environment", "staging")
# Or using OpenTelemetry convention
span.set_attribute("deployment.environment.name", "staging")
# Flush events in short-lived applications
abv.flush()
```
# Filtering
In the ABV UI, you can filter events by environment using the environment filter in the navigation bar. This filter applies across all views in ABV.
See our [API Reference](/developer/platform/api-data-platform/public-api) for details on how to filter by environment on our API.
--------------------------------------------------------------------------------
## Multi-Modality and Attachments
URL: /developer/basic-features/multi-modality-and-attachments
ABV supports multi-modal traces including text, images, audio, and other attachments.
By default, [base64 encoded data URIs](https://developer.mozilla.org/en-US/docs/Web/URI/Schemes/data#syntax) are handled automatically by the ABV SDKs. They are extracted from the payloads commonly used in multi-modal LLMs, uploaded to ABV's object storage, and linked to the trace.
This also works if you:
1. Reference media files via external URLs.
2. Customize the handling of media files in the SDKs via the `ABVMedia` class.
3. Integrate via the ABV API directly.
Learn more on how to get started and how this works under the hood below.
*Examples:*
# Supported media formats
ABV supports:
- **Images**: .png, .jpg, .webp
- **Audio files**: .mpeg, .mp3, .wav
- **Other attachments**: .pdf, plain text
# Get Started
## Base64 data URI encoded media
If you use base64 encoded images, audio, or other files in your LLM applications, upgrade to the latest version of the ABV SDKs. The ABV SDKs automatically detect and handle base64 encoded media by extracting it, uploading it separately as a ABV Media file, and including a reference in the trace.
This works with standard Data URI ([MDN](https://developer.mozilla.org/en-US/docs/Web/URI/Schemes/data#syntax)) formatted media (like those used by OpenAI and other LLMs).
For examples using the OpenAI SDK and LangChain, refer to the [Python SDK instrumentation guide](/developer/sdks/python/instrumentation) or [JavaScript/TypeScript SDK instrumentation guide](/developer/sdks/js-ts/instrumentation).
## External media (URLs)
ABV supports in-line rendering of media files via URLs if they follow common formats. In this case, the media file is not uploaded to ABV's object storage but simply rendered in the UI directly from the source.
### Supported formats:
**Markdown images**
```md
```
**OpenAI content parts**
```json
{
"content": [
{
"role": "system",
"content": "You are an AI trained to describe and interpret images. Describe the main objects and actions in the image."
},
{
"role": "user",
"content": [
{
"type": "text",
"text": "What's happening in this image?"
},
{
"type": "image_url",
"image_url": {
"url": "https://example.com/image.jpg"
}
}
]
}
]
}
```
## Custom attachments
If you want to have more control or your media is not base64 encoded, you can upload arbitrary media attachments to ABV via the SDKs using the new `ABVeMedia` class. Wrap media with ABVMedia before including it in trace inputs, outputs, or metadata. See the multi-modal documentation for examples.
### Python SDK
Install package
```bash
pip install abvdev
```
```python
from abvdev import ABV, observe
from abvdev.media import ABVMedia
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# Create a ABVMedia object from a file
with open("static/bitcoin.pdf", "rb") as pdf_file:
pdf_bytes = pdf_file.read()
# Wrap media in ABVMedia class
pdf_media = ABVMedia(content_bytes=pdf_bytes, content_type="application/pdf")
# Using with the decorator
@observe()
def process_document():
# Update the current trace with the media file
abv.update_current_trace(
metadata={"document": pdf_media}
)
# Or update the current span
abv.update_current_span(
input={"document": pdf_media}
)
# Using with context managers
with abv.start_as_current_span(name="analyze-document") as span: # Include media in the span input, output, or metadata
span.update(
input={"document": pdf_media},
metadata={"file_size": len(pdf_bytes)}
)
# Process document...
# Add results with media to the output
span.update(output={
"summary": "This document explains Bitcoin...",
"original": pdf_media
})
```
### JS/TS SDK
**Install packages**
```bash
npm install @abvdev/tracing @abvdev/otel @opentelemetry/sdk-node dotenv
```
```bash
npm install @abvdev/core
```
**Add credentials**
Add your ABV credentials to your environment variables. Make sure that you have a `.env` file in your project root and a package like dotenv to load the variables.
.env
```typescript
ABV_API_KEY = "sk-abv-..."
ABV_BASE_URL = "https://app.abv.dev" # US region
//ABV_BASE_URL = "https://eu.app.abv.dev" # EU region
```
Create`instrumentation.ts` file and use `dotenv` package to load the variables.
Additional parameters are provided to get trace visible in the UI immediately.
instrumentation.ts
```typescript
import dotenv from "dotenv";
dotenv.config();
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
const sdk = new NodeSDK({
spanProcessors: [
new ABVSpanProcessor({
apiKey: process.env.ABV_API_KEY,
baseUrl: process.env.ABV_BASE_URL,
exportMode: "immediate",
flushAt: 1,
flushInterval: 1,
additionalHeaders: {
"Content-Type": "application/json",
"Accept": "application/json"
}
})
],
});
sdk.start();
```
Import the `instrumentation.ts` file at the top of your application.
```typescript
import fs from "fs";
import { ABVMedia } from "@abvdev/core";
import { startObservation } from "@abvdev/tracing";
// Wrap media in ABVMedia class
const wrappedMedia = new ABVMedia({
source: "bytes",
contentBytes: fs.readFileSync(new URL("./bitcoin.pdf", import.meta.url)),
contentType: "application/pdf",
});
// Optionally, access media via wrappedMedia.obj
console.log(wrappedMedia.obj);
// Include media in any trace or observation
const span3 = startObservation("media-pdf-generation");
const generation3 = span3.startObservation('llm-call', {
model: 'gpt-4',
input: wrappedMedia,
}, {asType: "generation"});
generation3.end();
span3.end();
```
## API
If you use the API directly to log traces to ABV, you need to follow these steps:
### 1) Upload media to ABV
1. If you use base64 encoded media: you need to extract it from the trace payloads similar to how the ABV SDKs do it.
2. Initialize the upload and get a `mediaId` and `presignedURL`: `POST /api/public/media`.
3. Upload media file: `PUT [presignedURL]`.
See the [Public API documentation](/developer/platform/api-data-platform/public-api) for details on using the API directly to upload media files.
### 2) Add reference to mediaId in trace/observation
Use the *ABV Media Token* to reference the `mediaId` in the trace or observation `input`, `output`, or `metadata`.
# How does it work?
When using media files (that are not referenced via external URLs), ABV handles them in the following way:
## 1. Media Upload Process
### Detection and Extraction
- ABV supports media files in traces and observations on `input`, `output`, and `metadata` fields
- SDKs separate media from tracing data client-side for performance optimization
- Media files are uploaded directly to object storage (AWS S3 or compatible)
- Original media content is replaced with a reference string
### Security and Optimization
- Uploads use presigned URLs with content validation (content length, content type, content SHA256 hash)
- Deduplication: Files are simply replaced by their `mediaId` reference string if already uploaded
- File uniqueness determined by project, content type, and content SHA256 hash
### Implementation Details
- Python SDK: Background thread handling for non-blocking execution
- JS/TS SDKs: Asynchronous, non-blocking implementation
- API support for direct uploads (see [Public API documentation](/developer/platform/api-data-platform/public-api))
## 2. Media Reference System
The base64 data URIs and the wrapped `ABVMedia` objects in ABV traces are replaced by references to the `mediaId` in the following standardized token format, which helps reconstruct the original payload if needed:
```text
@@@abvdevMedia:type={MIME_TYPE}|id={ABV_MEDIA_ID}|source={SOURCE_TYPE}@@@
```
- `MIME_TYPE`: MIME type of the media file, e.g., `image/jpeg`
- `ABV_MEDIA_ID`: ID of the media file in ABV's object storage
- `SOURCE_TYPE`: Source type of the media file, can be `base64_data_uri`, `bytes`, or `file`
Based on this token, the ABV UI can automatically detect the `mediaId` and render the media file inline. The `ABVMedia` class provides utility functions to extract the `mediaId` from the reference string.
## 3. Resolving Media References
When dealing with traces, observations, ~~or ~~dataset items that include media references, you can convert them back to their base64 data URI format using the `resolve_media_references` utility method provided by the ABV client. This is particularly useful for reinserting the original content during fine-tuning, dataset runs, or replaying a generation. The utility method traverses the parsed object and returns a deep copy with all media reference strings replaced by the corresponding base64 data URI representations.
### Python SDK
```python
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# Example object with media references
obj = {
"summary": "This document explains Bitcoin...",
"original": "@@@abvMedia:type=application/pdf|id=your_attached_document_id|source=bytes@@@"
}
# Resolve media references to base64 data URIs
resolved_obj = abv.resolve_media_references(
obj=obj,
resolve_with="base64_data_uri"
)
print(resolved_obj)
# Result:
# {
# "summary": "This document explains Bitcoin...",
# "original": "data:application/pdf;base64,JVBERi0xLjQK..."
# }
```
### JS/TS SDK
```bash
npm install @abvdev/client
```
```typescript
import "./instrumentation"; // Must be the first import
import { ABVClient } from "@abvdev/client";
const abv = new ABVClient()
// Example object with media references
const obj = {
summary: "This document explains Bitcoin...",
pdf: "@@@abvdevMedia:type=application/pdf|id=sWdBkaiOxc3XM-QkCoGAMQ|source=base64_data_uri@@@",
};
async function main() {
// Resolve media references to base64 data URIs
const resolvedTrace = await abv.media.resolveReferences({
obj: obj,
resolveWith: "base64DataUri",
});
console.log(resolvedTrace);
// Result:
// {
// summary: 'This document explains Bitcoin...',
// pdf: 'data:application/pdf;base64,JVBERi0x...'
// }
}
main();
```
--------------------------------------------------------------------------------
## Agent Graphs
URL: /developer/basic-features/agent-graphs
Agent graphs in ABV provide a visual representation of complex AI agent workflows, helping you understand and debug multi-step reasoning processes and agent interactions.
# Get Started
The graph view is currently in beta and supports LangGraph. We are working on a more generalized implementation and expanding support for other frameworks in the future.
**LangGraph**: To integrate LangGraph with ABV for LLM Agent tracing, refer to the [Python SDK instrumentation guide](/developer/sdks/python/instrumentation) for integration details.
--------------------------------------------------------------------------------
## ABV Prompt Playground
URL: /developer/basic-features/prompt-playground
Test and iterate on your prompts directly in the ABV Prompt Playground. Tweak the prompt and model parameters to see how different models respond to these input changes. This allows you to quickly iterate on your prompts and optimize them for the best results in your LLM app without having to switch between tools or use any code.
# Core features
## Side-by-Side Comparison View
Compare multiple prompt variants alongside each other. Execute them all at once or focus on a single variant. Each variant keeps its own LLM settings, variables, tool definitions, and placeholders so you can immediately see the impact of every change.
## Open your prompt in the playground
You can open a prompt you created with [ABV Prompt Management ](/developer/prompt-management/get-started) in the playground.
## Save your prompt to Prompt Management
When you're satisfied with your prompt, you can save it to Prompt Management by clicking the save button.
## Open a generation in the playground
You can open a generation from [ABV Observability](/developer/prompt-management/get-started) in the playground by clicking the `Open in Playground` button in the generation details page.
## Tool calling and structured outputs
The ABV Playground supports tool calling and structured output schemas, enabling you to define, test, and validate LLM executions that rely on tool calls and enforce specific response formats.
**Tool Calling**
- Define custom tools with JSON schema definitions
- Test prompts relying on tools in real-time by mocking tool responses
- Save tool definitions to your project
**Structured Output**
- Enforce response formats using JSON schemas
- Save schemas to your project
- Jump into the playground from your OpenAI generation using structured output
## Add prompt variables
You can add prompt variables in the playground to simulate different inputs to your prompt.
## Use your favorite model
You can use your favorite model by adding the API key for the model you want to use in the ABV project settings. You can learn how to set up an LLM connection [here](/developer/platform/administration/llm-connections).
Optionally, many LLM providers allow for additional parameters when invoking a model. You can pass these parameters in the playground when toggling “Additional Options” in the model selection dropdown. *Read this documentation about additional provider options* for more information.
--------------------------------------------------------------------------------
## Comments on Objects
URL: /developer/basic-features/comments-on-objects
Comments enable teams to add contextual notes and discussions directly to traces, observations, sessions, and prompts within ABV. This feature facilitates collaboration by allowing team members to:
- Flag issues or anomalies in specific traces
- Share insights about particular model outputs
- Document edge cases and debugging notes
- Coordinate on prompt improvements
- Leave feedback during development and review cycles
# Supported Object Types
Comments can be added to the following ABV objects:
- **Traces** - Comment on complete execution flows
- **Observations** - Add notes to specific LLM calls, spans, or events
- **Sessions** - Discuss user interaction patterns
- **Prompts** - Collaborate on prompt versions and improvements
# Adding Comments
## Via UI
Each supported object page displays a comment button in the interface. The button shows:
- The current comment count (capped at "99+" for readability)
- A disabled state if you don't have read permissions
- An active state when comments are available or you can create them
Clicking the comment button opens a side drawer containing:
1. **Comment Thread** - All existing comments displayed chronologically
2. **Composer** - Text area for writing new comments (if you have write permissions)
3. **Markdown Preview** - Real-time preview of your formatted comment
Comment authors can only delete their own comments. Project admins cannot delete other users' comments through the UI.
## With API
The Comments API allows programmatic access to create and retrieve comments. All endpoints follow the standard ABV API patterns.
```bash
POST /api/public/comments
```
--------------------------------------------------------------------------------
## Overview
URL: /developer/llm-gateway/overview
# AI Gateway Introduction
The ABV AI Gateway is a unified interface that lets you call different LLM providers through a single, consistent API. Think of it as a universal adapter for LLMs - you write your code once, and it works across OpenAI, Anthropic, and Google Gemini.
## Why Use the AI Gateway?
When you build AI-powered applications, you face several common challenges. The AI Gateway was designed to solve these specific problems.
Imagine you build your application using OpenAI's API directly. Your code is full of OpenAI-specific patterns, error handling, and assumptions. Then one day you discover that Claude produces better results for your use case, or that GPT-4 is too expensive for your scale, or that OpenAI is having an outage. Switching providers means rewriting significant portions of your application.
The AI Gateway solves this by providing a consistent interface. You change one parameter - the provider name - and your code continues working. The gateway translates your request into whatever format that provider expects and normalizes the response back to a standard format.
When you call provider APIs directly, you lose visibility into what's happening. How many tokens did that request use? How long did it take? What were the exact parameters? Did it fail, and if so, why? Building comprehensive logging for every AI call adds complexity to your codebase.
The gateway automatically traces every request with detailed metadata. You don't write a single line of logging code, yet you get complete visibility into token usage, latency, costs, errors, and parameter configurations. This observability is critical for debugging, optimization, and cost management.
Each provider has their own SDK, their own way of structuring requests, their own error types, and their own quirks. OpenAI's streaming works differently than Anthropic's. Parameter names vary. Authentication mechanisms differ. Managing multiple SDKs increases your dependency footprint and maintenance burden.
The gateway provides an OpenAI-compatible interface that works with all providers. If you know how to use OpenAI's API, you already know how to use the gateway. This reduces the learning curve and keeps your codebase simpler.
## How the Gateway Works
Understanding the gateway's architecture helps you use it more effectively. When you make a request through the gateway, here's what happens:
Your client library sends your request to ABV's gateway service. This request includes which provider you want to use (OpenAI, Anthropic, or Gemini), which model you want, and your messages. The request uses a standardized format that's compatible with OpenAI's API structure.
The gateway receives your request and immediately begins tracing it. Before the request even goes to the provider, ABV creates a trace record that captures:
- **Input data**: Your messages, temperature, max tokens, and other parameters
- **Timing information**: Request start time
- **Contextual metadata**: Provider, model, user identifier, and session information
This happens automatically without any instrumentation code in your application. You don't need to manually log anything or set up monitoring infrastructure.
The gateway translates your request into the specific format that your chosen provider expects. If you're calling Anthropic, it converts your OpenAI-formatted request into Anthropic's format. If you're calling Gemini, it performs a different translation. This translation happens transparently - you don't see it or think about it.
The gateway makes the actual API call to your chosen provider using your provider's credentials (which you've configured in ABV).
When the provider responds, the gateway translates that response back into the standardized OpenAI-compatible format. It also extracts key metrics like token usage and latency.
The gateway completes the trace with all the data collected during the request:
- **Output data**: The model's response, token usage, and any errors that occurred
- **Performance metrics**: Total request duration and latency breakdown
- **Final status**: Success or failure indicators
The complete trace is immediately available in your ABV dashboard where you can filter by provider, model, user, time range, or error status. You can see patterns like which models are slowest, which requests use the most tokens, or where errors are occurring.
Finally, the gateway returns the standardized response to your application.
Throughout this entire process, your application only sees the standardized interface. The complexity of dealing with different providers is handled by the gateway, while comprehensive observability happens automatically.
## When to Use the Gateway
The gateway is particularly valuable in several scenarios:
**Production applications:** When cost control, performance monitoring, and reliability matter, the automatic tracing gives you the visibility you need to optimize and debug.
**Provider flexibility:** If you want to experiment with different providers or models without rewriting your application, the consistent interface makes switching trivial.
**Avoiding vendor lock-in:** When you're concerned about provider lock-in or want redundancy in case a provider has an outage, the gateway makes it easy to route requests to different providers.
**Multi-model support:** If you need to support multiple models or providers in the same application (perhaps letting users choose, or routing different request types to different models), the gateway provides a single, unified interface.
**Simplified error handling:** When you want consistent error handling without learning the quirks of each provider's error types and retry logic, the gateway normalizes these concerns.
## What the Gateway Doesn't Do
It's equally important to understand what the gateway is not.
**The gateway is not an LLM itself.** It doesn't train models, host models, or process your requests with its own AI. Every request goes to the provider you specify (OpenAI, Anthropic, or Gemini).
**The gateway doesn't add latency through complex processing.** It performs efficient request translation and tracing, but the vast majority of your request time is spent waiting for the AI provider to generate tokens. The gateway's overhead is minimal.
**The gateway doesn't store your data beyond what's needed for tracing.** Your messages flow through the gateway to the provider and back. The traces store metadata and request/response information for observability, but ABV doesn't use your data to train models or for any purpose beyond providing you visibility into your own requests.
**The gateway doesn't replace provider SDKs for every use case. **If you're using advanced provider-specific features that don't have cross-provider equivalents, you may need to call those providers directly. However, for the common case of chat completions (which covers most AI applications), the gateway provides everything you need.
--------------------------------------------------------------------------------
## Quickstart
URL: /developer/llm-gateway/quickstart
# Quick Start Guide
This guide will get you making your first AI Gateway request in under five minutes. We'll cover both TypeScript and Python, so jump to whichever language you're using.
## Before You Begin
You need an ABV API key. If you don't have one yet, sign up at https\://app.abv.dev to get your key. It will look something like sk\_....
## TypeScript/JavaScript Quick Start
Install the ABV client library in your project. This single package includes everything you need for the gateway.
```bash
npm install @abvdev/client
```
Create a file called first-request.ts (or .js if you're not using TypeScript) and add this code:
```typescript
import { ABVClient } from '@abvdev/client';
// Initialize the client with your API key
const abv = new ABVClient({
apiKey: 'sk_...' // Replace with your actual API key
});
// Make your first request
const response = await abv.gateway.chat.completions.create({
provider: 'openai',
model: 'gpt-4o-mini',
messages: [
{ role: 'user', content: 'Explain what an API gateway is in one sentence.' }
]
});
// Print the response
console.log(response.choices[0].message.content);
```
Run your code with node first-request.js (or ts-node first-request.ts for TypeScript). You should see an explanation of what an API gateway is appear in your terminal.
Now go to your ABV dashboard at https\://app.abv.dev and click on the Traces section. You'll see a trace for the request you just made. Click into it to see details like token usage, latency, and the full conversation. This trace appeared automatically without you writing any logging code.
## Python Quick Start
Install the ABV client library using pip:
```bash
pip install abvdev
```
Create a file called first\_request.py and add this code:
```python
from abvdev import ABV
# Initialize the client with your API key
abv = ABV(api_key='sk_...') # Replace with your actual API key
# Make your first request
response = abv.gateway.chat.completions.create(
provider='openai',
model='gpt-4o-mini',
messages=[
{'role': 'user', 'content': 'Explain what an API gateway is in one sentence.'}
]
)
# Print the response
print(response['choices'][0]['message']['content'])
```
Run your code with python first\_request.py. You should see an explanation appear in your terminal.
Visit your ABV dashboard at https\://app.abv.dev and navigate to Traces. Your request appears there with complete details about what happened, including token counts and timing information. This observability is built in automatically.
## Understanding What Just Happened
Let's break down what happened in that simple example, because understanding these pieces will help you use the gateway effectively.
When you created the ABV client, you provided your API key. This key authenticates your requests to ABV's gateway service. The client handles all the authentication details for you automatically.
When you called abv.gateway.chat.completions.create(), you provided three key pieces of information. First, you specified the provider as openai, which tells the gateway to route this request to OpenAI's API. Second, you specified the model as gpt-4o-mini, which is an efficient and fast GPT-4 model. Third, you provided messages, which is an array containing the conversation. In this case, there's only one message from the user role.
The gateway received your request, created a trace to record what you're asking for, translated your request into OpenAI's expected format (though in this case it's already OpenAI-compatible), sent the request to OpenAI, received OpenAI's response, translated it back to the standard format, completed the trace with all the response details, and returned the response to your code.
The response object contains a choices array because the API supports generating multiple completions for a single request (controlled by the n parameter, which defaults to 1). Each choice has a message object, which contains the content that the model generated.
Meanwhile, the trace captured everything about this interaction and made it available in your dashboard. You can see what you asked, what the model responded with, how many tokens were used (which affects cost), and how long it took.
## Trying Different Providers
Now that you've made a successful request, try switching providers to see how easy it is. Change your provider parameter from openai to anthropic and the model to `claude-3-5-sonnet-20241022`:
**TypeScript:**
```typescript
const response = await abv.gateway.chat.completions.create({
provider: 'anthropic', // Changed from 'openai'
model: 'claude-3-5-sonnet-20241022', // Changed from 'gpt-4o-mini'
messages: [
{ role: 'user', content: 'Explain what an API gateway is in one sentence.' }
]
});
```
**Python:**
```python
response = abv.gateway.chat.completions.create(
provider='anthropic', # Changed from 'openai'
model='claude-3-5-sonnet-20241022', # Changed from 'gpt-4o-mini'
messages=[
{'role': 'user', 'content': 'Explain what an API gateway is in one sentence.'}
]
)
```
Run your code again. Notice that you didn't need to change anything else. The same code structure works with both providers. The response format is identical. This is the power of the gateway's unified interface - you get provider flexibility without code rewrites.
Check your ABV dashboard again and you'll see a second trace, this time showing that the request went to Anthropic instead of OpenAI. You can compare the responses, token usage, and performance of the two providers directly in the dashboard.
## Using Environment Variables
Hardcoding your API key in your code is not a good practice, especially if you're committing code to version control. Both client libraries support reading your API key from an environment variable, which is more secure and flexible.
Set your API key as an environment variable:
```bash
export ABV_API_KEY=sk_...
```
Then initialize the client without explicitly passing the key:
**TypeScript:**
```typescript
const abv = new ABVClient(); // Automatically uses ABV_API_KEY
```
**Python:**
```python
abv = ABV() # Automatically uses ABV_API_KEY
```
This pattern keeps your credentials out of your source code and makes it easy to use different keys in different environments (development, staging, production).
## Adding a System Message
Most real applications don't just send a single user message. They set up a system message that configures the model's behavior. The system message acts like instructions to the model about how it should respond.
Try modifying your messages array to include a system message:
**TypeScript:**
```typescript
const response = await abv.gateway.chat.completions.create({
provider: 'openai',
model: 'gpt-4o-mini',
messages: [
{
role: 'system',
content: 'You are a helpful assistant who explains technical concepts clearly and concisely.'
},
{
role: 'user',
content: 'What is an API gateway?'
}
]
});
```
**Python:**
```python
response = abv.gateway.chat.completions.create(
provider='openai',
model='gpt-4o-mini',
messages=[
{
'role': 'system',
'content': 'You are a helpful assistant who explains technical concepts clearly and concisely.'
},
{
'role': 'user',
'content': 'What is an API gateway?'
}
]
)
```
The system message doesn't change the structure of your code, but it does influence how the model responds. You'll typically use system messages to set the tone, persona, or constraints for your AI application.
## Next Steps
You've now made your first gateway requests, switched between providers, and seen how automatic tracing works. You understand the basic structure of a request and how to read the response.
From here, you can dive deeper into the specifics of your chosen language. The TypeScript guide covers streaming responses, error handling, type safety, and advanced patterns for TypeScript users. The Python guide covers the same topics but with Python-specific idioms and patterns.
You might also want to understand more about choosing between providers and models. The providers guide explains the strengths and tradeoffs of each provider, helping you make informed decisions about which one to use for different use cases.
Or if you want to understand the deeper concepts behind parameters like temperature and max\_tokens, the concepts guide explains how these parameters affect model behavior and when to adjust them.
--------------------------------------------------------------------------------
## Models Available
URL: /developer/llm-gateway/models-available
# Available Models
The AI Gateway supports a comprehensive range of models from OpenAI, Anthropic, and Google Gemini. Each provider offers models optimized for different use cases, with varying capabilities, performance characteristics, and pricing tiers.
Understanding the models available helps you choose the right one for your specific needs. This page lists all supported models along with their pricing, so you can make informed decisions about cost versus capability for your application.
## Don't See Your Model?
Looking for a model that's not listed here? We're always adding new models to the gateway based on user needs and provider availability. Contact our support team to let us know which model you'd like to see added, and we'll work to make it available.
## OpenAI Models
OpenAI provides several model families, each designed for different use cases. The GPT-4 and GPT-5 series excel at general-purpose tasks, while the O-series models are optimized for complex reasoning and problem-solving.
### GPT-4 Series
The GPT-4 family represents OpenAI's mature, production-ready models with strong performance across a wide range of tasks. These models balance capability with cost-effectiveness.
- **gpt-4** - $0.03 per 1K input tokens | $0.06 per 1K output tokens
- **gpt-4-turbo** - $0.01 per 1K input tokens | $0.03 per 1K output tokens
- **gpt-4o** - $0.0025 per 1K input tokens | $0.01 per 1K output tokens
- **gpt-4o-mini** - $0.00015 per 1K input tokens | $0.0006 per 1K output tokens
- **gpt-4.1** - $0.002 per 1K input tokens | $0.008 per 1K output tokens
- **gpt-4.1-mini** - $0.0004 per 1K input tokens | $0.0016 per 1K output tokens
- **gpt-4.1-nano** - $0.0001 per 1K input tokens | $0.0004 per 1K output tokens
### GPT-5 Series
The GPT-5 family represents OpenAI's next-generation models with improved reasoning, understanding, and generation capabilities.
- **gpt-5** - $0.00125 per 1K input tokens | $0.01 per 1K output tokens
- **gpt-5-mini** - $0.00025 per 1K input tokens | $0.002 per 1K output tokens
- **gpt-5-nano** - $0.00005 per 1K input tokens | $0.0004 per 1K output tokens
- **gpt-5-pro** - Available via ChatGPT Pro subscription ($200/month)
### O-Series (Reasoning Models)
The O-series models are specifically designed for complex reasoning tasks, mathematical problem-solving, and scenarios requiring careful step-by-step thinking.
- **o1** - $0.015 per 1K input tokens | $0.06 per 1K output tokens
- **o1-pro** - $0.15 per 1K input tokens | $0.60 per 1K output tokens
- **o3** - $0.002 per 1K input tokens | $0.008 per 1K output tokens
- **o3-mini** - $0.0011 per 1K input tokens | $0.0044 per 1K output tokens
- **o4-mini** - $0.0011 per 1K input tokens | $0.0044 per 1K output tokens
### Specialized Models
OpenAI offers several specialized models optimized for specific use cases beyond general conversation.
- **codex-mini-latest** - $0.0015 per 1K input tokens | $0.006 per 1K output tokens
- **gpt-4o-mini-search-preview** - Uses gpt-4o-mini pricing + web search tool fees
- **gpt-4o-search-preview** - Uses gpt-4o pricing + web search tool fees
## Anthropic Models
Anthropic's Claude models are known for their careful reasoning, strong instruction following, and large context windows. The Claude family emphasizes helpfulness, harmlessness, and honesty in its responses.
### Claude 4 Series
The Claude 4 family represents Anthropic's latest generation, offering significant improvements in reasoning, knowledge, and task performance.
- **claude-opus-4-0** - $0.015 per 1K input tokens | $0.075 per 1K output tokens
- **claude-opus-4-1** - $0.015 per 1K input tokens | $0.075 per 1K output tokens
- **claude-sonnet-4-0** - $0.003 per 1K input tokens | $0.015 per 1K output tokens
- **claude-sonnet-4-5** - $0.003 per 1K input tokens | $0.015 per 1K output tokens
### Claude 3 Series
The Claude 3 family continues to be widely used for production applications, offering proven reliability and strong performance.
- **claude-3-7-sonnet** - $0.003 per 1K input tokens | $0.015 per 1K output tokens
- **claude-3-5-sonnet** - $0.003 per 1K input tokens | $0.015 per 1K output tokens
- **claude-3-5-haiku** - $0.001 per 1K input tokens | $0.005 per 1K output tokens
- **claude-3-opus** - $0.015 per 1K input tokens | $0.075 per 1K output tokens
### Claude Haiku Series
The Haiku models prioritize speed and efficiency, making them ideal for applications where fast responses matter.
- **claude-haiku-4-5** - $0.001 per 1K input tokens | $0.005 per 1K output tokens
## Google Gemini Models
Google's Gemini models offer competitive performance with particularly strong multimodal capabilities and cost-effective pricing. The Gemini family includes multiple generations, each optimized for different performance and cost profiles.
### Gemini 2.5 Series
The latest Gemini generation offers enhanced capabilities across understanding, generation, and multimodal processing.
- **gemini-2.5-flash** - $0.0003 per 1K input tokens | $0.0025 per 1K output tokens
- **gemini-2.5-flash-lite** - $0.0001 per 1K input tokens | $0.0004 per 1K output tokens
- **gemini-2.5-pro** - $0.00125 per 1K input tokens | $0.01 per 1K output tokens
### Gemini 2.0 Series
The Gemini 2.0 family provides strong performance with particularly attractive pricing for cost-sensitive applications.
- **gemini-2.0-flash** - $0.0001 per 1K input tokens | $0.0004 per 1K output tokens
- **gemini-2.0-flash-lite** - $0.00007 per 1K input tokens | $0.0003 per 1K output tokens
### Gemini 1.5 Series
The proven Gemini 1.5 family continues to serve production applications reliably with excellent cost-effectiveness.
- **gemini-1.5-flash** - $0.000075 per 1K input tokens | $0.0003 per 1K output tokens
- **gemini-1.5-flash-8b** - $0.0000375 per 1K input tokens | $0.00015 per 1K output tokens
- **gemini-1.5-pro** - $0.00125 per 1K input tokens | $0.005 per 1K output tokens
## Choosing the Right Model
Selecting the appropriate model involves balancing several factors including capability requirements, cost constraints, latency needs, and the specific characteristics of your use case. For guidance on making these decisions, see the providers guide which explains how to match models to different application scenarios.
Understanding pricing helps you optimize costs while maintaining the quality your application needs. Input and output pricing differs because generation requires more computation than processing. Applications with long outputs benefit from models with lower output costs, while applications processing large inputs should consider input pricing carefully.
For detailed implementation guidance on using these models with the gateway, refer to the TypeScript guide or Python guide depending on your development environment.
--------------------------------------------------------------------------------
## Typescript Guide
URL: /developer/llm-gateway/typescript-guide
## Installation and Setup
The ABV client library comes as a single npm package that includes everything you need. Install it in your project with npm, yarn, or pnpm:
```bash
npm install @abvdev/client
```
If you're working with TypeScript, the type definitions are included in the package automatically. You don't need to install separate @types packages.
For projects that want explicit control over tracing behavior, you can also install the tracing package, though it's already included as a dependency of the client:
```bash
npm install @abvdev/tracing
```
## Client Initialization Patterns
The way you initialize the ABV client affects how you use it throughout your application. Let's explore different patterns and when to use each.
The simplest initialization creates a client with your API key directly:
```typescript
import { ABVClient } from '@abvdev/client';
const abv = new ABVClient({
apiKey: 'sk_...'
});
```
This works for quick prototypes or scripts, but hardcoding credentials isn't recommended for production applications. Instead, use environment variables:
```typescript
const abv = new ABVClient({
apiKey: process.env.ABV_API_KEY
});
```
The client also checks for the ABV_API_KEY environment variable automatically, so you can simplify this further:
```typescript
const abv = new ABVClient();
```
This pattern is cleaner and makes it impossible to accidentally commit credentials to version control.
For applications that need to support multiple regions, specify the region during initialization:
```typescript
const abv = new ABVClient({
region: 'eu' // 'us' (default) or 'eu'
});
```
Region selection affects which ABV infrastructure your requests route through. Choose the region closest to your users or the region that matches your data residency requirements.
## Client Lifecycle and Reuse
Creating an ABV client is lightweight, but you should generally create one client instance and reuse it throughout your application. The client maintains connection pooling and handles request optimization internally.
For a Node.js server application, create the client once at startup and export it for use across your application:
```typescript
// abv-client.ts
import { ABVClient } from '@abvdev/client';
export const abv = new ABVClient();
```
Then import this client wherever you need it:
```typescript
// route-handler.ts
import { abv } from './abv-client';
export async function handleRequest(userMessage: string) {
const response = await abv.gateway.chat.completions.create({
provider: 'openai',
model: 'gpt-4o-mini',
messages: [{ role: 'user', content: userMessage }]
});
return response.choices[0].message.content;
}
```
This pattern ensures efficient connection management and simplifies testing since you can mock the exported client in your test files.
## Working with Types
TypeScript's type system helps catch errors before runtime. The ABV client library provides comprehensive type definitions that make working with the gateway safer and more productive.
Import types from the client package to annotate your code:
```typescript
import type {
ChatCompletionParams,
ChatCompletionResponse,
ChatMessage
} from '@abvdev/client';
```
These types help you build functions that work with gateway responses:
```typescript
function extractResponse(response: ChatCompletionResponse): string {
return response.choices[0].message.content;
}
function buildMessages(
systemPrompt: string,
userMessage: string
): ChatMessage[] {
return [
{ role: 'system', content: systemPrompt },
{ role: 'user', content: userMessage }
];
}
```
When building the parameters object for a request, TypeScript ensures you provide valid values:
```typescript
const params: ChatCompletionParams = {
provider: 'openai',
model: 'gpt-4o-mini',
messages: [{ role: 'user', content: 'Hello' }],
temperature: 0.7,
max_tokens: 500
};
// TypeScript catches errors here
const response = await abv.gateway.chat.completions.create(params);
```
If you make a typo in the provider name or forget a required field, TypeScript flags it immediately. This catches bugs during development rather than in production.
## Handling Streaming Responses
Streaming responses arrive as an asynchronous iterable, which you consume using a for-await-of loop. This pattern feels natural in modern JavaScript and TypeScript:
```typescript
const stream = await abv.gateway.chat.completions.create({
provider: 'openai',
model: 'gpt-4o-mini',
messages: [{ role: 'user', content: 'Tell me a story' }],
stream: true
});
for await (const chunk of stream) {
const content = chunk.choices[0]?.delta?.content;
if (content) {
process.stdout.write(content);
}
}
```
Notice the optional chaining (?.) when accessing the content. Early chunks might not have content, or the structure might vary slightly. Optional chaining prevents runtime errors from null or undefined values.
For applications that need to both display tokens as they arrive and store the complete response, accumulate chunks while streaming:
```typescript
const stream = await abv.gateway.chat.completions.create({
provider: 'openai',
model: 'gpt-4o-mini',
messages: [{ role: 'user', content: 'Explain TypeScript generics' }],
stream: true
});
let fullResponse = '';
for await (const chunk of stream) {
const content = chunk.choices[0]?.delta?.content;
if (content) {
fullResponse += content;
process.stdout.write(content);
}
}
console.log('\n\nComplete response length:', fullResponse.length);
```
This pattern works well for chatbots that display streaming responses to users but also need to save conversation history.
## Error Handling Strategies
Gateway requests can fail for various reasons, and handling these failures gracefully improves your application's reliability. TypeScript's type system helps you handle errors correctly.
The most basic error handling uses try-catch:
```typescript
try {
const response = await abv.gateway.chat.completions.create({
provider: 'openai',
model: 'gpt-4o-mini',
messages: [{ role: 'user', content: 'Hello' }]
});
console.log(response.choices[0].message.content);
} catch (error) {
console.error('Gateway request failed:', error);
}
```
For production applications, you want more sophisticated error handling that distinguishes between error types. Errors might be network issues, rate limits, authentication problems, or invalid requests. Each type suggests different handling:
```typescript
async function makeRequest(
messages: ChatMessage[]
): Promise {
try {
const response = await abv.gateway.chat.completions.create({
provider: 'openai',
model: 'gpt-4o-mini',
messages
});
return response.choices[0].message.content;
} catch (error) {
if (error instanceof Error) {
// Check error message or properties to determine type
if (error.message.includes('rate limit')) {
console.error('Rate limited, need to slow down');
throw new Error('Service temporarily unavailable');
} else if (error.message.includes('authentication')) {
console.error('Authentication failed, check API key');
throw new Error('Configuration error');
}
}
// Generic error handling for unexpected failures
console.error('Unexpected error:', error);
throw new Error('Request failed');
}
}
```
For applications that need retry logic, implement exponential backoff. This pattern waits increasingly longer between retries, reducing load on the service:
```typescript
async function makeRequestWithRetry(
messages: ChatMessage[],
maxRetries: number = 3
): Promise {
let lastError: Error | null = null;
for (let attempt = 0; attempt < maxRetries; attempt++) {
try {
const response = await abv.gateway.chat.completions.create({
provider: 'openai',
model: 'gpt-4o-mini',
messages
});
return response.choices[0].message.content;
} catch (error) {
lastError = error instanceof Error ? error : new Error(String(error));
// Don't retry on authentication errors
if (lastError.message.includes('authentication')) {
throw lastError;
}
// On last attempt, throw the error
if (attempt === maxRetries - 1) {
throw lastError;
}
// Wait before retrying, with exponential backoff
const delay = Math.pow(2, attempt) * 1000;
await new Promise(resolve => setTimeout(resolve, delay));
}
}
throw lastError || new Error('Max retries exceeded');
}
```
This retry logic handles transient failures like network blips while avoiding endless loops on permanent failures.
## Building Conversation Context
Most AI applications involve multi-turn conversations where the model needs context from previous exchanges. Managing this context is a core part of implementing chat applications.
The straightforward approach maintains an array of messages that grows with the conversation:
```typescript
class Conversation {
private messages: ChatMessage[] = [];
constructor(systemPrompt: string) {
this.messages.push({
role: 'system',
content: systemPrompt
});
}
async sendMessage(userMessage: string): Promise {
// Add user message to history
this.messages.push({
role: 'user',
content: userMessage
});
// Get response from model
const response = await abv.gateway.chat.completions.create({
provider: 'openai',
model: 'gpt-4o-mini',
messages: this.messages
});
const assistantMessage = response.choices[0].message;
// Add assistant response to history
this.messages.push(assistantMessage);
return assistantMessage.content;
}
getHistory(): ChatMessage[] {
return [...this.messages]; // Return copy to prevent external modification
}
}
```
This pattern encapsulates conversation state and ensures the message history stays synchronized. Using it looks like this:
```typescript
const conversation = new Conversation(
'You are a helpful assistant that explains programming concepts clearly.'
);
console.log(await conversation.sendMessage('What is a closure in JavaScript?'));
console.log(await conversation.sendMessage('Can you give me an example?'));
console.log(await conversation.sendMessage('How is this different from a regular function?'));
```
Each message sees the full conversation history, allowing the model to reference earlier exchanges and maintain context.
For long-running conversations, you might need to manage the context window size. Here's a more sophisticated approach that limits history:
```typescript
class Conversation {
private messages: ChatMessage[];
private readonly maxMessages: number;
constructor(systemPrompt: string, maxMessages: number = 20) {
this.messages = [{
role: 'system',
content: systemPrompt
}];
this.maxMessages = maxMessages;
}
async sendMessage(userMessage: string): Promise {
this.messages.push({
role: 'user',
content: userMessage
});
// Keep only recent messages (but always keep system message)
if (this.messages.length > this.maxMessages) {
this.messages = [
this.messages[0], // System message
...this.messages.slice(-(this.maxMessages - 1)) // Recent messages
];
}
const response = await abv.gateway.chat.completions.create({
provider: 'openai',
model: 'gpt-4o-mini',
messages: this.messages
});
const assistantMessage = response.choices[0].message;
this.messages.push(assistantMessage);
return assistantMessage.content;
}
}
```
This approach prevents the conversation from exceeding token limits by keeping only recent messages. The tradeoff is that the model loses access to earlier context.
## Implementing Function Calling
Function calling lets models invoke functions you define, enabling structured outputs and tool use. While the gateway supports function calling, the implementation details vary by provider. This example shows the general pattern:
```typescript
interface WeatherParams {
location: string;
unit: 'celsius' | 'fahrenheit';
}
async function getWeather(params: WeatherParams): Promise {
// In a real application, this would call a weather API
return `The weather in ${params.location} is 72°${params.unit === 'celsius' ? 'C' : 'F'} and sunny`;
}
const tools = [{
type: 'function' as const,
function: {
name: 'get_weather',
description: 'Get the current weather for a location',
parameters: {
type: 'object',
properties: {
location: {
type: 'string',
description: 'The city and state, e.g. San Francisco, CA'
},
unit: {
type: 'string',
enum: ['celsius', 'fahrenheit']
}
},
required: ['location']
}
}
}];
const response = await abv.gateway.chat.completions.create({
provider: 'openai',
model: 'gpt-4o-mini',
messages: [
{ role: 'user', content: 'What\'s the weather in New York?' }
],
tools
});
// Check if model wants to call a function
const message = response.choices[0].message;
if (message.tool_calls) {
for (const toolCall of message.tool_calls) {
if (toolCall.function.name === 'get_weather') {
const params = JSON.parse(toolCall.function.arguments) as WeatherParams;
const result = await getWeather(params);
console.log(result);
}
}
}
```
This pattern defines functions the model can call, parses the model's function call requests, executes the functions, and returns results to the model.
## Framework Integration
The gateway integrates naturally with popular TypeScript frameworks. Here are patterns for common frameworks.
For Express.js applications, add an endpoint that handles AI requests:
```typescript
import express from 'express';
import { abv } from './abv-client';
const app = express();
app.use(express.json());
app.post('/api/chat', async (req, res) => {
try {
const { message } = req.body;
const response = await abv.gateway.chat.completions.create({
provider: 'openai',
model: 'gpt-4o-mini',
messages: [{ role: 'user', content: message }]
});
res.json({
response: response.choices[0].message.content
});
} catch (error) {
console.error('Chat error:', error);
res.status(500).json({ error: 'Failed to generate response' });
}
});
```
For Next.js API routes, the pattern is similar but uses Next.js request handlers:
```typescript
// app/api/chat/route.ts
import { NextResponse } from 'next/server';
import { abv } from '@/lib/abv-client';
export async function POST(request: Request) {
try {
const { message } = await request.json();
const response = await abv.gateway.chat.completions.create({
provider: 'openai',
model: 'gpt-4o-mini',
messages: [{ role: 'user', content: message }]
});
return NextResponse.json({
response: response.choices[0].message.content
});
} catch (error) {
console.error('Chat error:', error);
return NextResponse.json(
{ error: 'Failed to generate response' },
{ status: 500 }
);
}
}
```
For Next.js applications that need streaming responses to the browser, use server-sent events:
```typescript
export async function POST(request: Request) {
const { message } = await request.json();
const stream = await abv.gateway.chat.completions.create({
provider: 'openai',
model: 'gpt-4o-mini',
messages: [{ role: 'user', content: message }],
stream: true
});
const encoder = new TextEncoder();
const readable = new ReadableStream({
async start(controller) {
for await (const chunk of stream) {
const content = chunk.choices[0]?.delta?.content;
if (content) {
controller.enqueue(encoder.encode(content));
}
}
controller.close();
}
});
return new Response(readable, {
headers: { 'Content-Type': 'text/plain; charset=utf-8' }
});
}
```
## Testing Strategies
Testing code that calls AI models requires different strategies than testing deterministic functions. You can't assert exact outputs since model responses vary, but you can test the structure of your code and ensure error handling works correctly.
Mock the ABV client for unit tests to avoid making actual API calls:
```typescript
// __tests__/chat-handler.test.ts
import { jest } from '@jest/globals';
// Mock the ABV client
jest.mock('./abv-client', () => ({
abv: {
gateway: {
chat: {
completions: {
create: jest.fn()
}
}
}
}
}));
import { abv } from './abv-client';
import { handleChatRequest } from './chat-handler';
test('handleChatRequest processes messages correctly', async () => {
const mockResponse = {
choices: [{
message: {
role: 'assistant',
content: 'This is a test response'
}
}]
};
(abv.gateway.chat.completions.create as jest.Mock).mockResolvedValue(mockResponse);
const result = await handleChatRequest('Hello');
expect(result).toBe('This is a test response');
});
```
For integration tests where you want to verify actual API behavior, make real requests but structure tests to be flexible about specific outputs:
```typescript
test('gateway returns valid responses', async () => {
const response = await abv.gateway.chat.completions.create({
provider: 'openai',
model: 'gpt-4o-mini',
messages: [{ role: 'user', content: 'Say hello' }]
});
// Assert structure, not specific content
expect(response.choices).toHaveLength(1);
expect(response.choices[0].message.content).toBeTruthy();
expect(response.choices[0].message.role).toBe('assistant');
expect(response.usage).toBeDefined();
expect(response.usage.total_tokens).toBeGreaterThan(0);
});
```
This approach verifies that the API integration works without depending on specific model outputs.
## Next Steps
You now understand how to implement the gateway in TypeScript applications, handle errors, manage conversations, and integrate with frameworks. Check the concepts guide to deepen your understanding of parameters and model behavior, or explore the providers guide to learn which providers and models work best for different use cases.
For production deployment considerations and best practices, see the best practices guide.
--------------------------------------------------------------------------------
## Python Guide
URL: /developer/llm-gateway/python-guide
# Python Implementation Guide
This guide covers implementation specifics for using the AI Gateway with Python. It assumes you've already read the introduction and understand the core concepts. We'll focus on Python-specific patterns, async/await usage, error handling, and practical implementation details.
## Installation and Setup
The ABV client library installs via pip and works with Python 3.8 or later. Install it in your project or virtual environment:
```bash
pip install abvdev
```
The package includes both synchronous and asynchronous clients, so you don't need separate packages for async support. Type stubs are included for better IDE support if you're using type checkers like mypy or Pylance.
## Client Initialization Patterns
How you initialize the ABV client affects your application's structure. Let's explore different patterns and their use cases.
The simplest initialization provides your API key directly:
```python
from abvdev import ABV
abv = ABV(api_key='sk_...')
```
This works for quick prototypes or scripts, but storing credentials in code isn't recommended for production. Instead, use environment variables:
```python
import os
from abvdev import ABV
abv = ABV(api_key=os.environ.get('ABV_API_KEY'))
```
The client checks for the ABV_API_KEY environment variable automatically, so you can simplify further:
```python
from abvdev import ABV
abv = ABV() # Automatically uses ABV_API_KEY from environment
```
This pattern keeps credentials out of your codebase and makes it easy to use different keys in different environments.
For applications that need to support multiple regions, specify the region during initialization:
```python
from abvdev import ABV
abv = ABV(region='eu') # 'us' (default) or 'eu'
```
Region selection determines which ABV infrastructure handles your requests. Choose the region closest to your users or matching your data residency requirements.
## Client Lifecycle and Module Organization
Creating an ABV client is lightweight, but you should generally create one client instance and reuse it throughout your application. Python's module system makes this pattern natural.
Create a module that initializes and exports the client:
```python
# abv_client.py
from abvdev import ABV
abv = ABV() # Uses ABV_API_KEY from environment
```
Then import this client wherever you need it:
```python
# chat_handler.py
from abv_client import abv
def handle_chat_request(user_message: str) -> str:
"""Process a chat request and return the response."""
response = abv.gateway.chat.completions.create(
provider='openai',
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': user_message}]
)
return response['choices'][0]['message']['content']
```
This pattern ensures you're reusing the same client instance across your application, which is more efficient and simplifies testing since you can mock the imported client in test files.
## Working with Type Hints
Python's type hints help catch errors during development and improve code readability. While the ABV client library works fine without type hints, adding them makes your code more maintainable.
Use type hints to document function signatures that work with gateway responses:
```python
from typing import Dict, List, Any
def extract_response(response: Dict[str, Any]) -> str:
"""Extract the text content from a gateway response."""
return response['choices'][0]['message']['content']
def build_messages(system_prompt: str, user_message: str) -> List[Dict[str, str]]:
"""Build a messages array with system and user messages."""
return [
{'role': 'system', 'content': system_prompt},
{'role': 'user', 'content': user_message}
]
```
Type hints help your IDE provide better autocomplete and catch type errors before runtime. They also serve as documentation for other developers reading your code.
For more sophisticated type checking, you can define TypedDict classes that represent the structure of gateway requests and responses:
```python
from typing import TypedDict, List, Literal
class Message(TypedDict):
role: Literal['system', 'user', 'assistant']
content: str
class ChatRequest(TypedDict):
provider: Literal['openai', 'anthropic', 'gemini']
model: str
messages: List[Message]
temperature: float
max_tokens: int
def create_completion(request: ChatRequest) -> Dict[str, Any]:
"""Create a chat completion with type-checked parameters."""
return abv.gateway.chat.completions.create(**request)
```
This level of type safety catches more errors during development, though it requires more upfront definition effort.
## Handling Streaming Responses
Streaming responses arrive as an iterator that yields chunks as the model generates tokens. Python's iteration protocol makes working with streams natural.
The basic streaming pattern uses a for loop to process chunks:
```python
stream = abv.gateway.chat.completions.create(
provider='openai',
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': 'Tell me a story'}],
stream=True
)
for chunk in stream:
content = chunk.get('choices', [{}])[0].get('delta', {}).get('content')
if content:
print(content, end='', flush=True)
```
The nested .get() calls with default values handle the varying structure of chunks safely. Early chunks might not have content, and this pattern avoids KeyError exceptions.
For applications that need both real-time display and the complete response, accumulate chunks while iterating:
```python
stream = abv.gateway.chat.completions.create(
provider='openai',
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': 'Explain Python decorators'}],
stream=True
)
full_response = ''
for chunk in stream:
content = chunk.get('choices', [{}])[0].get('delta', {}).get('content')
if content:
full_response += content
print(content, end='', flush=True)
print(f'\n\nComplete response length: {len(full_response)} characters')
```
This pattern works well for chatbots that display streaming text while also saving the complete conversation history.
## Async/Await for Concurrent Requests
Python's asyncio support enables efficient concurrent processing of multiple AI requests. The ABV client provides async methods for applications built with asyncio.
For async streaming, use the create_async method and async for to iterate:
```python
import asyncio
async def stream_response():
"""Stream a response asynchronously."""
stream = await abv.gateway.chat.completions.create_async(
provider='openai',
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': 'Tell me a story'}],
stream=True
)
async for chunk in stream:
content = chunk.get('choices', [{}])[0].get('delta', {}).get('content')
if content:
print(content, end='', flush=True)
# Run the async function
asyncio.run(stream_response())
```
The real power of async comes when processing multiple requests concurrently. This is much faster than processing them sequentially:
```python
import asyncio
async def process_prompt(prompt: str) -> str:
"""Process a single prompt asynchronously."""
response = await abv.gateway.chat.completions.create_async(
provider='openai',
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': prompt}]
)
return response['choices'][0]['message']['content']
async def process_multiple_prompts():
"""Process multiple prompts concurrently."""
prompts = [
'What is Python?',
'Explain asyncio',
'What are decorators?',
'How does type hinting work?'
]
# Create tasks for all prompts and run them concurrently
tasks = [process_prompt(prompt) for prompt in prompts]
results = await asyncio.gather(*tasks)
for prompt, result in zip(prompts, results):
print(f'\nPrompt: {prompt}')
print(f'Response: {result}')
asyncio.run(process_multiple_prompts())
```
Using asyncio.gather processes all requests concurrently, which is much faster than processing them one by one. This pattern is particularly valuable for batch processing or applications that need to make multiple AI requests in response to a single user action.
## Error Handling Strategies
Gateway requests can fail for various reasons, and handling these failures appropriately improves application reliability. Python's exception system gives you several approaches to error handling.
The basic pattern uses try-except to catch failures:
```python
try:
response = abv.gateway.chat.completions.create(
provider='openai',
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': 'Hello'}]
)
print(response['choices'][0]['message']['content'])
except Exception as error:
print(f'Gateway request failed: {error}')
```
For production applications, distinguish between different error types to handle them appropriately. Rate limit errors need different handling than authentication errors:
```python
def make_request_safe(messages: List[Dict[str, str]]) -> str:
"""Make a gateway request with comprehensive error handling."""
try:
response = abv.gateway.chat.completions.create(
provider='openai',
model='gpt-4o-mini',
messages=messages
)
return response['choices'][0]['message']['content']
except Exception as error:
error_message = str(error).lower()
if 'rate limit' in error_message:
print('Rate limited - need to slow down requests')
raise ValueError('Service temporarily unavailable')
elif 'authentication' in error_message:
print('Authentication failed - check API key')
raise ValueError('Configuration error')
elif 'invalid' in error_message:
print('Invalid request parameters')
raise ValueError('Invalid request')
else:
print(f'Unexpected error: {error}')
raise ValueError('Request failed')
```
This pattern examines the error message to determine the failure type and responds appropriately. Different error types map to different user-facing messages or retry strategies.
For applications requiring retry logic, implement exponential backoff to handle transient failures:
```python
import time
def make_request_with_retry(
messages: List[Dict[str, str]],
max_retries: int = 3
) -> str:
"""Make a gateway request with exponential backoff retry logic."""
last_error = None
for attempt in range(max_retries):
try:
response = abv.gateway.chat.completions.create(
provider='openai',
model='gpt-4o-mini',
messages=messages
)
return response['choices'][0]['message']['content']
except Exception as error:
last_error = error
error_message = str(error).lower()
# Don't retry authentication errors - they won't succeed
if 'authentication' in error_message:
raise error
# On final attempt, give up
if attempt == max_retries - 1:
raise error
# Wait before retrying, with exponential backoff
delay = (2 ** attempt) * 1.0
print(f'Retry {attempt + 1}/{max_retries} after {delay}s')
time.sleep(delay)
raise last_error or Exception('Max retries exceeded')
```
This retry logic handles temporary failures like network issues while avoiding endless loops on permanent failures. The exponential backoff prevents overwhelming the service with rapid retries.
## Building Conversation Context
Most AI applications involve multi-turn conversations where the model needs context from previous messages. Managing this context is essential for building chat applications.
The straightforward approach maintains a list of messages that grows with the conversation:
```python
class Conversation:
"""Manages a conversation with context history."""
def __init__(self, system_prompt: str):
"""Initialize conversation with a system prompt."""
self.messages = [
{'role': 'system', 'content': system_prompt}
]
def send_message(self, user_message: str) -> str:
"""Send a message and get a response, maintaining context."""
# Add user message to history
self.messages.append({
'role': 'user',
'content': user_message
})
# Get response from model
response = abv.gateway.chat.completions.create(
provider='openai',
model='gpt-4o-mini',
messages=self.messages
)
assistant_message = response['choices'][0]['message']
# Add assistant response to history
self.messages.append(assistant_message)
return assistant_message['content']
def get_history(self) -> List[Dict[str, str]]:
"""Get a copy of the conversation history."""
return self.messages.copy()
```
This class encapsulates conversation state and ensures history stays synchronized. Using it looks like this:
```python
conversation = Conversation(
'You are a helpful assistant that explains programming concepts clearly.'
)
print(conversation.send_message('What is a Python decorator?'))
print(conversation.send_message('Can you show me an example?'))
print(conversation.send_message('How is this different from a regular function?'))
```
Each message includes the full conversation history, enabling the model to reference earlier exchanges and maintain context.
For long-running conversations, manage the context window size to avoid token limits:
```python
class Conversation:
"""Manages a conversation with automatic history truncation."""
def __init__(self, system_prompt: str, max_messages: int = 20):
"""Initialize conversation with history limit."""
self.messages = [
{'role': 'system', 'content': system_prompt}
]
self.max_messages = max_messages
def send_message(self, user_message: str) -> str:
"""Send a message with automatic history management."""
self.messages.append({
'role': 'user',
'content': user_message
})
# Keep only recent messages (always preserve system message)
if len(self.messages) > self.max_messages:
self.messages = [
self.messages[0], # System message
*self.messages[-(self.max_messages - 1):] # Recent messages
]
response = abv.gateway.chat.completions.create(
provider='openai',
model='gpt-4o-mini',
messages=self.messages
)
assistant_message = response['choices'][0]['message']
self.messages.append(assistant_message)
return assistant_message['content']
```
This approach prevents conversations from exceeding token limits by keeping only recent messages. The tradeoff is losing access to earlier context.
## Working with Dataclasses
Python's dataclasses provide a clean way to structure data for AI applications. They're particularly useful for managing conversation state and request parameters:
```python
from dataclasses import dataclass, field
from typing import List, Dict
@dataclass
class ChatMessage:
"""Represents a single message in a conversation."""
role: str
content: str
def to_dict(self) -> Dict[str, str]:
"""Convert to dictionary for API requests."""
return {'role': self.role, 'content': self.content}
@dataclass
class Conversation:
"""Manages a conversation with structured message handling."""
system_prompt: str
messages: List[ChatMessage] = field(default_factory=list)
max_history: int = 20
def __post_init__(self):
"""Initialize with system message."""
self.messages.append(
ChatMessage(role='system', content=self.system_prompt)
)
def send_message(self, content: str) -> str:
"""Send a user message and get a response."""
user_message = ChatMessage(role='user', content=content)
self.messages.append(user_message)
# Manage history size
if len(self.messages) > self.max_history:
self.messages = [
self.messages[0], # System message
*self.messages[-(self.max_history - 1):]
]
# Convert to API format
api_messages = [msg.to_dict() for msg in self.messages]
response = abv.gateway.chat.completions.create(
provider='openai',
model='gpt-4o-mini',
messages=api_messages
)
assistant_content = response['choices'][0]['message']['content']
assistant_message = ChatMessage(role='assistant', content=assistant_content)
self.messages.append(assistant_message)
return assistant_content
```
Dataclasses make the structure explicit and provide useful methods like __repr__ automatically, which helps with debugging.
## Framework Integration
The gateway integrates naturally with popular Python web frameworks. Here are patterns for common frameworks.
For Flask applications, create an endpoint that handles AI requests:
```python
from flask import Flask, request, jsonify
from abv_client import abv
app = Flask(__name__)
@app.route('/api/chat', methods=['POST'])
def chat():
"""Handle chat requests."""
try:
data = request.get_json()
user_message = data.get('message')
if not user_message:
return jsonify({'error': 'Message is required'}), 400
response = abv.gateway.chat.completions.create(
provider='openai',
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': user_message}]
)
return jsonify({
'response': response['choices'][0]['message']['content']
})
except Exception as error:
print(f'Chat error: {error}')
return jsonify({'error': 'Failed to generate response'}), 500
```
For FastAPI applications, the pattern is similar but uses FastAPI's async support:
```python
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from abv_client import abv
app = FastAPI()
class ChatRequest(BaseModel):
message: str
class ChatResponse(BaseModel):
response: str
@app.post('/api/chat', response_model=ChatResponse)
async def chat(request: ChatRequest):
"""Handle chat requests asynchronously."""
try:
response = await abv.gateway.chat.completions.create_async(
provider='openai',
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': request.message}]
)
return ChatResponse(
response=response['choices'][0]['message']['content']
)
except Exception as error:
print(f'Chat error: {error}')
raise HTTPException(
status_code=500,
detail='Failed to generate response'
)
```
FastAPI's async support and Pydantic models provide type safety and automatic validation, making it an excellent choice for AI-powered APIs.
For Django applications, create a view that processes AI requests:
```python
from django.http import JsonResponse
from django.views.decorators.http import require_http_methods
from django.views.decorators.csrf import csrf_exempt
import json
from abv_client import abv
@csrf_exempt
@require_http_methods(['POST'])
def chat_view(request):
"""Handle chat requests in Django."""
try:
data = json.loads(request.body)
user_message = data.get('message')
if not user_message:
return JsonResponse(
{'error': 'Message is required'},
status=400
)
response = abv.gateway.chat.completions.create(
provider='openai',
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': user_message}]
)
return JsonResponse({
'response': response['choices'][0]['message']['content']
})
except Exception as error:
print(f'Chat error: {error}')
return JsonResponse(
{'error': 'Failed to generate response'},
status=500
)
```
These patterns integrate the gateway into your existing framework naturally without requiring architectural changes.
## Testing Strategies
Testing code that calls AI models requires different approaches than testing deterministic functions. You can't assert exact outputs since model responses vary, but you can test your code's structure and error handling.
Mock the ABV client for unit tests to avoid making actual API calls:
```python
from unittest.mock import Mock, patch
import pytest
from chat_handler import handle_chat_request
def test_handle_chat_request():
"""Test chat request handling with mocked client."""
mock_response = {
'choices': [{
'message': {
'role': 'assistant',
'content': 'This is a test response'
}
}]
}
with patch('abv_client.abv.gateway.chat.completions.create') as mock_create:
mock_create.return_value = mock_response
result = handle_chat_request('Hello')
assert result == 'This is a test response'
mock_create.assert_called_once()
```
This approach tests your code's logic without depending on external services, making tests fast and deterministic.
For integration tests where you want to verify actual API behavior, make real requests but structure tests to be flexible about specific outputs:
```python
import pytest
from abv_client import abv
def test_gateway_returns_valid_responses():
"""Test that gateway returns properly structured responses."""
response = abv.gateway.chat.completions.create(
provider='openai',
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': 'Say hello'}]
)
# Assert structure, not specific content
assert 'choices' in response
assert len(response['choices']) == 1
assert 'message' in response['choices'][0]
assert 'content' in response['choices'][0]['message']
assert response['choices'][0]['message']['content']
assert 'usage' in response
assert response['usage']['total_tokens'] > 0
```
This approach verifies that your API integration works without depending on specific model outputs, making tests more robust.
## Next Steps
You now understand how to implement the gateway in Python applications, handle errors, manage conversations, and integrate with frameworks. Check the concepts guide to deepen your understanding of parameters and model behavior, or explore the providers guide to learn which providers and models work best for different use cases.
For production deployment considerations and best practices, see the best practices guide.
--------------------------------------------------------------------------------
## Overview
URL: /developer/guardrails/overview
# Guardrails
Guardrails automatically check your LLM's inputs and outputs to keep them safe, compliant, and high-quality. Think of them as safety checks that run before content reaches your users or your language model.
## What Problems Do Guardrails Solve?
When you're building with GenAI, you need to ensure that both user inputs and AI outputs meet your standards. Without guardrails, you might encounter issues like users submitting toxic messages that poison your AI's context, your AI generating biased content that damages your brand, structured outputs from your AI that break your application, or regulatory compliance failures that create legal risk.
Guardrails solve these problems by providing **automated validation** that runs in milliseconds or seconds, depending on what you're checking. Every check creates an **observation** in your ABV dashboard so you can monitor patterns and tune your validation strategy over time.
## How Guardrails Work
Using guardrails follows a simple pattern. First, you send text to a guardrail along with your configuration. The guardrail analyzes the text and returns a **result** that tells you whether the content passed or failed, explains why it made that decision, and provides a **confidence score** between 0.0 and 1.0. Based on this result, you decide whether to allow the content, block it, or flag it for human review.
Here's what this looks like in practice:
```typescript
import { ABVClient } from "@abvdev/client";
const abv = new ABVClient({ apiKey: process.env.ABV_API_KEY });
// Check user input for toxic content
const result = await abv.guardrails.toxicLanguage.validate(
userMessage,
{ sensitivity: "medium" }
);
if (result.status === "pass") {
// Safe to process
await sendToAI(userMessage);
} else {
// Block the message
return "Your message violates our community guidelines.";
}
```
## Available Guardrails
Guardrails come in two categories based on how they work. **AI-powered guardrails** use language models to understand context and nuance, taking about one to three seconds per check. **Rule-based guardrails** use deterministic logic for instant results in under ten milliseconds.
### AI-Powered Guardrails
**Toxic Language** detects hate speech, threats, insults, and other harmful content that could damage your community or poison your AI's context. Use this when you need to understand intent and tone, not just keywords.
**Biased Language** identifies stereotypes, discriminatory assumptions, and coded language that could harm your brand or violate compliance requirements. This guardrail is essential for HR content, job postings, and customer-facing communications.
### Rule-Based Guardrails
**Contains String** validates the presence or absence of specific text, perfect for ensuring legal disclaimers appear in AI outputs or blocking sensitive information like passwords and credit card numbers in user inputs.
**Valid JSON** ensures your AI's structured outputs are properly formatted and match your expected schema, preventing application errors from malformed data.
## Common Usage Patterns
Guardrails typically fit into two places in your application architecture. **Input validation** runs before you send user content to your AI, protecting your AI from toxic prompts and ensuring you only process valid requests. **Output validation** runs after your AI generates content but before you show it to users, maintaining brand safety and ensuring compliance.
Most applications use a **layered approach**. You might first run a fast rule-based check to catch obvious problems, then only run expensive AI-powered checks if the content passes the initial screening. This pattern minimizes cost while maintaining thorough validation.
## Performance and Cost
Understanding the tradeoffs between different guardrails helps you build efficient validation pipelines. **Rule-based guardrails** like Contains String and Valid JSON are instant and free because they use simple pattern matching. **AI-powered guardrails** like Toxic Language and Biased Language take one to three seconds and consume tokens because they use language models to understand context.
The best practice is to use rule-based guardrails first when possible, then fall back to AI-powered guardrails only when you need semantic understanding. This minimizes both **latency** and **cost** while maintaining thorough validation.
## Next Steps
Ready to implement guardrails in your application? Start here:
- [Quickstart](/developer/guardrails/quickstart): Get up and running with your first guardrail in minutes
- [Concepts](/developer/guardrails/concepts): Understand sensitivity levels, confidence scores, and validation strategies
- [Best Practices](/developer/guardrails/best-practices): Learn optimal patterns for layering and combining guardrails
- [Security Guardrails](/developer/guardrails/llm-security-guardrails-overview): Explore security-focused guardrails
### Available Guardrails
- [Biased Language](/developer/guardrails/available/biased-language): Detect stereotypes and discriminatory content
- [Contains String](/developer/guardrails/available/contains-string): Validate presence or absence of specific text
- [Valid JSON](/developer/guardrails/available/valid-json): Ensure properly formatted JSON outputs
## Integration with Other Features
Guardrails work seamlessly with other ABV features:
- **Observability**: Every guardrail check creates an [observation](/developer/basic-features/observability-tracing) in your traces for monitoring
- **SDKs**: Use guardrails with [Python SDK](/developer/sdks/python/overview) or [JS/TS SDK](/developer/sdks/js-ts/overview)
- **Evaluations**: Track guardrail performance over time using [Evaluations](/developer/evaluations/overview)
- **Metrics**: Monitor guardrail patterns in the [Metrics Dashboard](/developer/platform/metrics/overview)
--------------------------------------------------------------------------------
## Quickstart
URL: /developer/guardrails/quickstart
# Quickstart
This guide will have you running guardrails in under five minutes. You'll learn how to install the SDK, validate your first piece of content, and understand the result.
## Installation
First, install the ABV client library for your language.
```typescript
npm install @abvdev/client
```
```python
pip install abvdev
```
## Get Your API Key
You'll need an API key from your ABV dashboard. If you don't have one yet, sign up at app.abv.dev and create a new API key from your account settings.
## Your First Validation
Let's check if a user message contains toxic content. This is one of the most common uses of guardrails since it protects both your AI from poisoned context and your users from harmful responses.
```typescript
import { ABVClient } from "@abvdev/client";
// Initialize the client with your API key
const abv = new ABVClient({
apiKey: process.env.ABV_API_KEY,
});
// Check a user message for toxic content
const result = await abv.guardrails.toxicLanguage.validate(
"I really disagree with your approach to this problem.",
{ sensitivity: "medium" }
);
// The result tells you if the content passed or failed
console.log("Status:", result.status); // "pass", "fail", or "unsure"
console.log("Confidence:", result.confidence); // 0.0 to 1.0
console.log("Reason:", result.reason); // Explanation of the decision
```
```python
from abvdev import ABV
# Initialize the client with your API key
abv = ABV(api_key="your_api_key_here")
# Check a user message for toxic content
result = abv.guardrails.toxic_language.validate(
"I really disagree with your approach to this problem.",
{"sensitivity": "medium"}
)
# The result tells you if the content passed or failed
print("Status:", result["status"]) # "pass", "fail", or "unsure"
print("Confidence:", result["confidence"]) # 0.0 to 1.0
print("Reason:", result["reason"]) # Explanation of the decision
```
The message in this example expresses disagreement but does so professionally, so it should pass with high confidence. Try changing the message to something more hostile and see how the result changes.
## Understanding the Result
Every guardrail returns the same structure, which makes it easy to build consistent validation logic across your application. The status field tells you the outcome and will be pass if the content meets your criteria, fail if it violates them, or unsure for ambiguous cases. The confidence field gives you a score between zero and one indicating how certain the guardrail is about its decision. The reason field provides a human-readable explanation of why the guardrail made its decision.
You'll typically use the status field to make decisions in your code. Here's a common pattern:
```typescript
if (result.status === "pass") {
// Content is safe, continue processing
await processUserMessage(message);
} else if (result.status === "fail") {
// Content violated guidelines, block it
return { error: "Your message violates our guidelines." };
} else {
// Ambiguous case - you choose how to handle
// Conservative: treat as fail
// Permissive: treat as pass
// Balanced: flag for human review
await flagForReview(message, result);
}
```
```python
if result["status"] == "pass":
# Content is safe, continue processing
process_user_message(message)
elif result["status"] == "fail":
# Content violated guidelines, block it
return {"error": "Your message violates our guidelines."}
else:
# Ambiguous case - you choose how to handle
# Conservative: treat as fail
# Permissive: treat as pass
# Balanced: flag for human review
flag_for_review(message, result)
```
## Validating AI Outputs
Guardrails work equally well for checking AI-generated content before you show it to users. This helps maintain brand safety and compliance. Here's how you'd validate an AI response:
```typescript
// Generate a response from your AI
const aiResponse = await generateResponse(userPrompt);
// Check if the AI's response contains biased language
const validation = await abv.guardrails.biasedLanguage.validate(
aiResponse,
{ sensitivity: "high" }
);
// Only show the response if it passes
if (validation.status === "pass") {
return aiResponse;
} else {
// Either regenerate or use a fallback message
return "I apologize, I need to reconsider my response.";
}
```
```python
# Generate a response from your AI
ai_response = generate_response(user_prompt)
# Check if the AI's response contains biased language
validation = abv.guardrails.biased_language.validate(
ai_response,
{"sensitivity": "high"}
)
# Only show the response if it passes
if validation["status"] == "pass":
return ai_response
else:
# Either regenerate or use a fallback message
return "I apologize, I need to reconsider my response."
```
## Working with Structured Outputs
If your AI generates JSON, you can validate both the format and schema in one step. This is crucial for applications that parse AI outputs into structured data.
```typescript
// Your AI generated this response
const aiOutput = await generateStructuredResponse(prompt);
// Validate it matches your expected schema
const validation = await abv.guardrails.validJson.validate(
aiOutput,
{
schema: {
name: "string",
age: "number",
email: "string",
},
}
);
if (validation.status === "pass") {
// Safe to parse and use
const data = JSON.parse(aiOutput);
await saveToDatabase(data);
} else {
console.error("Invalid format:", validation.reason);
// Retry with a more explicit prompt
}
```
```python
# Your AI generated this response
ai_output = generate_structured_response(prompt)
# Validate it matches your expected schema
validation = abv.guardrails.valid_json.validate(
ai_output,
{
"schema": {
"name": "string",
"age": "number",
"email": "string"
}
}
)
if validation["status"] == "pass":
# Safe to parse and use
data = json.loads(ai_output)
save_to_database(data)
else:
print("Invalid format:", validation["reason"])
# Retry with a more explicit prompt
```
## Running Multiple Guardrails
You'll often want to check content against multiple criteria. Run guardrails in parallel to minimize latency.
```typescript
// Check multiple things at once
const [toxicCheck, biasCheck, formatCheck] = await Promise.all([
abv.guardrails.toxicLanguage.validate(content),
abv.guardrails.biasedLanguage.validate(content),
abv.guardrails.validJson.validate(content),
]);
// All must pass for content to be approved
const allPassed = [toxicCheck, biasCheck, formatCheck].every(
(result) => result.status === "pass"
);
```
```python
import asyncio
# Check multiple things at once
toxic_check, bias_check, format_check = await asyncio.gather(
abv.guardrails.toxic_language.validate_async(content),
abv.guardrails.biased_language.validate_async(content),
abv.guardrails.valid_json.validate_async(content)
)
# All must pass for content to be approved
all_passed = all(
result["status"] == "pass"
for result in [toxic_check, bias_check, format_check]
)
```
## Monitoring Your Guardrails
Every time you run a guardrail, ABV automatically creates an observation in your dashboard. This lets you monitor how often guardrails are failing, what reasons they're giving, and how confident they are in their decisions. You can use this data to tune your sensitivity settings and improve your validation strategy over time.
Visit app.abv.dev to view your observations and analyze patterns in your validation results.
## Next Steps
Now that you understand the basics, you can explore specific guardrails to learn about their unique configuration options. The toxic language documentation explains sensitivity levels and when to use each one. The biased language guide covers the different categories of bias you can check for. The contains string reference shows how to use rule-based checks for maximum performance. The valid JSON guide details schema validation and strict mode.
You should also read the concepts page to understand how sensitivity levels and confidence scores work across all guardrails.
--------------------------------------------------------------------------------
## Concepts
URL: /developer/guardrails/concepts
# Core Concepts
Understanding a few fundamental concepts will help you use guardrails effectively. This guide explains how guardrails work, what their results mean, and how to make decisions based on those results.
## How Guardrails Make Decisions
Guardrails fall into two categories based on how they analyze content. AI-powered guardrails use language models to understand context, nuance, and intent. When you send text to an AI-powered guardrail like toxic language or biased language, the guardrail asks a language model to analyze the content and make a judgment. This takes about one to three seconds but gives you sophisticated analysis that understands things like sarcasm, coded language, and cultural context.
Rule-based guardrails use deterministic logic like pattern matching and schema validation. When you send text to a rule-based guardrail like contains string or valid JSON, the guardrail applies fixed rules that always produce the same result for the same input. This takes under ten milliseconds and costs nothing, but it can only check for explicitly defined patterns.
The key difference is that AI-powered guardrails understand meaning while rule-based guardrails match patterns. If someone writes "people like you are the problem," an AI-powered guardrail recognizes this as hostile even though it doesn't contain any explicit profanity. A rule-based guardrail would only catch it if you explicitly listed that exact phrase.
## Understanding Results
Every guardrail returns a result with three essential pieces of information. The status tells you the outcome of the validation, the confidence score tells you how certain the guardrail is about its decision, and the reason provides a human-readable explanation.
The status field will be one of three values. When status is pass, the content meets your validation criteria and is safe to use. When status is fail, the content violates your criteria and should be blocked or regenerated. When status is unsure, the guardrail cannot make a confident determination and you need to decide how to handle this ambiguity.
The unsure status only appears with AI-powered guardrails since rule-based guardrails make binary decisions. You'll see unsure when content is genuinely ambiguous or when it sits right on the boundary between acceptable and unacceptable. For example, mild sarcasm might be hard to judge definitively, or a comment that could be interpreted as either critical feedback or a personal attack.
## Confidence Scores Explained
The confidence field ranges from zero to one and indicates how certain the guardrail is about its decision. Rule-based guardrails always return one because their logic is deterministic. There's no uncertainty when checking if a string contains another string or if JSON is valid.
AI-powered guardrails return variable confidence based on how clear-cut the case is. Very high confidence between 0.9 and 1.0 means the guardrail found clear, unambiguous indicators. High confidence between 0.7 and 0.9 means strong evidence supports the decision but with some minor ambiguity. Moderate confidence between 0.5 and 0.7 indicates the case has notable ambiguity. Low confidence under 0.5 means the content is borderline and difficult to judge definitively.
Several factors affect confidence scores. Content with obvious violations or clear acceptability produces high confidence. Ambiguous phrasing, sarcasm, or context-dependent meaning reduces confidence. Very short text with little context makes it harder to judge intent. Cultural or linguistic nuances that might be misinterpreted also reduce confidence.
You can use confidence scores to create more sophisticated decision logic. High confidence failures might be automatically blocked while low confidence failures are flagged for human review. This lets you balance automation with accuracy.
## Sensitivity Levels
AI-powered guardrails support sensitivity settings that adjust how strict they are. Think of sensitivity like a threshold. With low sensitivity, only severe violations trigger a failure. With medium sensitivity, clear violations trigger failures but borderline cases often pass. With high sensitivity, even mild or potential violations trigger failures.
The same content can pass at low sensitivity and fail at high sensitivity. This isn't inconsistency, it's flexibility. Different applications have different risk tolerances. A children's app needs high sensitivity to create a safe environment, while a professional forum for technical discussions needs low sensitivity to allow robust debate.
Low sensitivity works well when you expect strong opinions and professional disagreement. Use it for technical forums, professional feedback tools, and B2B communications where directness is valued. This setting flags only severe violations like explicit threats and hate speech.
Medium sensitivity is the default and works well for most applications. Use it for customer service, social media platforms, collaborative tools, and general business communications. This setting flags clear violations including insults and hostile language while allowing professional disagreement.
High sensitivity creates the safest environment by flagging any potentially harmful content. Use it for children's apps, educational platforms, mental health communities, and compliance-critical contexts. This setting blocks even mild rudeness or dismissive language.
## Making Decisions with Results
The simplest decision strategy uses only the status field. If status is pass, allow the content. If status is fail, block it. If status is unsure, you choose whether to treat it as pass or fail based on your risk tolerance.
A more sophisticated strategy considers both status and confidence. You might automatically block high-confidence failures above 0.8 but flag low-confidence failures for human review. This balances automation with accuracy by having humans make the final call on ambiguous cases.
Your application's risk tolerance should guide your strategy. High risk tolerance applications like entertainment or casual chat can be permissive and only block high-confidence failures. Medium risk tolerance applications like business tools should block all failures and flag unsure results. Low risk tolerance applications like children's apps should block both failures and unsure results, only allowing high-confidence passes.
## Response Times and Costs
Understanding the performance characteristics of different guardrails helps you build efficient validation pipelines. Rule-based guardrails respond in under ten milliseconds and consume zero tokens since they use local computation. AI-powered guardrails take one to three seconds and consume tokens since they make API calls to language models.
The best practice for performance is to run rule-based guardrails first. If a rule-based check fails, you can skip the expensive AI-powered checks entirely. This pattern is especially useful when you're running multiple guardrails. Check for forbidden strings first, then only run toxic language detection if the content passes the string check.
For cost optimization, use rule-based guardrails whenever possible and fall back to AI-powered guardrails only when you need semantic understanding. If you can enumerate the patterns you want to catch, contains string will be both faster and cheaper than an AI-powered guardrail.
## Observations and Monitoring
Every time you run a guardrail through the SDK, ABV automatically creates an observation in your dashboard. These observations capture the input text, the result including status and confidence, the guardrail configuration you used, how long the check took, and how many tokens were consumed for AI-powered guardrails.
Observations serve several purposes. You can monitor patterns over time to see how often your guardrails are failing and for what reasons. You can analyze confidence distributions to understand where ambiguity occurs in your validation. You can tune sensitivity settings based on real data from your application. You can debug unexpected results by examining the full context of a validation.
Visit your ABV dashboard regularly to review these observations. If you're seeing too many false positives where good content gets blocked, consider lowering your sensitivity. If you're seeing harmful content slip through, consider raising it. The observations give you the data you need to make these decisions.
## Best Practices for Combining Guardrails
Most applications use multiple guardrails together. The key to doing this efficiently is understanding dependencies between checks. Some guardrails are independent and can run in parallel. For example, checking for toxic language and checking for valid JSON don't depend on each other, so run them at the same time.
Other guardrails have a dependency relationship. If you want to check both for forbidden strings and for toxic language, run the string check first since it's instant. Only run the toxic language check if the string check passes. This creates a validation pipeline where fast checks act as a filter before expensive checks.
When running multiple independent guardrails in parallel, all checks happen simultaneously and you get results in the time it takes for the slowest check to complete. This is much faster than running checks sequentially. Just make sure you're actually checking independent criteria, not redundant ones.
## Next Steps
Now that you understand these core concepts, you're ready to explore specific guardrails. Each guardrail has its own documentation page that covers its unique configuration options, use cases, and examples. The toxic language page explains sensitivity levels in detail with specific examples. The biased language page covers the different categories of bias. The contains string page shows patterns for blocking or requiring specific text. The valid JSON page details schema validation.
For practical implementation advice, read the best practices guide which covers performance optimization, cost management, security considerations, and error handling patterns.
--------------------------------------------------------------------------------
## Best Practices
URL: /developer/guardrails/best-practices
# Best Practices
This guide covers strategies for using guardrails effectively in production applications. You'll learn how to optimize performance and costs, implement robust error handling, maintain security, and monitor your validation pipeline.
## Choosing Between AI-Powered and Rule-Based Guardrails
The first and most important decision in building your validation pipeline is understanding when to use AI-powered guardrails versus rule-based ones. This choice fundamentally affects your application's performance, cost, and accuracy.
Rule-based guardrails like contains string and valid JSON use deterministic logic. They check for exact patterns or validate against fixed rules. This makes them extremely fast, responding in under ten milliseconds, and completely free since they run locally without making any API calls. The tradeoff is that they can only catch what you explicitly define. If you're checking whether text contains the word "password", it will catch that exact word but won't catch "my pw" or "my login credentials" unless you've added those to your list.
AI-powered guardrails like toxic language and biased language use language models to understand context and meaning. They can recognize that "people like you are the problem" is hostile even though it contains no profanity. They understand sarcasm, coded language, and subtle implications. This power comes at a cost of about one to three seconds per check and token consumption for the API call.
The key insight for building efficient systems is that you should use rule-based guardrails whenever your requirements can be expressed as explicit patterns. Save AI-powered guardrails for cases where you need semantic understanding. In many scenarios, you can use both in sequence, with the fast rule-based check acting as a filter before the expensive AI check.
## Building Efficient Validation Pipelines
Understanding how to sequence and combine guardrails is essential for building performant applications. The pattern that works best in most cases involves running fast checks first to catch obvious issues, then running expensive checks only when necessary.
Consider a scenario where you want to validate user input for both forbidden terms and toxic content. Your intuition might be to run both checks in parallel to minimize latency. However, if the content contains an explicitly forbidden term, there's no point in spending one to three seconds and token cost to analyze its tone. Instead, check for forbidden terms first using contains string, which takes under ten milliseconds. Only if that passes do you run the toxic language check.
This sequential pattern creates a validation pipeline where each stage filters out content before more expensive stages run. The earlier stages should be faster and cheaper, acting as gatekeepers. Here's what this looks like in practice:
```typescript
async function validateContent(content: string): Promise<{ valid: boolean; reason?: string }> {
// Stage 1: Fast rule-based checks (< 10ms total, $0)
// Check for forbidden terms first
const forbiddenCheck = await abv.guardrails.containsString.validate(content, {
strings: ["forbidden-term", "explicit-slur", "banned-phrase"],
mode: "none"
});
if (forbiddenCheck.status === "fail") {
return { valid: false, reason: "Contains forbidden content" };
}
// Check JSON structure if relevant
if (expectingJson) {
const jsonCheck = await abv.guardrails.validJson.validate(content);
if (jsonCheck.status === "fail") {
return { valid: false, reason: "Invalid format" };
}
}
// Stage 2: AI-powered checks (only if stage 1 passed)
// These can run in parallel since they're independent
const [toxicCheck, biasCheck] = await Promise.all([
abv.guardrails.toxicLanguage.validate(content, { sensitivity: "medium" }),
abv.guardrails.biasedLanguage.validate(content, { sensitivity: "medium" })
]);
if (toxicCheck.status === "fail") {
return { valid: false, reason: "Contains toxic content" };
}
if (biasCheck.status === "fail") {
return { valid: false, reason: "Contains biased language" };
}
return { valid: true };
}
```
```python
async def validate_content(content: str) -> dict:
# Stage 1: Fast rule-based checks (< 10ms total, $0)
# Check for forbidden terms first
forbidden_check = await abv.guardrails.contains_string.validate_async(content, {
"strings": ["forbidden-term", "explicit-slur", "banned-phrase"],
"mode": "none"
})
if forbidden_check["status"] == "fail":
return {"valid": False, "reason": "Contains forbidden content"}
# Check JSON structure if relevant
if expecting_json:
json_check = await abv.guardrails.valid_json.validate_async(content)
if json_check["status"] == "fail":
return {"valid": False, "reason": "Invalid format"}
# Stage 2: AI-powered checks (only if stage 1 passed)
# These can run in parallel since they're independent
toxic_check, bias_check = await asyncio.gather(
abv.guardrails.toxic_language.validate_async(content, {"sensitivity": "medium"}),
abv.guardrails.biased_language.validate_async(content, {"sensitivity": "medium"})
)
if toxic_check["status"] == "fail":
return {"valid": False, "reason": "Contains toxic content"}
if bias_check["status"] == "fail":
return {"valid": False, "reason": "Contains biased language"}
return {"valid": True}
```
This pattern minimizes cost and latency by failing fast when possible. Most forbidden content gets caught in under ten milliseconds by the rule-based checks. Only content that passes those hurdles gets the full AI analysis. When you do run AI-powered checks, you run them in parallel since they're independent analyses, which means your total AI check time is the maximum of the individual checks rather than the sum.
## Selecting the Right Sensitivity Level
Sensitivity level is one of the most important configuration choices for AI-powered guardrails, and it requires careful thought about your application's context and risk tolerance. The right sensitivity balances safety against false **positive**s.
Consider starting with medium sensitivity in development and adjusting based on your observations. Medium represents a balanced default that catches clear violations while allowing professional disagreement. As you review your guardrail observations in the ABV dashboard, you'll see patterns that help you tune your sensitivity.
If you're seeing too many false **positive**s where acceptable content gets blocked, you're probably using sensitivity that's too high for your context. For example, if you're building a technical forum where people debate code quality and you're seeing technical critiques getting flagged, consider lowering your sensitivity. Technical discussions naturally include strong opinions about approaches and implementations.
If you're seeing problematic content slip through, your sensitivity is probably too low. This is particularly important to watch for in consumer-facing applications or applications used by children. Any hostile content that reaches users damages trust and potentially exposes you to liability.
You can also implement context-aware sensitivity where different parts of your application or different user types get different sensitivity levels. A children's section gets high sensitivity automatically. Content from authenticated users with good history might get medium sensitivity while anonymous users get high sensitivity. Customer service queries might get lower sensitivity since you expect frustrated customers to express strong emotions.
## Implementing Sophisticated Decision Logic
The simplest decision logic uses only the status field. If status is pass you allow the content, if it's fail you block it, and if it's unsure you choose whether to be conservative and block or permissive and allow. This works for many applications, but you can build more sophisticated logic using confidence scores and multiple guardrails together.
Consider implementing tiered responses based on confidence. When a guardrail returns fail with high confidence above 0.9, you can automatically block the content without human review. When it returns fail with moderate confidence between 0.6 and 0.9, you might flag it for human review rather than automatically blocking. When it returns fail with low confidence below 0.6 or returns unsure, you definitely want human eyes on it before making a final decision.
This approach recognizes that AI-powered guardrails are probabilistic systems. They're very good at catching clear violations but less certain about edge cases. By using confidence scores, you can automate the obvious cases while routing ambiguous cases to humans. This gives you both efficiency and accuracy.
Here's what tiered decision logic looks like:
```typescript
async function handleContent(content: string, context: "public" | "private") {
// Adjust sensitivity based on context
const sensitivity = context === "public" ? "high" : "medium";
const result = await abv.guardrails.toxicLanguage.validate(content, { sensitivity });
// Tiered decision logic based on confidence
if (result.status === "pass") {
// Always allow clear passes
return { action: "approve", content };
}
if (result.status === "fail" && result.confidence > 0.9) {
// High-confidence violations: auto-block
await logRejection(content, result.reason, "auto");
return { action: "reject", message: "Content violates guidelines" };
}
if (result.status === "fail" && result.confidence > 0.6) {
// Medium-confidence violations: flag for review
await queueForReview(content, result);
return { action: "pending", message: "Content under review" };
}
// Low confidence or unsure: always review
await queueForReview(content, result);
return { action: "pending", message: "Content under review" };
}
```
```python
async def handle_content(content: str, context: str):
# Adjust sensitivity based on context
sensitivity = "high" if context == "public" else "medium"
result = await abv.guardrails.toxic_language.validate_async(content, {"sensitivity": sensitivity})
# Tiered decision logic based on confidence
if result["status"] == "pass":
# Always allow clear passes
return {"action": "approve", "content": content}
if result["status"] == "fail" and result["confidence"] > 0.9:
# High-confidence violations: auto-block
await log_rejection(content, result["reason"], "auto")
return {"action": "reject", "message": "Content violates guidelines"}
if result["status"] == "fail" and result["confidence"] > 0.6:
# Medium-confidence violations: flag for review
await queue_for_review(content, result)
return {"action": "pending", "message": "Content under review"}
# Low confidence or unsure: always review
await queue_for_review(content, result)
return {"action": "pending", "message": "Content under review"}
```
## Security and Privacy Considerations
Guardrails are part of your security posture, and you need to think carefully about how you use them to avoid creating vulnerabilities. The most important security principle is to never expose the detailed reason field to end users. When a guardrail blocks content, the reason explains exactly why, and this information helps attackers understand your validation logic so they can evade it.
Instead of showing the detailed reason, use generic error messages to users while logging the detailed reason internally for monitoring and debugging. This prevents adversarial learning where users iterate on their inputs to find ways around your guardrails.
Another security consideration is validating in both directions. You should validate user inputs before sending them to your AI to prevent prompt injection and context poisoning. You should also validate AI outputs before showing them to users to maintain brand safety and compliance. Don't assume that because your AI is generally helpful it will never generate problematic content. AIs can be manipulated through cleverly crafted prompts, and even without manipulation they sometimes generate content you wouldn't want to show users.
Rate limiting is another security measure worth implementing. Guardrail checks have a cost, and malicious actors could potentially run up your bill by making many validation requests. Implement rate limits on how many validations a single user or IP address can request in a given time window. This protects both your costs and your service availability.
Consider implementing monitoring and alerting for unusual patterns. If you suddenly see a spike in failed validations from a particular user or for a particular type of content, that might indicate an attack or a system misconfiguration. The ABV dashboard gives you observability into these patterns, and you should regularly review it for anomalies.
## Error Handling and Resilience
Guardrails make network requests to external services, which means they can fail in ways beyond your control. Network issues, service outages, and rate limits can all cause validation requests to fail. You need to decide how to handle these failure modes based on your application's risk tolerance.
The two fundamental strategies are failing open and failing closed. Failing open means that when validation fails due to an error, you allow the content through. This prioritizes availability over security. Use failing open when blocking legitimate content due to a service outage would be worse than allowing some potentially problematic content through temporarily. Failing closed means that when validation fails, you block the content. This prioritizes security over availability. Use failing closed when allowing problematic content through could cause serious harm or legal liability.
Most applications should fail closed for user-facing content and fail open for internal content. If an end user is posting content that will be seen by other users, fail closed to protect your community. If an employee is using an internal tool and validation fails, fail open to avoid disrupting their work.
Implement retry logic with exponential backoff for transient failures. Network blips and temporary service issues often resolve themselves quickly. If a validation request fails, wait a moment and try again. If it fails again, wait longer and try again. After a few attempts, accept the failure and fall back to your fail-open or fail-closed strategy.
Here's what robust error handling looks like:
```typescript
async function validateWithRetry(
content: string,
maxRetries: number = 3
): Promise<{ valid: boolean; error?: string }> {
let lastError: Error | undefined;
for (let attempt = 0; attempt < maxRetries; attempt++) {
try {
const result = await abv.guardrails.toxicLanguage.validate(content);
// Success - return the result
return {
valid: result.status === "pass"
};
} catch (error) {
lastError = error as Error;
// Log the error
console.error(`Validation attempt ${attempt + 1} failed:`, error);
// If this isn't our last attempt, wait before retrying
if (attempt < maxRetries - 1) {
const delay = Math.pow(2, attempt) * 1000; // Exponential backoff
await new Promise(resolve => setTimeout(resolve, delay));
}
}
}
// All retries failed - implement fail-closed strategy
// For production, you'd make this configurable
console.error("All validation attempts failed, blocking content");
return {
valid: false,
error: "Unable to validate content, please try again"
};
}
```
```python
async def validate_with_retry(
content: str,
max_retries: int = 3
) -> dict:
last_error = None
for attempt in range(max_retries):
try:
result = await abv.guardrails.toxic_language.validate_async(content)
# Success - return the result
return {"valid": result["status"] == "pass"}
except Exception as error:
last_error = error
# Log the error
print(f"Validation attempt {attempt + 1} failed: {error}")
# If this isn't our last attempt, wait before retrying
if attempt < max_retries - 1:
delay = (2 ** attempt) * 1.0 # Exponential backoff
await asyncio.sleep(delay)
# All retries failed - implement fail-closed strategy
# For production, you'd make this configurable
print("All validation attempts failed, blocking content")
return {
"valid": False,
"error": "Unable to validate content, please try again"
}
```
## Monitoring and Continuous Improvement
Guardrails aren't set-it-and-forget-it systems. They require ongoing monitoring and tuning based on real data from your application. The ABV dashboard provides observability into every guardrail execution, and you should develop a practice of regularly reviewing this data.
Look for patterns in failures. If you're seeing the same types of content repeatedly flagged, that might indicate that your sensitivity is miscalibrated or that you need to adjust your prompts if you're validating AI outputs. If you're seeing very few failures, you might be using sensitivity that's too low and missing problematic content.
Pay attention to confidence distributions. If most of your failures have low confidence, that suggests you're on the edge of ambiguity and might benefit from human review workflows. If most of your failures have high confidence, your validation is clear-cut and you can confidently automate decisions.
Monitor your false **positive** rate by sampling content that was blocked and manually reviewing whether it should have been allowed. This helps you tune sensitivity and adjust your configuration to reduce cases where good content gets incorrectly blocked. Similarly, monitor your false **negative** rate by sampling content that was allowed and checking whether any of it should have been blocked. This is harder to do systematically but is important for maintaining safety.
Track your costs and performance metrics. How much are you spending on guardrail validation? What's your average latency? Are there specific patterns where validation is particularly slow or expensive? Use this data to optimize your validation pipeline by implementing caching where appropriate, using rule-based pre-filters more aggressively, or adjusting which guardrails you run in which scenarios.
## Caching and Performance Optimization
For applications with high throughput, consider implementing caching for validation results. If you're checking the same content repeatedly, there's no need to validate it each time. Cache the result and reuse it. This is particularly valuable for template content or common phrases that appear frequently.
Be thoughtful about cache invalidation. If you change your guardrail configuration, particularly sensitivity levels, you need to invalidate cached results since they might no longer be accurate. Similarly, if you're caching AI-powered guardrail results, consider setting a TTL since language models can be updated and results might change over time.
Another performance optimization is batching validations when possible. If you have multiple pieces of content to validate and they're independent, validate them in parallel. If you're validating a document with multiple sections, you might validate each section in parallel rather than sequentially. This multiplies your throughput without increasing your per-item latency.
## Next Steps
The patterns and principles in this guide give you a foundation for using guardrails effectively in production. As you gain experience with guardrails in your specific application, you'll develop intuition for which configurations work best for your use case. The key is to treat guardrails as a system that requires ongoing attention and tuning, not as a static configuration you set once and forget about.
Make sure to explore the specific guardrail documentation pages for detailed information about each guardrail's unique characteristics and configuration options. The toxic language page covers sensitivity levels in depth. The biased language page explains the different categories of bias. The contains string page shows efficient pattern matching strategies. The valid JSON page details schema validation options. Understanding these specifics helps you make informed decisions about which guardrails to use and how to configure them.
--------------------------------------------------------------------------------
## LLM Security & Guardrails
URL: /developer/guardrails/llm-security-guardrails-overview
There are a host of potential safety risks involved with LLM-based applications. These include prompt injection, leakage of personally identifiable information (PII), or harmful prompts. ABV can be used to monitor and protect against these security risks, and investigate incidents when they occur.
# What is LLM Security?
LLM Security involves implementing protective measures to safeguard LLMs and their infrastructure from unauthorized access, misuse, and adversarial attacks, ensuring the integrity and confidentiality of both the model and data. This is crucial in AI/ML systems to maintain ethical usage, prevent security risks like prompt injections, and ensure reliable operation under safe conditions.
# How does LLM Security work?
LLM Security can be addressed with a combination of
- LLM Security libraries for run-time security measures
- ABV for the ex-post evaluation of the effectiveness of these measures
## 1. Run-time security measures
There are several popular security libraries that can be used to mitigate security risks in LLM-based applications. These include: [LLM Guard](https://llm-guard.com), [Prompt Armor](https://promptarmor.com), [NeMo Guardrails](https://github.com/NVIDIA/NeMo-Guardrails), [Microsoft Azure AI Content Safety](https://azure.microsoft.com/en-us/products/ai-services/ai-content-safety), [Lakera](https://www.lakera.ai). These libraries help with security measures in the following ways:
1. Catching and blocking a potentially harmful or inappropriate prompt before sending to the model
2. Redacting sensitive PII before being sending into the model and then un-redacting in the response
3. Evaluating prompts and completions on toxicity, relevance, or sensitive material at run-time and blocking the response if necessary
## 2. Monitoring and evaluation of security measures with ABV
Use ABV [tracing](/developer/basic-features/observability-tracing) to gain visibility and confidence in each step of the security mechanism. These are common workflows:
1. Manually inspect traces to investigate security issues.
2. Monitor security scores over time in the ABV Dashboard.
3. Validate security checks. You can use ABV [scores](/developer/evaluations/overview) to evaluate the effectiveness of security tools. Integrating ABV into your team's workflow can help teams identify which security risks are most prevalent and build more robust tools around those specific issues. There are two main workflows to consider:
- [Annotations (in UI)](/developer/evaluations/human-annotation). If you establish a baseline by annotating a share of production traces, you can compare the security scores returned by the security tools with these annotations.
- [Automated evaluations](/developer/evaluations/llm-as-a-judge). ABV's model-based evaluations will run asynchronously and can scan traces for things such as toxicity or sensitivity to flag potential risks and identify any gaps in your LLM security setup. Check out the docs to learn more about how to set up these evaluations.
4. Track Latency. Some LLM security checks need to be awaited before the model can be called, others block the response to the user. Thus they quickly are an essential driver of overall latency of an LLM application. ABV can help disect the latencies of these checks within a trace to understand whether the checks are worth the wait.
# Getting Started
**Example: Anonymizing Personally Identifiable Information (PII)**
Exposing PII to LLMs can pose serious security and privacy risks, such as violating contractual obligations or regulatory compliance requirements, or mitigating the risks of data leakage or a data breach.
Personally Identifiable Information (PII) includes:
- Credit card number
- Full name
- Phone number
- Email address
- Social Security number
- IP Address
The example below shows a simple application that summarizes a given court transcript. For privacy reasons, the application wants to anonymize PII before the information is fed into the model, and then un-redact the response to produce a coherent summary.
To read more about other security risks, including prompt injection, banned topics, or malicious URLs, please check out the docs of the various libraries or read our which includes more examples.
## 1) Install packages
In this example we use the open source library [LLM Guard](https://llm-guard.com/) for run-time security checks. All examples easily translate to other libraries such as [Prompt Armor](https://promptarmor.com), [NeMo Guardrails](https://github.com/NVIDIA/NeMo-Guardrails), [Microsoft Azure AI Content Safety](https://azure.microsoft.com/en-us/products/ai-services/ai-content-safety), and [Lakera](https://www.lakera.ai).
First, import the security packages and ABV tools.
```bash
pip install llm-guard abv openai
```
```python
from llm_guard.input_scanners import Anonymize
from llm_guard.input_scanners.anonymize_helpers import BERT_LARGE_NER_CONF
from abvdev import observe
from llm_guard.output_scanners import Deanonymize
from llm_guard.vault import Vault
```
## 2) Anonymize and deanonymize PII and trace with ABV
We break up each step of the process into its own function so we can track each step separately in ABV.
By decorating the functions with `@observe()`, we can trace each step of the process and monitor the risk scores returned by the security tools. This allows us to see how well the security tools are working and whether they are catching the PII as expected.
```python
vault = Vault()
@observe()
def anonymize(input: str):
scanner = Anonymize(vault, preamble="Insert before prompt", allowed_names=["John Doe"], hidden_names=["Test LLC"],
recognizer_conf=BERT_LARGE_NER_CONF, language="en")
sanitized_prompt, is_valid, risk_score = scanner.scan(input)
return sanitized_prompt
@observe()
def deanonymize(sanitized_prompt: str, answer: str):
scanner = Deanonymize(vault)
sanitized_model_output, is_valid, risk_score = scanner.scan(sanitized_prompt, answer)
return sanitized_model_output
```
## 3) Instrument LLM call
Note: ABV natively integrates with Python and JS/TS frameworks and you can easily instrument any LLM via the [SDKs](/developer/sdks/overview).
```python
@observe()
def summarize_transcript(prompt: str):
sanitized_prompt = anonymize(prompt)
with abv.start_as_current_observation(as_type="generation", name="OpenAI-gen"):
answer = OpenAI().chat.completions.create(
model="gpt-5-2025-08-07",
max_tokens=100,
messages=[
{"role": "system", "content": "Summarize the given court transcript."},
{"role": "user", "content": sanitized_prompt}
],
).choices[0].message.content
sanitized_model_output = deanonymize(sanitized_prompt, answer)
return sanitized_model_output
```
## 4) Execute the application
Run the function. In this example, we input a section of a court transcript. Applications that handle sensitive information will often need to use anonymize and deanonymize functionality to comply with data privacy policies such as HIPAA or GDPR.
```python
prompt = """
Plaintiff, Jane Doe, by and through her attorneys, files this complaint
against Defendant, Big Corporation, and alleges upon information and belief,
except for those allegations pertaining to personal knowledge, that on or about
July 15, 2023, at the Defendant's manufacturing facility located at 123 Industrial Way, Springfield, Illinois, Defendant negligently failed to maintain safe working conditions,
leading to Plaintiff suffering severe and permanent injuries. As a direct and proximate
result of Defendant's negligence, Plaintiff has endured significant physical pain, emotional distress, and financial hardship due to medical expenses and loss of income. Plaintiff seeks compensatory damages, punitive damages, and any other relief the Court deems just and proper.
"""
summarize_transcript(prompt)
```
## 5) Inspect trace in ABV
In this trace (), we can see how the name of the plaintiff is anonymized before being sent to the model, and then un-redacted in the response. We can now evaluate run evaluations in ABV to control for the effectiveness of these measures.
# More Examples
Find more examples of LLM security monitoring in our cookbook.
Cookbook: Observing LLM Security
--------------------------------------------------------------------------------
## Biased Language
URL: /developer/guardrails/available/biased-language
# Biased Language
The biased language guardrail detects discriminatory content, stereotypes, and prejudiced assumptions that could harm your brand or violate compliance requirements. This guardrail uses AI to identify not just explicit bias but also subtle coded language and implicit assumptions that keyword filters would miss.
## When to Use This Guardrail
You should use biased language detection when you're creating content that could have legal or compliance implications, when you need to maintain your brand's commitment to diversity and inclusion, or when you're generating content that will be seen by diverse audiences. This guardrail is essential for HR systems and recruiting platforms where biased language can expose you to legal liability, for marketing and advertising content where bias can damage your brand reputation, for educational platforms where you want to model inclusive language, and for AI-generated content where the AI might inadvertently reproduce biases from its training data.
The key difference between this guardrail and simple keyword filtering is that bias often appears in subtle forms. Someone might never use an explicit slur but still express bias through coded language like "culture fit" or "digital native" that excludes certain groups. The guardrail analyzes assumptions and implications, not just word choice.
## Understanding Bias Categories
Biased language detection can focus on specific categories of bias relevant to your use case. You can check all categories or narrow your focus to the types of bias most important for your application.
Gender bias includes discrimination based on gender identity, biological sex, or gender expression. This appears in contexts like assuming technical roles are for men or caregiving roles are for women, using gendered language when gender is irrelevant, or making assumptions about capabilities based on gender. For example, "We need a strong leader for this role" might subtly code for masculine traits, while "looking for a friendly receptionist" might code for feminine ones.
Race and ethnicity bias involves discrimination based on race, ethnic background, or national origin. This can appear as stereotypes about work ethic or intelligence, assumptions about cultural background based on names, or preferences for certain ethnic groups disguised as "culture fit." A job posting requiring "native English speaker" might unintentionally discriminate based on national origin when language proficiency is what actually matters.
Religious bias includes discrimination based on religious beliefs or lack of religious beliefs. This might appear as assumptions about availability for work on certain days, preferences for certain religious backgrounds, or judgments about character based on religious affiliation. For example, requiring attendance at weekend events without acknowledging that this might conflict with religious observance.
Age bias involves discrimination based on age or generation. This frequently appears in job postings through terms like "recent graduate," "digital native," or "energetic," which all code for younger workers. On the flip side, terms like "overqualified" can discriminate against older workers. The guardrail catches both ends of age discrimination.
Nationality bias relates to discrimination based on citizenship status or national origin. This might appear as unnecessary citizenship requirements, preferences for people from certain countries, or assumptions about work authorization. For example, "Must be a US citizen" in a job posting when citizenship isn't legally required for the role.
Political bias involves discrimination based on political beliefs or affiliations. While less common in most business contexts, this can appear when companies imply political litmus tests for employment or when content makes assumptions about audience political views.
Disability bias includes discrimination based on physical or mental abilities and disabilities. This often appears as assumptions about what disabled people can or cannot do, requirements for abilities that aren't actually essential to the job, or language that characterizes disability negatively. For example, "must be able to lift 50 pounds" when the job rarely requires lifting, or using disability as a metaphor for negative traits.
Socioeconomic bias involves discrimination based on economic status or social class. This might appear as assumptions about educational background, requirements for unpaid internships that exclude those who can't afford to work without pay, or judgments about neighborhoods or schools people come from.
## Focusing Your Detection
You can configure the guardrail to check specific categories rather than all categories. This is useful when you know which types of bias are most relevant to your context or when you want to reduce processing time by focusing on your primary concerns.
```typescript
// Check all categories (default)
await abv.guardrails.biasedLanguage.validate(text, {
sensitivity: "medium"
});
// Check specific categories for a job posting
await abv.guardrails.biasedLanguage.validate(jobDescription, {
sensitivity: "high",
categories: ["gender", "race", "age", "disability"]
});
// Focus on just gender bias
await abv.guardrails.biasedLanguage.validate(content, {
categories: ["gender"]
});
```
```python
# Check all categories (default)
await abv.guardrails.biased_language.validate_async(text, {
"sensitivity": "medium"
})
# Check specific categories for a job posting
await abv.guardrails.biased_language.validate_async(job_description, {
"sensitivity": "high",
"categories": ["gender", "race", "age", "disability"]
})
# Focus on just gender bias
await abv.guardrails.biased_language.validate_async(content, {
"categories": ["gender"]
})
```
## Understanding Sensitivity Levels
Like toxic language detection, bias detection supports sensitivity **level**s that control how strict the guardrail is. The right sensitivity depends on your risk tolerance and the context where the content will appear.
Low sensitivity flags only explicit discrimination and severe stereotypes. At this level, the guardrail looks for clear, unambiguous bias that most people would immediately recognize. A statement like "Women aren't suited for engineering roles" would fail, but subtle implications might pass. Use low sensitivity when you're checking general content where mild stereotypes are less critical and when you want to avoid false **positive**s that might block acceptable content.
Medium sensitivity is the default and represents a balanced approach. At this level, the guardrail flags clear stereotypes and coded language while still allowing neutral demographic references and objective descriptions. For example, "Looking for a digital native for this role" would fail because "digital native" is coded language for young workers, but "Must have 3 years of experience" would pass. Use medium sensitivity for most professional content, customer-facing communications, and general business contexts.
High sensitivity creates the most inclusive environment by flagging any potential bias including subtle stereotypes and generalizations about groups. At this level, even statements that might seem neutral could fail if they make assumptions about groups. For example, "Seeking recent graduates with fresh perspectives" would fail because it implies age bias. Use high sensitivity for contexts where inclusion is paramount, such as DEI-focused applications, HR systems, compliance-critical content, and any context where you want to avoid even subtle bias.
## Real-World Examples
Let's examine specific examples to understand how the guardrail evaluates different types of content.
Consider a job posting that says "We're looking for software engineers with strong technical skills." This passes at all sensitivity **level**s because it focuses on skills without making assumptions about who possesses those skills. The language is neutral and objective.
Now consider "We're looking for a rockstar engineer who can hit the ground running." This might pass at low sensitivity but could fail at medium or high sensitivity. "Rockstar" can code for young, and "hit the ground running" might imply you won't provide training, potentially excluding career changers or those returning to the workforce.
A posting that says "Seeking recent graduates with fresh ideas" would fail at medium and high sensitivity. "Recent graduates" explicitly discriminates by age, and "fresh ideas" implies older workers don't have fresh ideas. At low sensitivity this might pass since it's not severely discriminatory, but it's still problematic.
Content that says "Must be a cultural fit with our young, dynamic team" would fail at all sensitivity **level**s. This combines multiple forms of bias, describing the team as young which signals age discrimination, and "cultural fit" is well-known coded language often used to discriminate against people who are different from the existing team.
For marketing content, consider "Stay-at-home moms will love this product." This fails at medium and high sensitivity because it assumes caregiving is done by mothers rather than parents, reinforcing gender stereotypes. Better phrasing would be "Parents managing household responsibilities will appreciate this product."
## Implementation Patterns
Here's how you'd typically implement bias detection in different scenarios. For job posting validation, you'd check content before publishing and provide feedback for revision:
```typescript
async function validateJobPosting(description: string) {
const result = await abv.guardrails.biasedLanguage.validate(
description,
{
sensitivity: "high",
categories: ["gender", "race", "age", "disability"]
}
);
if (result.status === "pass") {
return { approved: true };
}
// Don't expose the detailed reason to the user
// Instead, provide generic guidance
return {
approved: false,
feedback: "The job posting may contain biased language. Please review for inclusive phrasing."
};
}
```
```python
async def validate_job_posting(description: str):
result = await abv.guardrails.biased_language.validate_async(
description,
{
"sensitivity": "high",
"categories": ["gender", "race", "age", "disability"]
}
)
if result["status"] == "pass":
return {"approved": True}
# Don't expose the detailed reason to the user
# Instead, provide generic guidance
return {
"approved": False,
"feedback": "The job posting may contain biased language. Please review for inclusive phrasing."
}
```
For AI-generated content, you'd validate outputs before showing them to ensure your AI doesn't inadvertently reproduce biases from its training data:
```typescript
async function generateInclusiveContent(prompt: string): Promise {
// Generate initial content
let content = await callAI(prompt);
// Validate for bias
const validation = await abv.guardrails.biasedLanguage.validate(
content,
{ sensitivity: "high" }
);
// If biased, regenerate with explicit instruction
if (validation.status === "fail") {
content = await callAI(
prompt + "\n\nIMPORTANT: Use inclusive, unbiased language that doesn't make assumptions about any demographic groups."
);
}
return content;
}
```
```python
async def generate_inclusive_content(prompt: str) -> str:
# Generate initial content
content = await call_ai(prompt)
# Validate for bias
validation = await abv.guardrails.biased_language.validate_async(
content,
{"sensitivity": "high"}
)
# If biased, regenerate with explicit instruction
if validation["status"] == "fail":
content = await call_ai(
f"{prompt}\n\nIMPORTANT: Use inclusive, unbiased language that doesn't make assumptions about any demographic groups."
)
return content
```
For high-stakes content like HR communications, you might implement a review workflow where ambiguous cases get flagged for human review:
```typescript
async function reviewHRContent(content: string) {
const result = await abv.guardrails.biasedLanguage.validate(
content,
{ sensitivity: "high" }
);
if (result.status === "pass" && result.confidence > 0.8) {
// High confidence pass - approve automatically
return { action: "approve" };
} else if (result.status === "fail" && result.confidence > 0.8) {
// High confidence failure - reject automatically
return { action: "reject", reason: "Contains biased language" };
} else {
// Low confidence or unsure - flag for review
return { action: "review", confidence: result.confidence };
}
}
```
```python
async def review_hr_content(content: str):
result = await abv.guardrails.biased_language.validate_async(
content,
{"sensitivity": "high"}
)
if result["status"] == "pass" and result["confidence"] > 0.8:
# High confidence pass - approve automatically
return {"action": "approve"}
elif result["status"] == "fail" and result["confidence"] > 0.8:
# High confidence failure - reject automatically
return {"action": "reject", "reason": "Contains biased language"}
else:
# Low confidence or unsure - flag for review
return {"action": "review", "confidence": result["confidence"]}
```
## Choosing Categories for Your Context
Different application types benefit from focusing on specific bias categories. For recruiting and HR systems, focus on gender, race, age, and disability since these are the categories most commonly involved in employment discrimination cases. For marketing and advertising, consider gender, race, and socioeconomic status since these affect how different audiences perceive your brand. For educational content, check all categories since you want to model inclusive thinking across all dimensions. For political content or news, focus on political, religious, and nationality bias to maintain neutrality.
You can also adjust your focus based on your geographic context. Different regions have different legal requirements and cultural sensitivities around various forms of bias. Make sure you're covering the categories that matter most for your jurisdiction and audience.
## Combining with Other Guardrails
Bias detection often works alongside other content safety measures. You might combine it with toxic language detection to catch both hostile content and biased content. You might use contains string to quickly block explicitly prohibited terms before running the more sophisticated bias detection. You might validate both for bias and for proper JSON structure if you're generating structured content from an AI.
When combining multiple guardrails, run them in parallel when they're independent checks. Both bias detection and toxic language detection analyze the same text for different criteria, so run them at the same time. If you're also checking for valid JSON structure, that check is independent too and should happen in parallel. This minimizes your total latency compared to running checks sequentially.
## Next Steps
For more sophisticated implementation patterns including caching strategies, error handling, and monitoring approaches, see the best practices guide. If you're specifically working on job postings or HR content, pay special attention to the sections on high sensitivity configuration and compliance considerations.
You might also want to explore toxic language detection which catches different types of harmful content and complements bias detection well. Together, these two guardrails provide comprehensive content safety for most applications.
--------------------------------------------------------------------------------
## Contains String
URL: /developer/guardrails/available/contains-string
# Overview
The contains string guardrail validates text against specific strings using pattern matching. Unlike AI-powered guardrails that interpret meaning, this guardrail simply checks whether certain strings appear in your text. This makes it extremely fast, completely predictable, and free since it runs locally without any AI calls.
## When to Use This Guardrail
You should use contains string when you can enumerate exactly what you're looking for and when speed and cost matter. This guardrail excels at several specific tasks. It can ensure legal disclaimers appear in AI-generated content, block sensitive information like passwords or credit card numbers in user inputs, verify that all required sections exist in documents or emails, prevent competitor mentions in your marketing content, and catch explicitly forbidden terms before running more expensive AI-powered checks.
The key insight is that this guardrail works best when you know exactly what you're looking for. If someone needs to include "not financial advice" in their content, you can check for that exact string. If you never want to see "password" or "credit card" in user messages, you can block those exact strings. This precision makes the guardrail extremely reliable but also limited to the patterns you explicitly define, and ideal to be used as a "front-line" guardrail meant to filter out obvious violations.
## Understanding Match Modes
The contains string guardrail supports three modes that change its behavior fundamentally. Understanding these modes is essential to using the guardrail effectively.
ANY mode passes when at least one string from your list appears in the text. Think of this as an OR operation. If you provide the strings "not financial advice" and "consult a professional," the text passes if it contains either one. This mode is perfect for ensuring disclaimers where you want to give the content creator some flexibility in their exact wording. For example, you might accept several different phrasings of a disclaimer as long as at least one appears.
ALL mode passes when every string from your list appears in the text. Think of this as an AND operation. If you provide "Unsubscribe" and "Privacy Policy," both must appear for the text to pass. This mode is ideal for validating that documents contain all required elements. For instance, a marketing email might need to include an unsubscribe link, a privacy policy link, and a physical address to comply with regulations. Use ALL mode to ensure nothing is missing.
NONE mode passes when none of your strings appear in the text. Think of this as a NOT operation. If you provide "password" and "credit card," the text fails if either one appears. This mode is perfect for blocking forbidden content. You might use it to prevent users from sharing sensitive information, to block competitor mentions in your content, or to catch explicitly prohibited terms before running more expensive validation.
## Configuration and Examples
The configuration for contains string is straightforward. You provide a list of strings to check for, specify which mode to use, and optionally indicate whether matching should be case-sensitive.
```typescript
// ANY mode: at least one disclaimer must be present
await abv.guardrails.containsString.validate(aiOutput, {
strings: ["not financial advice", "consult a professional", "for informational purposes only"],
mode: "any"
});
// ALL mode: all required elements must be present
await abv.guardrails.containsString.validate(email, {
strings: ["Unsubscribe", "Privacy Policy", "© 2024"],
mode: "all"
});
// NONE mode: forbidden terms must not be present
await abv.guardrails.containsString.validate(userMessage, {
strings: ["password", "credit card", "SSN", "social security"],
mode: "none"
});
// Case-sensitive matching for brand names
await abv.guardrails.containsString.validate(content, {
strings: ["iPhone", "MacBook"], // Must match exact capitalization
mode: "any",
caseSensitive: true
});
```
```python
# ANY mode: at least one disclaimer must be present
await abv.guardrails.contains_string.validate_async(ai_output, {
"strings": ["not financial advice", "consult a professional", "for informational purposes only"],
"mode": "any"
})
# ALL mode: all required elements must be present
await abv.guardrails.contains_string.validate_async(email, {
"strings": ["Unsubscribe", "Privacy Policy", "© 2024"],
"mode": "all"
})
# NONE mode: forbidden terms must not be present
await abv.guardrails.contains_string.validate_async(user_message, {
"strings": ["password", "credit card", "SSN", "social security"],
"mode": "none"
})
# Case-sensitive matching for brand names
await abv.guardrails.contains_string.validate_async(content, {
"strings": ["iPhone", "MacBook"], # Must match exact capitalization
"mode": "any",
"caseSensitive": True
})
```
By default, matching is case-insensitive, which means "PASSWORD", "Password", and "password" all match the string "password". This is usually what you want since people write in many different ways. However, when you're checking for specific brands, product names, or technical terms where capitalization matters, enable case-sensitive matching.
## Real-World Implementation Patterns
Let's explore concrete patterns for common use cases. For legal disclaimer enforcement, you typically want to ensure AI-generated content includes appropriate warnings. Using ANY mode gives content creators flexibility while ensuring compliance:
```typescript
async function ensureDisclaimer(content: string): Promise<{ valid: boolean; error?: string }> {
const result = await abv.guardrails.containsString.validate(content, {
strings: [
"not financial advice",
"not legal advice",
"consult a professional",
"for informational purposes only"
],
mode: "any"
});
if (result.status === "pass") {
return { valid: true };
}
return {
valid: false,
error: "Content must include an appropriate disclaimer"
};
}
```
```python
async def ensure_disclaimer(content: str) -> dict:
result = await abv.guardrails.contains_string.validate_async(content, {
"strings": [
"not financial advice",
"not legal advice",
"consult a professional",
"for informational purposes only"
],
"mode": "any"
})
if result["status"] == "pass":
return {"valid": True}
return {
"valid": False,
"error": "Content must include an appropriate disclaimer"
}
```
For preventing sensitive information leakage, you want to block user inputs that contain personal data. Using NONE mode ensures this information never enters your system:
```typescript
async function blockSensitiveInfo(userInput: string): Promise<{ allowed: boolean; message?: string }> {
const result = await abv.guardrails.containsString.validate(userInput, {
strings: [
"password",
"credit card",
"ssn",
"social security",
"bank account",
"routing number",
"pin",
"cvv"
],
mode: "none",
caseSensitive: false // Catch any variation
});
if (result.status === "pass") {
return { allowed: true };
}
return {
allowed: false,
message: "Please don't share sensitive personal information in your message"
};
}
```
```python
async def block_sensitive_info(user_input: str) -> dict:
result = await abv.guardrails.contains_string.validate_async(user_input, {
"strings": [
"password",
"credit card",
"ssn",
"social security",
"bank account",
"routing number",
"pin",
"cvv"
],
"mode": "none",
"caseSensitive": False # Catch any variation
})
if result["status"] == "pass":
return {"allowed": True}
return {
"allowed": False,
"message": "Please don't share sensitive personal information in your message"
}
```
For email compliance validation, you need to ensure all required elements appear. Using ALL mode guarantees nothing is missing:
```typescript
async function validateEmailCompliance(emailBody: string): Promise<{ compliant: boolean; missing?: string[] }> {
const requiredElements = [
"Unsubscribe",
"Privacy Policy",
"Contact Us",
"© 2024" // Current year copyright
];
const result = await abv.guardrails.containsString.validate(emailBody, {
strings: requiredElements,
mode: "all"
});
if (result.status === "pass") {
return { compliant: true };
}
// In production, you might parse the reason to determine which specific
// elements are missing, but for now we'll just indicate non-compliance
return {
compliant: false,
missing: requiredElements // You could get more specific by checking each individually
};
}
```
```python
async def validate_email_compliance(email_body: str) -> dict:
required_elements = [
"Unsubscribe",
"Privacy Policy",
"Contact Us",
"© 2024" # Current year copyright
]
result = await abv.guardrails.contains_string.validate_async(email_body, {
"strings": required_elements,
"mode": "all"
})
if result["status"] == "pass":
return {"compliant": True}
# In production, you might parse the reason to determine which specific
# elements are missing, but for now we'll just indicate non-compliance
return {
"compliant": False,
"missing": required_elements # You could get more specific by checking each individually
}
```
## Using Contains String as a Fast Pre-Filter
One of the most powerful patterns is using contains string as a fast pre-filter before expensive AI-powered guardrails. Since contains string responds in under ten milliseconds and costs nothing, it can catch obvious violations instantly without making an API call. This pattern significantly reduces both latency and cost.
```typescript
async function efficientValidation(content: string): Promise {
// Fast check first: block explicitly forbidden terms
const quickCheck = await abv.guardrails.containsString.validate(content, {
strings: ["explicit-slur", "forbidden-term", "banned-phrase"],
mode: "none"
});
// If quick check fails, we're done - no need for expensive AI check
if (quickCheck.status === "fail") {
console.log("Blocked by fast pre-filter");
return false;
}
// Only run expensive AI-powered check if content passed pre-filter
// This saves 1-3 seconds and token costs when quick check catches issues
const deepCheck = await abv.guardrails.toxicLanguage.validate(content);
return deepCheck.status === "pass";
}
```
```python
async def efficient_validation(content: str) -> bool:
# Fast check first: block explicitly forbidden terms
quick_check = await abv.guardrails.contains_string.validate_async(content, {
"strings": ["explicit-slur", "forbidden-term", "banned-phrase"],
"mode": "none"
})
# If quick check fails, we're done - no need for expensive AI check
if quick_check["status"] == "fail":
print("Blocked by fast pre-filter")
return False
# Only run expensive AI-powered check if content passed pre-filter
# This saves 1-3 seconds and token costs when quick check catches issues
deep_check = await abv.guardrails.toxic_language.validate_async(content)
return deep_check["status"] == "pass"
```
This pattern is particularly effective when you have a small list of terms that you absolutely never want to allow. Rather than making every piece of content go through the slow AI analysis, you catch the obvious violations instantly and only use AI for the more nuanced cases.
## Building Effective String Lists
The effectiveness of contains string depends entirely on the quality of your string lists. For blocking sensitive information, cast a wide net by including variations. Rather than just "password", include "pwd", "pass", "passcode". For credit cards, include "credit card", "cc", "card number", "mastercard", "visa". The goal is to catch every reasonable way someone might express the concept.
For requiring disclaimers, offer multiple phrasings that accomplish the same legal purpose. This gives content creators flexibility while ensuring compliance. For example, for financial advice disclaimers, you might accept "not financial advice", "consult a financial professional", "for informational purposes only", or "not investment advice". All of these accomplish the same goal of limiting liability.
For competitor blocking, be comprehensive. Include company names, product names, domain names, and common misspellings. If you're blocking mentions of "CompetitorCo", also include "competitor-co", "competitorco", "competitor.com", and any common variations.
--------------------------------------------------------------------------------
## Valid JSON
URL: /developer/guardrails/available/valid-json
# Overview
The valid JSON guardrail ensures text is properly formatted JSON with optional **schema validation**. When you're working with AI-generated structured outputs, this guardrail prevents the frustrating errors that happen when your AI returns malformed JSON or when the JSON structure doesn't match what your application expects.
## When to Use This Guardrail
You should use valid JSON whenever your application needs to parse AI output as structured data. This is incredibly common in modern AI applications. Perhaps your AI generates function calls where the parameters must be valid JSON. Maybe you're extracting structured information from unstructured text and need to ensure the output matches your database schema. You might be building a pipeline where multiple systems pass JSON messages and any malformation breaks the chain. Or you could be validating configuration files before loading them into your application.
The key insight is that AI models, despite being very capable, sometimes make syntax errors in their JSON outputs. They might include a trailing comma, forget a closing bracket, use the wrong quote type, or add fields you weren't expecting. Without validation, these errors crash your application with parsing exceptions. With validation, you catch them immediately and can either regenerate the output or handle the error gracefully.
## Understanding Validation Modes
The valid JSON guardrail operates in three modes depending on how you configure it. Understanding these modes helps you choose the right level of strictness for your use case.
Basic JSON validation checks only whether the text can be parsed as valid JSON. This mode doesn't care about the structure or fields, only that the syntax is correct. Use basic validation when you just need syntactically valid JSON and don't care about the specific structure. For example, if you're storing AI output in a JSON column in your database, you need it to be valid JSON but the structure can be flexible.
Schema validation with non-strict mode checks that specific fields exist with the correct types but allows extra fields that aren't in your schema. This is the default mode when you provide a schema. It works well with AI outputs because language models often include helpful additional information beyond what you explicitly requested. For example, you might ask for name and age, and the AI might also include email and phone number. Non-strict mode considers this a pass because your required fields are present.
Schema validation with strict mode checks that the JSON exactly matches your schema with no extra fields. Any field that isn't explicitly in your schema causes validation to fail. Use strict mode when you're working with security-sensitive contexts where unexpected data could be a security risk, when you're interfacing with external APIs that expect exact formats, or when you want to enforce discipline in your AI's outputs.
## Configuration and Schema Definition
Configuring valid JSON is straightforward. You provide the text to validate and optionally specify a schema and validation mode. The schema is a simple object that maps field names to type strings.
```typescript
// Basic JSON validation (no schema)
await abv.guardrails.validJson.validate(aiOutput);
// Schema validation (non-strict by default)
await abv.guardrails.validJson.validate(aiOutput, {
schema: {
name: "string",
age: "number",
email: "string"
}
});
// Strict schema validation (reject extra fields)
await abv.guardrails.validJson.validate(aiOutput, {
strictMode: true,
schema: {
id: "number",
title: "string",
completed: "boolean"
}
});
// Complex nested structures
await abv.guardrails.validJson.validate(aiOutput, {
schema: {
user: "object", // Nested object
tags: "array", // Array of values
metadata: "object", // Another nested object
count: "number"
}
});
```
```python
# Basic JSON validation (no schema)
await abv.guardrails.valid_json.validate_async(ai_output)
# Schema validation (non-strict by default)
await abv.guardrails.valid_json.validate_async(ai_output, {
"schema": {
"name": "string",
"age": "number",
"email": "string"
}
})
# Strict schema validation (reject extra fields)
await abv.guardrails.valid_json.validate_async(ai_output, {
"strictMode": True,
"schema": {
"id": "number",
"title": "string",
"completed": "boolean"
}
})
# Complex nested structures
await abv.guardrails.valid_json.validate_async(ai_output, {
"schema": {
"user": "object", # Nested object
"tags": "array", # Array of values
"metadata": "object", # Another nested object
"count": "number"
}
})
```
The available type strings are string for text values, number for integers and floats, boolean for true and false values, object for nested JSON objects, and array for JSON arrays. Note that the **schema validation** checks the type of each field but doesn't recursively validate the contents of nested objects or arrays. So if you specify that user should be an object, the validation checks that it is indeed an object but doesn't examine what fields that object contains.
## Common Validation Failures
Understanding why JSON validation fails helps you write better prompts for your AI and handle errors appropriately. The most common failures are syntax errors where the JSON has invalid format. This includes missing closing brackets or braces, trailing commas after the last element in arrays or objects, using single quotes instead of double quotes for strings, or having malformed escape sequences.
Type mismatches occur when a field exists but has the wrong type. For example, your schema expects age to be a number but the AI returned it as a string like "25" instead of 25. Missing required fields happen when your schema specifies a field but it doesn't appear in the JSON at all. Extra fields in strict mode cause failure when the AI includes fields that aren't in your schema and you've enabled strict mode.
Understanding these failure modes helps you improve your prompts. If you're getting type mismatches, your prompt should be more explicit about data types. If you're getting extra fields in strict mode, either disable strict mode or make your prompt more explicit about returning only the specified fields.
## Real-World Implementation Patterns
Let's explore how you'd actually use valid JSON in different scenarios. For validating structured AI output, you typically generate content and validate it before parsing:
```typescript
async function extractStructuredData(text: string): Promise {
// Ask AI to extract structured information
const aiOutput = await callAI(
`Extract the following information from this text: ${text}\n\n` +
`Return a JSON object with: name (string), age (number), email (string)`
);
// Validate the output matches our expected schema
const validation = await abv.guardrails.validJson.validate(aiOutput, {
schema: {
name: "string",
age: "number",
email: "string"
}
});
if (validation.status === "pass") {
// Safe to parse and use
return JSON.parse(aiOutput);
}
// If validation failed, log the issue and return null
// In production, you might retry with a more explicit prompt
console.error("JSON validation failed:", validation.reason);
return null;
}
```
```python
async def extract_structured_data(text: str):
# Ask AI to extract structured information
ai_output = await call_ai(
f"Extract the following information from this text: {text}\n\n"
f"Return a JSON object with: name (string), age (number), email (string)"
)
# Validate the output matches our expected schema
validation = await abv.guardrails.valid_json.validate_async(ai_output, {
"schema": {
"name": "string",
"age": "number",
"email": "string"
}
})
if validation["status"] == "pass":
# Safe to parse and use
return json.loads(ai_output)
# If validation failed, log the issue and return None
# In production, you might retry with a more explicit prompt
print(f"JSON validation failed: {validation['reason']}")
return None
```
For function calling validation where your AI needs to generate function calls in a specific format:
```typescript
async function validateFunctionCall(aiOutput: string): Promise<{ valid: boolean; call?: any }> {
// AI should return function calls in this format:
// {"function": "function_name", "arguments": {"param1": "value1"}}
const validation = await abv.guardrails.validJson.validate(aiOutput, {
strictMode: true, // Don't allow extra fields
schema: {
function: "string",
arguments: "object"
}
});
if (validation.status === "pass") {
const call = JSON.parse(aiOutput);
return { valid: true, call };
}
return {
valid: false,
// Don't expose validation details to prevent adversarial learning
};
}
```
```python
async def validate_function_call(ai_output: str) -> dict:
# AI should return function calls in this format:
# {"function": "function_name", "arguments": {"param1": "value1"}}
validation = await abv.guardrails.valid_json.validate_async(ai_output, {
"strictMode": True, # Don't allow extra fields
"schema": {
"function": "string",
"arguments": "object"
}
})
if validation["status"] == "pass":
call = json.loads(ai_output)
return {"valid": True, "call": call}
return {
"valid": False
# Don't expose validation details to prevent adversarial learning
}
```
For implementing retry logic when validation fails, you can regenerate with more explicit instructions:
```typescript
async function getValidJsonWithRetry(
prompt: string,
schema: Record,
maxRetries: number = 3
): Promise {
let currentPrompt = prompt;
for (let attempt = 0; attempt < maxRetries; attempt++) {
// Generate output
const output = await callAI(currentPrompt);
// Validate
const validation = await abv.guardrails.validJson.validate(output, { schema });
if (validation.status === "pass") {
return JSON.parse(output);
}
// Make prompt more explicit for next retry
currentPrompt = prompt +
`\n\nIMPORTANT: Respond ONLY with valid JSON. ` +
`Do not include any explanatory text before or after the JSON. ` +
`The JSON must match this exact structure: ${JSON.stringify(schema)}`;
}
// All retries failed
console.error("Failed to get valid JSON after", maxRetries, "attempts");
return null;
}
```
```python
async def get_valid_json_with_retry(
prompt: str,
schema: dict,
max_retries: int = 3
):
current_prompt = prompt
for attempt in range(max_retries):
# Generate output
output = await call_ai(current_prompt)
# Validate
validation = await abv.guardrails.valid_json.validate_async(output, {"schema": schema})
if validation["status"] == "pass":
return json.loads(output)
# Make prompt more explicit for next retry
current_prompt = (
f"{prompt}\n\n"
f"IMPORTANT: Respond ONLY with valid JSON. "
f"Do not include any explanatory text before or after the JSON. "
f"The JSON must match this exact structure: {json.dumps(schema)}"
)
# All retries failed
print(f"Failed to get valid JSON after {max_retries} attempts")
return None
```
## Prompt Engineering for Valid JSON
Getting consistently valid JSON from your AI requires good prompt engineering. The quality of your validation is only as good as the AI's ability to produce valid JSON in the first place. Here are patterns that work well.
Be extremely explicit in your instructions. Don't just ask for information, specify exactly the JSON format you want. Instead of "extract the name and age", say "return a JSON object with two fields: name as a string and age as a number". The more explicit you are, the more likely the AI is to comply.
Show an example in your prompt. AIs are excellent at pattern matching, so if you show them exactly what you want, they'll replicate that format. Include a template like "respond in this exact format: \{name: string, age: number, email: string}".
Emphasize that nothing except JSON should be in the response. AIs often want to be helpful and add explanations before or after the JSON. Tell them explicitly "respond ONLY with the JSON object, no other text before or after it".
Specify types clearly. Rather than just saying "age", say "age as a number, not a string". This prevents the common error where AIs return "25" instead of 25.
## When to Use Strict Mode
Strict mode has specific use cases where it's valuable and others where it creates unnecessary friction. Use strict mode when you're integrating with external systems that expect exact formats and any deviation could cause errors. Use it in security-critical contexts where unexpected fields could represent injection attacks or data leakage. Use it when you're trying to enforce discipline in your AI outputs and want to train your system to produce exactly what you specify.
Don't use strict mode when you're iterating on your application and the schema might change. Don't use it when your AI often provides helpful additional context that you want to preserve. Don't use it when you're okay with the AI being more verbose as long as your required fields are present. The default non-strict mode works well for most applications.
## Combining with Other Guardrails
Valid JSON is often the first guardrail in a validation pipeline because you need to ensure you have parseable data before you check its content. Once you've confirmed the JSON is valid, you might then check whether the content itself meets your other requirements. For example, you might validate that the JSON is well-formed, then check whether the text fields in that JSON contain toxic or biased language.
This creates a natural pipeline: first check JSON validity which is instant and free, then parse the JSON and check the content of specific fields with other guardrails. You only run expensive content checks after confirming you have valid data to check.
## Next Steps
For more advanced patterns including caching strategies, sophisticated error handling, and monitoring approaches, see the best practices guide. That guide shows you how to build robust validation pipelines that handle edge cases gracefully while maintaining good performance and reasonable costs.
You might also explore how valid JSON combines with other guardrails. After validating JSON structure, you often want to validate the content of text fields for toxicity or bias, creating a comprehensive validation strategy that ensures both structure and content quality.
--------------------------------------------------------------------------------
## Evaluation Overview
URL: /developer/evaluations/overview
Evaluation is a critical aspect of developing and deploying LLM applications. Usually, teams use a multitude of different evaluation methods to score the performance of their AI application depending on the use case and the stage of the development process.
# Why use LLM Evaluation?
LLM evaluation is crucial for improving the accuracy and robustness of language models, ultimately enhancing the user experience and trust in your AI application. Here are the key benefits:
- **Quality Assurance**: Detect hallucinations, factual inaccuracies, and inconsistent outputs to ensure your AI app delivers reliable results
- **Performance Monitoring**: Measure response quality, relevance, and user satisfaction across different scenarios and edge cases
- **Continuous Improvement**: Identify areas for enhancement and track improvements over time through structured evaluation metrics
- **User Trust**: Build confidence in your AI application by demonstrating consistent, high-quality outputs through systematic evaluation
- **Risk Mitigation**: Catch potential issues before they reach production users, reducing the likelihood of poor user experiences or reputational damage
# Online & Offline Evaluation
**Offline Evaluation involves**
- Evaluating the application in a controlled setting
- Typically using curated test Datasets instead of live user queries
- Heavily used during development (can be part of CI/CD pipelines) to measure improvements / regressions
- Repeatable and you can get clear accuracy metrics since you have ground truth.
**Online Evaluation involves**
- Evaluating the application in a live, real-world environment, i.e. during actual usage in production.
- Use Evaluation Methods that track success rates, user satisfaction scores, or other metrics on live traffic
- Advantage of online evaluation is that it captures things you might not anticipate in a lab setting
- Can include collecting implicit and explicit user feedback, and possibly running shadow tests or A/B tests
**In practice, successful evaluation blends online and offline evaluations.** Many teams adopt a loop-like approach. This way, evaluation is continuous and ever-improving.
# Core Concepts
| Concept | Description |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| **Scores** | Scores are a flexible data object that can be used to store any evaluation metric and link it to other objects in ABV. |
| **Evaluation Methods** | Evaluation methods are functions or tools to assign scores to other objects. |
| **Datasets** | Datasets are a collection of inputs and, optionally, expected outputs that can be during Dataset runs. |
| **Dataset Runs** | Dataset runs are used to run a dataset through your LLM application and optionally apply evaluation methods to the results. |
# Evaluation Methods
Evaluation methods are functions or tools to assign evaluation `Score`s to other objects. ABV uses the Scores to store evaluation metrics, it is meant to be flexible to represent any evaluation metric.
ABV currently supports: automatic scoring through **LLM-as-a-Judge**, manual **Human Annotations** or fully **Custom Scoring via API/SDKs**. We keep adding more evaluation methods fast, so stay tuned!
[LLM-as-a-Judge](./llm-as-a-judge)
[Human Annotation](./human-annotation)
[Custom Scores](./custom-scores)
Learn more about the [Scores Data Model](./scores-data-model).
# Dataset Runs
**Dataset Runs** are used to loop your LLM application through **Datasets** and **optionally apply Evaluation Methods** to the results. This lets you strategically evaluate your application and compare the performance of different inputs, prompts, models, or other parameters side-by-side against controlled conditions.
In ABV we differentiate between **Native** vs. **Remote** Dataset Runs. Native Dataset Runs rely on Dataset, Prompts and optionally LLM-as-a-Judge Evaluators all being on the ABV platform. Remote Dataset Runs rely only on Datasets being on the ABV platform, prompts and evaluation methods can managed off platform – they are run via code.
All require managing the [Datasets](/developer/evaluations/datasets) on the ABV platform.
[Create a Dataset](/developer/evaluations/datasets)
[Remote Dataset Runs](./remote-dataset-runs)
[Native Dataset Runs](/developer/evaluations/prompt-experiments)
Learn more about the [Dataset Runs Data Model](./dataset-runs-data-model).
## Integration with Other Features
Evaluations work seamlessly with other ABV features to provide comprehensive testing and monitoring:
- **Prompt Management**: Test different prompt versions using [Prompt Experiments](/developer/evaluations/prompt-experiments) to find the best performing prompts
- **Observability**: Evaluation scores appear directly on [traces](/developer/basic-features/observability-tracing) for real-time quality monitoring
- **SDK Support**: Create and manage evaluations programmatically with the [Python SDK](/developer/sdks/python/evaluations) and [JS/TS SDK](/developer/sdks/js-ts/overview)
- **Metrics Dashboard**: Aggregate evaluation scores in the [Metrics](/developer/platform/metrics/overview) section to track quality trends over time
- **Data Export**: Export evaluation results via the [Public API](/developer/platform/api-data-platform/public-api) for further analysis
## Getting Started
1. **Set up tracing**: Start by instrumenting your application with [Python SDK](/developer/quickstart-python) or [JS/TS SDK](/developer/quickstart-js-ts)
2. **Create datasets**: Build test datasets with representative inputs and expected outputs
3. **Choose evaluation methods**: Select from LLM-as-a-Judge, Human Annotation, or Custom Scores
4. **Run evaluations**: Execute dataset runs to evaluate your application systematically
5. **Monitor and iterate**: Track scores over time and improve based on insights
--------------------------------------------------------------------------------
## Evaluations Quickstart Tutorial
URL: /developer/evaluations/quickstart-tutorial
# Evaluations Quickstart Tutorial
This tutorial walks you through a complete evaluation workflow with a practical example. You'll create a simple dataset, run evaluations, and view results—giving you hands-on experience before diving into advanced concepts.
**Time investment:** 20-30 minutes
**Prerequisites:**
- ABV SDK installed ([Python](/developer/sdks/python/setup) or [JS/TS](/developer/sdks/js-ts/setup))
- API key configured
- Basic understanding of [traces](/developer/observability-data-model)
---
## The Example: Email Response Quality
We'll build an evaluation system for an AI that generates customer service email responses. Our goal is to measure:
- **Politeness:** Is the tone professional and courteous?
- **Completeness:** Does it address all customer questions?
- **Accuracy:** Is the information correct?
This is a common real-world use case that demonstrates key evaluation concepts.
---
## Step 1: Create a Simple Dataset
First, let's create a small dataset with 3 test cases. Each case has an input (customer email) and expected qualities.
```python
from abvdev import ABV
abv = ABV()
# Create a dataset for email response evaluation
dataset = abv.datasets.create(
name="email-responses-v1",
description="Test cases for customer service email quality"
)
# Add test cases
test_cases = [
{
"input": {
"customer_email": "My order #12345 arrived damaged. I need a refund.",
"context": "Product: Coffee Maker, Price: $89.99"
},
"expected_output": {
"polite": True,
"addresses_refund": True,
"mentions_order_number": True
}
},
{
"input": {
"customer_email": "How do I reset my password? I've tried 3 times.",
"context": "Customer tier: Premium"
},
"expected_output": {
"polite": True,
"provides_steps": True,
"offers_additional_help": True
}
},
{
"input": {
"customer_email": "Your website is terrible! Nothing works!",
"context": "Customer tier: Free"
},
"expected_output": {
"polite": True, # Must stay professional even when customer isn't
"acknowledges_frustration": True,
"offers_specific_help": True
}
}
]
# Add each test case to the dataset
for i, case in enumerate(test_cases):
dataset.create_item(
input=case["input"],
expected_output=case["expected_output"],
metadata={"case_number": i + 1}
)
print(f"✅ Created dataset with {len(test_cases)} test cases")
print(f"Dataset ID: {dataset.id}")
```
```typescript
import { ABV } from "@abvdev/client";
const abv = new ABV();
// Create a dataset for email response evaluation
const dataset = await abv.datasets.create({
name: "email-responses-v1",
description: "Test cases for customer service email quality"
});
// Add test cases
const testCases = [
{
input: {
customer_email: "My order #12345 arrived damaged. I need a refund.",
context: "Product: Coffee Maker, Price: $89.99"
},
expected_output: {
polite: true,
addresses_refund: true,
mentions_order_number: true
}
},
{
input: {
customer_email: "How do I reset my password? I've tried 3 times.",
context: "Customer tier: Premium"
},
expected_output: {
polite: true,
provides_steps: true,
offers_additional_help: true
}
},
{
input: {
customer_email: "Your website is terrible! Nothing works!",
context: "Customer tier: Free"
},
expected_output: {
polite: true, // Must stay professional even when customer isn't
acknowledges_frustration: true,
offers_specific_help: true
}
}
];
// Add each test case to the dataset
for (let i = 0; i < testCases.length; i++) {
await dataset.createItem({
input: testCases[i].input,
expected_output: testCases[i].expected_output,
metadata: { case_number: i + 1 }
});
}
console.log(`✅ Created dataset with ${testCases.length} test cases`);
console.log(`Dataset ID: ${dataset.id}`);
```
**What we did:**
- Created a dataset named `email-responses-v1`
- Added 3 test cases covering different scenarios (damaged product, password help, frustrated customer)
- Each case has input data and expected qualities we want to verify
---
## Step 2: Run Your First Evaluation
Now let's run our AI model on each test case and evaluate the results.
```python
# Your AI function that generates email responses
def generate_email_response(customer_email: str, context: str) -> str:
"""
This is your actual AI function. For this tutorial, we'll use a simple mock.
In production, this would call your LLM (OpenAI, Anthropic, etc.)
"""
# Mock response for demonstration
# Replace this with your actual LLM call
return f"Dear valued customer, thank you for contacting us about: {customer_email[:50]}..."
# Run the dataset through your AI
dataset_run = dataset.run(
name="Initial evaluation",
description="Testing baseline email response quality",
metadata={"model": "gpt-4", "temperature": 0.7}
)
# Process each test case
for item in dataset.items:
# Generate AI response
input_data = item.input
ai_response = generate_email_response(
customer_email=input_data["customer_email"],
context=input_data["context"]
)
# Record the output
dataset_run.create_observation(
dataset_item_id=item.id,
output=ai_response,
metadata={"input_case": item.metadata["case_number"]}
)
print(f"✅ Completed evaluation run: {dataset_run.id}")
print("View results in the ABV platform!")
```
```typescript
// Your AI function that generates email responses
function generateEmailResponse(customerEmail: string, context: string): string {
/**
* This is your actual AI function. For this tutorial, we'll use a simple mock.
* In production, this would call your LLM (OpenAI, Anthropic, etc.)
*/
// Mock response for demonstration
// Replace this with your actual LLM call
return `Dear valued customer, thank you for contacting us about: ${customerEmail.slice(0, 50)}...`;
}
// Run the dataset through your AI
const datasetRun = await dataset.run({
name: "Initial evaluation",
description: "Testing baseline email response quality",
metadata: { model: "gpt-4", temperature: 0.7 }
});
// Process each test case
const items = await dataset.items();
for (const item of items) {
// Generate AI response
const inputData = item.input;
const aiResponse = generateEmailResponse(
inputData.customer_email,
inputData.context
);
// Record the output
await datasetRun.createObservation({
dataset_item_id: item.id,
output: aiResponse,
metadata: { input_case: item.metadata.case_number }
});
}
console.log(`✅ Completed evaluation run: ${datasetRun.id}`);
console.log("View results in the ABV platform!");
```
**What we did:**
- Created a dataset run to track this specific evaluation
- Generated AI responses for each test case
- Recorded outputs so we can score them
---
## Step 3: Score the Results
Now we'll add scores to evaluate quality. We'll use both automated and manual scoring.
### Option A: LLM-as-a-Judge (Automated)
Use another LLM to evaluate the responses:
```python
# Score politeness using LLM-as-a-Judge
def score_politeness_llm(response: str) -> dict:
"""
Use an LLM to judge if the response is polite
"""
# In production, call your LLM API
# For demo, we'll return a mock score
score_value = 0.85 # 0-1 scale
return {
"value": score_value,
"comment": "Response uses professional language and courteous tone"
}
# Apply scores to each observation
for observation in dataset_run.observations:
# Get politeness score
politeness = score_politeness_llm(observation.output)
# Record the score
observation.score(
name="politeness",
value=politeness["value"],
comment=politeness["comment"]
)
print("✅ Scored all responses for politeness")
```
```typescript
// Score politeness using LLM-as-a-Judge
function scorePolitenesLLM(response: string): { value: number; comment: string } {
/**
* Use an LLM to judge if the response is polite
*/
// In production, call your LLM API
// For demo, we'll return a mock score
const scoreValue = 0.85; // 0-1 scale
return {
value: scoreValue,
comment: "Response uses professional language and courteous tone"
};
}
// Apply scores to each observation
const observations = await datasetRun.observations();
for (const observation of observations) {
// Get politeness score
const politeness = scorePolitenesLLM(observation.output);
// Record the score
await observation.score({
name: "politeness",
value: politeness.value,
comment: politeness.comment
});
}
console.log("✅ Scored all responses for politeness");
```
### Option B: Custom Scoring (Rule-Based)
Create simple rules to check specific criteria:
```python
def score_completeness(response: str, expected: dict) -> float:
"""
Check if response addresses expected points
Simple keyword matching for demonstration
"""
score = 0
checks = 0
# Check each expected criterion
for key, should_have in expected.items():
checks += 1
if key == "addresses_refund" and "refund" in response.lower():
score += 1
elif key == "mentions_order_number" and "#" in response:
score += 1
elif key == "provides_steps" and any(word in response.lower() for word in ["step", "first", "then"]):
score += 1
return score / checks if checks > 0 else 0
# Apply custom scores
for observation in dataset_run.observations:
# Get the expected output for this case
item = dataset.get_item(observation.dataset_item_id)
expected = item.expected_output
# Calculate completeness score
completeness = score_completeness(observation.output, expected)
# Record the score
observation.score(
name="completeness",
value=completeness,
comment=f"Addressed {int(completeness * 100)}% of expected points"
)
print("✅ Scored all responses for completeness")
```
```typescript
function scoreCompleteness(response: string, expected: Record): number {
/**
* Check if response addresses expected points
* Simple keyword matching for demonstration
*/
let score = 0;
let checks = 0;
// Check each expected criterion
for (const [key, shouldHave] of Object.entries(expected)) {
checks += 1;
if (key === "addresses_refund" && response.toLowerCase().includes("refund")) {
score += 1;
} else if (key === "mentions_order_number" && response.includes("#")) {
score += 1;
} else if (key === "provides_steps" && ["step", "first", "then"].some(word => response.toLowerCase().includes(word))) {
score += 1;
}
}
return checks > 0 ? score / checks : 0;
}
// Apply custom scores
for (const observation of observations) {
// Get the expected output for this case
const item = await dataset.getItem(observation.dataset_item_id);
const expected = item.expected_output;
// Calculate completeness score
const completeness = scoreCompleteness(observation.output, expected);
// Record the score
await observation.score({
name: "completeness",
value: completeness,
comment: `Addressed ${Math.round(completeness * 100)}% of expected points`
});
}
console.log("✅ Scored all responses for completeness");
```
---
## Step 4: View Results in the Platform
Now go to the ABV platform to see your results:
1. Navigate to **Evaluations** → **Datasets**
2. Find your dataset: `email-responses-v1`
3. Click on the run: "Initial evaluation"
4. Review scores for each test case
You'll see:
- Overall score statistics (average, min, max)
- Individual test case results
- Scores breakdown by dimension (politeness, completeness)
- Comments explaining each score
**What to look for:**
- Which test cases scored lowest? Those need attention.
- Are scores consistent across dimensions?
- Do comments explain the reasoning clearly?
---
## Step 5: Iterate and Improve
Based on the results, let's improve our AI and re-evaluate:
```python
# Improved version with better prompting
def generate_email_response_v2(customer_email: str, context: str) -> str:
"""
Version 2: Improved prompt engineering
"""
# In production, use a better prompt like:
prompt = f"""Generate a professional customer service email response.
Customer email: {customer_email}
Context: {context}
Requirements:
- Be polite and professional
- Address all customer concerns
- Offer specific next steps
- Show empathy when appropriate
Response:"""
# Mock improved response
return "Dear Customer, I sincerely apologize for the issue with your order..."
# Run evaluation again with v2
dataset_run_v2 = dataset.run(
name="Improved model (v2)",
description="Testing with better prompting",
metadata={"model": "gpt-4", "temperature": 0.7, "version": "v2"}
)
for item in dataset.items:
input_data = item.input
ai_response = generate_email_response_v2(
customer_email=input_data["customer_email"],
context=input_data["context"]
)
dataset_run_v2.create_observation(
dataset_item_id=item.id,
output=ai_response
)
# Score the new version
# ... (same scoring code as before)
print("✅ Completed v2 evaluation - compare results in the platform!")
```
```typescript
// Improved version with better prompting
function generateEmailResponseV2(customerEmail: string, context: string): string {
/**
* Version 2: Improved prompt engineering
*/
// In production, use a better prompt like:
const prompt = `Generate a professional customer service email response.
Customer email: ${customerEmail}
Context: ${context}
Requirements:
- Be polite and professional
- Address all customer concerns
- Offer specific next steps
- Show empathy when appropriate
Response:`;
// Mock improved response
return "Dear Customer, I sincerely apologize for the issue with your order...";
}
// Run evaluation again with v2
const datasetRunV2 = await dataset.run({
name: "Improved model (v2)",
description: "Testing with better prompting",
metadata: { model: "gpt-4", temperature: 0.7, version: "v2" }
});
for (const item of await dataset.items()) {
const inputData = item.input;
const aiResponse = generateEmailResponseV2(
inputData.customer_email,
inputData.context
);
await datasetRunV2.createObservation({
dataset_item_id: item.id,
output: aiResponse
});
}
// Score the new version
// ... (same scoring code as before)
console.log("✅ Completed v2 evaluation - compare results in the platform!");
```
Now you can **compare** the two runs side-by-side in the platform to see improvements!
---
## What You've Learned
Congratulations! You've completed a full evaluation workflow:
✅ **Created a dataset** with realistic test cases
✅ **Ran evaluations** on your AI outputs
✅ **Applied scores** using both LLM-as-a-Judge and custom rules
✅ **Viewed results** in the ABV platform
✅ **Iterated** by running a second evaluation with improvements
---
## Next Steps
Now that you understand the basics, explore advanced topics:
### Expand Your Dataset
- Add 10-20 more test cases covering edge cases
- Include examples from production errors
- Balance positive and negative examples
**Read:** [Datasets Best Practices](/developer/evaluations/datasets)
### Advanced Scoring
- Use [LLM-as-a-Judge](/developer/evaluations/llm-as-a-judge) with custom criteria
- Add [Human Annotation](/developer/evaluations/human-annotation) for nuanced scoring
- Create [Custom Scores](/developer/evaluations/custom-scores) for domain-specific metrics
### Automation
- Set up [Remote Dataset Runs](/developer/evaluations/remote-dataset-runs) to run nightly
- Use [Prompt Experiments](/developer/evaluations/prompt-experiments) for A/B testing
- Track [Scores Data Model](/developer/evaluations/scores-data-model) over time
### Integration
- Link evaluations to [production traces](/developer/prompt-management/link-prompts-to-traces)
- Export data for [model fine-tuning](/developer/platform/api-data-platform/export-for-fine-tuning)
- Build [custom dashboards](/developer/platform/metrics/custom-dashboards) tracking quality metrics
---
## Common Questions
Start with 10-20 high-quality test cases covering key scenarios. Add more as you find gaps. Quality > quantity—focus on representative, real-world examples.
Use both! Custom scoring is fast and deterministic for clear criteria (e.g., "contains word X"). LLM-as-a-Judge is better for nuanced qualities like tone or helpfulness.
- **During development:** Every time you change prompts or models
- **In production:** Daily or weekly automated runs
- **For experiments:** Before and after each change to measure impact
Yes! You can score production traces directly without creating datasets. This is called "online evaluation." See [Evaluations Overview](/developer/evaluations/overview) for details.
---
## Troubleshooting
**Dataset not showing up?**
- Wait a few seconds for the data to sync
- Check your API key permissions
- Verify you're looking in the correct project
**Scores not calculating?**
- Ensure observations are created before scoring
- Check that score values are between 0 and 1
- Verify your scoring function returns the correct format
**Can't compare runs?**
- Both runs must be on the same dataset
- Scores must have the same names across runs
- Allow time for score aggregation to complete
For more help, see [Evaluations Troubleshooting FAQ](/developer/evaluations/troubleshooting-faq)
---
## Complete Example Code
Want the full working example? Here's everything in one place:
```python
from abvdev import ABV
abv = ABV()
# 1. Create dataset
dataset = abv.datasets.create(
name="email-responses-tutorial",
description="Quickstart tutorial dataset"
)
# 2. Add test cases
test_cases = [
{
"input": {"customer_email": "Order damaged, need refund", "context": "Product: Coffee Maker"},
"expected_output": {"polite": True, "addresses_refund": True}
},
{
"input": {"customer_email": "Password reset help", "context": "Premium customer"},
"expected_output": {"polite": True, "provides_steps": True}
}
]
for case in test_cases:
dataset.create_item(input=case["input"], expected_output=case["expected_output"])
# 3. Run evaluation
dataset_run = dataset.run(name="Tutorial run")
for item in dataset.items:
response = f"Mock response to: {item.input['customer_email']}"
dataset_run.create_observation(dataset_item_id=item.id, output=response)
# 4. Add scores
for observation in dataset_run.observations:
observation.score(name="quality", value=0.8, comment="Good response")
print(f"✅ Complete! View results for dataset run: {dataset_run.id}")
```
```typescript
import { ABV } from "@abvdev/client";
const abv = new ABV();
async function runTutorial() {
// 1. Create dataset
const dataset = await abv.datasets.create({
name: "email-responses-tutorial",
description: "Quickstart tutorial dataset"
});
// 2. Add test cases
const testCases = [
{
input: { customer_email: "Order damaged, need refund", context: "Product: Coffee Maker" },
expected_output: { polite: true, addresses_refund: true }
},
{
input: { customer_email: "Password reset help", context: "Premium customer" },
expected_output: { polite: true, provides_steps: true }
}
];
for (const testCase of testCases) {
await dataset.createItem({
input: testCase.input,
expected_output: testCase.expected_output
});
}
// 3. Run evaluation
const datasetRun = await dataset.run({ name: "Tutorial run" });
const items = await dataset.items();
for (const item of items) {
const response = `Mock response to: ${item.input.customer_email}`;
await datasetRun.createObservation({
dataset_item_id: item.id,
output: response
});
}
// 4. Add scores
const observations = await datasetRun.observations();
for (const observation of observations) {
await observation.score({
name: "quality",
value: 0.8,
comment: "Good response"
});
}
console.log(`✅ Complete! View results for dataset run: ${datasetRun.id}`);
}
runTutorial();
```
---
You're now ready to build comprehensive evaluation systems for your LLM applications! 🎉
--------------------------------------------------------------------------------
## Datasets
URL: /developer/evaluations/datasets
A dataset is a collection of inputs and expected outputs and is used to test your application. Before executing your first dataset run, you need to create a dataset.
# Why use datasets?
- **Datasets prerequisite for Dataset Runs**, they serve as the data input of Dataset Runs
- Create test cases for your application with real production traces
- Collaboratively create and collect dataset items with your team
- Have a single source of truth for your test data
# Get Started
## 1) Creating a dataset
Datasets have a name which is unique within a project.
### ABV UI
1. **Navigate to** `Your Project` > `Datasets`
2. **Click on** `+ New dataset` to create a new dataset.
### Python SDK
Install package
```bash
pip install abvdev
```
```python
from abvdev import ABV
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
abv.create_dataset(
name="",
# optional description
description="My first dataset",
# optional metadata
metadata={
"author": "Alice",
"date": "2025-01-01",
"type": "benchmark"
}
)
```
*See *[Python SDK](/developer/sdks/python/overview)* docs for details on how to initialize the Python client.*
### JS/TS SDK
```bash
npm i @abvdev/client
```
**Environment variables**
Add your ABV credentials as environment variables, e.g. use .env file and dotenv package to load variable values.
```bash
npm install dotenv
```
```bash title=".env"
ABV_API_KEY = "sk-abv-...";
ABV_BASEURL = "https://app.abv.dev"; US region
# ABV_BASEURL = "https://eu.app.abv.dev"; EU region
```
```typescript
import { ABVClient } from "@abvdev/client";
const abv = new ABVClient();
```
**alternatively use Constructor parameters**
```typescript
import { ABVClient } from "@abvdev/client";
const abv = new ABVClient({
apiKey: "sk-abv-...",
baseUrl: "https://app.abv.dev", // US region
// baseUrl: "https://eu.app.abv.dev", // EU region
});
```
**Create dataset**
```typescript
import { ABVClient } from "@abvdev/client"
import dotenv from "dotenv";
dotenv.config();
const abv = new ABVClient()
async function main(){
await abv.api.datasets.create({
name: "dataset1", // set dataset name
// optional description
description: "My first dataset",
// optional metadata
metadata: {
author: "Alice",
date: "2025-01-01",
type: "benchmark",
},
});
}
main();
```
*See *[JS/TS SDK](/developer/sdks/js-ts/overview)* docs for details on how to initialize the JS/TS client.*
## 2) Create new dataset items
Dataset items can be added to a dataset by providing the input and optionally the expected output.
### ABV UI
- Add item - Add item manually via UI
- Import CSV - Import CSV file
- Add from trace - Add from the trace view
### Python SDK
```python
from abvdev import ABV
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
abv.create_dataset_item(
dataset_name="",
# any python object or value, optional
input={
"text": "hello world"
},
# any python object or value, optional
expected_output={
"text": "hello world"
},
# metadata, optional
metadata={
"model": "llama3",
}
)
```
*See *[Python SDK](/developer/sdks/python/overview)* docs for details on how to initialize the Python client.*
### JS/TS SDK
```typescript
import { ABVClient } from "@abvdev/client";
import dotenv from "dotenv";
dotenv.config();
const abv = new ABVClient();
async function main() {
await abv.api.datasetItems.create({
datasetName: "dataset1",
// any JS object or value
input: {
text: "hello world",
},
// any JS object or value, optional
expectedOutput: {
text: "hello world",
},
// metadata, optional
metadata: {
model: "llama3",
},
});
}
main();
```
*See *[JS/TS SDK](/developer/sdks/js-ts/overview)* docs for details on how to initialize the JS/TS client.*
# Create synthetic datasets
Frequently, you want to create synthetic examples to test your application to bootstrap your dataset. LLMs are great at generating these by prompting for common questions/tasks.
To get started have a look at this cookbook for examples on how to generate synthetic datasets:
# Create items from production data
A common workflow is to select production traces where the application did not perform as expected. Then you let an expert add the expected output to test new versions of your application on the same data.
### ABV UI
In the UI, use `+ Add to dataset`on any observation (span, event, generation) of a production trace.
### Python SDK
```python
from abvdev import ABV
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
abv.create_dataset_item(
dataset_name="",
input={ "text": "hello world" },
expected_output={ "text": "hello world" },
# link to a trace
source_trace_id="",
# optional: link to a specific span, event, or generation
source_observation_id=""
)
```
### JS/TS SDK
```typescript
import { ABVClient } from "@abvdev/client";
import dotenv from "dotenv";
dotenv.config();
const abv = new ABVClient();
async function main() {
await abv.api.datasetItems.create({
datasetName: "dataset1",
input: { text: "hello world" },
expectedOutput: { text: "hello world" },
// link to a trace
sourceTraceId: "",
// optional: link to a specific span, event, or generation
sourceObservationId: "",
});
}
main();
```
# Edit/archive dataset items
You can edit or archive dataset items. Archiving items will remove them from future experiment runs.
### ABV UI
In the UI, you can edit the item by clicking on the item id. To archive or delete the item, click on the dots next to the item and select `Archive` or `Delete`.
### Python SDK
You can upsert items by providing the `id` of the item you want to update.
```python
from abvdev import ABV
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
abv.create_dataset_item(
dataset_name="",
id="",
# example: update status to "ARCHIVED"
status="ARCHIVED"
)
```
### JS/TS SDK
You can upsert items by providing the `id` of the item you want to update.
```typescript
import { ABVClient } from "@abvdev/client";
import dotenv from "dotenv";
dotenv.config();
const abv = new ABVClient();
async function main() {
await abv.api.datasetItems.create({
datasetName: "dataset1",
id: "dataset_item_id_here",
// example: update status to "ARCHIVED"
status: "ARCHIVED",
});
}
main();
```
# Dataset runs
Once you created a dataset, you can test and evaluate your application based on it.
[Native Dataset Runs](/developer/evaluations/prompt-experiments)
[Remote Dataset Runs](./remote-dataset-runs)
--------------------------------------------------------------------------------
## Dataset Runs Data Model
URL: /developer/evaluations/dataset-runs-data-model
This page describes the data model of datasets and dataset runs. For detailed reference please refer to the .
## Datasets
Datasets are a collection of inputs and, optionally, expected outputs that can be during Dataset runs.
`Dataset`s are a collection of `DatasetItem`s.
### Dataset object
| Attribute | Type | Required | Description |
| ------------- | ------ | -------- | ----------------------------------- |
| `name` | string | Yes | Name of the dataset |
| `description` | string | No | Description of the dataset |
| `metadata` | object | No | Additional metadata for the dataset |
### DatasetItem object
| Attribute | Type | Required | Description |
| --------------------- | ------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `datasetName` | string | Yes | Name of the dataset to add the item to |
| `input` | object | No | Input data for the dataset item |
| `expectedOutput` | object | No | Expected output data for the dataset item |
| `metadata` | object | No | Additional metadata for the dataset item |
| `sourceTraceId` | string | No | ID of the source trace to link this dataset item to |
| `sourceObservationId` | string | No | ID of the source observation to link this dataset item to |
| `id` | string | No | Unique identifier for the dataset item. Dataset items are upserted on their id. Id needs to be unique (project-level) and cannot be reused across datasets. |
| `status` | DatasetStatus | No | Status of the dataset item. Defaults to ACTIVE for newly created items. Possible values: `ACTIVE`, `ARCHIVED` |
## DatasetRun
Dataset runs are used to run a dataset through your LLM application and optionally apply evaluation methods to the results.
### DatasetRun object
| Attribute | Type | Required | Description |
| ------------- | ------ | -------- | ----------------------- |
| `datasetName` | string | Yes | Name of the dataset |
| `runName` | string | Yes | Name of the dataset run |
### DatasetRunItem object
| Attribute | Type | Required | Description |
| ---------------- | ------ | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `runName` | string | Yes | Name of the dataset run to add the item to |
| `runDescription` | string | No | Description of the run. If run exists, description will be updated |
| `metadata` | object | No | Metadata of the dataset run, updates run if run already exists |
| `datasetItemId` | string | Yes | ID of the dataset item to link to this run |
| `observationId` | string | No | ID of the observation to link to this run |
| `traceId` | string | No | ID of the trace to link to this run. traceId should always be provided. For compatibility with older SDK versions it can also be inferred from the provided observationId. |
Most of the time, we recommend that DatasetRunItems reference TraceIDs directly. The reference to ObservationID exists for backwards compatibility with older SDK versions.
## End to end data relations
DataSetRuns can combine a few ABV objects:
- `DatasetRuns` are created by looping through all or selected `DatasetItem`s of a `Dataset` with your LLM application.
- For each `DatasetItem` passed into the LLM application as an Input a `DatasetRunItem` & a `Trace` are created.
- Optionally `Score`s can be added to the `Trace`s to evaluate the output of the LLM application during the `DatasetRun`.
--------------------------------------------------------------------------------
## Remote Dataset Runs
URL: /developer/evaluations/remote-dataset-runs
Once you [created a dataset](/developer/evaluations/datasets), you can use the dataset to test how your application performs on different inputs.
Remote Dataset Runs are used to programmatically loop your applications or prompts through a dataset and optionally apply Evaluation Methods to the results.
They are called "Remote Dataset Runs" because they can make use of "remote" or external logic and code.
Optionally, you can also trigger Remote Dataset Runs via the ABV UI which will call them via a webhook.
# Why use Remote Dataset Runs?
- Full flexibility to use your own application logic
- Use custom scoring functions to evaluate the outputs
- Run multiple experiments on the same dataset in parallel
- Easy to integrate with your existing evaluation infrastructure
# Setup & Run via SDK
### Sequence Diagram
```mermaid
sequenceDiagram
actor Person as User
participant User as Experiment Runner
participant App as LLM Application
participant LF as ABV Server
Note over Person, LF: Setup & Execute Remote Dataset Run
%% Trigger dataset run
Person->>User: Trigger dataset run
%% Load Dataset
User->>LF: get_dataset("dataset_name")
LF-->>User: Return dataset with items
%% Loop through dataset items
loop For each dataset item
Note over User, LF: Process dataset item
%% Start dataset run context
User->>LF: item.run(run_name, description, metadata)
LF-->>User: Return run context (root_span)
%% Execute LLM application
User->>App: Execute LLM app with item.input
Note over App: Create ABV trace
for execution
App->>LF: App is instrumented with ABV and reports traces
App-->>User: Return application output
%% Optional: Add evaluation scores from experiment runner
opt Add Custom Scores
Note over User: Run evaluation function locally
User->>LF: root_span.score_trace(name, value, comment)
end
%% Link execution to dataset item
Note over User, LF: Trace automatically linked
to dataset run
end
%% Flush data to server
User->>LF: flush() - Send all pending data
LF-->>User: Confirm data received
User-->>Person: Dataset run complete
%% Optional: ABV server-side evaluations
opt Server-side Evaluations
Note over LF: Run configured evaluations
(e.g., LLM-as-a-judge)
LF->>LF: Add evaluation scores to dataset run
end
%% View results in UI
Note over Person, LF: Analyze dataset run results
Person->>LF: Access dataset runs UI
LF-->>Person: Display aggregated scores and comparisons
```
## **1) Instrument your application**
First we create our application runner helper function. This function will be called for every dataset item in the next step. If you use ABV for production observability, you do not need to change your application code.
For a dataset run, it is important that your application creates ABV traces for each execution so they can be linked to the dataset item. Please refer to the integrations page for details on how to instrument the framework you are using.
### Python SDK
Install packages
```bash
pip install abvdev, openai
```
Assume you already have a ABV-instrumented LLM-app:
```python title="app.py"
from abvdev import ABV, observe
from openai import OpenAI
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
openai_client = OpenAI(api_key="sk-proj-...")
@observe
def my_llm_function(question: str):
with abv.start_as_current_observation(as_type="generation", name="OpenAI-gen"):
response = openai_client.chat.completions.create(
model="gpt-4o", messages=[{"role": "user", "content": question}]
)
output = response.choices[0].message.content
# Update trace input / output
abv.update_current_trace(input=question, output=output)
return output
```
*See *[Python SDK](/developer/sdks/python/overview)* docs for more details.*
### JS/TS SDK
Please make sure you have the [JS/TS SDK set up](/developer/sdks/js-ts/setup) for tracing of your application. If you use ABV for [observability](/developer/basic-features/observability-tracing), this is the same setup.
**Install packages**
```bash
npm install @abvdev/tracing @abvdev/otel @abvdev/client openai
```
**Add credentials**
Add your ABV credentials to your environment variables. Make sure that you have a `.env` file in your project root and a package like dotenv to load the variables.
```bash title=".env"
ABV_API_KEY = "sk-abv-..."
ABV_BASE_URL = "https://app.abv.dev" # US region
//ABV_BASE_URL = "https://eu.app.abv.dev" # EU region
OPENAI_API_KEY = "sk-proj-..." // added to use openai llm
```
**Initialize OpenTelemetry**
Install the OpenTelemetry Node SDK package:
```bash
npm install @opentelemetry/sdk-node
```
Create a `instrumentation.ts` file that initializes the OpenTelemetry `NodeSDK` and registers the `ABVSpanProcessor`.
```typescript title="instrumentation.ts"
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
const sdk = new NodeSDK({
spanProcessors: [new ABVSpanProcessor()],
});
sdk.start();
```
Modify `instrumentation.ts` file to use `dotenv` package to load the variables.
Additional parameters are provided to get trace visible in the UI immediately.
```bash
npm install dotenv
```
```typescript title="instrumentation.ts"
import dotenv from "dotenv";
dotenv.config();
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
const sdk = new NodeSDK({
spanProcessors: [
new ABVSpanProcessor({
apiKey: process.env.ABV_API_KEY,
baseUrl: process.env.ABV_BASE_URL,
exportMode: "immediate",
flushAt: 1,
flushInterval: 1,
additionalHeaders: {
"Content-Type": "application/json",
"Accept": "application/json"
}
})
],
});
sdk.start();
```
Import the `instrumentation.ts` file at the top of your application.
```typescript title="index.ts"
import "./instrumentation"; // Must be the first import
```
Instrumentation:
```typescript title="app.ts"
import { ABVClient } from "@abvdev/client";
import { startActiveObservation } from "@abvdev/tracing";
import OpenAI from "openai";
import dotenv from "dotenv";
dotenv.config();
const openai = new OpenAI({
apiKey: process.env.OPENAI_API_KEY,
});
const myLLMApplication = async (input: string) => {
return startActiveObservation("my-llm-application", async (span) => {
const output = await openai.chat.completions.create({
model: "gpt-4o",
messages: [{ role: "user", content: input }],
});
span.update({ input, output: output.choices[0].message.content });
// return reference to span and output
// will be simplified in a future version of the SDK
return [span, output] as const;
})
};
```
## 2) Run experiment on dataset
When running an experiment on a dataset, the application that shall be tested is executed for each item in the dataset. The execution trace is then linked to the dataset item. This allows you to compare different runs of the same application on the same dataset. Each experiment is identified by a `run_name`.
### Python SDK
You may then execute that LLM-app for each dataset item to create a dataset run:
execute\_dataset.py
```python
from abvdev import get_client
from .app import my_llm_application
# Load the dataset
dataset = get_client().get_dataset("")
# Loop over the dataset items
for item in dataset.items:
# Use the item.run() context manager for automatic trace linking
with item.run(
run_name="",
run_description="My first run",
run_metadata={"model": "llama3"},
) as root_span:
# Execute your LLM-app against the dataset item input
output = my_llm_application.run(item.input)
# Optionally: Add scores computed in your experiment runner, e.g. json equality check
root_span.score_trace(
name="",
value=my_eval_fn(item.input, output, item.expected_output),
comment="This is a comment", # optional, useful to add reasoning
)
# Flush the abv client to ensure all data is sent to the server at the end of the experiment run
get_client().flush()
```
*See *[Python SDK](/developer/sdks/python/overview)* docs for details on the new OpenTelemetry-based SDK.*
### JS/TS SDK
```typescript
import { ABVClient } from "@abvdev/client";
const abv = new ABVClient();
for (const item of dataset.items) {
// execute application function and get abvObject (trace/span/generation/event)
// output also returned as it is used to evaluate the run
// you can also link using ids, see sdk reference for details
const [span, output] = await myLlmApplication.run(item.input);
// link the execution trace to the dataset item and give it a run_name
await item.link(span, "", {
description: "My first run", // optional run description
metadata: { model: "llama3" }, // optional run metadata
});
// Optionally: Add scores
abv.score.trace(span, {
name: "",
value: myEvalFunction(item.input, output, item.expectedOutput),
comment: "This is a comment", // optional, useful to add reasoning
});
}
// Flush the abv client to ensure all score data is sent to the server at the end of the experiment run
await abv.flush();
```
If you want to learn more about how adding evaluation scores from the code works, please refer to the docs:
[Add custom scores](./custom-scores)
## 3) Optionally: Run Evals in ABV
In the code above, we show how to add scores to the dataset run from your experiment code.
Alternatively, you can run evals in ABV. This is useful if you want to use the [LLM-as-a-Judge](./llm-as-a-judge) feature to evaluate the outputs of the dataset runs. We have recorded a on how this works end-to-end.
[Set up LLM-as-a-judge](./llm-as-a-judge)
## 4) Compare dataset runs
After each experiment run on a dataset, you can check the aggregated score in the dataset runs table and compare results side-by-side.
# Optional: Trigger Remote Dataset Runs via UI
When setting up Remote Dataset Runs via the SDK, it can be useful to expose a trigger in the ABV UI that helps you trigger the experiment runs.
You need to set up a webhook to receive the trigger request from ABV.
## 1) Navigate to the dataset
- **Navigate to** `Your Project` > `Datasets`
- **Click on** the dataset you want to set up a remote experiment trigger for
## 2) Open the setup page
**Click on** `Start Experiment` to open the setup page
**Click on** `⚡` below `Custom Experiment`
## 3) Configure the webhook
**Enter** the URL of your external evaluation service that will receive the webhook when experiments are triggered.
**Specify** a default config that will be sent to your webhook. Users can modify this when triggering experiments.
## 4) Trigger experiments
Once configured, team members can trigger remote experiments via the `Run` button under the **Custom Experiment** option. ABV will send the dataset metadata (ID and name) along with any custom configuration to your webhook.
**Typical workflow**: Your webhook receives the request, fetches the dataset from ABV, runs your application against the dataset items, evaluates the results, and ingests the scores back into ABV as a new Dataset Run.
--------------------------------------------------------------------------------
## Prompt Experiments
URL: /developer/evaluations/prompt-experiments
You can execute Prompt Experiments in the ABV UI to test different prompt versions from [Prompt Management](/developer/prompt-management/overview) or language models and compare the results side-by-side.
Optionally, you can use [LLM-as-a-Judge Evaluators](./llm-as-a-judge) to automatically score the responses based on the expected outputs to further analyze the results on an aggregate level.
# Why use Prompt Experiments?
- Quickly test different prompt versions or models
- Structure your prompt testing by using a dataset to test different prompt versions and models
- Quickly iterate on prompts through Dataset Runs
- Optionally use LLM-as-a-Judge Evaluators to score the responses based on the expected outputs from the dataset
- Prevent regressions by running tests when making prompt changes
# Prerequisites
## 1) Create a usable prompt
Create a prompt that you want to test and evaluate. [How to create a prompt?](/developer/prompt-management/get-started)
A prompt is usable when:
**Example: Prompt Variables & Dataset Item Keys Mapping**
**Prompt:**
```none
You are an ABV expert. Answer based on:
{{documentation}}
Question: {{question}}
```
**Dataset Item:**
```json
{
"documentation": "ABV is an LLM Engineering Platform",
"question": "What is ABV?"
}
```
In this example:
- The prompt variable `{{documentation}}` maps to the JSON key `"documentation"`
- The prompt variable `{{question}}` maps to the JSON key `"question"`
- Both keys must exist in the dataset item's input JSON for the experiment to run successfully
**Example: Chat Message Placeholder Mapping**
In addition to variables, you can also map placeholders in chat message prompts to dataset item keys.
This is useful when the dataset item also contains for example a chat message history to use.
Your chat prompt needs to contain a placeholder with a name. Variables within placeholders are not resolved.
**Chat Prompt:**
Placeholder named: `message_history`
**Dataset Item:**
```json
{
"message_history": [
{
"role": "user",
"content": "What is ABV?"
},
{
"role": "assistant",
"content": "ABV is a tool for tracking and analyzing the performance of language models."
}
],
"question": "What is ABV?"
}
```
In this example:
- The chat prompt placeholder `message_history` maps to the JSON key `"message_history"`.
- The prompt variable `{{question}}` maps to the JSON key `"question"` in a variable not within a placeholder message.
- Both keys must exist in the dataset item's input JSON for the experiment to run successfully
## 2) Create a usable dataset
Create a dataset with the inputs and expected outputs you want to use for your prompt experiments. [How to create a dataset?](/developer/evaluations/datasets)
**A dataset is usable when: **\[1] the dataset items have JSON objects as input and \[2] these objects have JSON keys that matches the prompt variables of the prompt(s) you will use. See the example below.
**Example: Prompt Variables & Dataset Item Keys Mapping**
**Prompt:**
```none
You are an ABV expert. Answer based on:
{{documentation}}
Question: {{question}}
```
**Dataset Item:**
```json
{
"documentation": "ABV is an LLM Engineering Platform",
"question": "What is ABV?"
}
```
In this example:
- The prompt variable `{{documentation}}` maps to the JSON key `"documentation"`
- The prompt variable `{{question}}` maps to the JSON key `"question"`
- Both keys must exist in the dataset item's input JSON for the experiment to run successfully
## 3) Configure LLM connection
As your prompt will be executed for each dataset item, you need to configure an LLM connection in the project settings. [How to configure an LLM connection?](/developer/platform/administration/llm-connections)
## 4) Optional: Set up LLM-as-a-judge
You can set up an LLM-as-a-judge evaluator to score the responses based on the expected outputs. Make sure to set the target of the LLM-as-a-Judge to "Experiment runs" and filter for the dataset you want to use. [How to set up LLM-as-a-judge?](./llm-as-a-judge)
# Trigger a Prompt Experiment
## 1) Navigate to the dataset
`Dataset Runs are currently started from the detail page of a dataset.`
- **Navigate to** `Your Project` > `Datasets`
- **Click on** the dataset you want to start a Dataset Run for
## 2) Open the setup page
**Click on** `Start Experiment` to open the setup page
**Click on** `Create` below `prompt Experiment`
## 3) Configure the Dataset Run
1. **Set** a Dataset Run name
2. **Select** the prompt you want to use
3. **Set up or select** the LLM connection you want to use
4. **Select** the dataset you want to use
5. **Optionally select** the evaluator you want to use
6. **Click on** `Create` to trigger the Dataset Run
This will trigger the Dataset Run and you will be redirected to the Dataset Runs page. The run might take a few seconds or minutes to complete depending on the prompt complexity and dataset size.
## 4) Compare runs
After each experiment run, you can check the aggregated score in the Dataset Runs table and compare results side-by-side.
--------------------------------------------------------------------------------
## LLM-as-a-Judge
URL: /developer/evaluations/llm-as-a-judge
LLM-as-a-judge is a technique to evaluate the quality of LLM applications by using an LLM as a judge. The LLM is given a trace or a dataset entry and asked to score and reason about the output. The scores and reasoning are stored as [scores](./scores-data-model) in ABV.
# Why use LLM-as-a-judge?
- **Scalable & cost‑effective:** Judge thousands of outputs quickly and cheaply versus human panels.
- **Human‑like judgments:** Captures nuance (helpfulness, safety, coherence) better than simple metrics, especially when rubric‑guided.
- **Repeatable comparisons:** With a fixed rubric, you can rerun the same prompts to get consistent scores and short rationales.
# Set up step-by-step
## 1) Create a new LLM-as-a-Judge evaluator
Navigate to the Evaluators page and click on the `Create Evaluator` button.
## 2) Set the default model
Next, you'll define the default model used for conducting the evaluations. The default is used by every managed evaluator; custom templates may override it.
This step requires an LLM Connection to be set up. Please see [LLM Connections](/developer/platform/administration/llm-connections) for more information.
- **Setup**: This default model needs to be set up once, though it can be changed at any point if needed.
- **Change**: Existing evaluators keep evaluating with the new model—historic results stay preserved.
- **Structured Output Support**: It's crucial that the chosen default model supports structured output. This is essential for our system to correctly interpret the evaluation results from the LLM judge.
## 3) Pick an Evaluator
Now we select an evaluator. There are two main ways:
### Managed Evaluator
ABV ships a growing catalog of evaluators built and maintained by us and partners like **Ragas**. Each evaluator captures best-practice evaluation prompts for a specific quality dimension—e.g. *Hallucination*, *Context-Relevance*, *Toxicity*, *Helpfulness*.
- **Ready to use**: no prompt writing required.
- **Continuously expanded**: by adding OSS partner-maintained evaluators and more evaluator types in the future (e.g. regex-based).
### Custom Evaluator
When the library doesn't fit your specific needs, add your own:
1. Draft an evaluation prompt with `{{variables}}` placeholders (`input`, `output`, `ground_truth` …).
2. Optional: Customize the **score** (0-1) and **reasoning** prompts to guide the LLM in scoring.
3. Optional: Pin a custom dedicated model for this evaluator. If no custom model is specified, it will use the default evaluation model (see Section 2).
4. Save → the evaluator can now be reused across your project.
## 4) Choose which Data to Evaluate
With your evaluator and model selected, you now specify which data to run the evaluations on. You can chose between running on **production tracing data** or **Datasets during Dataset Runs**.
### Live Data
Evaluating live production traffic allows you to monitor the performance of your LLM application in real-time.
- **Scope**: Choose whether to run on *new* traces only and/or *existing* traces once (for backfilling). When in doubt, we recommend running on *new* traces.
- **Filter**: Narrow down the evaluation to a specific subset of data you're interested in. You can filter by trace name, tags, `userId` and may more. Combine filters freely.
- **Preview**: ABV shows a sample of traces from the last 24 hours that match your current filters, allowing you to sanity-check your selection.
- **Sampling**: To manage costs and evaluation throughput, you can configure the evaluator to run on a percentage (e.g., 5%) of the matched traces.
### Dataset Runs
LLM-as-a-Judge evaluators can score the results of your Dataset Runs.
[Native Dataset Runs](/developer/evaluations/prompt-experiments): When running Native Dataset Runs through the UI, you can simply select which evaluators you want to run. These selected evaluators will then automatically execute on the data generated by your next Dataset Run.
[Remote Dataset Runs](./remote-dataset-runs): Before running Remote Dataset Runs through the SDKs, you will need to set up which evaluators you want to run in UI. You will need to configure a running evaluator in the following format:
- **Dataset**: Filter which source dataset the evaluator should run on.
- **Scope**: Choose whether to target only new Dataset Runs and/or execute the evaluator on past Dataset Runs (for backfilling).
- **Sampling**: To manage costs and evaluation throughput, you can configure the evaluator to run on a percentage (e.g., 5%) of Dataset Run items.
## 5) Map Variables & preview Evaluation Prompt
You now need to teach ABV *which properties* of your trace or dataset item represent the actual data to populate these variables for a sensible evaluation. For instance, you might map your system's logged trace input to the prompt's `{{input}}` variable, and the LLM response ie trace output to the prompt's `{{output}}` variable. This mapping is crucial for ensuring the evaluation is sensible and relevant.
### Live Data
- **Prompt Preview**: As you configure the mapping, ABV shows a **live preview of the evaluation prompt populated with actual data**. This preview uses historical traces from the last 24 hours that matched your filters (from Step 3). You can navigate through several example traces to see how their respective data fills the prompt, helping you build confidence that the mapping is correct.
- **JSONPath**: If the data is nested (e.g., within a JSON object), you can use a JSONPath expression (like `$.choices[0].message.content`) to precisely locate it.
### Dataset Runs
- **Suggested mappings**: The system will often be able to autocomplete common mappings based on typical field names in datasets. For example, if you're evaluating for correctness, and your prompt includes `{{input}}`, `{{output}}`, and `{{ground_truth}}` variables, we would likely suggest mapping these to the trace input, trace output, and the dataset item's expected\_output respectively.
- **Edit mappings**: You can easily edit these suggestions if your dataset schema differs. You can map any properties of your dataset item (e.g., `input`, `expected_output`). Further, as dataset runs create traces under the hood, using the trace input/output as the evaluation input/output is a common pattern. Think of the trace output as your experiment run's outpu
✨ Done! You have successfully set up an evaluator which will run on your data.
Need custom logic? Use the SDK instead—see [Custom Scores](./custom-scores) or an external pipeline example.
# Monitor & Iterate
As our system evaluates your data it writes the results as [scores](./scores-data-model). You can then:
- **View Logs**: Check detailed logs for each evaluation, including status, any retry errors, and the full request/response bodies sent to the evaluation model.
- **Use Dashboards**: Aggregate scores over time, filter by version or environment, and track the performance of your LLM application.
- **Take Actions**: Pause, resume, or delete an evaluator.
--------------------------------------------------------------------------------
## Human Annotation
URL: /developer/evaluations/human-annotation
Human Annotation is a manual evaluation method. It is used to collaboratively annotate traces, sessions and observations with scores.
In ABV you can use [Annotation Queues](./#annotation-queues) to streamline working through reviewing larger batches of of [single interactions (trace-level)](/developer/basic-features/observability-tracing), [multiple interactions (session-level)](/developer/basic-features/sessions) or even [single observations](/developer/basic-features/observability-tracing), below the trace level.
# Why use Human Annotation?
- **Collaboration**: Enable team collaboration by inviting other internal members to annotate a subset of traces and observations. This manual evaluation can enhance the overall accuracy and reliability of your results by incorporating diverse perspectives and expertise.
- **Annotation data consistency**: Create score configurations for annotation workflows to ensure that all team members are using standardized scoring criteria. Hereby configure categorical, numerical or binary score types to capture different aspects of your data.
- **Evaluation of new product features**: This feature can be useful for new use cases where no other scores have been allocated yet.
- **Benchmarking of other scores**: Establish a human baseline score that can be used as a benchmark to compare and evaluate other scores. This can provide a clear standard of reference and enhance the objectivity of your performance evaluations.
# Annotation of Single Traces, Sessions and Observations
Manual Annotation of single traces, sessions and observations is available in the trace, session and observation detail view.
### 1) **Prerequisite: Create a Score Config**
To use Human Annotation, you **need to have at least one score configuration (Score Config) set up**. See for details.
### 2) Trigger Annotation on a Trace, Session or Observation
On a Trace, Session or Observation detail view **click on** `Annotate` to open the annotation form.
### 3) Select Score Configs to use
### 4) Set Score values
### 5) See newly added Scores
To see your newly added scores on traces or observations, **click on** the `Scores` tab on the trace or observation detail view.
To see your newly added scores on sessions, **click on** the `Scores` tab on the session detail view.
All scores are also available in the traces, sessions and observations table views respectively.
# Annotation Queues
Annotation queues allow you to manage and prioritize your annotation tasks in a structured way. This feature is particularly useful for large-scale projects that benefit of human-in-the-loop evaluation at some scale. Queues streamline this process by allowing for specifying which traces, sessions or observations you'd like to annotate on which dimensions.
## Create Annotation Queues
### 1) **Prerequisite: Create a Score Config**
To use Human Annotation, you **need to have at least one score configuration (Score Config) set up**. See how to create and manage Score Configs for details.
### 2) Go to Annotation Queues View
- **Navigate to** `Your Project` > `Human Annotation` to see all your annotation queues.
- **Click on** `New queue` to create a new queue.
### 3) Fill out Create Queue Form
- **Select** the `Score Configs` you want to use for this queue.
- **Set** the `Queue name` and `Description` (optional).
- **Click on** `Create queue` to create the queue.
## Assign Users to Annotation Queues
### 1) **Go to Annotation Queues View**
- **Navigate to** `Your Project` > `Human Annotation` to see all your annotation queues.
- **Click on** `New queue `to assign users to a new queue or click the three dots on the right of an existing queue to assign users to it.
### 2) **Assign Users to the Queue**
- **Expand** the `User Assignment` section under the `Advanced settings` section.
- **Search** for the users you want to assign to the queue.
- **Click on** the user to assign them to the queue. You can also assign multiple users at once by selecting them in the search results.
- **Review** the users you have selected to be assigned to the queue.
- **Click on** `Save` to save the queue.
### 3) **Manage Assignments**
- **Click on** the three dots on the right of the queue or the `Edit queue` button in the queue detail view to manage the assignments.
- **View** the users assigned to the queue by expanding the `User Assignment` section.
- **Assign** new users by clicking the `Assign users` button and searching for the users you want to assign.
- **Unassign** current users by clicking the `X` icon on the right of the user.
## Run Annotation Queues
### 1) Populate Annotation Queues
Once you have created annotation queues, you can assign traces, sessions or observations to them.
**Bulk Selection**
To add multiple traces, sessions or observations to a queue:
1. **Navigate to the respective table view** and optionally adjust the filters
2. **Select** Traces, Sessions or Observations via the checkboxes.
3. **Click on** the "Actions" dropdown menu
4. **Click on** `Add to queue` to add the selected traces, sessions or observations to the queue.
5. **Select** the queue you want to add the traces, sessions or observations to.
**Single Item**
To add single traces, sessions or observations to a queue:
1. **Navigate** to the trace, session or observation detail view
2. **Click on** the `Annotate` dropdown
3. **Select** the queue you want to add the trace, session or observation to
### 2) Navigate to Annotation Queue
1. **Navigate to** `Your Project` > `Human Annotation`
2. Option 1: **Click on** the queue name to view the associated annotation tasks
3. Option 2: **Click on** "Process queue" to start processing the queue
### 3) Process Annotation Tasks
You will see an annotation task for each item in the queue.
1. On the `Annotate` Card **add scores** on the defined dimensions
2. **Click on** `Complete + next` to move to the next annotation task or finish the queue
## Manage Annotation Queues via API
You can enqueue, manage and dequeue annotation tasks via the [API](/developer/platform/api-data-platform/public-api). This allows for scaling and automating your annotation workflows.
--------------------------------------------------------------------------------
## Custom Scores
URL: /developer/evaluations/custom-scores
Custom Scores are the most flexible way to implement evaluation workflows using ABV. As any other evaluation method the purpose of custom scores is to assign evaluations metrics to `Traces`, `Observations`, `Sessions`, or `DatasetRuns` via the `Score` object (see [Scores Data Model](./scores-data-model)).
This is achieved by ingesting scores via the ABV SDKs or API.
# Common Use Cases
- **Collecting user feedback**: collect in-app feedback from your users on application quality or performance. Can be captured in the frontend via our Browser SDK.
->
- **Custom evaluation data pipeline**: continuously monitor the quality by fetching traces from ABV, running custom evaluations, and ingesting scores back into ABV.
->
- **Custom internal workflow tooling**: build custom internal tooling that helps you manage human-in-the-loop workflows. Ingest scores back into ABV, optionally following your custom schema by referencing a config.
- **Custom run-time evaluations**: e.g. track whether the generated SQL code actually worked, or if the structured output was valid JSON.
# Ingesting Scores via API/SDKs
You can add scores via the ABV SDKs or API. Scores can take one of three data types: **Numeric**, **Categorical** or **Boolean**.
If a score is ingested manually using a `trace_id` to link the score to a trace, it is not necessary to wait until the trace has been created. The score will show up in the scores table and will be linked to the trace once the trace with the same `trace_id` is created.
Here are examples by `Score` data types
## Python SDK
Install package
```bash
pip install abvdev
```
Numeric
Numeric score values must be provided as float.
```python
from abvdev import ABV
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# Method 1: Score via low-level method
abv.create_score(
name="correctness",
value=0.9,
trace_id="trace_id_here",
observation_id="observation_id_here", # optional
data_type="NUMERIC", # optional, inferred if not provided
comment="Factually correct", # optional
)
# Method 2: Score current span/generation (within context)
with abv.start_as_current_span(name="my-operation") as span:
# Score the current span
span.score(
name="correctness",
value=0.9,
data_type="NUMERIC",
comment="Factually correct"
)
# Score the trace
span.score_trace(
name="overall_quality",
value=0.95,
data_type="NUMERIC"
)
# Method 3: Score via the current context
with abv.start_as_current_span(name="my-operation"):
# Score the current span
abv.score_current_span(
name="correctness",
value=0.9,
data_type="NUMERIC",
comment="Factually correct"
)
# Score the trace
abv.score_current_trace(
name="overall_quality",
value=0.95,
data_type="NUMERIC"
)
```
### Categorical
Categorical score values must be provided as strings.
```python
from abvdev import ABV
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# Method 1: Score via low-level method
abv.create_score(
name="accuracy",
value="partially correct",
trace_id="trace_id_here",
observation_id="observation_id_here", # optional
data_type="CATEGORICAL", # optional, inferred if not provided
comment="Some factual errors", # optional
)
# Method 2: Score current span/generation (within context)
with abv.start_as_current_span(name="my-operation") as span:
# Score the current span
span.score(
name="accuracy",
value="partially correct",
data_type="CATEGORICAL",
comment="Some factual errors"
)
# Score the trace
span.score_trace(
name="overall_quality",
value="partially correct",
data_type="CATEGORICAL"
)
# Method 3: Score via the current context
with abv.start_as_current_span(name="my-operation"):
# Score the current span
abv.score_current_span(
name="accuracy",
value="partially correct",
data_type="CATEGORICAL",
comment="Some factual errors"
)
# Score the trace
abv.score_current_trace(
name="overall_quality",
value="partially correct",
data_type="CATEGORICAL"
)
```
### Boolean
Boolean scores must be provided as a float. The value’s string equivalent will be automatically populated and is accessible on read. See for more details on POST/GET scores endpoints.
```python
from abvdev import ABV
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# Method 1: Score via low-level method
abv.create_score(
name="helpfulness",
value=0, # 0 or 1
trace_id="trace_id_here",
observation_id="observation_id_here", # optional
data_type="BOOLEAN", # required, numeric values
#without data type would be inferred as NUMERIC
comment="Incorrect answer", # optional
)
# Method 2: Score current span/generation (within context)
with abv.start_as_current_span(name="my-operation") as span:
# Score the current span
span.score(
name="helpfulness",
value=1, # 0 or 1
data_type="BOOLEAN",
comment="Very helpful response"
)
# Score the trace
span.score_trace(
name="overall_quality",
value=1, # 0 or 1
data_type="BOOLEAN"
)
# Method 3: Score via the current context
with abv.start_as_current_span(name="my-operation"):
# Score the current span
abv.score_current_span(
name="helpfulness",
value=1, # 0 or 1
data_type="BOOLEAN",
comment="Very helpful response"
)
# Score the trace
abv.score_current_trace(
name="overall_quality",
value=1, # 0 or 1
data_type="BOOLEAN"
)
```
## JS/TS SDK
```bash
npm i @abvdev/client
```
**Environment variables**
Add your ABV credentials as environment variables, e.g. use .env file and dotenv package to load variable values.
```bash
npm install dotenv
```
```bash title=".env"
ABV_API_KEY = "sk-abv-...";
ABV_BASEURL = "https://app.abv.dev"; US region
# ABV_BASEURL = "https://eu.app.abv.dev"; EU region
```
```typescript
import { ABVClient } from "@abvdev/client";
const abv = new ABVClient();
```
**alternatively use Constructor parameters**
```typescript
import { ABVClient } from "@abvdev/client";
const abv = new ABVClient({
apiKey: "sk-abv-...",
baseUrl: "https://app.abv.dev", // US region
// baseUrl: "https://eu.app.abv.dev", // EU region
});
```
### Numeric
Numeric score values must be provided as float.
```typescript
import { ABVClient } from "@abvdev/client";
import dotenv from "dotenv";
dotenv.config();
const abv = new ABVClient();
abv.score.create({
id: "unique_id", // optional, can be used as an idempotency
// key to update the score subsequently
traceId: "target_trace_id_here",
observationId: "target_observation_id_here", // optional
name: "correctness",
value: 0.9,
dataType: "NUMERIC", // optional, inferred if not provided
comment: "Factually correct", // optional
});
async function main() {
// Flush the scores in short-lived environments
await abv.flush();
}
main();
```
### Categorical
Categorical score values must be provided as strings.
```typescript
import { ABVClient } from "@abvdev/client";
import dotenv from "dotenv";
dotenv.config();
const abv = new ABVClient();
abv.score.create({
id: "unique_id", // optional, can be used
// as an idempotency key to update the score subsequently
traceId: "target_trace_id_here",
observationId: "target_observation_id_here", // optional
name: "accuracy",
value: "partially correct",
dataType: "CATEGORICAL", // optional, inferred if not provided
comment: "Factually correct", // optional
});
async function main() {
// Flush the scores in short-lived environments
await abv.flush();
}
main();
```
### Boolean
Boolean scores must be provided as a float. The value’s string equivalent will be automatically populated and is accessible on read. See API reference for more details on POST/GET scores endpoints.
```typescript
import { ABVClient } from "@abvdev/client";
import dotenv from "dotenv";
dotenv.config();
const abv = new ABVClient();
abv.score.create({
id: "unique_id", // optional, can be used as an
// idempotency key to update the score subsequently
traceId: "target_trace_id_here",
observationId: "target_observation_id_here", // optional
name: "helpfulness",
value: 0, // 0 or 1
dataType: "BOOLEAN", // required, numeric values without
// data type would be inferred as NUMERIC
comment: "Incorrect answer", // optional
});
async function main() {
// Flush the scores in short-lived environments
await abv.flush();
}
main();
```
→ More details in [Python SDK docs ](/developer/sdks/python/overview) and [JS/TS SDK docs](/developer/sdks/js-ts/overview). See [API reference](/developer/platform/api-data-platform/public-api) for more details on POST/GET score configs endpoints.
## Preventing Duplicate Scores
By default, ABV allows for multiple scores of the same `name` on the same trace. This is useful if you'd like to track the evolution of a score over time or if e.g. you've received multiple user feedback scores on the same trace.
In some cases, you want to prevent this behavior or update an existing score. This can be achieved by creating an **idempotency key** on the score and add this as an `id` when creating the score, e.g. `-`.
## Enforcing a Score Config
Score configs are helpful when you want to standardize your scores for future analysis.
To enforce a score config, you can provide a `configId` when creating a score to reference a `ScoreConfig` that was previously created. `Score Configs` can be defined in the ABV UI or via our API. .
Whenever you provide a `ScoreConfig`, the score data will be validated against the config. The following rules apply:
- **Score Name**: Must equal the config's name
- **Score Data Type**: When provided, must match the config's data type
- **Score Value when Type is numeric**: Value must be within the min and max values defined in the config (if provided, min and max are optional and otherwise are assumed as -∞ and +∞ respectively)
- **Score Value when Type is categorical**: Value must map to one of the categories defined in the config
- **Score Value when Type is boolean**: Value must equal `0` or `1`
### Python SDK
**Numeric Scores**
When ingesting numeric scores, you can provide the value as a float. If you provide a configId, the score value will be validated against the config's numeric range, which might be defined by a minimum and/or maximum value.
```python
from abvdev import ABV
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# Method 1: Score via low-level method
abv.create_score(
trace_id="trace_id_here",
observation_id="observation_id_here", # optional
session_id="session_id_here", # optional, Id of the session the score relates to
name="accuracy",
value=0.9,
comment="Factually correct", # optional
score_id="unique_id", # optional, can be used
# as an idempotency key to update the score subsequently
config_id="78545-6565-3453654-43543", # optional,
# to ensure that the score follows a specific min/max value range
data_type="NUMERIC" # optional, possibly inferred
)
# Method 2: Score within context
with abv.start_as_current_span(name="my-operation") as span:
span.score(
name="accuracy",
value=0.9,
comment="Factually correct",
config_id="78545-6565-3453654-43543",
data_type="NUMERIC"
)
```
**Categorical Scores**
Categorical scores are used to evaluate data that falls into specific categories. When ingesting categorical scores, you can provide the value as a string. If you provide a configId, the score value will be validated against the config’s categories.
```python
from abvdev import ABV
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# Method 1: Score via low-level method
abv.create_score(
trace_id="trace_id_here",
observation_id="observation_id_here", # optional
name="correctness",
value="correct",
comment="Factually correct", # optional
score_id="unique_id", # optional, can be used as an idempotency
# key to update the score subsequently
config_id="12345-6565-3453654-43543", # optional, to ensure that
# the score maps to a specific category defined in a score config
data_type="CATEGORICAL" # optional, possibly inferred
)
# Method 2: Score within context
with abv.start_as_current_span(name="my-operation") as span:
span.score(
name="correctness",
value="correct",
comment="Factually correct",
config_id="12345-6565-3453654-43543",
data_type="CATEGORICAL"
)
```
**Boolean Scores**
When ingesting boolean scores, you can provide the value as a float. If you provide a configId, the score’s name and config’s name must match as well as their data types.
```python
from abvdev import ABV
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# Method 1: Score via low-level method
abv.create_score(
trace_id="trace_id_here",
observation_id="observation_id_here", # optional
name="helpfulness",
value=1,
comment="Factually correct", # optional
score_id="unique_id", # optional, can be used as an
# idempotency key to update the score subsequently
config_id="93547-6565-3453654-43543", # optional, can
# be used to infer the score data type and validate the score value
data_type="BOOLEAN" # optional, possibly inferred
)
# Method 2: Score within context
with abv.start_as_current_span(name="my-operation") as span:
span.score(
name="helpfulness",
value=1,
comment="Factually correct",
config_id="93547-6565-3453654-43543",
data_type="BOOLEAN"
)
```
### **JS/TS SDK**
**Numeric Scores**
When ingesting numeric scores, you can provide the value as a float. If you provide a configId, the score value will be validated against the config's numeric range, which might be defined by a minimum and/or maximum value.
```typescript
import { ABVClient } from "@abvdev/client";
import dotenv from "dotenv";
dotenv.config();
const abv = new ABVClient();
abv.score.create({
traceId: "target_trace_id_here",
observationId: "target_observation_id_here", // optional
name: "accuracy",
value: 7,
comment: "Factually correct", // optional
id: "unique_id", // optional, can be used as an
// idempotency key to update the score subsequently
configId: "config-id-here", // optional,
// to ensure that the score follows a specific min/max value range
dataType: "NUMERIC", // optional, possibly inferred
});
async function main() {
// Flush the scores in short-lived environments
await abv.flush();
}
main();
```
**Categorical Scores**
Categorical scores are used to evaluate data that falls into specific categories. When ingesting categorical scores, you can provide the value as a string. If you provide a configId, the score value will be validated against the config’s categories.
```typescript
import { ABVClient } from "@abvdev/client";
import dotenv from "dotenv";
dotenv.config();
const abv = new ABVClient();
abv.score.create({
id: "unique_id", // optional, can be used
// as an idempotency key to update the score subsequently
traceId: "target_trace_id_here",
observationId: "target_observation_id_here", // optional
name: "accuracy",
value: "partially correct",
// key to update the score subsequently
configId: "config-id-here", // optional, to ensure that
// a score maps to a specific category defined in a score config
dataType: "CATEGORICAL", // optional, inferred if not provided
comment: "Factually correct", // optional
});
async function main() {
// Flush the scores in short-lived environments
await abv.flush();
}
```
**Boolean Scores**
When ingesting boolean scores, you can provide the value as a float. If you provide a configId, the score’s name and config’s name must match as well as their data types.
```typescript
import { ABVClient } from "@abvdev/client";
import dotenv from "dotenv";
dotenv.config();
const abv = new ABVClient();
abv.score.create({
id: "unique_id", // optional, can be used as an
// idempotency key to update the score subsequently
traceId: "cb35f468686ad95603029f404004d456",
observationId: "f7145d410802f3fe", // optional
name: "helpfulness",
value: 0, // 0 or 1
configId: "config-id-here", // optional,
// can be used to infer the score data type and validate the score value
dataType: "BOOLEAN", // required, numeric values without
// data type would be inferred as NUMERIC
comment: "Incorrect answer", // optional
});
async function main() {
// Flush the scores in short-lived environments
await abv.flush();
}
main();
```
→ More details in [Python SDK docs ](/developer/sdks/python/overview) and [JS/TS SDK docs](/developer/sdks/js-ts/overview). See [API reference](/developer/platform/api-data-platform/public-api) for more details on POST/GET score configs endpoints.
## Inferred Score Properties
Certain score properties might be inferred based on your input:
- **If you don't provide a score data type** it will always be inferred. See tables below for details.
- **For boolean and categorical scores**, we will provide the score value in both numerical and string format where possible. The score value format that is not provided as input, i.e. the translated value is referred to as the inferred value in the tables below.
- **On read for boolean scores both** numerical and string representations of the score value will be returned, e.g. both 1 and True.
- **For categorical scores**, the string representation is always provided and a numerical mapping of the category will be produced only if a `ScoreConfig` was provided.
Detailed Examples:
### Numeric Scores
For example, let's assume you'd like to ingest a numeric score to measure **accuracy**. We have included a table of possible score ingestion scenarios below.
| Value | Data Type | Config Id | Description | Inferred Data Type | Valid |
| ------- | --------- | --------- | ----------------------------------------------------------- | ------------------ | -------------------------------- |
| `0.9` | `Null` | `Null` | Data type is inferred | `NUMERIC` | Yes |
| `0.9` | `NUMERIC` | `Null` | No properties inferred | | Yes |
| `depth` | `NUMERIC` | `Null` | Error: data type of value does not match provided data type | | No |
| `0.9` | `NUMERIC` | `78545` | No properties inferred | | Conditional on config validation |
| `0.9` | `Null` | `78545` | Data type inferred | `NUMERIC` | Conditional on config validation |
| `depth` | `NUMERIC` | `78545` | Error: data type of value does not match provided data type | | No |
### Categorical Scores
For example, let’s assume you’d like to ingest a categorical score to measure **correctness**. We have included a table of possible score ingestion scenarios below.
| Value | Data Type | Config Id | Description | Inferred Data Type | Inferred Value representation | Valid |
| --------- | ------------- | --------- | ----------------------------------------------------------- | ------------------ | ----------------------------------- | -------------------------------- |
| `correct` | `Null` | `Null` | Data type is inferred | `CATEGORICAL` | | Yes |
| `correct` | `CATEGORICAL` | `Null` | No properties inferred | | | Yes |
| `1` | `CATEGORICAL` | `Null` | Error: data type of value does not match provided data type | | | No |
| `correct` | `CATEGORICAL` | `12345` | Numeric value inferred | | `4` numeric config category mapping | Conditional on config validation |
| `correct` | `NULL` | `12345` | Data type inferred | `CATEGORICAL` | | Conditional on config validation |
| `1` | `CATEGORICAL` | `12345` | Error: data type of value does not match provided data type | | | No |
### **Boolean Scores**
For example, let’s assume you’d like to ingest a boolean score to measure **helpfulness**. We have included a table of possible score ingestion scenarios below.
| Value | Data Type | Config Id | Description | Inferred Data Type | Inferred Value representation | Valid |
| ------- | --------- | --------- | ----------------------------------------------------------- | ------------------ | ----------------------------- | -------------------------------- |
| `1` | `BOOLEAN` | `Null` | Value's string equivalent inferred | | `True` | Yes |
| `true` | `BOOLEAN` | `Null` | Error: data type of value does not match provided data type | | | No |
| `3` | `BOOLEAN` | `Null` | Error: boolean data type expects `0` or `1` as input value | | | No |
| `0.9` | `Null` | `93547` | Data type and value's string equivalent inferred | `BOOLEAN` | `True` | Conditional on config validation |
| `depth` | `BOOLEAN` | `93547` | Error: data type of value does not match provided data type | | | No |
# Update Existing Scores via API/SDKs
When creating a score, you can provide an optional `id` parameter. This will update the score if it already exists within your project.
If you want to update a score without needing to fetch the list of existing scores from ABV, you can set your own `id` parameter as an idempotency key when initially creating the score.
--------------------------------------------------------------------------------
## Scores Data Model
URL: /developer/evaluations/scores-data-model
In ABV `Scores` are the data object to store evaluations. They are a flexible object that is used across all [evaluation methods](/developer/evaluations/overview) to assign evaluation scores to different objects across the ABV platform. High level you can think of Scores as the output of an evaluation method.
# Scores
`Scores` serve as objects for storing evaluation metrics in ABV. Here are its core properties:
- Scores reference a `Trace`, `Observation`, `Session`, or `DatasetRun`
- Each Score references **exactly one** of the above objects.
- Scores are either **numeric**, **categorical**, or **boolean**.
- Scores can **optionally be linked to a **`ScoreConfig` to ensure they comply with a specific schema.
## Common Use
| Level | Description |
| ----------- | -------------------------------------------------------------------------------------------------------------- |
| Trace | Used for evaluation of a single interaction. (most common) |
| Observation | Used for evaluation of a single observation below the trace level. |
| Session | Used for comprehensive evaluation of outputs across multiple interactions. |
| Dataset Run | Used for performance scores of a Dataset Run. [See Dataset Runs for context.](./dataset-runs-data-model) |
## Score object
| Attribute | Type | Description |
| --------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name` | string | Name of the score, e.g. user\_feedback, hallucination\_eval |
| `value` | number | Optional: Numeric value of the score. Always defined for numeric and boolean scores. Optional for categorical scores. |
| `stringValue` | string | Optional: String equivalent of the score's numeric value for boolean and categorical data types. Automatically set for categorical scores based on the config if the `configId` is provided. |
| `traceId` | string | Optional: Id of the trace the score relates to |
| `observationId` | string | Optional: Observation (e.g. LLM call) the score relates to |
| `sessionId` | string | Optional: Id of the session the score relates to |
| `datasetRunId` | string | Optional: Id of the dataset run the score relates to |
| `comment` | string | Optional: Evaluation comment, commonly used for user feedback, eval output or internal notes |
| `id` | string | Unique identifier of the score. Auto-generated by SDKs. Optionally can also be used as an indempotency key to update scores. |
| `source` | string | Automatically set based on the souce of the score. Can be either `API`, `EVAL`, or `ANNOTATION` |
| `dataType` | string | Automatically set based on the config data type when the `configId` is provided. Otherwise can be defined manually as `NUMERIC`, `CATEGORICAL` or `BOOLEAN` |
| `configId` | string | Optional: Score config id to ensure that the score follows a specific schema. Can be defined in the ABV UI or via API. When provided the score's `dataType` is automatically set based on the config |
# Score Config
Score configs are used to ensure that your scores follow a specific schema.
Using score configs allows you to standardize your scoring schema across your team and ensure that scores are consistent and comparable for future analysis.
You can define a `scoreConfig` in the ABV UI or via our API. Configs are immutable but can be archived (and restored anytime).
A score config includes:
- **Score name**
- **Data type:** `NUMERIC`, `CATEGORICAL`, `BOOLEAN`
- **Constraints on score value range** (Min/Max for numerical, Custom categories for categorical data types
## Score Config object
| Attribute | Type | Description |
| ------------- | ------- | ----------------------------------------------------------------------------------------------- |
| `id` | string | Unique identifier of the score config. |
| `name` | string | Name of the score config, e.g. user\_feedback, hallucination\_eval |
| `dataType` | string | Can be either `NUMERIC`, `CATEGORICAL` or `BOOLEAN` |
| `isArchived` | boolean | Whether the score config is archived. Defaults to false |
| `minValue` | number | Optional: Sets minimum value for numerical scores. If not set, the minimum value defaults to -∞ |
| `maxValue` | number | Optional: Sets maximum value for numerical scores. If not set, the maximum value defaults to +∞ |
| `categories` | list | Optional: Defines categories for categorical scores. List of objects with label value pairs |
| `description` | string | Optional: Provides further description of the score configuration |
--------------------------------------------------------------------------------
## Evals Troubleshooting and FAQ
URL: /developer/evaluations/troubleshooting-faq
FAQ
- How to create and manage Score Configs in ABV?
- I have setup ABV, but I do not see any traces in the dashboard. How to solve this?
- Ten Reasons to Use ABV for LLM Observability, Evaluations and Promp Management
- How to capture User Feedback for Evaluation of LLM apps?
- I have setup ABV, but I do not see any traces in the dashboard. How to solve this?
--------------------------------------------------------------------------------
## Prompt Management Overview
URL: /developer/prompt-management/overview
Prompt management is a systematic approach to storing, versioning and retrieving prompts in LLM applications.
# Why Prompt Management?
- Involve non-technical users through UI
- Decoupling: edit prompts without redeploying
- Quickly rollback to a previous version of a prompt
- Compare different prompt versions side-by-side
- No latency/availability impact due to [client-side caching](/developer/prompt-management/caching-prompts)
# Where to start
Follow the quickstart to create your first prompt and use it in your application.
[Quickstart](/developer/prompt-management/get-started)
[Data Model](/developer/prompt-management/prompts-data-model)
# Core Features
[Prompt Version Control](/developer/prompt-management/version-control)
[Message Placeholders in Chat Prompts](/developer/prompt-management/message-placeholders)
[Prompt Composability](/developer/prompt-management/composability)
[LLM Playground](/developer/basic-features/prompt-playground)
# Advanced Features
Extended capabilities for sophisticated prompt management and analysis:
[Link Prompts to Traces](/developer/prompt-management/link-prompts-to-traces)
[Prompt Config](/developer/prompt-management/config)
[Caching of Prompts in Client SDKs](/developer/prompt-management/caching-prompts)
[Prompt Folders](/developer/prompt-management/folders)
[A/B Testing of LLM Prompts](/developer/prompt-management/ab-testing-llm-prompts)
[Guaranteed Availability](/developer/prompt-management/guaranteed-availability)
--------------------------------------------------------------------------------
## Get Started with Prompt Management
URL: /developer/prompt-management/get-started
This quickstart helps you to create your first prompt and use it in your application.
## 1) Get API key
1. [Create ABV account](https://app.abv.dev/auth/sign-up).
2. Create new API credentials in the project settings.
## 2) Create a prompt
### **ABV UI**
****Use the ABV UI to create a new prompt or update an existing one.
### Python SDK
```bash
pip install abvdev
```
Add your ABV credentials as environment variables, e.g. use .env file.
```bash title=".env"
ABV_API_KEY = "sk-abv-..."
ABV_HOST = "https://app.abv.dev" # US region
# ABV_HOST = "https://eu.app.abv.dev" # EU region
```
Install package to get credentials from .env file
```bash
pip install python-dotenv
```
Use the Python SDK to create a new prompt or update an existing one.
```python
# Import the function to load the .env file
from dotenv import load_dotenv
from abvdev import get_client
# Load the environment variables from the .env file
load_dotenv()
abv = get_client()
# Create a text prompt
abv.create_prompt(
name="movie-critic",
type="text",
prompt="As a {{criticlevel}} movie critic, do you like {{movie}}?",
labels=["production"], # directly promote to production
config={
"model": "gpt-5-2025-08-07",
"temperature": 0.7,
"supported_languages": ["en", "fr"],
}, # optionally, add configs (e.g. model parameters or model tools) or tags
)
# Create a chat prompt
abv.create_prompt(
name="movie-critic-chat",
type="chat",
prompt=[
{ "role": "system", "content": "You are an {{criticlevel}} movie critic" },
{ "role": "user", "content": "Do you like {{movie}}?" },
],
labels=["production"], # directly promote to production
config={
"model": "gpt-5-2025-08-07",
"temperature": 0.7,
"supported_languages": ["en", "fr"],
}, # optionally, add configs (e.g. model parameters or model tools) or tags
)
```
If you already have a prompt with the same name, the prompt will be added as a new version.
### JS/TS SDK
```bash
npm i @abvdev/client
```
**Environment variables**
Add your ABV credentials as environment variables, e.g. use .env file and dotenv package to load variable values.
```bash
npm install dotenv
```
```bash title=".env"
ABV_API_KEY = "sk-abv-..."
ABV_BASEURL = "https://app.abv.dev" # US region
# ABV_BASEURL = "https://eu.app.abv.dev" # EU region
```
```typescript
import { ABVClient } from "@abvdev/client";
const abv = new ABVClient();
```
**alternatively use Constructor parameters**
```typescript
import { ABVClient } from "@abvdev/client";
const abv = new ABVClient({
apiKey: "sk-abv-...",
baseUrl: "https://app.abv.dev", // US region
// baseUrl: "https://eu.app.abv.dev", // EU region
});
```
Use the JS/TS SDK to create a new prompt or update an existing one.
```typescript
import { ABVClient } from "@abvdev/client";
import dotenv from "dotenv";
dotenv.config();
const abv = new ABVClient();
async function main() {
// Create a text prompt
await abv.prompt.create({
name: "movie-critic",
type: "text",
prompt: "As a {{criticlevel}} critic, do you like {{movie}}?",
labels: ["production"], // directly promote to production
config: {
model: "gpt-4o",
temperature: 0.7,
supported_languages: ["en", "fr"],
}, // optionally, add configs (e.g. model parameters or model tools) or tags
});
// Create a chat prompt
await abv.prompt.create({
name: "movie-critic-chat",
type: "chat",
prompt: [
{ role: "system", content: "You are an {{criticlevel}} movie critic" },
{ role: "user", content: "Do you like {{movie}}?" },
],
labels: ["production"], // directly promote to production
config: {
model: "gpt-5-2025-08-07",
temperature: 0.7,
supported_languages: ["en", "fr"],
}, // optionally, add configs (e.g. model parameters or model tools) or tags
});
}
main();
```
If you already have a prompt with the same name, the prompt will be added as a new version.
**Run your application**
Execute your application. You should see prompts appeared / updated in the ABV UI.
```bash
npx tsx filename.ts
```
### API
Use the [Public API](/developer/platform/api-data-platform/public-api) to create a new prompt or update an existing one.
```bash
curl https://app.abv.devapi/public/v2/prompts \
--request POST \
--header 'Content-Type: application/json' \
--data '{
"type": "chat",
"name": "",
"prompt": [
{
"type": "chatmessage",
"role": "",
"content": ""
}
],
"config": null,
"labels": [
""
],
"tags": [
""
],
"commitMessage": null
}'
```
## 3) Use prompt
At runtime, you can fetch the latest production version from ABV. Learn more about control (versions/labels) [here](/developer/prompt-management/version-control).
### Python SDK
```python
from abvdev import get_client
# Initialize ABV client
abv = get_client()
```
**Text prompt**
```python
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# Get current `production` version of a text prompt
prompt = abv.get_prompt("movie-critic")
# Insert variables into prompt template
compiled_prompt = prompt.compile(criticlevel="expert", movie="Dune 2")
# -> "As an expert movie critic, do you like Dune 2?"
print(compiled_prompt)
```
**Chat prompt**
```python
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# Get current `production` version of a chat prompt
chat_prompt = abv.get_prompt("movie-critic-chat", type="chat") # type arg infers the prompt type (default is 'text')
# Insert variables into chat prompt template
compiled_chat_prompt = chat_prompt.compile(criticlevel="expert", movie="Dune 2")
# -> [{"role": "system", "content": "You are an expert movie critic"}, {"role": "user", "content": "Do you like Dune 2?"}]
print(compiled_chat_prompt)
```
**Optional parameters**
```python
# Get specific version
prompt = abv.get_prompt("movie-critic", version=1)
# Get specific label
prompt = abv.get_prompt("movie-critic", label="staging")
# Get latest prompt version. The 'latest' label is automatically maintained by ABV.
prompt = abv.get_prompt("movie-critic", label="latest")
```
**Attributes .prompt and .config**
```python
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# Get current `production` version of a chat prompt
chat_prompt = abv.get_prompt("movie-critic-chat", type="chat") # type arg infers the prompt type (default is 'text')
print(chat_prompt.prompt)
print(chat_prompt.config)
```
### JS/TS SDK
**Text prompt**
```typescript
import { ABVClient } from "@abvdev/client";
import dotenv from "dotenv";
dotenv.config();
const abv = new ABVClient();
async function main() {
// Get current `production` version
const prompt = await abv.prompt.get("movie-critic");
// Insert variables into prompt template
const compiledPrompt = prompt.compile({
criticlevel: "expert",
movie: "Dune 2",
});
// -> "As an expert movie critic, do you like Dune 2?"
console.log(compiledPrompt);
}
main();
```
**Chat prompt**
```typescript
import { ABVClient } from "@abvdev/client";
import dotenv from "dotenv";
dotenv.config();
const abv = new ABVClient();
async function main() {
// Get current `production` version of a chat prompt
const chatPrompt = await abv.prompt.get("movie-critic-chat", {
type: "chat",
}); // type option infers the prompt type (default is 'text')
// Insert variables into chat prompt template
const compiledChatPrompt = chatPrompt.compile({
criticlevel: "expert",
movie: "Dune 2",
});
// -> [{"role": "system", "content": "You are an expert movie critic"}, {"role": "user", "content": "Do you like Dune 2?"}]
console.log(compiledChatPrompt);
}
main();
```
**Optional parameters**
```typescript
import { ABVClient } from "@abvdev/client";
import dotenv from "dotenv";
dotenv.config();
const abv = new ABVClient();
async function main() {
// Get specific version of a prompt (here version 1)
const prompt1 = await abv.prompt.get("movie-critic", {
version: 1
});
// Get specific label
const prompt2 = await abv.prompt.get("movie-critic", {
label: "staging", // use any label that is available / added in the prompt versions
});
// Get latest prompt version. The 'latest' label is automatically maintained by ABV.
const prompt3 = await abv.prompt.get("movie-critic", {
label: "latest",
});
// Insert variables into prompt template
const compiledPrompt = prompt3.compile({
criticlevel: "expert",
movie: "Dune 2",
});
// -> "As an expert movie critic, do you like Dune 2?"
console.log(compiledPrompt);
}
main();
```
**Attributes**
```typescript
import { ABVClient } from "@abvdev/client";
import dotenv from "dotenv";
dotenv.config();
const abv = new ABVClient();
async function main() {
// Get specific version of a prompt (here version 1)
const prompt1 = await abv.prompt.get("movie-critic", {
version: 1
});
// Raw prompt including {{variables}}. For chat prompts, this is a list of chat messages.
console.log(prompt1.prompt);
// Config object
console.log(prompt1.config);
}
main();
```
## 4) Link with ABV Tracing (optional)
You can link the prompt to the LLM `generation` span that used the prompt. This linkage enables tracking of metrics by prompt version and name directly in the ABV UI and see which prompt performed best.
### Python SDK
**Decorators**
```python
from abvdev import ABV, observe
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
@observe(as_type="generation")
def nested_generation():
prompt = abv.get_prompt("movie-critic")
abv.update_current_generation(
prompt=prompt,
)
@observe()
def main():
nested_generation()
main()
```
**Context Managers**
```python
from abvdev import ABV, observe
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
prompt = abv.get_prompt("movie-critic")
with abv.start_as_current_observation(
as_type='generation',
name="movie-generation",
model="gpt-5-2025-08-07",
prompt=prompt
) as generation:
# Your LLM call here
generation.update(output="LLM response")
```
If a [fallback prompt](./guaranteed-availability) is used, no link will be created.
### JS/TS SDK
**install additional packages**
```bash
npm install @abvdev/tracing @abvdev/otel @opentelemetry/sdk-node
```
Create a `instrumentation.ts` file that initializes the OpenTelemetry `NodeSDK` and registers the `ABVSpanProcessor`.
```typescript title="instrumentation.ts"
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
const sdk = new NodeSDK({
spanProcessors: [new ABVSpanProcessor()],
});
sdk.start();
```
Modify `instrumentation.ts` file to use `dotenv` package to load the variables.
Additional parameters are provided to get trace visible in the UI immediately.
```typescript title="instrumentation.ts"
import dotenv from "dotenv";
dotenv.config();
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
const sdk = new NodeSDK({
spanProcessors: [
new ABVSpanProcessor({
apiKey: process.env.ABV_API_KEY,
baseUrl: process.env.ABV_BASE_URL,
exportMode: "immediate",
flushAt: 1,
flushInterval: 1,
additionalHeaders: {
"Content-Type": "application/json",
"Accept": "application/json"
}
})
],
});
sdk.start();
```
Import the `instrumentation.ts` file at the top of your application.
```typescript title="index.ts"
import "./instrumentation"; // Must be the first import
```
**Use prompt with Manual observations**
```typescript
import "./instrumentation"; // Must be the first import
import { ABVClient } from "@abvdev/client";
import { startObservation, updateActiveObservation } from "@abvdev/tracing";
const prompt = new ABVClient().prompt.get("movie-critic");
async function main() {
const generation = startObservation(
"llm",
{
input: (await prompt).prompt,
},
{ asType: "generation" },
);
generation.end();
}
main();
```
**With Context manager**
```typescript
import "./instrumentation"; // Must be the first import
import { ABVClient } from "@abvdev/client";
import { startActiveObservation } from "@abvdev/tracing";
const abv = new ABVClient();
startActiveObservation(
"llm",
async (generation) => {
const prompt = abv.prompt.get("movie-critic");
generation.update({ input: (await prompt).prompt });
},
{ asType: "generation" },
);
```
**With Observe wrapper**
```typescript
import { ABVClient } from "@abvdev/client";
import { observe, updateActiveObservation } from "@abvdev/tracing";
const abv = new ABVClient();
const callLLM = async (input: string) => {
const prompt = abv.prompt.get("my-prompt");
updateActiveObservation({ prompt }, { asType: "generation" });
return await invokeLLM(input);
};
export const observedCallLLM = observe(callLLM);
```
If a [fallback prompt](./guaranteed-availability) is used, no link will be created.
## Next Steps
Now that you've created and fetched your first prompt, explore these advanced features:
- [Version Control](/developer/prompt-management/version-control): Learn how to manage prompt versions and labels for safe deployments
- [Link Prompts to Traces](/developer/prompt-management/link-prompts-to-traces): Track metrics and performance for each prompt version
- [A/B Testing](/developer/prompt-management/ab-testing-llm-prompts): Compare different prompt versions in production
- [Message Placeholders](/developer/prompt-management/message-placeholders): Use dynamic placeholders in chat prompts
- [Prompt Experiments](/developer/evaluations/prompt-experiments): Test prompts systematically with datasets
- [Caching](/developer/prompt-management/caching-prompts): Learn about client-side caching for zero-latency prompt fetching
- [Prompt Playground](/developer/basic-features/prompt-playground): Test prompts interactively in the UI
## Integration with Other Features
Prompt Management works seamlessly with other ABV features:
- **Observability**: [Link prompts to traces](/developer/prompt-management/link-prompts-to-traces) to see which prompt version generated each response
- **Evaluations**: Use [Prompt Experiments](/developer/evaluations/prompt-experiments) to compare prompt performance on test datasets
- **SDKs**: Fetch prompts programmatically with [Python SDK](/developer/sdks/python/overview) or [JS/TS SDK](/developer/sdks/js-ts/overview)
- **Metrics**: Track prompt performance over time in the [Metrics Dashboard](/developer/platform/metrics/overview)
--------------------------------------------------------------------------------
## Prompt Config
URL: /developer/prompt-management/config
This config is **versioned** together with the prompt and allows you to manage all parameters associated with a prompt in one place, making it easier to deploy, update, and track prompt behavior and performance across different versions and environments.
# Use cases
- Change the model your application uses without touching code
- Manage and version the function/tool JSON schemas your application uses
- Define the JSON schema for your application's responses
- Set and manage you model parameters such as temperature, top\_p, max\_tokens, etc.
--------------------------------------------------------------------------------
## Prompt Folders
URL: /developer/prompt-management/folders
Prompts can be organized into virtual folders to group prompts with similar purposes.
To create a folder, add slashes (`/`) to a prompt name. The UI shows every segment ending with a `/` as a folder automatically.
# Create a folder
Use the ABV UI to create a folder by adding a slash (`/`) to a prompt name.
--------------------------------------------------------------------------------
## Prompt Version Control
URL: /developer/prompt-management/version-control
# Versions & Labels
Each prompt version is automatically assigned a `version ID`. Additionally, you can assign `labels` to follow your own versioning scheme.
Labels can be used to assign prompts to environments (staging, production), tenants (tenant-1, tenant-2), or experiments (prod-a, prod-b).
## UI
Use the ABV UI to assign labels to a prompt.
## Python SDK
Install package
```bash
pip install abvdev
```
Use the Python SDK to assign labels to a prompt when creating a new prompt version.
```python
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
abv.create_prompt(
name="movie-critic",
type="text",
prompt="As a {{criticlevel}} movie critic, do you like {{movie}}?",
labels=["production"], # add the label "production" to the prompt version
)
```
Alternatively, you can also update the labels of an existing prompt version using the Python SDK:
```python
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
abv.update_prompt(
name="movie-critic",
version=1,
new_labels=["john", "doe"], # assign these labels to the prompt version
)
```
## JS/TS SDK
```bash
npm i @abvdev/client
```
**Environment variables**
Add your ABV credentials as environment variables, e.g. use .env file and dotenv package to load variable values.
```bash
npm install dotenv
```
```bash title=".env"
ABV_API_KEY = "sk-abv-...";
ABV_BASEURL = "https://app.abv.dev"; US region
# ABV_BASEURL = "https://eu.app.abv.dev"; EU region
```
```typescript
import { ABVClient } from "@abvdev/client";
const abv = new ABVClient();
```
**alternatively use Constructor parameters**
```typescript
import { ABVClient } from "@abvdev/client";
const abv = new ABVClient({
apiKey: "sk-abv-...",
baseUrl: "https://app.abv.dev", // US region
// baseUrl: "https://eu.app.abv.dev", // EU region
});
```
Use the JS/TS SDK to assign labels to a prompt when creating a new prompt version.
```typescript
import { ABVClient } from "@abvdev/client";
import dotenv from "dotenv";
dotenv.config();
const abv = new ABVClient();
async function main() {
await abv.prompt.create({
name: "movie-critic",
type: "text",
prompt: "As a {{criticlevel}} critic, do you like {{movie}}?",
labels: ["production"], // add the label "production" to the prompt version
});
}
main();
```
Alternatively, you can also update the labels of an existing prompt version using the JS/TS SDK:
```typescript
import { ABVClient } from "@abvdev/client";
import dotenv from "dotenv";
dotenv.config();
const abv = new ABVClient();
async function main() {
await abv.prompt.update({
name: "movie-critic",
version: 1,
newLabels: ["john", "doe"],
});
}
main();
```
# Fetching by Label or Version
When fetching prompts to use them in your application you can either do you by fetching a specific version or label.
Here are code examples for fetching prompts by label or version.
**To "deploy" a prompt version**, you have to assign the label `production` or any environment label you created to that prompt version.
Some notes on fetching prompts:
- The `latest` label points to the most recently created version.
- When using a prompt without specifying a label, ABV will serve the version with the `production` label.
## Python SDK
```python
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# Get specific version
prompt = abv.get_prompt("movie-critic", version=1)
# Get specific label
prompt = abv.get_prompt("movie-critic", label="doe")
# Get latest prompt version. The 'latest' label is automatically maintained by ABV.
prompt = abv.get_prompt("movie-critic", label="latest")
```
## JS/TS SDK
```typescript
import { ABVClient } from "@abvdev/client";
import dotenv from "dotenv";
dotenv.config();
const abv = new ABVClient();
async function main() {
// Get specific version of a prompt (here version 1)
const prompt1 = await abv.prompt.get("movie-critic", {
version: 1,
});
// Get specific label
const prompt2 = await abv.prompt.get("movie-critic", {
label: "stage", // use any existing label here
});
// Get latest prompt version. The 'latest' label is automatically maintained by ABV.
const prompt3 = await abv.prompt.get("movie-critic", {
label: "latest",
});
console.log(prompt3.prompt)
}
main();
```
# Rollbacks
When a prompt has a `production` label, then that version will be served by default in the SDKs. You can quickly rollback to a previous version by setting the `production` label to that previous version in the ABV UI.
# Prompt Diffs
The prompt version diff view shows you the changes you made to the prompt over time. This helps you understand how the prompt has evolved and what changes have been made to debug issues or understand the impact of changes.
# Protected prompt labels
Protected prompt labels give project admins and owners ([RBAC docs](/developer/platform/administration/role-based-access-controls)) the ability to prevent labels from being modified or deleted, ensuring better control over prompt deployment.
Once a label such as `production` is marked as protected:
- `viewer` and `member` roles cannot modify or delete the label from prompts, preventing changes to the `production` prompt version. This also blocks the deletion of the prompt.
- `admin` and `owner` roles can still modify or delete the label, effectively changing the `production` prompt version.
Admins and owners can update a label's protection status in the project settings.
--------------------------------------------------------------------------------
## ABV Prompts Data Model
URL: /developer/prompt-management/prompts-data-model
# Prompt Object
Example prompt in ABV with custom config
```json
{
"name": "movie-critic",
"type": "text",
"prompt": "As a {{criticLevel}} movie critic, do you like {{movie}}?",
"config": {
"model": "gpt-4o",
"temperature": 0.5,
"supported_languages": ["en", "fr"]
},
"version": 1,
"labels": ["production", "staging", "latest"],
"tags": ["movies"]
}
```
- `name`: Unique name of the prompt within a ABV project.
- `type`: The [type](./#chat-vs-text-prompts) of the prompt content (`text` or `chat`). Default is `text`.
- `prompt`: The text template with variables (e.g. `This is a prompt with a {{variable}}`). For chat prompts, this is a list of chat messages each with `role` and `content` but can also contain [placeholders](./#prompt-object).
- `config`: Optional JSON object to store any parameters (e.g. model parameters or model tools).
- `version`: Integer to indicate the version of the prompt. The version is automatically incremented when creating a new prompt version.
- `labels`: [Labels](./#versioning-and-labels) that can be used to fetch specific prompt versions in the SDKs.
- When using a prompt without specifying a label, ABV will serve the version with the `production` label.
- `latest` points to the most recently created version.
- You can create any additional labels, e.g. for different environments (`staging`, `production`) or tenants (`tenant-1`, `tenant-2`).
- `tags`: Use tags to categorize prompts, e.g. to filter them in the UI or SDKs. All versions of a prompt share the same tags.
# Chat vs Text prompts
ABV supports two types of prompts: **text** and **chat**. The key difference is in how the content is structured:
## Text Prompts
Text prompts contain a single string with optional variables. They are ideal for simple completions or when using models that expect a single text input.
Text prompt example
```json
{
"name": "movie-critic",
"type": "text",
"prompt": "As a {{criticLevel}} movie critic, do you like {{movie}}?",
"version": 1
}
```
When compiled, text prompts return a single string:
```text
# Returns: "As an expert movie critic, do you like Dune 2?"
```
## Chat Prompts
Chat prompts contain an array of messages, each with a `role` and `content`. They are designed for conversational models and multi-turn interactions.
Chat prompt example
```json
{
"name": "movie-critic-chat",
"type": "chat",
"prompt": [
{
"role": "system",
"content": "You are a {{criticLevel}} movie critic"
},
{
"role": "user",
"content": "Do you like {{movie}}?"
}
],
"version": 1
}
```
When compiled, chat prompts return an array of message objects:
```text
# Returns: [
# {"role": "system", "content": "You are an expert movie critic"},
# {"role": "user", "content": "Do you like Dune 2?"}
# ]
```
Both prompt types support variable substitution using `{{variable}}` syntax and can include custom configuration parameters.
# Versioning and Labels
ABV prompt [versioning](/developer/prompt-management/version-control) automatically creates incremental versions (1, 2, 3...) each time you update a prompt with the same name, while labels act as pointers to specific versions for deployment management. The `production` label serves as the default version returned by SDKs when no specific label is requested, `latest` automatically points to the most recently created version, and you can create custom labels (like `staging`, `tenant-1`, or `prod-a`) to manage different environments, tenants, or A/B testing scenarios - allowing you to safely deploy, test, and rollback prompts by simply reassigning labels rather than changing code.
--------------------------------------------------------------------------------
## Prompt Composability
URL: /developer/prompt-management/composability
# Why Use Composed Prompts?
- Create modular **prompt components** that can be reused across multiple prompts
- **Maintain** common instructions, examples, or context in a single place
- **Update dependent prompts** automatically when base prompts change
# Get started
## ABV UI
When creating the prompt via the ABV UI, you can use the `Add prompt reference` button to insert a prompt reference into your prompt.
## SDKs/API
You can reference other **text** prompts in your prompts the following format:
```text
@@@abvdevPrompt:name=PromptName|version=1@@@
```
You can also use a label instead of a specific version for dynamic resolution:
```text
@@@abvdevPrompt:name=PromptName|label=production@@@
```
--------------------------------------------------------------------------------
## Message Placeholders in Chat Prompts
URL: /developer/prompt-management/message-placeholders
You can define multiple placeholders in a prompt and resolve them with different values at runtime.
Message Placeholders are also supported in the [Playground](/developer/basic-features/prompt-playground) and [Prompt Experiments](/developer/evaluations/prompt-experiments).
## 1) Create prompt with placeholders
### via UI
1. Create a placeholder in any prompt by using the `Add message placeholder` button.
2. Select a `name` for the placeholder that will be used to reference it in your application.
### with Python SDK
Install package
```bash
pip install abvdev
```
```python
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
abv.create_prompt(
name="movie-critic-chat",
type="chat",
prompt=[
{ "role": "system", "content": "You are an {{criticlevel}} movie critic" },
{ "type": "placeholder", "name": "chat_history" },
{ "role": "user", "content": "What should I watch next?" },
],
labels=["production"], # directly promote to production
)
```
### JS/TS SDK
```bash
npm i @abvdev/client
```
**Environment variables**
Add your ABV credentials as environment variables, e.g. use .env file and dotenv package to load variable values.
```bash
npm install dotenv
```
```bash title=".env"
ABV_API_KEY = "sk-abv-...";
ABV_BASEURL = "https://app.abv.dev"; US region
# ABV_BASEURL = "https://eu.app.abv.dev"; EU region
```
```typescript
import { ABVClient } from "@abvdev/client";
const abv = new ABVClient();
```
**alternatively use Constructor parameters**
```typescript
import { ABVClient } from "@abvdev/client";
const abv = new ABVClient({
apiKey: "sk-abv-...",
baseUrl: "https://app.abv.dev", // US region
// baseUrl: "https://eu.app.abv.dev", // EU region
});
```
**create prompt with placeholders**
```typescript
import { ABVClient } from "@abvdev/client";
import dotenv from "dotenv";
dotenv.config();
const abv = new ABVClient();
async function main() {
await abv.prompt.create({
name: "movie-critic-chat",
type: "chat",
prompt: [
{ role: "system", content: "You are an {{criticlevel}} movie critic" },
{ type: "placeholder", name: "chat_history" },
{ role: "user", content: "What should I watch next?" },
],
labels: ["production"], // directly promote to production
});
}
main();
```
## 2) Resolve placeholders at runtime
In the SDKs, use the `.compile(variables, placeholders)` method on a `ChatPromptClient` to set the values to be filled in for the placeholders.
The filled in messages should be of the `ChatMessage` format with a `role` and `content` property, but custom formats are also accepted as `compile` does not validate the format of the messages.
### Python SDK
```python
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# Use prompt with placeholders in your application
prompt = abv.get_prompt("movie-critic-chat")
# Compile the variable and resolve the placeholder with a list of messages.
compiled_prompt = prompt.compile(criticlevel="expert", chat_history=[
{"role": "user", "content": "I love Ron Fricke movies like Baraka"},
{"role": "user", "content": "Also, the Korean movie Memories of a Murderer"}
])
print(compiled_prompt)
# -> compiled_prompt = [
# { "role": "system", "content": "You are an expert movie critic" },
# { "role": "user", "content": "I love Ron Fricke movies like Baraka" },
# { "role": "user", "content": "Also, the Korean movie Memories of a Murderer" },
# { "role": "user", "content": "What should I watch next?" },
# ]
```
### JS/TS SDK
```typescript
import { ABVClient } from "@abvdev/client";
import dotenv from "dotenv";
dotenv.config();
const abv = new ABVClient();
async function main() {
const prompt = await abv.prompt.get("movie-critic-chat", {
type: "chat",
});
// Compile the variable and resolve the placeholder with a list of messages.
const compiledPrompt = prompt.compile(
// variables
{ criticlevel: "expert" },
// placeholders
{
chat_history: [
{ role: "user", content: "I love Ron Fricke movies like Baraka" },
{
role: "user",
content: "Also, the Korean movie Memories of a Murderer",
},
],
}
);
console.log(compiledPrompt);
// -> compiledPrompt = [
// { role: "system", content: "You are an expert movie critic" },
// { role: "user", content: "I love Ron Fricke movies like Baraka" },
// { role: "user", content: "Also, the Korean movie Memories of a Murderer" },
// { role: "user", content: "What should I watch next?" },
// ]
}
main();
```
--------------------------------------------------------------------------------
## A/B Testing of LLM Prompts
URL: /developer/prompt-management/ab-testing-llm-prompts
**When to use A/B testing?**
A/B testing helps you see how different prompt versions work in real situations, adding to what you learn from testing on datasets. This works best when:
- Your app has good ways to measure success, deals with many different kinds of user inputs, and can handle some ups and downs in performance. This usually works for consumer apps where mistakes aren't a big deal.
- You've already tested thoroughly on your test data and want to try your changes with a small group of users before rolling out to everyone (also called canary deployment).
# Implementation
## 1) Label your Prompt Versions
Label your prompt versions (e.g., `prod-a` and `prod-b`) to identify different variants for testing.
## 2) Fetch Prompts and Run A/B Test
### Python SDK
Install package
```bash
pip install abvdev
```
```python
from abvdev import ABV
from openai import OpenAI
import random
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
openai_client = OpenAI(api_key="sk-proj-...")
# Fetch prompt versions
prompt_a = abv.get_prompt("movie-critic", label="prod-a")
prompt_b = abv.get_prompt("movie-critic", label="prod-b")
# Randomly select version
selected_prompt = random.choice([prompt_a, prompt_b])
# Use in LLM call
with abv.start_as_current_observation(as_type="generation", name="OpenAI-gen", prompt=selected_prompt):
response = openai_client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": selected_prompt.compile(variable="value")}],
)
result_text = response.choices[0].message.content
print(result_text)
# Flush events to ABV in short-lived applications
abv.flush()
```
### JS/TS SDK
**Install packages**
```bash
npm install @abvdev/tracing @abvdev/otel @abvdev/client
```
**Add credentials**
Add your ABV credentials to your environment variables. Make sure that you have a `.env` file in your project root and a package like dotenv to load the variables.
```bash title=".env"
ABV_API_KEY = "sk-abv-..."
ABV_BASE_URL = "https://app.abv.dev" # US region
//ABV_BASE_URL = "https://eu.app.abv.dev" # EU region
OPENAI_API_KEY = "sk-proj-..." // added to use openai llm
```
**Initialize OpenTelemetry**
Install the OpenTelemetry Node SDK package:
```bash
npm install @opentelemetry/sdk-node
```
Create a `instrumentation.ts` file that initializes the OpenTelemetry `NodeSDK` and registers the `ABVSpanProcessor`.
```typescript title="instrumentation.ts"
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
const sdk = new NodeSDK({
spanProcessors: [new ABVSpanProcessor()],
});
sdk.start();
```
Modify `instrumentation.ts` file to use `dotenv` package to load the variables.
Additional parameters are provided to get trace visible in the UI immediately.
```bash
npm install dotenv
```
```typescript title="instrumentation.ts"
import dotenv from "dotenv";
dotenv.config();
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
const sdk = new NodeSDK({
spanProcessors: [
new ABVSpanProcessor({
apiKey: process.env.ABV_API_KEY,
baseUrl: process.env.ABV_BASE_URL,
exportMode: "immediate",
flushAt: 1,
flushInterval: 1,
additionalHeaders: {
"Content-Type": "application/json",
"Accept": "application/json"
}
})
],
});
sdk.start();
```
Import the `instrumentation.ts` file at the top of your application.
```typescript title="index.ts"
import "./instrumentation"; // Must be the first import
```
**fetch prompts and run a/b test**
```typescript
import "./instrumentation"; // Must be the first import
import { ABVClient } from "@abvdev/client";
import { startObservation } from "@abvdev/tracing";
import OpenAI from "openai";
import dotenv from "dotenv";
dotenv.config();
const openai = new OpenAI({
apiKey: process.env.OPENAI_API_KEY,
});
// Requires environment variables for initialization
const abv = new ABVClient();
async function main(){
// Fetch prompt versions
const promptA = await abv.prompt.get("movie-critic-chat", {
label: "prod-a",
});
const promptB = await abv.prompt.get("movie-critic-chat", {
label: "prod-b",
});
// Randomly select version
const selectedPrompt = Math.random() < 0.5 ? promptA : promptB;
const model = "gpt-4o";
// Use in LLM call
// Start a trace without knowing the user yet
const generation = startObservation(
"llm-call-a-b",
{
model: model,
input: [{ role: "user", content: "What is the capital of France?" }],
prompt: selectedPrompt
},
{ asType: "generation" }
);
const completion = await openai.chat.completions.create({
model: model,
messages: selectedPrompt.compile({ criticlevel: "expert", movie: "The Lord of the Rings"}),
});
const resultText = completion.choices[0].message.content;
console.log(resultText);
generation.update({
output: { content: resultText },
});
generation.end();
}
main();
```
Refer to [prompt management documentation](/developer/prompt-management/get-started) for additional examples on how to fetch and use prompts.
## 3) Analyze Results
Compare metrics for each prompt version in the ABV UI:
**Key metrics available for comparison:**
- Response latency and token usage
- Cost per request
- Quality evaluation scores
- Custom metrics you define
--------------------------------------------------------------------------------
## Link Prompts to Traces
URL: /developer/prompt-management/link-prompts-to-traces
After linking prompts and traces, navigating to a generation span in ABV will highlight the prompt that was used to generate the response. To access the metrics, navigate to your prompt and click on the `Metrics` tab.
# How to Link Prompts to Traces
## Python SDK
Install package
```bash
pip install abvdev, openai
```
**Decorators**
```python
from abvdev import ABV, observe
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
@observe(as_type="generation")
def nested_generation():
prompt = abv.get_prompt("movie-critic")
abv.update_current_generation(
prompt=prompt,
)
@observe()
def main():
nested_generation()
main()
```
**Context Managers**
```python
from abvdev import ABV
from openai import OpenAI
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
openai_client = OpenAI(api_key = "sk-proj-...") # your openai api key here
prompt = abv.get_prompt("movie-critic")
compiled_prompt = prompt.compile(criticlevel="expert", movie="The Lord of the Rings")
with abv.start_as_current_observation(
as_type='generation',
name="movie-generation",
model="gpt-4o",
prompt=prompt
) as generation:
# ... make LLM call ...
response = openai_client.chat.completions.create(
messages=[{"role": "user", "content": compiled_prompt}],
model="gpt-4o",
)
generation.update(output=response.choices[0].message.content)
```
If a [fallback prompt](./guaranteed-availability) is used, no link will be created.
## JS/TS SDK
```bash
npm i @abvdev/client @abvdev/tracing @abvdev/otel @opentelemetry/sdk-node
```
**Environment variables**
Add your ABV credentials as environment variables, e.g. use .env file and dotenv package to load variable values.
```bash
npm install dotenv
```
```bash title=".env"
ABV_API_KEY = "sk-abv-...";
ABV_BASEURL = "https://app.abv.dev"; US region
# ABV_BASEURL = "https://eu.app.abv.dev"; EU region
```
Create a `instrumentation.ts` file that initializes the OpenTelemetry `NodeSDK` and registers the `ABVSpanProcessor`.
```typescript title="instrumentation.ts"
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
const sdk = new NodeSDK({
spanProcessors: [new ABVSpanProcessor()],
});
sdk.start();
```
Modify `instrumentation.ts` file to use `dotenv` package to load the variables.
Additional parameters are provided to get trace visible in the UI immediately.
```typescript title="instrumentation.ts"
import dotenv from "dotenv";
dotenv.config();
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
const sdk = new NodeSDK({
spanProcessors: [
new ABVSpanProcessor({
apiKey: process.env.ABV_API_KEY,
baseUrl: process.env.ABV_BASE_URL,
exportMode: "immediate",
flushAt: 1,
flushInterval: 1,
additionalHeaders: {
"Content-Type": "application/json",
"Accept": "application/json"
}
})
],
});
sdk.start();
```
Import the `instrumentation.ts` file at the top of your application.
```typescript title="index.ts"
import "./instrumentation"; // Must be the first import
```
**Use prompt with Manual observations**
```typescript
import "./instrumentation"; // Must be the first import
import { ABVClient } from "@abvdev/client";
import { startObservation, updateActiveObservation } from "@abvdev/tracing";
const prompt = new ABVClient().prompt.get("movie-critic");
async function main() {
const generation = startObservation(
"llm",
{
input: (await prompt).prompt,
},
{ asType: "generation" },
);
generation.end();
}
main();
```
**With Context manager**
```typescript
import "./instrumentation"; // Must be the first import
import { ABVClient } from "@abvdev/client";
import { startActiveObservation } from "@abvdev/tracing";
const abv = new ABVClient();
startActiveObservation(
"llm",
async (generation) => {
const prompt = abv.prompt.get("movie-critic");
generation.update({ input: (await prompt).prompt });
},
{ asType: "generation" },
);
```
**With Observe wrapper**
```typescript
import { ABVClient } from "@abvdev/client";
import { observe, updateActiveObservation } from "@abvdev/tracing";
const abv = new ABVClient();
const callLLM = async (input: string) => {
const prompt = abv.prompt.get("my-prompt");
updateActiveObservation({ prompt }, { asType: "generation" });
return await invokeLLM(input);
};
export const observedCallLLM = observe(callLLM);
```
If a [fallback prompt](./guaranteed-availability) is used, no link will be created.
# Metrics Reference
- Median generation latency
- Median generation input tokens
- Median generation output tokens
- Median generation costs
- Generation count
- Median [score](/developer/evaluations/scores-data-model) value
- First and last generation timestamp
--------------------------------------------------------------------------------
## GitHub Integration for ABV Prompts
URL: /developer/prompt-management/github-integration
- [GitHub Actions Webhook](./#trigger-github-actions) - Trigger CI/CD workflows when prompts change. This does not require additional infrastructure.
- [Sync ABV Prompts to a repository](./#sync-abv-prompts-to-a-repository) - Store prompts in a specific file in your repository. This involves a webhook server that listens for prompt version changes and commits them to the repository.
***
# Trigger GitHub Actions
Trigger GitHub Actions workflows when ABV prompts change using `repository_dispatch` events.
## 1. Create GitHub Workflow
`.github/workflows/abv-ci.yml`:
```yaml
name: ABV Prompt CI
on:
repository_dispatch:
types: [abv-prompt-update]
workflow_dispatch:
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run tests
run: |
echo "Testing prompt: ${{ github.event.client_payload.prompt.name }} v${{ github.event.client_payload.prompt.version }}"
# Add your test commands
# npm test
# python -m pytest
deploy:
needs: test
runs-on: ubuntu-latest
if: contains(github.event.client_payload.prompt.labels, 'production')
steps:
- uses: actions/checkout@v4
- name: Deploy to production
run: |
echo "Deploying ${{ github.event.client_payload.prompt.name }} v${{ github.event.client_payload.prompt.version }}"
# Your deployment commands
```
**Accessing webhook data:** Use `github.event.client_payload.*` to access prompt data:
```yaml
# Example: Access webhook data in your workflow
- name: Process prompt data
run: |
echo "Action: ${{ github.event.client_payload.action }}"
echo "Prompt: ${{ github.event.client_payload.prompt.name }}"
echo "Version: ${{ github.event.client_payload.prompt.version }}"
echo "Labels: ${{ github.event.client_payload.prompt.labels }}"
- name: Deploy only production prompts
if: contains(github.event.client_payload.prompt.labels, 'production')
run: echo "Deploying production prompt"
```
## 2. Create GitHub Token for Actions
**Steps:**
1. **GitHub Settings > Developer settings > Personal access tokens**
2. **Generate new token (classic or fine-grained)**
3. **Select scope** (see table below)
| Token Type | Required Permissions |
| ------------------------------- | ------------------------------------------------------------------ |
| Personal Access Token (classic) | `repo` scope (public repos) or `public_repo` scope (private repos) |
| Fine-grained PAT or GitHub App | `read` and `write` to `actions` |
## 3. Configure ABV Webhook for Actions
1. Go to **Prompts > Webhooks** in your ABV project
2. Click **Create Webhook**
3. Set endpoint URL: `https://api.github.com/repos/{owner}/{repo}/dispatches`
4. Add headers:
- `Accept: application/vnd.github+json`
- `Authorization: Bearer {your_github_token}`
**Important:** Store your GitHub token securely in the Authorization header. ABV encrypts webhook headers and stores them securely.
## 4. Test GitHub Actions Integration
1. **Update a prompt** in ABV with the `production` label
2. **Check GitHub Actions** tab for triggered workflow
3. **Verify** that both test and deploy jobs run successfully
***
# Sync ABV Prompts to a repository
Automatically sync prompt changes from ABV to GitHub. This enables version control for prompts and can trigger CI/CD workflows.
## Overview of the Sync Workflow
Whenever you save a new prompt version in ABV, it's automatically committed to your GitHub repository. With this setup, you can also trigger CI/CD workflows when prompts change.
## Prerequisites for Sync
1. **ABV Project:** [Prompt setup](/developer/prompt-management/get-started) with Project Owner access
2. **GitHub Repository:** Public or private repo to store prompts
3. **GitHub PAT:** Personal Access Token with minimum required permissions (see Step 2 for details)
4. **Python 3.9+ (for the example below, can be any language)** with FastAPI, Uvicorn, httpx, Pydantic
5. **Public HTTPS endpoint** for your webhook server (Render, Fly.io, Heroku, etc.)
## Step 1: Configure a Prompt Webhook in ABV
1. Go to **Prompts > Webhooks** in your ABV project
2. Click **Create Webhook**
3. (optional) filter events: filter by which prompt version events to receive webhooks (default: `created`, `updated`, `deleted`)
4. Set endpoint URL: `https:///webhook/prompt`
5. Save and copy the **Signing Secret**
**Note:** Your endpoint must return 2xx status codes. ABV retries failed webhooks with exponential backoff.
### Sample Webhook Payload
Sample webhook payload:
```json
{
"id": "550e8400-e29b-41d4-a716-446655440000",
"timestamp": "2024-07-10T10:30:00Z",
"type": "prompt-version",
"action": "created",
"prompt": {
"id": "prompt_abc123",
"name": "movie-critic",
"version": 3,
"projectId": "xyz789",
"labels": ["production", "latest"],
"prompt": "As a {{criticLevel}} movie critic, rate {{movie}} out of 10.",
"type": "text",
"config": { "...": "..." },
"commitMessage": "Improved critic persona",
"tags": ["entertainment"],
"createdAt": "2024-07-10T10:30:00Z",
"updatedAt": "2024-07-10T10:30:00Z"
}
}
```
## Step 2: Prepare Your GitHub Repo and Token for Sync
Create a `.env` file with your GitHub credentials:
```bash
GITHUB_TOKEN=
GITHUB_REPO_OWNER=
GITHUB_REPO_NAME=
# (Optional) GITHUB_FILE_PATH=abv_prompt.json
# (Optional) GITHUB_BRANCH=main
# (Optional) REQUIRED_LABEL=production
```
Replace placeholders with your actual values. The server will commit prompts to `abv_prompt.json` on the `main` branch by default. If `REQUIRED_LABEL` is set, only prompts with that specific label will be synced to GitHub.
### GitHub PAT Permissions for Sync
For the webhook to work, your GitHub Personal Access Token needs **minimal permissions**:
| Permission Type | Required Permissions |
| -------------------- | ------------------------------------------------------------------------------------ |
| Required Permissions | Contents: Read and write, Metadata: Read-only |
| Legacy Token Scopes | For public repositories: `public_repo` scope, For private repositories: `repo` scope |
## Step 3: Implement the FastAPI Webhook Server
Create `main.py` with this FastAPI server:
```python
from typing import Any, Dict
from uuid import UUID
import json
import base64
import httpx
from pydantic import BaseModel, Field
from pydantic_settings import BaseSettings, SettingsConfigDict
from fastapi import FastAPI, HTTPException, Body
class GitHubSettings(BaseSettings):
"""GitHub repository configuration."""
GITHUB_TOKEN: str
GITHUB_REPO_OWNER: str
GITHUB_REPO_NAME: str
GITHUB_FILE_PATH: str = "abv_prompt.json"
GITHUB_BRANCH: str = "main"
REQUIRED_LABEL: str = "" # Optional: only sync prompts with this label
model_config = SettingsConfigDict(
env_file=".env",
env_file_encoding="utf-8",
case_sensitive=True
)
config = GitHubSettings()
class ABVEvent(BaseModel):
"""ABV webhook event structure."""
id: UUID = Field(description="Event identifier")
timestamp: str = Field(description="Event timestamp")
type: str = Field(description="Event type")
action: str = Field(description="Performed action")
prompt: Dict[str, Any] = Field(description="Prompt content")
async def sync(event: ABVEvent) -> Dict[str, Any]:
"""Synchronize prompt data to GitHub repository."""
# Check if prompt has required label (if specified)
if config.REQUIRED_LABEL:
prompt_labels = event.prompt.get("labels", [])
if config.REQUIRED_LABEL not in prompt_labels:
return {"skipped": f"Prompt does not have required label '{config.REQUIRED_LABEL}'"}
api_endpoint = f"https://api.github.com/repos/{config.GITHUB_REPO_OWNER}/{config.GITHUB_REPO_NAME}/contents/{config.GITHUB_FILE_PATH}"
request_headers = {
"Authorization": f"Bearer {config.GITHUB_TOKEN}",
"Accept": "application/vnd.github.v3+json"
}
content_json = json.dumps(event.prompt, indent=2)
encoded_content = base64.b64encode(content_json.encode("utf-8")).decode("utf-8")
name = event.prompt.get("name", "unnamed")
version = event.prompt.get("version", "unknown")
message = f"{event.action}: {name} v{version}"
payload = {
"message": message,
"content": encoded_content,
"branch": config.GITHUB_BRANCH
}
async with httpx.AsyncClient() as http_client:
try:
existing = await http_client.get(api_endpoint, headers=request_headers, params={"ref": config.GITHUB_BRANCH})
if existing.status_code == 200:
payload["sha"] = existing.json().get("sha")
except Exception:
pass
try:
response = await http_client.put(api_endpoint, headers=request_headers, json=payload)
response.raise_for_status()
return response.json()
except Exception as e:
raise HTTPException(status_code=500, detail=f"Repository sync failed: {str(e)}")
app = FastAPI(title="ABV GitHub Sync", version="1.0")
@app.post("/webhook/prompt", status_code=201)
async def receive_webhook(event: ABVEvent = Body(...)):
"""Process ABV webhook and sync to GitHub."""
result = await sync(event)
return {
"status": "synced",
"commit_info": result.get("commit", {}),
"file_info": result.get("content", {})
}
@app.get("/status")
async def health_status():
"""Service health check."""
return {"healthy": True}
```
The server validates webhook payloads, retrieves existing file SHAs if needed, and commits prompt changes to GitHub with descriptive commit messages.
### Dependencies
Install dependencies:
```bash
pip install fastapi uvicorn pydantic-settings httpx
```
### Running Locally
Run locally:
```bash
uvicorn main:app --reload --port 8000
```
Test the health endpoint at `http://localhost:8000/health`. Use ngrok or similar to expose localhost for webhook testing.
## Step 4: Deploy and Connect the Server
1. **Deploy:** Use Render, Fly.io, Heroku, or similar. Set environment variables and ensure HTTPS is enabled.
2. **Update Webhook:** In ABV, edit your webhook and set the URL to `https://your-domain.com/webhook/prompt`.
3. **Test:** Update a prompt in ABV and verify a new commit appears in your GitHub repository.
## Security Considerations
- **Verify signatures:** Use the signing secret and `x-abv-signature` header to validate requests
- **Limit PAT scope:** Use fine-grained tokens restricted to specific repositories
- **Handle retries:** The implementation is idempotent - duplicate events won't create conflicting commits
--------------------------------------------------------------------------------
## Webhooks & Slack Integration
URL: /developer/prompt-management/webhooks-slack-integration
# Why use webhooks?
- **Production Monitoring**: Get alerted when production prompts are updated
- **Team Coordination**: Keep everyone informed about prompt changes
- **Syncing**: Sync prompt catalogues with other systems
# Get started
Navigate to `Prompts` and click on `Webhooks`.
Click on `Create Webhook`.
Select events to watch.
Choose the prompt‑version actions that should fire the webhook:
- **Created:** a new version is added.
- **Updated:** labels or tags change (two events fire: one for the version that gains a label/tag, one for the version that loses it).
- **Deleted:** a version is removed.
5\. (Optional) filter to only trigger on specific prompts.
### **1) Configure the request**
- **URL**: HTTPS endpoint that accepts POST requests.
- **Headers**: Default headers include:
- `Content-Type: application/json`
- `User-Agent: ABV/1.0`
- `x-abv-signature: ` (see note on HMAC signature verification below)
- **Add custom static headers if required.**
### 2) Inspect the payload
Your endpoint receives a JSON body like:
webhook-payload.json
```json
{
"id": "550e8400-e29b-41d4-a716-446655440000",
"timestamp": "2024-07-10T10:30:00Z",
"type": "prompt-version",
"apiVersion": "v1",
"action": "created",
"prompt": {
"id": "prompt_abc123",
"name": "movie-critic",
"version": 3,
"projectId": "xyz789",
"labels": ["production", "latest"],
"prompt": "As a {{criticLevel}} movie critic, rate {{movie}} out of 10.",
"type": "text",
"config": { "key": "value" },
"commitMessage": "Improved critic persona",
"tags": ["entertainment"],
"createdAt": "2024-07-10T10:30:00Z",
"updatedAt": "2024-07-10T10:30:00Z"
}
}
```
### 3) Acknowledge delivery
Your handler must:
- Return an HTTP 2xx status to confirm receipt.
- Be idempotent—ABV may retry (exponential back‑off) until it receives a success response.
### 4) Verify authenticity (recommended)
Each request carries an HMAC SHA‑256 signature in `x-abv-signature`.
Retrieve the secret when you create the webhook (you can regenerate it later).
```python
import hmac
import hashlib
from typing import Optional
def verify_abv_signature(
raw_body: str,
signature_header: str,
secret: str,
) -> bool:
"""
Validate a ABV webhook/event signature.
Parameters
----------
raw_body : str
The request body exactly as received (no decoding or reformatting).
signature_header : str
The value of the `ABV-Signature` header, e.g. "t=1720701136,s=0123abcd...".
secret : str
Your ABV signing secret.
Returns
-------
bool
True if the signature is valid, otherwise False.
"""
# Split "t=timestamp,s=signature" into the two expected key/value chunks
try:
ts_pair, sig_pair = signature_header.split(",", 1)
except ValueError: # wrong format / missing comma
return False
# Extract values (everything after the first "=")
if "=" not in ts_pair or "=" not in sig_pair:
return False
timestamp = ts_pair.split("=", 1)[1]
received_sig_hex = sig_pair.split("=", 1)[1]
# Recreate the message and compute the expected HMAC-SHA256 hex digest
message = f"{timestamp}.{raw_body}".encode("utf-8")
expected_sig_hex = hmac.new(
secret.encode("utf-8"), message, hashlib.sha256
).hexdigest()
# Use constant-time comparison on the *decoded* byte strings
try:
return hmac.compare_digest(
bytes.fromhex(received_sig_hex), bytes.fromhex(expected_sig_hex)
)
except ValueError: # received_sig_hex isn't valid hex
return False
```
```typescript
import crypto from "crypto";
export function verifyABVSignature(
rawBody: string,
signatureHeader: string,
secret: string
): boolean {
const [tsPair, sigPair] = signatureHeader.split(",");
if (!tsPair || !sigPair) return false;
const timestamp = tsPair.split("=")[1];
const receivedSig = sigPair.split("=")[1];
const expectedSig = crypto
.createHmac("sha256", secret)
.update(`${timestamp}.${rawBody}`, "utf8")
.digest("hex");
return crypto.timingSafeEqual(
Buffer.from(receivedSig, "hex"),
Buffer.from(expectedSig, "hex")
);
}
```
### **Authenticate Slack with ABV**
- ABV connects to Slack via OAuth.
- We store secrets to Slack encrypted in our database.
### **Select channels to send notifications to**
- You can select a channel where you want to send notifications.
- You can run a dry run to see that messages arrive in your channel.
### **See the message in Slack**
--------------------------------------------------------------------------------
## Caching of Prompts in Client SDKs
URL: /developer/prompt-management/caching-prompts
### Cache Hit
When the SDK cache contains a fresh prompt, it's returned **immediately** without any network requests.
### Background Revalidation
When the cache TTL has expired, stale prompts are served **immediately** while it **revalidates in the background**.
### Cache Miss
When no cached prompt exists (e.g., first application startup), the prompt is fetched from the API. The API caches prompts in a Redis cache to ensure low latency.
Multiple fallback layers ensure **resilience** - if Redis is unavailable, the database serves as backup.
### Optional: Pre-fetch
Pre-fetching prompts during application startup ensures that the cache is populated before runtime requests.
This step is optional and often unnecessary. Typically, the minimal latency experienced during the first use after a service starts is acceptable. See examples below on how to set this up.
### Optional: Fallback
When both the local cache is empty and the ABV API is unavailable, a fallback prompt can be used to ensure 100% availability.
This is rarely necessary because the prompts API is highly available, and we closely monitor its performance ([status page](https://statuspage.incident.io/abvdev)). In the event of a brief service disruption, the SDK-level prompt cache typically ensures that applications remain unaffected.
# Optional: Customize caching duration (TTL)
The caching duration is configurable if you wish to reduce network overhead of the ABV Client. The default cache TTL is 60 seconds. After the TTL expires, the SDKs will refetch the prompt in the background and update the cache. Refetching is done asynchronously and does not block the application.
### Python SDK
Install package
```bash
pip install abvdev
```
```python
from abvdev import ABV
# Initialize
abv = ABV(
api_key="sk-abv-...",
host="https://app.abv.dev", # or host="https://eu.app.abv.dev", for EU
)
# Get current `production` prompt version and cache for 5 minutes
prompt = abv.get_prompt("movie-critic", cache_ttl_seconds=300)
```
### JS/TS SDK
```bash
npm i @abvdev/client
```
**Environment variables**
Add your ABV credentials as environment variables, e.g. use .env file and dotenv package to load variable values.
```bash
npm install dotenv
```
```bash title=".env"
ABV_API_KEY = "sk-abv-...";
ABV_BASEURL = "https://app.abv.dev"; US region
# ABV_BASEURL = "https://eu.app.abv.dev"; EU region
```
```typescript
import { ABVClient } from "@abvdev/client";
const abv = new ABVClient();
```
**alternatively use Constructor parameters**
```typescript
import { ABVClient } from "@abvdev/client";
const abv = new ABVClient({
apiKey: "sk-abv-...",
baseUrl: "https://app.abv.dev", // US region
// baseUrl: "https://eu.app.abv.dev", // EU region
});
```
Customize caching duration (TTL)
```typescript
import { ABVClient } from "@abvdev/client";
import dotenv from "dotenv";
dotenv.config();
const abv = new ABVClient();
async function main(){
// Get current `production` version and cache prompt for 5 minutes
const prompt = await abv.prompt.get("movie-critic", {
cacheTtlSeconds: 300,
});
}
main();
```
# Optional: Disable caching
You can disable caching by setting the `cacheTtlSeconds` to `0`. This will ensure that the prompt is fetched from the ABV API on every call. This is recommended for non-production use cases where you want to ensure that the prompt is always up to date with the latest version in ABV.
### Python SDK
```python
from abvdev import ABV
# Initialize
abv = ABV(
api_key="sk-abv-...",
host="https://app.abv.dev", # or host="https://eu.app.abv.dev", for EU
)
prompt = abv.get_prompt("movie-critic", cache_ttl_seconds=0)
# Common in non-production environments, no cache + latest version
prompt = abv.get_prompt("movie-critic", cache_ttl_seconds=0, label="latest")
```
### JS/TS SDK
```typescript
import { ABVClient } from "@abvdev/client";
import dotenv from "dotenv";
dotenv.config();
const abv = new ABVClient();
async function main(){
// Get current `production` version and cache prompt for 5 minutes
const prompt1 = await abv.prompt.get("movie-critic", {
cacheTtlSeconds: 0,
});
// Common in non-production environments, no cache + latest version
const prompt2 = await abv.prompt.get("movie-critic", {
cacheTtlSeconds: 0,
label: "latest",
});
}
main();
```
# Optional: Guaranteed availability of prompts
While usually not necessary, you can ensure 100% availability of prompts by pre-fetching them on application startup and providing a fallback prompt. Please follow this [guide](./guaranteed-availability) for more information.
# Performance measurement of inital fetch
We measured the execution time of the following snippet with fully disabled caching. You can run [this notebook](https://drive.google.com/file/d/1P2ltQySDW3NBCA2c3Rwv2IP40lMoDKbT/view?usp=sharing) yourself to verify the results.
```python
prompt = abv.get_prompt("perf-test", cache_ttl_seconds=0)
prompt.compile(input="test")
```
Results from 1000 sequential executions using ABV (includes network latency):
--------------------------------------------------------------------------------
## Guaranteed Availability
URL: /developer/prompt-management/guaranteed-availability
Implementing this is usually not necessary as it adds complexity to your application. The ABV Prompt Management is highly available due to multiple [caching layers](/developer/prompt-management/caching-prompts) and we closely monitor its performance ([status page](https://status.abv.dev)). However, if you require 100% availability, you can use the following options.
The ABV API has high uptime and prompts are [cached locally](/developer/prompt-management/caching-prompts) in the SDKs to prevent network issues from affecting your application.
However, `get_prompt()`/`getPrompt()` will throw an exception if:
- No local (fresh or stale) cached prompt is available -> new application instance fetching prompt for the first time
- *and* network request fails -> networking or ABV API issue (after retries)
To guarantee 100% availability, there are two options:
1. Pre-fetch prompts on application startup and exit the application if the prompt is not available.
2. Provide a `fallback` prompt that will be used in these cases.
# Option 1: Pre-fetch prompts
Pre-fetch prompts on application startup and exit the application if the prompt is not available.
### Python (Flask)
Install packages
```bash
pip install abvdev, flask
```
```python
from flask import Flask, jsonify
from abvdev import ABV
import sys
# Initialize the Flask app and ABV client
app = Flask(__name__)
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
def fetch_prompts_on_startup():
try:
# Fetch and cache the production version of the prompt
abv.get_prompt("movie-critic")
except Exception as e:
print(f"Failed to fetch prompt on startup: {e}")
sys.exit(1) # Exit the application if the prompt is not available
# Call the function during application startup
fetch_prompts_on_startup()
@app.route('/get-movie-prompt/', methods=['GET'])
def get_movie_prompt(movie):
prompt = abv.get_prompt("movie-critic")
compiled_prompt = prompt.compile(criticlevel="expert", movie=movie)
return jsonify({"prompt": compiled_prompt})
if __name__ == '__main__':
app.run(debug=True)
```
### JS/TS (Express)
```bash
npm i @abvdev/client express
```
**Environment variables**
Add your ABV credentials as environment variables, e.g. use .env file and dotenv package to load variable values.
```bash
npm install dotenv
```
```bash title=".env"
ABV_API_KEY = "sk-abv-...";
ABV_BASEURL = "https://app.abv.dev"; US region
# ABV_BASEURL = "https://eu.app.abv.dev"; EU region
```
```typescript
import { ABVClient } from "@abvdev/client";
const abv = new ABVClient();
```
**alternatively use Constructor parameters**
```typescript
import { ABVClient } from "@abvdev/client";
const abv = new ABVClient({
apiKey: "sk-abv-...",
baseUrl: "https://app.abv.dev", // US region
// baseUrl: "https://eu.app.abv.dev", // EU region
});
```
**application**
```typescript
import express from "express";
import { ABVClient } from "@abvdev/client";
import dotenv from "dotenv";
dotenv.config();
// Initialize the Express app and ABV client
const app = express();
const abv = new ABVClient();
async function fetchPromptsOnStartup() {
try {
// Fetch and cache the production version of the prompt
await abv.prompt.get("movie-critic");
} catch (error) {
console.error("Failed to fetch prompt on startup:", error);
process.exit(1); // Exit the application if the prompt is not available
}
}
// Call the function during application startup
fetchPromptsOnStartup();
app.get("/get-movie-prompt/:movie", async (req, res) => {
const movie = req.params.movie;
const prompt = await abv.prompt.get("movie-critic");
const compiledPrompt = prompt.compile({ criticlevel: "expert", movie });
res.json({ prompt: compiledPrompt });
});
app.listen(3000, () => {
console.log("Server is running on port 3000");
});
```
**Run your application**
Execute your application. You should see prompts appeared / updated in the ABV UI.
```bash
npx tsx filename.ts
```
**Check the application by the link**
****[http://localhost:3000/get-movie-prompt/The-Lord-of-the-Rings](http://localhost:3000/get-movie-prompt/The-Lord-of-the-Rings)
# Option 2: Fallback
Provide a fallback prompt that will be used in these cases:
### Python SDK
```python
from abvdev import ABV
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# Get `text` prompt with fallback
prompt = abv.get_prompt(
"movie-critic",
fallback="Do you like {{movie}}?"
)
# Get `chat` prompt with fallback
chat_prompt = abv.get_prompt(
"movie-critic-chat",
type="chat",
fallback=[{"role": "system", "content": "You are an expert on {{movie}}"}]
)
# True if the prompt is a fallback
print(prompt.is_fallback)
```
### JS/TS SDK
```typescript
import { ABVClient } from "@abvdev/client";
import dotenv from "dotenv";
dotenv.config();
const abv = new ABVClient();
async function main() {
// Get `text` prompt with fallback
const prompt = await abv.prompt.get("movie-critic", {
fallback: "Do you like {{movie}}?",
});
console.log(prompt);
// Get `chat` prompt with fallback
const chatPrompt = await abv.prompt.get("movie-critic-chat", {
type: "chat",
fallback: [{ role: "system", content: "You are an expert on {{movie}}" }],
});
console.log(chatPrompt);
// True if the prompt is a fallback
console.log(prompt.isFallback);
}
main();
```
--------------------------------------------------------------------------------
## Prompt Troubleshooting and FAQ
URL: /developer/prompt-management/troubleshooting-faq
# FAQ
ABV prompts are cached client-side by default, so network-related issues are minimized after the first fetch. However, you can configure network behavior:
**Caching Configuration:**
The default cache TTL is 60 seconds. You can customize this:
**Python SDK:**
```python
from abvdev import ABV
abv = ABV(api_key="sk-abv-...", host="https://app.abv.dev")
# Increase cache duration to reduce network calls
prompt = abv.get_prompt("my-prompt", cache_ttl_seconds=300) # 5 minutes
# Disable caching for development
prompt = abv.get_prompt("my-prompt", cache_ttl_seconds=0)
```
**JS/TS SDK:**
```typescript
import { ABVClient } from "@abvdev/client";
const abv = new ABVClient();
// Increase cache duration
const prompt = await abv.prompt.get("my-prompt", {
cacheTtlSeconds: 300 // 5 minutes
});
// Disable caching for development
const prompt = await abv.prompt.get("my-prompt", {
cacheTtlSeconds: 0
});
```
**Guaranteed Availability:**
For critical applications requiring 100% availability:
1. **Pre-fetch prompts on startup** to populate the cache
2. **Provide fallback prompts** for when the API is unavailable
**Python example:**
```python
from abvdev import ABV
abv = ABV(api_key="sk-abv-...", host="https://app.abv.dev")
# Pre-fetch during application startup
startup_prompts = ["critical-prompt-1", "critical-prompt-2"]
for prompt_name in startup_prompts:
abv.get_prompt(prompt_name)
# Use with fallback
try:
prompt = abv.get_prompt("my-prompt")
except Exception as e:
# Use fallback prompt
prompt = "As a helpful assistant, {{input}}"
```
**How caching works:**
- **Cache hit**: Prompt returned immediately (no network call)
- **Stale cache**: Old prompt returned immediately while revalidating in background
- **Cache miss**: Prompt fetched from API (Redis cache on ABV side for low latency)
ABV provides comprehensive metrics when you link prompts to traces. This enables tracking performance by prompt version.
**Step 1: Link prompts to generations**
**Python SDK:**
```python
from abvdev import ABV, observe
abv = ABV(api_key="sk-abv-...", host="https://app.abv.dev")
# With decorator
@observe(as_type="generation")
def call_llm():
prompt = abv.get_prompt("my-prompt")
abv.update_current_generation(
prompt=prompt,
model="gpt-4o"
)
# Your LLM call here
response = openai_client.chat.completions.create(...)
abv.update_current_generation(
output=response.choices[0].message.content,
usage_details={
"input": response.usage.input_tokens,
"output": response.usage.output_tokens,
}
)
# With context manager
prompt = abv.get_prompt("my-prompt")
with abv.start_as_current_observation(
as_type='generation',
name="llm-call",
model="gpt-4o",
prompt=prompt
) as generation:
# Your LLM call
generation.update(output="response")
```
**JS/TS SDK:**
```typescript
import { ABVClient } from "@abvdev/client";
import { startObservation } from "@abvdev/tracing";
const abv = new ABVClient();
const prompt = await abv.prompt.get("my-prompt");
const generation = startObservation(
"llm-call",
{
model: "gpt-4o",
input: prompt.prompt,
prompt: prompt // Link the prompt
},
{ asType: "generation" }
);
// ... LLM call ...
generation.update({
output: "response",
usageDetails: { /* ... */ }
}).end();
```
**Step 2: View metrics in ABV UI**
Navigate to your prompt in the ABV UI and click the **Metrics** tab to see:
**Available Metrics:**
- Median generation latency
- Median input tokens
- Median output tokens
- Median generation costs
- Generation count
- Median score values (from evaluations)
- First and last generation timestamps
**Compare versions:** Use the UI to compare metrics across different prompt versions or A/B test variants.
**Custom metrics:** Add custom scores via the Scores API to track domain-specific metrics (accuracy, relevance, user satisfaction, etc.).
Linking prompts to traces enables you to track which prompt version was used for each LLM call and analyze performance by prompt.
**Python SDK:**
```python
from abvdev import ABV, observe
from openai import OpenAI
abv = ABV(api_key="sk-abv-...", host="https://app.abv.dev")
openai_client = OpenAI(api_key="sk-proj-...")
# Method 1: With @observe decorator
@observe(as_type="generation")
def generate_response():
prompt = abv.get_prompt("movie-critic")
# Link prompt to current generation
abv.update_current_generation(prompt=prompt)
compiled = prompt.compile(criticlevel="expert", movie="Dune 2")
response = openai_client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": compiled}]
)
return response.choices[0].message.content
# Method 2: With context manager
prompt = abv.get_prompt("movie-critic")
with abv.start_as_current_observation(
as_type='generation',
name="movie-generation",
model="gpt-4o",
prompt=prompt # Link here
) as generation:
compiled = prompt.compile(criticlevel="expert", movie="Dune 2")
response = openai_client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": compiled}]
)
generation.update(output=response.choices[0].message.content)
```
**JS/TS SDK:**
```typescript
import "./instrumentation"; // Must be first
import { ABVClient } from "@abvdev/client";
import { startObservation, observe, updateActiveObservation } from "@abvdev/tracing";
const abv = new ABVClient();
// Method 1: Manual observation
async function example1() {
const prompt = await abv.prompt.get("movie-critic");
const generation = startObservation(
"llm-call",
{
model: "gpt-4o",
input: prompt.prompt,
prompt: prompt // Link here
},
{ asType: "generation" }
);
// ... LLM call ...
generation.end();
}
// Method 2: With observe wrapper
const callLLM = async (input: string) => {
const prompt = await abv.prompt.get("my-prompt");
updateActiveObservation(
{ prompt },
{ asType: "generation" }
);
return await invokeLLM(input);
};
export const observedCallLLM = observe(callLLM);
```
**Benefits of linking:**
- See which prompt version was used in each trace
- Filter traces by prompt name/version
- Track metrics aggregated by prompt version
- Compare performance between prompt versions
- Identify which prompts lead to better outcomes
**Note:** If a fallback prompt is used (when API is unavailable), no link will be created.
ABV provides comprehensive prompt management through the UI, SDKs, and API.
**Creating Prompts:**
**Via UI:**
1. Sign in to ABV
2. Navigate to Prompts section
3. Click "Create Prompt"
4. Enter prompt content with {{variables}}
5. Add configuration (model, temperature, etc.)
6. Assign labels for deployment
**Via Python SDK:**
```python
from abvdev import ABV
abv = ABV(api_key="sk-abv-...", host="https://app.abv.dev")
# Create text prompt
abv.create_prompt(
name="movie-critic",
type="text",
prompt="As a {{criticlevel}} movie critic, do you like {{movie}}?",
labels=["production"],
config={
"model": "gpt-4o",
"temperature": 0.7,
"max_tokens": 1000
}
)
# Create chat prompt
abv.create_prompt(
name="chat-assistant",
type="chat",
prompt=[
{"role": "system", "content": "You are a {{persona}} assistant"},
{"role": "user", "content": "{{user_query}}"}
],
labels=["production"]
)
```
**Via JS/TS SDK:**
```typescript
import { ABVClient } from "@abvdev/client";
const abv = new ABVClient();
await abv.prompt.create({
name: "movie-critic",
type: "text",
prompt: "As a {{criticlevel}} critic, do you like {{movie}}?",
labels: ["production"],
config: {
model: "gpt-4o",
temperature: 0.7
}
});
```
**Fetching Prompts:**
```python
# Python
prompt = abv.get_prompt("movie-critic") # Gets production version
prompt = abv.get_prompt("movie-critic", version=1) # Specific version
prompt = abv.get_prompt("movie-critic", label="staging") # Specific label
prompt = abv.get_prompt("movie-critic", label="latest") # Latest version
# Compile with variables
compiled = prompt.compile(criticlevel="expert", movie="Dune 2")
```
```typescript
// JS/TS
const prompt = await abv.prompt.get("movie-critic");
const prompt2 = await abv.prompt.get("movie-critic", { version: 1 });
const prompt3 = await abv.prompt.get("movie-critic", { label: "staging" });
// Compile with variables
const compiled = prompt.compile({ criticlevel: "expert", movie: "Dune 2" });
```
**Updating Labels:**
```python
# Python
abv.update_prompt(
name="movie-critic",
version=2,
new_labels=["production", "experiment-a"]
)
```
```typescript
// JS/TS
await abv.prompt.update({
name: "movie-critic",
version: 2,
newLabels: ["production", "experiment-a"]
});
```
**Key Features:**
- Version control with automatic versioning
- Labels for deployment management
- Config versioning alongside prompts
- Diff view to see changes between versions
- Protected labels for production safety
- Rollback capability
- Variables with {{mustache}} syntax
Prompt engineering is the practice of designing and optimizing text prompts to get better outputs from Large Language Models (LLMs).
**Why it matters:**
- Better prompt = better LLM output quality
- Can significantly impact accuracy, relevance, and usefulness
- More cost-effective than fine-tuning models
- Faster iteration cycle than model training
**Key Techniques:**
1. **Clear Instructions:**
- Be specific about what you want
- Provide context and constraints
- Define the output format
```
Bad: "Write about dogs"
Good: "Write a 3-paragraph informative article about Golden Retrievers,
including their history, temperament, and care requirements.
Use a friendly tone suitable for first-time dog owners."
```
2. **Few-Shot Examples:**
- Show the model examples of desired output
- Helps establish patterns and format
```
Classify sentiment:
Text: "I love this product!"
Sentiment: Positive
Text: "This is terrible"
Sentiment: Negative
Text: "{{user_input}}"
Sentiment:
```
3. **Role/Persona:**
- Define who the AI should act as
- Influences tone and expertise level
```
You are an expert software architect with 15 years of experience
in distributed systems. Analyze the following system design...
```
4. **Chain of Thought:**
- Ask the model to think step-by-step
- Improves reasoning and accuracy
```
Solve this problem step by step, showing your reasoning:
{{problem}}
```
5. **Constraints and Format:**
- Specify output format (JSON, markdown, etc.)
- Set length limits
- Define what to avoid
```
Respond in JSON format with keys: "summary", "key_points", "confidence_score".
Keep summary under 100 words.
```
**ABV's Role in Prompt Engineering:**
ABV helps you iterate on prompts systematically:
- **Version control** to track changes
- **A/B testing** to compare variants
- **Metrics tracking** to measure improvements
- **Tracing** to see prompts in context
- **Team collaboration** via UI
- **Quick rollbacks** when changes don't work
**Best Practices:**
1. Start simple, then iterate
2. Test with diverse inputs
3. Measure performance metrics
4. Use version control
5. A/B test significant changes
6. Document what works and why
7. Keep prompts maintainable and readable
ABV provides built-in version control for all prompts with automatic versioning and label-based deployment.
**Automatic Versioning:**
Every time you create or update a prompt, ABV automatically assigns a version number:
```python
# Python
from abvdev import ABV
abv = ABV(api_key="sk-abv-...", host="https://app.abv.dev")
# First creation - becomes version 1
abv.create_prompt(
name="movie-critic",
prompt="Do you like {{movie}}?",
labels=["production"]
)
# Update - becomes version 2
abv.create_prompt(
name="movie-critic",
prompt="As a critic, do you like {{movie}}?",
labels=["staging"]
)
```
**Labels for Deployment:**
Use labels to manage which version is deployed where:
```python
# Assign labels to versions
abv.update_prompt(
name="movie-critic",
version=1,
new_labels=["production"]
)
abv.update_prompt(
name="movie-critic",
version=2,
new_labels=["staging", "experiment-a"]
)
```
**Fetching Specific Versions:**
```python
# Python
# Get production version (default)
prod_prompt = abv.get_prompt("movie-critic")
prod_prompt = abv.get_prompt("movie-critic", label="production")
# Get staging version
staging_prompt = abv.get_prompt("movie-critic", label="staging")
# Get specific version number
v1_prompt = abv.get_prompt("movie-critic", version=1)
# Get latest version
latest_prompt = abv.get_prompt("movie-critic", label="latest")
```
```typescript
// JS/TS
const prodPrompt = await abv.prompt.get("movie-critic");
const stagingPrompt = await abv.prompt.get("movie-critic", { label: "staging" });
const v1Prompt = await abv.prompt.get("movie-critic", { version: 1 });
const latestPrompt = await abv.prompt.get("movie-critic", { label: "latest" });
```
**Version Comparison:**
The ABV UI provides a diff view to compare prompt versions:
- See exactly what changed between versions
- Track who made changes and when
- Review config changes alongside prompt changes
**Rollback:**
To rollback to a previous version, simply reassign the `production` label:
```python
# Rollback: make version 1 the production version again
abv.update_prompt(
name="movie-critic",
version=1,
new_labels=["production"]
)
```
Or do it in the UI with one click.
**Protected Labels:**
For additional safety, admins can mark labels as "protected":
- Only admins/owners can modify protected labels
- Prevents accidental changes to production prompts
- Configure in project settings
**Best Practices:**
1. Always use `production` label for deployed versions
2. Use `staging` for testing before production
3. Use descriptive labels for experiments (e.g., `experiment-longer-context`)
4. The `latest` label is automatically maintained by ABV
5. Never delete old versions - keep history for debugging
6. Use commit messages to document why changes were made
7. Review diffs before promoting to production
ABV enables A/B testing by using labels to identify different prompt variants, then randomly selecting between them in your application.
**Step 1: Create Prompt Variants**
Create multiple versions and label them for your test:
```python
from abvdev import ABV
abv = ABV(api_key="sk-abv-...", host="https://app.abv.dev")
# Variant A - shorter prompt
abv.create_prompt(
name="movie-critic",
prompt="Do you like {{movie}}?",
labels=["prod-a"]
)
# Variant B - more detailed prompt
abv.create_prompt(
name="movie-critic",
prompt="As an expert film critic, provide your opinion on {{movie}}. Include analysis of the plot, acting, and cinematography.",
labels=["prod-b"]
)
```
**Step 2: Implement Random Selection**
**Python SDK:**
```python
from abvdev import ABV
from openai import OpenAI
import random
abv = ABV(api_key="sk-abv-...", host="https://app.abv.dev")
openai_client = OpenAI(api_key="sk-proj-...")
# Fetch both variants
prompt_a = abv.get_prompt("movie-critic", label="prod-a")
prompt_b = abv.get_prompt("movie-critic", label="prod-b")
# Randomly select
selected_prompt = random.choice([prompt_a, prompt_b])
# Use in LLM call with tracing
with abv.start_as_current_observation(
as_type="generation",
name="movie-review",
model="gpt-4o",
prompt=selected_prompt # Links to specific variant
) as generation:
compiled = selected_prompt.compile(movie="Dune 2")
response = openai_client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": compiled}]
)
generation.update(
output=response.choices[0].message.content,
usage_details={
"input": response.usage.input_tokens,
"output": response.usage.output_tokens
}
)
abv.flush()
```
**JS/TS SDK:**
```typescript
import { ABVClient } from "@abvdev/client";
import { startObservation } from "@abvdev/tracing";
import OpenAI from "openai";
const abv = new ABVClient();
const openai = new OpenAI();
async function runABTest() {
// Fetch both variants
const promptA = await abv.prompt.get("movie-critic", { label: "prod-a" });
const promptB = await abv.prompt.get("movie-critic", { label: "prod-b" });
// Randomly select (50/50 split)
const selectedPrompt = Math.random() < 0.5 ? promptA : promptB;
// Use in LLM call
const generation = startObservation(
"movie-review",
{
model: "gpt-4o",
prompt: selectedPrompt // Links to specific variant
},
{ asType: "generation" }
);
const compiled = selectedPrompt.compile({ movie: "Dune 2" });
const response = await openai.chat.completions.create({
model: "gpt-4o",
messages: [{ role: "user", content: compiled }]
});
generation.update({
output: response.choices[0].message.content,
usageDetails: {
prompt_tokens: response.usage.prompt_tokens,
completion_tokens: response.usage.completion_tokens
}
});
generation.end();
}
```
**Step 3: Analyze Results**
Navigate to your prompt in the ABV UI and view the Metrics tab:
**Compare Metrics:**
- Response latency by variant
- Token usage (input/output)
- Cost per request
- Quality scores (if you're scoring responses)
- Volume/distribution between variants
**Statistical Significance:**
- Run tests long enough to gather sufficient data
- Typical minimum: 100-1000 requests per variant
- Consider using staged rollout (90/10 split initially)
**Advanced: Weighted Distribution**
```python
import random
# 90% variant A, 10% variant B (canary deployment)
selected_prompt = prompt_a if random.random() < 0.9 else prompt_b
```
**Best Practices:**
1. Start with canary deployment (90/10 or 95/5)
2. Monitor error rates and user feedback
3. Use A/B testing for significant changes
4. Run tests long enough for statistical significance
5. Consider user segments (test on subset first)
6. Have rollback plan ready
7. Track multiple metrics (not just one)
8. Document test hypotheses and results
**When to Use A/B Testing:**
- Testing prompt improvements in production
- Validating changes before full rollout
- When evaluation datasets don't capture real usage
- For consumer apps where some variation is acceptable
- After thorough testing on evaluation datasets
ABV prompts are automatically cached client-side in the SDKs with intelligent background revalidation, ensuring minimal latency impact.
**How Caching Works:**
1. **Cache Hit** - Prompt in cache and fresh → returned immediately (0ms network)
2. **Stale Cache** - Prompt in cache but expired → returned immediately, revalidated in background
3. **Cache Miss** - First request → fetched from API (low latency, Redis-backed)
**Default Behavior:**
```python
# Python - Default 60-second cache
from abvdev import ABV
abv = ABV(api_key="sk-abv-...", host="https://app.abv.dev")
# First call - fetches from API and caches
prompt = abv.get_prompt("my-prompt")
# Subsequent calls within 60s - instant from cache
prompt = abv.get_prompt("my-prompt") # No network call
```
**Custom Cache Duration:**
**Python SDK:**
```python
# Cache for 5 minutes
prompt = abv.get_prompt("my-prompt", cache_ttl_seconds=300)
# Cache for 1 hour (for stable production prompts)
prompt = abv.get_prompt("my-prompt", cache_ttl_seconds=3600)
# Disable caching (development/testing)
prompt = abv.get_prompt("my-prompt", cache_ttl_seconds=0)
# Common pattern: no cache + latest version in development
prompt = abv.get_prompt(
"my-prompt",
cache_ttl_seconds=0,
label="latest"
)
```
**JS/TS SDK:**
```typescript
import { ABVClient } from "@abvdev/client";
const abv = new ABVClient();
// Cache for 5 minutes
const prompt1 = await abv.prompt.get("my-prompt", {
cacheTtlSeconds: 300
});
// Cache for 1 hour
const prompt2 = await abv.prompt.get("my-prompt", {
cacheTtlSeconds: 3600
});
// Disable caching
const prompt3 = await abv.prompt.get("my-prompt", {
cacheTtlSeconds: 0
});
// Development pattern
const devPrompt = await abv.prompt.get("my-prompt", {
cacheTtlSeconds: 0,
label: "latest"
});
```
**Pre-fetching for Zero Latency:**
Load prompts during application startup:
```python
# Python
from abvdev import ABV
abv = ABV(api_key="sk-abv-...", host="https://app.abv.dev")
# Pre-fetch during startup
critical_prompts = [
"user-greeting",
"error-handler",
"main-assistant"
]
for prompt_name in critical_prompts:
abv.get_prompt(prompt_name) # Populates cache
# Now runtime requests are instant
```
```typescript
// JS/TS
const abv = new ABVClient();
// Pre-fetch during startup
const criticalPrompts = [
"user-greeting",
"error-handler",
"main-assistant"
];
await Promise.all(
criticalPrompts.map(name => abv.prompt.get(name))
);
// Now runtime requests are instant
```
**Fallback for 100% Availability:**
```python
# Python
from abvdev import ABV
abv = ABV(api_key="sk-abv-...", host="https://app.abv.dev")
def get_prompt_with_fallback(name: str, fallback: str):
try:
return abv.get_prompt(name)
except Exception as e:
# Log error
print(f"Failed to fetch prompt: {e}")
# Return fallback
return fallback
# Usage
prompt = get_prompt_with_fallback(
"my-prompt",
fallback="You are a helpful assistant. {{user_input}}"
)
```
**Performance:**
From ABV's testing (1000 sequential requests with caching disabled):
- Median latency: ~50ms
- 95th percentile: ~100ms
- 99th percentile: ~150ms
With caching enabled (default):
- Cached requests: 0ms (instant)
- Stale-while-revalidate: 0ms (instant, updated in background)
**Best Practices:**
1. **Production**: Use default 60s cache or longer (5-10 minutes)
2. **Development**: Disable cache to see changes immediately
3. **Critical paths**: Pre-fetch prompts on startup
4. **High availability**: Implement fallback prompts
5. **Staging**: Use moderate cache (30-60s) for balance
6. **Monitor**: ABV status page for API availability
**When to Adjust Cache:**
- Increase TTL: Stable production prompts, reduce API calls
- Decrease TTL: Frequently updated prompts
- Disable: Local development, testing changes
- Pre-fetch: Startup-critical prompts, serverless cold starts
--------------------------------------------------------------------------------
## Overview of ABV SDKs
URL: /developer/sdks/overview
The ABV SDKs are the recommended way to integrate with ABV.
# Python SDK
Based on OpenTelemetry, so you can use any OTEL-based instrumentation library for your LLM:
```bash
pip install abvdev
```
Please see our [Python SDK documentation](/developer/sdks/python/overview) on how to get started.
# JS/TS SDK
The ABV JS/TS SDK is designed to be modular. Here’s an overview of the available packages:
```bash
npm install @abvdev/tracing @abvdev/otel
```
| Package | Description | Environment |
| --------------- | -------------------------------------------------------------------- | ------------ |
| @abvdev/core | Core utilities, types, and logger shared across packages. | Universal JS |
| @abvdev/client | Client for features like prompts, datasets, and scores. | Universal JS |
| @abvdev/tracing | Core OpenTelemetry-based tracing functions (startObservation, etc.). | Universal JS |
| @abvdev/otel | The ABVSpanProcessor to export traces to ABV. | Node.js ≥ 20 |
Please see our [TypeScript SDK documentation](/developer/sdks/js-ts/overview) on how to get started.
--------------------------------------------------------------------------------
## TypeScript SDK - Overview
URL: /developer/sdks/js-ts/overview
The modular ABV TypeScript SDK is built on [**OpenTelemetry**](https://opentelemetry.io/) for robust observability, better context management, and easy integration with third-party libraries.
Official Link: [https://www.npmjs.com/package/@abvdev/core](https://www.npmjs.com/package/@abvdev/core)
# Quickstart
Get your first trace into ABV in just a few minutes.
## 1) Install packages
Install the relevant packages to get started with tracing:
```bash
npm install @abvdev/tracing @abvdev/otel @opentelemetry/sdk-node
```
Learn more about the packages [here](./#packages).
## 2) Set up environment variables
Add your ABV credentials to your environment variables. Make sure that you have a `.env` file in your project root and a package like `dotenv` to load the variables.
```bash title=".env"
ABV_API_KEY = "sk-abv-..."
ABV_BASE_URL = "https://app.abv.dev" # US region
# ABV_BASE_URL = "https://eu.app.abv.dev" # EU region
```
## 3) Set up OpenTelemetry
Create a file named `instrumentation.ts` to initialize the OpenTelemetry SDK. The `ABVSpanProcessor` is the key component that sends traces to ABV.
```typescript title="instrumentation.ts"
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
const sdk = new NodeSDK({
spanProcessors: [new ABVSpanProcessor()],
});
sdk.start();
```
Modify `instrumentation.ts` file to use `dotenv` package to load the variables.
Additional parameters are provided to get trace visible in the UI immediately.
```bash
npm install dotenv
```
```typescript title="instrumentation.ts"
import dotenv from "dotenv";
dotenv.config();
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
const sdk = new NodeSDK({
spanProcessors: [
new ABVSpanProcessor({
apiKey: process.env.ABV_API_KEY,
baseUrl: process.env.ABV_BASE_URL,
exportMode: "immediate",
flushAt: 1,
flushInterval: 1,
additionalHeaders: {
"Content-Type": "application/json",
"Accept": "application/json"
}
})
],
});
sdk.start();
```
Import the `instrumentation.ts` file at the very top of your application's entry point (e.g., `index.ts`).
```typescript title="index.ts"
import "./instrumentation"; // Must be the first import
```
Learn more about setting up OpenTelemetry [here](/developer/sdks/js-ts/setup).
## 4) Instrument your application
Use one of the native ABV framework [integrations](/developer/sdks/js-ts/instrumentation) to automatically trace your application.
Alternatively, manually instrument your application, e.g. by using the `startActiveObservation`. This function takes a callback and automatically manages the observation's lifecycle and the OpenTelemetry context. Any observation created inside the callback will automatically be nested under the active observation, and the observation will be ended when the callback finishes.
This is just an example, check out the [instrumentation](/developer/sdks/js-ts/instrumentation) page for more details.
```typescript title="index.ts"
import "./instrumentation";
import { startActiveObservation } from "@abvdev/tracing";
async function main() {
await startActiveObservation("my-first-trace", async (span) => {
span.update({
input: "Hello, ABV!",
output: "This is my first trace!",
});
});
}
main();
```
## 5) Run your application
Execute your application. You should see your trace appear in the ABV UI.
```bash
npx tsx index.ts
```
# Packages
The ABV SDK is designed to be modular. Here's an overview of the available packages:
| Package | Description | Environment |
| ----------------- | ---------------------------------------------------------------------- | ------------ |
| `@abvdev/core` | Core utilities, types, and logger shared across packages. | Universal JS |
| `@abvdev/client` | Client for features like prompts, datasets, and scores. | Universal JS |
| `@abvdev/tracing` | Core OpenTelemetry-based tracing functions (`startObservation`, etc.). | Universal JS |
| `@abvdev/otel` | The `ABVSpanProcessor` to export traces to ABV. | Node.js ≥ 20 |
# OpenTelemetry foundation
Building on OpenTelemetry is a core design choice for this SDK. It offers several key advantages:
- **Standardization**: It aligns with the industry standard for observability, making it easier to integrate with existing monitoring and APM tools.
- **Robust Context Management**: OpenTelemetry provides reliable [context propagation](https://opentelemetry.io/docs/concepts/context-propagation/), ensuring that traces are correctly linked even in complex, asynchronous applications.
- **Ecosystem & Interoperability**: You can leverage a vast ecosystem of third-party instrumentations. If a library you use supports OpenTelemetry, its traces can be sent to ABV automatically.
# Learn more
[Setup](/developer/sdks/js-ts/setup)
[Instrumentation](/developer/sdks/js-ts/instrumentation)
[Advanced usage](/developer/sdks/js-ts/advanced-configuration)
[Troubleshooting & FAQ](/developer/sdks/js-ts/troubleshooting-faq)
## Key Features Supported
The JS/TS SDK provides full support for all ABV features:
- [Observability & Tracing](/developer/basic-features/observability-tracing): Comprehensive tracing of LLM applications
- [Sessions](/developer/basic-features/sessions): Group related traces by user journey
- [User Tracking](/developer/basic-features/user-tracking): Associate traces with specific users
- [Metadata](/developer/basic-features/metadata): Add structured context to traces
- [Tags](/developer/basic-features/tags): Label and categorize traces
- [Model Usage & Cost Tracking](/developer/basic-features/model-usage-cost-tracking): Monitor tokens and costs
- [Masking Sensitive Data](/developer/basic-features/masking-sensitive-data): Redact PII and secrets
- [Sampling](/developer/basic-features/sampling): Control trace volume
- [Prompt Management](/developer/prompt-management/get-started): Fetch and link prompts to traces
- [Evaluations](/developer/evaluations/overview): Create datasets and custom scores
- [Guardrails](/developer/guardrails/overview): Implement LLM security guardrails
--------------------------------------------------------------------------------
## TypeScript SDK - Setup
URL: /developer/sdks/js-ts/setup
The ABV TypeScript SDK offers two setup approaches:
1. [Tracing](./#tracing-setup) for [ABV Observability](/developer/basic-features/observability-tracing) using OpenTelemetry
2. [Client](./#client-setup) for other ABV features like prompt management, evaluation, or accessing the ABV API
# Tracing Setup
## 1) Installation
Install the relevant packages for a full tracing setup:
```bash
npm install @abvdev/tracing @abvdev/otel @opentelemetry/sdk-node
```
- `@abvdev/tracing`: Core tracing functions (`startObservation`, `startActiveObservation`, etc.)
- `@abvdev/otel`: The `ABVSpanProcessor` to export traces to ABV.
- `@opentelemetry/sdk-node`: The OpenTelemetry SDK for Node.js.
Learn more about the packages [here](/developer/sdks/js-ts/overview).
## 2) Register your credentials
Add your ABV credentials to your environment variables. Make sure that you have a `.env` file in your project root and a package like `dotenv` to load the variables.
```bash
ABV_API_KEY = "sk-abv-..."
ABV_BASE_URL = "https://app.abv.dev" # US region
# ABV_BASE_URL = "https://eu.app.abv.dev" # EU region
```
## 3) Initialize OpenTelemetry
The ABV TypeScript SDK's tracing is built on top of OpenTelemetry, so you need to set up the OpenTelemetry SDK. The `ABVSpanProcessor` is the key component that sends traces to ABV.
```typescript title="instrumentation.ts"
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
const sdk = new NodeSDK({
spanProcessors: [new ABVSpanProcessor()],
});
sdk.start();
```
The `ABVSpanProcessor` is the key component that sends traces to ABV.
Modify `instrumentation.ts` file to use `dotenv` package to load the variables.
Additional parameters are provided to get trace visible in the UI immediately.
```bash
npm install dotenv
```
```typescript title="instrumentation.ts"
import dotenv from "dotenv";
dotenv.config();
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
const sdk = new NodeSDK({
spanProcessors: [
new ABVSpanProcessor({
apiKey: process.env.ABV_API_KEY,
baseUrl: process.env.ABV_BASE_URL,
exportMode: "immediate",
flushAt: 1,
flushInterval: 1,
additionalHeaders: {
"Content-Type": "application/json",
"Accept": "application/json"
}
})
],
});
sdk.start();
```
Import the `instrumentation.ts` file at the top of your application.
```typescript title="index.ts"
import "./instrumentation"; // Must be the first import
```
For more options to configure the ABVSpanProcessor such as masking, filtering, and more, see [the advanced usage](/developer/sdks/js-ts/advanced-configuration).
You can learn more about setting up OpenTelemetry in your JS environment [here](https://opentelemetry.io/docs/languages/js/getting-started/nodejs/).
If you are using Next.js, please use the OpenTelemetry setup via the `NodeSDK` described above rather than via `registerOTel` from `@vercel/otel`. This is because [the `@vercel/otel` package does not yet support the OpenTelemetry JS SDK v2](https://github.com/vercel/otel/issues/154) on which the `@abvdev/tracing` and `@abvdev/otel` packages are based.
[See here for a full example for the Vercel AI SDK with NextJS on Vercel](/developer/sdks/js-ts/instrumentation).
# Client Setup
## 1) Installation
```bash
npm install @abvdev/client
```
## 2) Register your credentials
Add your ABV credentials to your environment variables. Make sure that you have a `.env` file in your project root and a package like `dotenv` to load the variables.
```bash
ABV_API_KEY = "sk-abv-..."
ABV_BASE_URL = "https://app.abv.dev" # US region
# ABV_BASE_URL = "https://eu.app.abv.dev" # EU region
```
## 3) Initialize the client
Initialize the `ABVClient` to interact with ABV. The client will automatically use the environment variables you set above.
```typescript title="client.ts"
import { ABVClient } from "@abvdev/client";
const abv = new ABVClient();
```
**Alternative: Configure via constructor**
You can also pass configuration options directly to the constructor:
```typescript title="client.ts"
import { ABVClient } from "@abvdev/client";
const abv = new ABVClient({
apiKey: "sk-abv-...",
baseUrl: "https://app.abv.dev", // US region
// baseUrl: "https://eu.app.abv.dev", // EU region
});
```
# Learn more
[Cookbook: Instrumentation](/developer/sdks/js-ts/cookbook)
[Advanced Usage](/developer/sdks/js-ts/advanced-configuration)
--------------------------------------------------------------------------------
## TypeScript SDK - Instrumentation
URL: /developer/sdks/js-ts/instrumentation
To instrument your application to send traces to ABV, you can use instrumentation methods for fine-grained control
1. Context manager: `startActiveObservation`
2. Wrapper: `observe`
3. Manual: `startObservation`
These components are interoperable.
# Setup
To get started with the ABV TypeScript SDK you need to [install the SDK and initialize the client](/developer/sdks/js-ts/setup).
# Instrumentation
You can add custom instrumentations to your application via
- the `observe` wrapper
- `startActiveObservation` context managers
- manually managing the observation lifecycle and its nesting with the `startObservation` function
For an end-to-end example, see the [JS Instrumentation Cookbook](/developer/sdks/js-ts/cookbook).
## Context management with callbacks
To simplify nesting and context management, you can use `startActiveObservation`. These functions take a callback and automatically manage the observation's lifecycle and the OpenTelemetry context. Any observation created inside the callback will automatically be nested under the active observation, and the observation will be ended when the callback finishes.
This is the recommended approach for most use cases as it prevents context leakage and ensures observations are properly ended.
```typescript
import "./instrumentation";
import { startActiveObservation, startObservation } from "@abvdev/tracing";
async function main() {
await startActiveObservation(
// name
"user-request",
// callback
async (span) => {
span.update({
input: { query: "What is the capital of France?" },
});
// Example child, could also use startActiveObservation
// This manually created generation (see docs below) will automatically be a child of "user-request"
const generation = startObservation(
"llm-call",
{
model: "gpt-4",
input: [{ role: "user", content: "What is the capital of France?" }],
},
{ asType: "generation" }
);
generation.update({
usageDetails: { input: 10, output: 5 },
output: { content: "The capital of France is Paris." },
});
generation.end();
span.update({ output: "Successfully answered." });
}
);
}
main();
```
## *observe* wrapper
The `observe` wrapper is a powerful tool for tracing existing functions without modifying their internal logic. It acts as a decorator that automatically creates a **span** or **generation** around the function call. You can use the `updateActiveObservation` function to add attributes to the observation from within the wrapped function.
```typescript
import { observe, updateActiveObservation } from "@abvdev/tracing";
// An existing function
async function fetchData(source: string) {
updateActiveObservation({ metadata: { source: "API" } });
// ... logic to fetch data
return { data: `some data from ${source}` };
}
// Wrap the function to trace it
const tracedFetchData = observe(
// method
fetchData,
// options, optional, see below
{}
);
// Now, every time you call tracedFetchData, a span is created.
// Its input and output are automatically populated with the
// function's arguments and return value.
const result = await tracedFetchData("API");
```
You can configure the `observe` wrapper by passing an options object as the second argument:
| Option | Description | Default |
| --------------- | -------------------------------------------------------------------------------------------------- | ----------------------------- |
| `name` | The name of the observation. | The original function's name. |
| `asType` | The type of observation to create (e.g. `span`, `generation`). | `"span"` |
| `captureInput` | Whether to capture the function's arguments as the `input` of the observation. | `true` |
| `captureOutput` | Whether to capture the function's return value or thrown error as the `output` of the observation. | `true` |
## Manual observations
The core tracing function (`startObservation`) gives you full control over creating observations. You can pass the `asType` option to specify the type of observation to create.
When you call one of these functions, the new observation is automatically linked as a child of the currently active operation in the OpenTelemetry context. However, it does **not** make this new observation the active one. This means any further operations you trace will still be linked to the *original* parent, not the one you just created.
To create nested observations manually, use the methods on the returned object (e.g., `parentSpan.startObservation(...)`).
```typescript
import { startObservation } from "@abvdev/tracing";
// Start a root span for a user request
const span = startObservation(
// name
"user-request",
// params
{
input: { query: "What is the capital of France?" },
}
);
// Create a nested span for a e.g. tool call
const toolCall = span.startObservation(
// name
"fetch-weather",
// params
{
input: { city: "Paris" },
},
// Specify observation type in asType
// This will type the attributes argument accordingly
// Default is 'span'
{ asType: "tool" }
);
// Simulate work and end the tool call span
await new Promise((resolve) => setTimeout(resolve, 100));
toolCall.update({ output: { temperature: "15°C" } }).end();
// Create a nested generation for the LLM call
const generation = span.startObservation(
"llm-call",
{
model: "gpt-4",
input: [{ role: "user", content: "What is the capital of France?" }],
},
{ asType: "generation" }
);
generation.update({
usageDetails: { input: 10, output: 5 },
output: { content: "The capital of France is Paris." },
});
generation.end();
// End the root span
span.update({ output: "Successfully answered user request." }).end();
```
## Updating Traces
Often, you might not have all the information about a trace (like a `userId` or `sessionId`) when you start it. The SDK lets you add or update trace-level attributes at any point during its execution.
### *.updateTrace()* on an observation
When you create an observation manually with `startObservation`, the returned object has an `.updateTrace()` method. You can call this at any time before the root span ends to apply attributes to the entire trace.
```typescript
import "./instrumentation";
import { startObservation } from "@abvdev/tracing";
// Start a trace without knowing the user yet
const rootSpan = startObservation("data-processing");
// ... some initial steps ...
// Later, once the user is authenticated, update the trace
const userId = "user-123";
const sessionId = "session-abc";
rootSpan.updateTrace({
userId: userId,
sessionId: sessionId,
tags: ["authenticated-user"],
metadata: { plan: "premium" },
});
// ... continue with the rest of the trace ...
const generation = rootSpan.startObservation(
"llm-call",
{},
{ asType: "generation" }
);
generation.end();
rootSpan.end();
```
### *updateActiveTrace()*
When you're inside a callback from `startActiveObservation`, or a function wrapped with `observe`, you might not have a direct reference to an observation object. In these cases, use the `updateActiveTrace()` function. It automatically finds the currently active trace in the context and applies the new attributes.
```typescript
import "./instrumentation";
import { startActiveObservation, updateActiveTrace } from "@abvdev/tracing";
async function main() {
await startActiveObservation("user-request", async (span) => {
// Initial part of the request
span.update({ input: { path: "/api/process" } });
// Simulate fetching user data
await new Promise((resolve) => setTimeout(resolve, 50));
const user = { id: "user-5678", name: "Jane Doe" };
// Update the active trace with the user's information
updateActiveTrace({
userId: user.id,
metadata: { userName: user.name },
});
// ... continue logic ...
span.update({ output: { status: "success" } }).end();
});
}
main();
```
--------------------------------------------------------------------------------
## TypeScript SDK - Advanced Configuration
URL: /developer/sdks/js-ts/advanced-configuration
# **Setup**
**Install packages**
```bash
npm install @abvdev/tracing @abvdev/otel
```
**Add credentials**
Add your ABV credentials to your environment variables. Make sure that you have a `.env` file in your project root and a package like dotenv to load the variables.
```bash title=".env"
ABV_API_KEY = "sk-abv-..."
ABV_BASE_URL = "https://app.abv.dev" # US region
//ABV_BASE_URL = "https://eu.app.abv.dev" # EU region
```
**Initialize OpenTelemetry**
Install the OpenTelemetry Node SDK package:
```bash
npm install @opentelemetry/sdk-node
```
Create a `instrumentation.ts` file that initializes the OpenTelemetry `NodeSDK` and registers the `ABVSpanProcessor`.
```typescript title="instrumentation.ts"
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
const sdk = new NodeSDK({
spanProcessors: [new ABVSpanProcessor()],
});
sdk.start();
```
Modify `instrumentation.ts` file to use `dotenv` package to load the variables.
Additional parameters are provided to get trace visible in the UI immediately.
```bash
npm install dotenv
```
```typescript title="instrumentation.ts"
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
const sdk = new NodeSDK({
spanProcessors: [
new ABVSpanProcessor({
apiKey: process.env.ABV_API_KEY,
baseUrl: process.env.ABV_BASE_URL,
exportMode: "immediate",
flushAt: 1,
flushInterval: 1,
additionalHeaders: {
"Content-Type": "application/json",
"Accept": "application/json"
}
})
],
});
sdk.start();
```
Import the `instrumentation.ts` file at the top of your application.
```typescript title="index.ts"
import "./instrumentation"; // Must be the first import
```
****
# Masking
To prevent sensitive data from being sent to ABV, you can provide a `mask` function to the `ABVSpanProcessor`. This function will be applied to the `input`, `output`, and `metadata` of every observation.
The function receives an object `{ data }`, where `data` is the stringified JSON of the attribute's value. It should return the masked data.
```typescript title="instrumentation_masked.ts"
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
import { TraceIdRatioBasedSampler } from "@opentelemetry/sdk-trace-base";
import dotenv from "dotenv";
dotenv.config();
const sdk = new NodeSDK({
spanProcessors: [
new ABVSpanProcessor({
apiKey: process.env.ABV_API_KEY,
baseUrl: process.env.ABV_BASE_URL,
exportMode: "immediate",
flushAt: 1,
flushInterval: 1,
additionalHeaders: {
"Content-Type": "application/json",
"Accept": "application/json"
},
mask: ({ data }) => {
// A simple regex to mask credit card numbers
const maskedData = data.replace(
/\b\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}\b/g,
"***MASKED_CREDIT_CARD***"
);
return maskedData;
},
})
],
});
sdk.start();
```
Simple example with masked data
```typescript
import "./instrumentation_masked";
import { startActiveObservation } from "@abvdev/tracing";
async function main() {
await startActiveObservation("my-masked-trace", async (span) => {
span.update({
input: "Hello, ABV! My card number is 5555-7777-3333-1111 ",
output: "This is my first trace! And my card number is 1111-1111-1111-1111",
});
});
}
main();
```
Masked data will be shown in UI:
*Input: "Hello, ABV! My card number is \*\*\*MASKED\_CREDIT\_CARD\*\*\* "*
*Output: "This is my first trace! And my card number is \*\*\*MASKED\_CREDIT\_CARD\*\*\*"*
# Filtering Spans
You can provide a predicate function `shouldExportSpan` to the `ABVSpanProcessor` to decide on a per-span basis whether it should be exported to ABV.
Filtering spans may break the parent-child relationships in your traces. For example, if you filter out a parent span but keep its children, you may see "orphaned" observations in the ABV UI. Consider the impact on trace structure when configuring `shouldExportSpan`.
```typescript title="instrumentation.ts"
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor, ShouldExportSpan } from "@abvdev/otel";
import dotenv from "dotenv";
dotenv.config();
// Example: Filter out all spans from the 'express' instrumentation
const shouldExportSpan: ShouldExportSpan = ({ otelSpan }) =>
otelSpan.instrumentationScope.name !== "express";
const sdk = new NodeSDK({
spanProcessors: [new ABVSpanProcessor({ shouldExportSpan })],
});
sdk.start();
```
If you want to include only LLM observability related spans, you can configure an allowlist like so:
```typescript title="instrumentation.ts"
import { ShouldExportSpan } from "@abvdev/otel";
const shouldExportSpan: ShouldExportSpan = ({ otelSpan }) =>
["abv-sdk", "ai"].includes(otelSpan.instrumentationScope.name);
```
If you would like to exclude ABV spans from being sent to third-party observability backends configured in your OpenTelemetry setup, see [the documentation on isolating the ABV tracer provider](./#isolated-tracer-provider).
# Sampling
ABV respects OpenTelemetry's sampling decisions. You can configure a sampler in your OTEL SDK to control which traces are sent to ABV. This is useful for managing costs and reducing noise in high-volume applications.
Here is an example of how to configure a `TraceIdRatioBasedSampler` to send only 20% of traces:
```typescript title="instrumentation.ts"
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
import { TraceIdRatioBasedSampler } from "@opentelemetry/sdk-trace-base";
import dotenv from "dotenv";
dotenv.config();
const sdk = new NodeSDK({
// Sample 20% of all traces
sampler: new TraceIdRatioBasedSampler(0.2),
spanProcessors: [
new ABVSpanProcessor({
apiKey: process.env.ABV_API_KEY,
baseUrl: process.env.ABV_BASE_URL,
exportMode: "immediate",
flushAt: 1,
flushInterval: 1,
additionalHeaders: {
"Content-Type": "application/json",
"Accept": "application/json"
}
})
],
});
sdk.start();
```
For more advanced sampling strategies, refer to the [OpenTelemetry JS Sampling Documentation](https://opentelemetry.io/docs/languages/js/sampling/).
# Managing trace and observation IDs
In ABV, every trace and observation has a unique identifier. Understanding their format and how to set them is useful for integrating with other systems.
- **Trace IDs** are 32-character lowercase hexadecimal strings, representing 16 bytes of data
- **Observation IDs** (also known as Span IDs in OpenTelemetry) are 16-character lowercase hexadecimal strings, representing 8 bytes
While the SDK handles ID generation automatically, you may manually set them to align with external systems or create specific trace structures. This is done using the `parentSpanContext` option in tracing methods.
When starting a new trace by setting a `traceId`, you must also provide an arbitrary parent-`spanId` for the parent observation. The parent span ID value is irrelevant as long as it is a valid 16-hexchar string as the span does not actually exist but is only used for trace ID inheritance of the created observation.
You can create valid, deterministic trace IDs from a seed string using `createTraceId`. This is useful for correlating ABV traces with IDs from external systems, like a support ticket ID.
```typescript
import "./instrumentation";
import { createTraceId, startObservation } from "@abvdev/tracing";
async function main() {
const externalId = "support-ticket-11112025";
// Generate a valid, deterministic traceId from the external ID
const abvTraceId = await createTraceId(externalId);
console.log(abvTraceId);
// You can now start a new trace with this ID
const rootSpan = startObservation(
"process-ticket",
{
input: "there is an input",
output: "here you are an output"
},
{
parentSpanContext: {
traceId: abvTraceId,
spanId: "0123456789abcdef", // A valid 16 hexchar string; value is irrelevant as parent span does not exist but only used for inheritance
traceFlags: 1, // mark trace as sampled
},
}
);
rootSpan.end();
// Later, you can regenerate the same traceId to score or retrieve the trace
const scoringTraceId = await createTraceId(externalId);
// scoringTraceId will be the same as abvTraceId
console.log(scoringTraceId);
}
main();
```
You may also access the current active trace ID via the `getActiveTraceId` function:
```typescript
import "./instrumentation";
import { startActiveObservation, getActiveTraceId } from "@abvdev/tracing";
async function main() {
await startActiveObservation("my-first-trace", async (span) => {
span.update({
input: "Hello, ABV!",
output: "This is my first trace!",
});
const traceId = getActiveTraceId();
console.log(`Current trace ID: ${traceId}`);
});
}
main();
```
# Logging
You can configure the global SDK logger to control the verbosity of log output. This is useful for debugging.
**In code:**
```typescript
import { configureGlobalLogger, LogLevel } from "@abvdev/core";
// Set the log level to DEBUG to see all log messages
configureGlobalLogger({ level: LogLevel.DEBUG });
import "./instrumentation";
import { startActiveObservation, getActiveTraceId } from "@abvdev/tracing";
async function main() {
await startActiveObservation("my-first-trace", async (span) => {
span.update({
input: "Hello, ABV!",
output: "This is my first trace!",
});
const traceId = getActiveTraceId();
console.log(`Current trace ID: ${traceId}`);
});
}
main();
```
Available log levels are `DEBUG`, `INFO`, `WARN`, and `ERROR`.
**Via environment variable:**
You can also set the log level using the `ABV_LOG_LEVEL` environment variable.
```bash title=".env"
ABV_LOG_LEVEL="DEBUG"
```
# Serverless environments
In short-lived environments such as serverless functions (e.g., Vercel Functions, AWS Lambda), you must explicitly flush the traces before the process exits or the runtime environment is frozen.
### Generic Serverless function
Export the processor from your OTEL SDK setup file in order to flush it later.
```typescript title="instrumentation.ts"
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
// Export the processor to be able to flush it
export const abvSpanProcessor = new ABVSpanProcessor();
const sdk = new NodeSDK({
spanProcessors: [abvSpanProcessor],
});
sdk.start();
```
In your serverless function handler, call `forceFlush()` on the span processor before the function exits.
```typescript title="handler.ts"
import { abvSpanProcessor } from "./instrumentation";
export async function handler(event, context) {
// ... your application logic ...
// Flush before exiting
await abvSpanProcessor.forceFlush();
}
```
### Vercel Cloud Functions
Export the processor from your `instrumentation.ts` file in order to flush it later.
```typescript title="instrumentation.ts"
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
// Export the processor to be able to flush it
export const abvSpanProcessor = new ABVSpanProcessor();
const sdk = new NodeSDK({
spanProcessors: [abvSpanProcessor],
});
sdk.start();
```
In Vercel Cloud Functions, please use the `after` utility to schedule a flush after the request has completed.
```typescript title="route.ts"
import { after } from "next/server";
import { abvSpanProcessor } from "./instrumentation.ts";
export async function POST() {
// ... existing request logic ...
// Schedule flush after request has completed
after(async () => {
await abvSpanProcessor.forceFlush();
});
// ... send response ...
}
```
# Isolated tracer provider
The ABV JS SDK uses the global OpenTelemetry TracerProvider to attach its span processor and create tracers that emit spans. This means that if you have an existing OpenTelemetry setup with another destination configured for your spans (e.g., Datadog), you will see ABV spans in those third-party observability backends as well.
If you'd like to avoid sending ABV spans to third-party observability backends in your existing OpenTelemetry setup, you will need to use an isolated OpenTelemetry TracerProvider that is separate from the global one.
If you would like to simply limit the spans that are sent to ABV and you have no third-party observability backend where you'd like to exclude ABV spans from, [see filtering spans instead](./#filtering-spans).
Using an isolated TracerProvider may break the parent-child relationships in your traces, as all TracerProviders still share the same active span context. For example, if you have an active parent span from the global TracerProvider but children from an isolated TracerProvider, you may see "orphaned" observations in the ABV UI. Consider the impact on trace structure when configuring an isolated tracer provider.
```typescript
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
import { setABVTracerProvider } from "@abvdev/tracing";
// Create a new TracerProvider and register the ABVSpanProcessor
// do not set this TracerProvider as the global TracerProvider
const abvTracerProvider = new NodeTracerProvider(
spanProcessors: [new ABVSpanProcessor()],
)
// Register the isolated TracerProvider
setABVTracerProvider(abvTracerProvider)
```
# Multi-project Setup
You can configure the SDK to send traces to multiple ABV projects. This is useful for multi-tenant applications or for sending traces to different environments. Simply register multiple `ABVSpanProcessor` instances, each with its own credentials.
```typescript title="instrumentation.ts"
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
import dotenv from "dotenv";
dotenv.config();
const sdk = new NodeSDK({
spanProcessors: [
new ABVSpanProcessor({
apiKey: "sk-abv-b14d5b78-de4e-4699-b48e-173d8dde1c12",
baseUrl: process.env.ABV_BASE_URL,
exportMode: "immediate",
flushAt: 1,
flushInterval: 1,
additionalHeaders: {
"Content-Type": "application/json",
"Accept": "application/json"
}
}),
new ABVSpanProcessor({
apiKey: "sk-abv-79dc2bb3-2c9e-4e11-80cd-f06b019b72f9",
baseUrl: process.env.ABV_BASE_URL,
exportMode: "immediate",
flushAt: 1,
flushInterval: 1,
additionalHeaders: {
"Content-Type": "application/json",
"Accept": "application/json"
}
})
],
});
sdk.start();
```
This configuration will send every trace to both projects. You can also configure a custom `shouldExportSpan` filter for each processor to control which traces go to which project.
# Custom scores from browser
Sending custom scores directly from the browser is not yet supported in the TypeScript SDK.
--------------------------------------------------------------------------------
## Cookbook: ABV JS/TS SDK
URL: /developer/sdks/js-ts/cookbook
This cookbook provides practical examples for using the ABV JS/TS SDK.
JS/TS applications can be traced via the [JS/TS SDK](/developer/sdks/js-ts/overview).
In this notebook, we will walk you through a **simple end-to-end example** that:
- Uses the core features of the ABV JS/TS SDK
- Shows how to log any LLM call via the low-level SDK
For this guide, we assume that you are already familiar with the ABV data model (traces, spans, generations, etc.). If not, please read the [conceptual introduction](/developer/basic-features/observability-tracing) to tracing.
# Set Up Environment
Get your ABV API key by signing up for ABV. You’ll also need your OpenAI API key.
***Note****: This cookbook uses ****Deno.js ****for execution, which requires different syntax for importing packages and setting environment variables. For Node.js applications, the setup process is similar but uses standard *`npm `*packages and *`process.env.`
Add your ABV credentials to your environment variables. Make sure that you have a `.env` file in your project root and a package like dotenv to load the variables.
```bash title=".env"
ABV_API_KEY = "sk-abv-..."
ABV_BASE_URL = "https://app.abv.dev" // US region
//ABV_BASE_URL = "https://eu.app.abv.dev" # EU region
OPENAI_API_KEY = "sk-proj-..." // added to use openai llm
```
With the environment variables set, we can now initialize the `abvSpanProcessor `which is passed to the main OpenTelemetry SDK that orchestrates tracing.
```typescript title="instrumentation.ts"
// Import required dependencies
import 'npm:dotenv/config';
import { NodeSDK } from "npm:@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "npm:@abvdev/otel";
// Export the processor to be able to flush it later
// This is important for ensuring all spans are sent to ABV
export const abvSpanProcessor = new ABVSpanProcessor({
apiKey: process.env.ABV_API_KEY!,
baseUrl: process.env.ABV_HOST ?? 'https://app.abv.dev', // Default if not specified
environment: process.env.NODE_ENV ?? 'development', // Default to development if not specified
});
// Initialize the OpenTelemetry SDK with our ABV processor
const sdk = new NodeSDK({
spanProcessors: [abvSpanProcessor],
});
// Start the SDK to begin collecting telemetry
// The warning about crypto module is expected in Deno and doesn't affect basic tracing functionality. Media upload features will be disabled, but all core tracing works normally
sdk.start();
```
The **ABVClient** provides additional functionality beyond OpenTelemetry tracing, such as scoring, prompt management, and data retrieval. It automatically uses the same environment variables we set earlier.
```typescript
import { ABVClient } from "npm:@abvdev/client";
const abv = new ABVClient();
```
# Log LLM Calls
You can use the SDK to log any LLM call or any of the integrations that are interoperable with it.
In the following, we will demonstrate how to log LLM calls using the SDK, LangChain, Vercel AI SDK, and OpenAI integrations.
## Option 1: Context Manager
To simplify nesting and context management, you can use startActiveObservation. These functions take a callback and automatically manage the observation’s lifecycle and the OpenTelemetry context. Any observation created inside the callback will automatically be nested under the active observation, and the observation will be ended when the callback finishes.
This is the recommended approach for most use cases as it prevents context leakage and ensures observations are properly ended.
```typescript title="apps.ts"
// Import necessary functions from the tracing package
import { startActiveObservation, startObservation, updateActiveTrace, updateActiveObservation } from "npm:@abvdev/tracing";
// Start a new span with automatic context management
await startActiveObservation("context-manager", async (span) => {
// Log the initial user query
span.update({
input: { query: "What is the capital of France?" }
});
// Create a new generation span that will automatically be a child of "context-manager"
const generation = startObservation(
"llm-call",
{
model: "gpt-4",
input: [{ role: "user", content: "What is the capital of France?" }],
},
{ asType: "generation" },
);
// ... LLM call logic would go here ...
// Update the generation with token usage statistics
generation.update({
usageDetails: {
input: 10, // Number of input tokens
output: 5, // Number of output tokens
cache_read_input_tokens: 2, // Tokens read from cache
some_other_token_count: 10, // Custom token metric
total: 17, // Optional: automatically calculated if not provided
},
});
// End the generation with the LLM response
generation.update({
output: { content: "The capital of France is Paris." },
}).end();
// Example user information
const user = { id: "user-5678", name: "Jane Doe", sessionId: "123" };
// Add an optional log level of type warning to the active span
updateActiveObservation(
{ level: "WARNING", statusMessage: "This is a warning" },
);
// Update the trace with user context
updateActiveTrace({
userId: user.id,
sessionId: user.sessionId,
metadata: { userName: user.name },
});
// Mark the span as complete with final output
span.update({ output: "Successfully answered." });
});
// Ensure all spans are sent to ABV
await abvSpanProcessor.forceFlush();
```
Public trace in the ABV UI
## Option 2: observe Decorator
The `observe `wrapper is a powerful tool for tracing existing functions without modifying their internal logic. It acts as a decorator that automatically creates a span or generation around the function call. You can use the `updateActiveObservation `function to add attributes to the observation from within the wrapped function.
```typescript
import { observe, updateActiveObservation } from "npm:@abvdev/tracing";
// An existing function
async function fetchData(source: string) {
updateActiveObservation({ usageDetails: {
// usage
input: 10,
output: 5,
}, { asType: 'generation' }
})
// ... logic to fetch data
return { data: `some data from ${source}` };
}
// Wrap the function to trace it
const tracedFetchData = observe(fetchData, {
name: "observe-wrapper",
asType: "generation",
});
// Now, every time you call tracedFetchData, a span is created.
// Its input and output are automatically populated with the
// function's arguments and return value.
const result = await tracedFetchData("API");
await abvSpanProcessor.forceFlush();
```
Public trace in the ABV UI
## Option 3: Manual Spans
This part shows how to log any LLM call by passing the model in and outputs via the ABV SDK.
Steps:
1. Create span to contain this section within the trace
2. Create generation, log input and model name as it is already known
3. Call the LLM SDK and log the output
4. End generation and span
Teams typically wrap their LLM SDK calls in a helper function that manages tracing internally. This implementation occurs once and is then reused for all LLM calls.
```typescript
// Import the startObservation function for manual span creation
import { startObservation } from 'npm:@abvdev/tracing';
// Create the root span for this operation
const span = startObservation('manual-observation', {
input: { query: 'What is the capital of France?' },
});
// Create a child span for a tool call (e.g., weather API)
const toolCall = span.startObservation(
'fetch-weather',
{ input: { city: 'Paris' } },
{ asType: "tool" },
);
// Simulate API call with timeout
await new Promise((r) => setTimeout(r, 100));
// End the tool call with its output
toolCall.update({ output: { temperature: '15°C' } }).end();
// Create a generation span for the LLM call
const generation = span.startObservation(
'llm-call',
{
model: 'gpt-4',
input: [{ role: 'user', content: 'What is the capital of France?' }],
output: { content: 'The capital of France is Paris.' },
},
{ asType: "generation" },
);
// Update the generation with token usage details
generation.update({
usageDetails: {
input: 10, // Input token count
output: 5, // Output token count
cache_read_input_tokens: 2, // Cached tokens used
some_other_token_count: 10, // Custom metric
total: 17, // Total tokens (optional)
},
});
// End the generation with final output
generation.update({
output: { content: 'The capital of France is Paris.' },
}).end();
// End the root span with final status and session ID
span.update({
output: 'Successfully answered user request.',
sessionId: '123'
}).end();
// Ensure all spans are flushed to ABV
await abvSpanProcessor.forceFlush();
```
Public trace in the ABV UI
# View the Trace in ABV
After ingesting your spans, you can view them in your ABV dashboard.
in the ABV UI.
# Learn More
- [ABV JS/TS SDK Guide](/developer/sdks/js-ts/overview)
- Support
--------------------------------------------------------------------------------
## TypeScript SDK - Troubleshooting and FAQ
URL: /developer/sdks/js-ts/troubleshooting-faq
If your issue is not covered here, please write to [support](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiJlZmZmN2JhYy0xMzZlLTNkOTUtOGIxYi1hMWQ3NWNkZWM3NGYiLCJpYXQiOjE3NTY4NDU1MTk1NzN9._OjNB4ESzMwKHLbUJPaQOjVQQqB7aCoKiCaFxhBrFoo).
# **No public key provided error appears or no traces in UI**
run the following command:
```bash
npm update
```
# Missing traces in serverless environments
Please see the documentation on [serverless environments](/developer/sdks/js-ts/advanced-configuration)
# Missing traces with *@vercel/otel*
If you are using Next.js, please use a manual OpenTelemetry setup via the `NodeTracerProvider` than via `registerOTel` from `@vercel/otel`. This is because [the @vercel/otel package does not yet support the OpenTelemetry JS SDK v2](https://github.com/vercel/otel/issues/154) on which the `@abvdev/tracing` and `@abvdev/otel` packages are based.
If you are still missing traces, please see the documentation on [serverless environments](/developer/sdks/js-ts/advanced-configuration)
--------------------------------------------------------------------------------
## Python SDK - Overview
URL: /developer/sdks/python/overview
The ABV Python SDK is designed for best in class developer experience and ease of use. Built on the robust OpenTelemetry Python SDK, it offers a more intuitive API for comprehensive tracing of your LLM application.
Official Link: [https://pypi.org/project/abvdev/](https://pypi.org/project/abvdev/)
The SDK introduces several key benefits:
- **Improved Developer Experience**: A more intuitive API means less code to write for tracing your application, simplifying the integration process.
- **Unified Context Sharing**: Seamlessly hook into the tracing context of the current span to update it or create child spans. This is particularly beneficial for integrating with other instrumented libraries.
- **Broad Third-Party Integrations**: Any library instrumented with OpenTelemetry will work out-of-the-box with the ABV SDK. Spans from these libraries are automatically captured and correctly nested within your ABV traces.
# Quickstart
There are three main ways of instrumenting your application with the new ABV SDK. All of them are fully interoperable with each other.
Install package
```bash
pip install abvdev
```
### Using Observe Decorator
The @observe decorator is the simplest way to instrument your application. It is a function decorator that can be applied to any function.
It sets the current span in the context for automatic nesting of child spans and automatically ends it when the function returns. It also automatically captures the function name, arguments, and return value.
```python
from abvdev import ABV, observe
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
@observe
def my_function():
return "Hello, world!" # Input/output and timings are automatically captured
my_function()
# Flush events in short-lived applications
abv.flush()
```
### Using Context Managers
Context managers are the recommended way to instrument chunks of work in your application as they automatically handle the start and end of spans, and set the current span in the context for automatic nesting of child spans. They provide more control than the @observe decorator.
```python
from abvdev import ABV
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# Create a span using a context manager
with abv.start_as_current_span(name="process-request") as span:
# Your processing logic here
span.update(output="Processing complete")
# Create a nested generation for an LLM call
with abv.start_as_current_observation(as_type='generation', name="llm-response", model="gpt-5-2025-08-07") as generation:
# Your LLM call logic here
generation.update(output="Generated response")
# All spans are automatically closed when exiting their context blocks
# Flush events in short-lived applications
abv.flush()
```
### Using Manual Observations
Manual observations give you control over when spans start and end and do not set the current span in the context for automatic nesting of child spans. You must explicitly call .end() when they’re complete.
```python
from abvdev import ABV
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# Create a span without a context manager
span = abv.start_span(name="user-request")
# Your processing logic here
span.update(output="Request processed")
# Child spans must be created using the parent span object
nested_span = span.start_span(name="nested-span")
nested_span.update(output="Nested span output")
# Important: Manually end the span
nested_span.end()
# Important: Manually end the parent span
span.end()
# Flush events in short-lived applications
abv.flush()
```
# Learn more
[Setup](/developer/sdks/python/setup)
[Instrumentation](/developer/sdks/python/instrumentation)
[Evaluation](/developer/sdks/python/evaluations)
[Advanced usage](/developer/sdks/python/advanced-usage)
[Troubleshooting & FAQ](/developer/sdks/python/troubleshooting)
## Key Features Supported
The Python SDK provides full support for all ABV features:
- [Observability & Tracing](/developer/basic-features/observability-tracing): Comprehensive tracing of LLM applications
- [Sessions](/developer/basic-features/sessions): Group related traces by user journey
- [User Tracking](/developer/basic-features/user-tracking): Associate traces with specific users
- [Metadata](/developer/basic-features/metadata): Add structured context to traces
- [Tags](/developer/basic-features/tags): Label and categorize traces
- [Model Usage & Cost Tracking](/developer/basic-features/model-usage-cost-tracking): Monitor tokens and costs
- [Masking Sensitive Data](/developer/basic-features/masking-sensitive-data): Redact PII and secrets
- [Sampling](/developer/basic-features/sampling): Control trace volume
- [Prompt Management](/developer/prompt-management/get-started): Fetch and link prompts to traces
- [Evaluations](/developer/evaluations/overview): Run evaluations and create datasets
- [Guardrails](/developer/guardrails/overview): Implement LLM security guardrails
# OTEL and ABV
The ABV SDK is built upon [OpenTelemetry (OTEL)](https://opentelemetry.io/), a standard for observability. Understanding the relation between OTEL and ABV is not required to use the SDK, but it is helpful to have a basic understanding of the concepts. OTEL related concepts are abstracted away and you can use the SDK without being deeply familiar with them.
- **OTEL Trace**: An OTEL-trace represents the entire lifecycle of a request or transaction as it moves through your application and its services. A trace is typically a sequence of operations, like an LLM generating a response followed by a parsing step. The root (first) span created in a sequence defines the OTEL-trace. OTEL-traces do not have a start and end time, they are defined by the root span.
- **OTEL Span**: A span represents a single unit of work or operation within a trace. Spans have a start and end time, a name, and can have attributes (key-value pairs of metadata). Spans can be nested to create a hierarchy, showing parent-child relationships between operations.
- **ABV Trace**: A ABV trace collects observations and holds trace attributes such as `session_id`, `user_id` as well as overall input and outputs. It shares the same ID as the OTEL trace and its attributes are set via specific OTEL span attributes that are automatically propagated to the ABV trace.
- **ABV Observation**: In ABV terminology, an "observation" is a ABV-specific representation of an OTEL span. It can be a generic span (ABV-span) or a specialized "generation" (ABV-generation) or a point in time event (ABV-event)
- **ABV Span**: A ABV-span is a generic OTEL-span in ABV, designed for non-LLM operations.
- **ABV Generation**: A ABV-generation is a specialized type of OTEL-span in ABV, designed specifically for Large Language Model (LLM) calls. It includes additional fields like `model`, `model_parameters`, `usage_details` (tokens), and `cost_details`.
- **ABV Event**: A ABV-event tracks a point in time action.
- **Context Propagation**: OpenTelemetry automatically handles the propagation of the current trace and span context. This means when you call another function (whether it's also traced by ABV, an OTEL-instrumented library, or a manually created span), the new span will automatically become a child of the currently active span, forming a correct trace hierarchy.
The ABV SDK provides wrappers around OTEL spans (`ABVSpan`, `ABVGeneration`) that offer convenient methods for interacting with ABV-specific features like scoring and media handling, while still being native OTEL spans under the hood. You can also use these wrapper objects to add ABV trace attributes.
--------------------------------------------------------------------------------
## Python SDK - Setup
URL: /developer/sdks/python/setup
To get started with the ABV Python SDK you need to install the SDK and initialize the client.
# **1) Installation**
To install the SDK, run:
```bash
pip install abvdev
```
# 2) Initialize Client
Begin by initializing the `ABV` client. You must provide your ABV api key. These can be passed as constructor arguments or set as environment variables (recommended).
If you are using a data region other than the default (EU, `https://eu.app.abv.dev`), ensure you configure the `host` argument or the `ABV_HOST` environment variable (recommended).
**via Environment Variables (recommended)**
```bash title=".env"
ABV_API_KEY="sk-abv-..."
ABV_HOST="https://app.abv.dev" # US region
# ABV_HOST="https://eu.app.abv.dev" # EU region
```
**via Constructor Arguments**
```python
from abvdev import ABV
# Initialize with constructor arguments
abv = ABV(
api_key="YOUR_API_KEY",
host="https://app.abv.dev" # US region
# host="https://eu.app.abv.dev" # EU region
)
```
If you are reinstantiating the ABV client with different constructor arguments in the same process, the new configuration will be used for all subsequent calls.
**Verify connection with abv.auth\_check()**
You can also verify your connection to the ABV server using `abv.auth_check()`. We do not recommend using this in production as this adds latency to your application.
```python
from abvdev import get_client
abv = get_client()
# Verify connection, do not use in production as this is a synchronous call
if abv.auth_check():
print("ABV client is authenticated and ready!")
else:
print("Authentication failed. Please check your credentials and host.")
```
Key configuration options:
| Constructor Argument | Environment Variable | Description | Default value |
| --------------------------- | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- |
| `api_key` | `ABV_API_KEY` | Your ABV project's secret API key. **Required.** | |
| `host` | `ABV_HOST` | The API host for your ABV instance. | `"https://app.abv.dev"` |
| `timeout` | `ABV_TIMEOUT` | Timeout in seconds for API requests. | `5` |
| `httpx_client` | - | Custom `httpx.Client` for making non-tracing HTTP requests. | |
| `debug` | `ABV_DEBUG` | Enables debug mode for more verbose logging. Set to `True` or `"True"`. | `False` |
| `tracing_enabled` | `ABV_TRACING_ENABLED` | Enables or disables the ABV client. If `False`, all observability calls become no-ops. | `True` |
| `flush_at` | `ABV_FLUSH_AT` | Number of spans to batch before sending to the API. | `512` |
| `flush_interval` | `ABV_FLUSH_INTERVAL` | Time in seconds between batch flushes. | `5` |
| `environment` | `ABV_TRACING_ENVIRONMENT` | Environment name for tracing (e.g., "development", "staging", "production"). Must be lowercase alphanumeric with hyphens/underscores. | `"default"` |
| `release` | `ABV_RELEASE` | [Release](/developer/basic-features/releases-versioning) version/hash of your application. Used for grouping analytics. | |
| `media_upload_thread_count` | `ABV_MEDIA_UPLOAD_THREAD_COUNT` | Number of background threads for handling media uploads. | `1` |
| `sample_rate` | `ABV_SAMPLE_RATE` | [Sampling](/developer/basic-features/sampling) rate for traces (float between 0.0 and 1.0). `1.0` means 100% of traces are sampled. | `1.0` |
| `mask` | - | A function `(data: Any) -> Any` to [mask](/developer/basic-features/masking-sensitive-data) sensitive data in traces before sending to the API. | |
| | `ABV_MEDIA_UPLOAD_ENABLED` | Whether to upload media files to ABV S3. | `True` |
# Accessing the Client Globally
The ABV client is a singleton. It can be accessed anywhere in your application using the `get_client` function.
Optionally, you can initialize the client via `ABV()` to pass in configuration options (see above). Otherwise, it is created automatically when you call `get_client()` based on environment variables.
```python
from abvdev import get_client
# Optionally, initialize the client with configuration options
# abv = ABV(api_key="sk-abv-...")
# Get the default client
client = get_client()
```
--------------------------------------------------------------------------------
## Python SDK - Instrumentation
URL: /developer/sdks/python/instrumentation
# Basic Tracing
ABV provides flexible ways to create and manage traces and their constituent observations (spans and generations).
## Install package
```bash
pip install abvdev
```
## ***@observe*** Decorator
The `@observe()` decorator provides a convenient way to automatically trace function executions, including capturing their inputs, outputs, execution time, and any errors. It supports both synchronous and asynchronous functions.
```python
from abvdev import ABV, observe
import asyncio
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
@observe()
def my_data_processing_function(data, parameter):
# ... processing logic ...
return {"processed_data": data, "status": "ok"}
@observe(name="llm-call", as_type="generation")
async def my_async_llm_call(prompt_text):
# ... async LLM call ...
await asyncio.sleep(0.1)
return "LLM response"
my_data_processing_function("test_data", "test_parameter")
asyncio.run(my_async_llm_call("test_prompt"))
```
### **Parameters:**
- `name: Optional[str]`: Custom name for the created span/generation. Defaults to the function name.
- `as_type: Optional[Literal["generation"]]`: If set to `"generation"`, a ABV generation object is created, suitable for LLM calls. Otherwise, a regular span is created.
- `capture_input: bool`: Whether to capture function arguments as input. Defaults to env var `ABV_OBSERVE_DECORATOR_IO_CAPTURE_ENABLED` or `True` if not set.
- `capture_output: bool`: Whether to capture function return value as output. Defaults to env var `ABV_OBSERVE_DECORATOR_IO_CAPTURE_ENABLED` or `True` if not set.
- `transform_to_string: Optional[Callable[[Iterable], str]]`: For functions that return generators (sync or async), this callable can be provided to transform the collected chunks into a single string for the `output` field. If not provided, and all chunks are strings, they will be concatenated. Otherwise, the list of chunks is stored.
### **Trace Context and Special Keyword Arguments:**
The `@observe` decorator automatically propagates the OTEL trace context. If a decorated function is called from within an active ABV span (or another OTEL span), the new observation will be nested correctly.
You can also pass special keyword arguments to a decorated function to control its tracing behavior:
- `abv_trace_id: str`: Explicitly set the trace ID for this function call. Must be a valid W3C Trace Context trace ID (32-char hex). If you have a trace ID from an external system, you can use `ABV.create_trace_id(seed=external_trace_id)` to generate a valid deterministic ID.
- `abv_parent_observation_id: str`: Explicitly set the parent observation ID. Must be a valid W3C Trace Context span ID (16-char hex).
```python
from abvdev import ABV, observe
abv = ABV(
api_key="sk-abv-", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
@observe()
def my_function(a, b):
return a + b
# Call with a specific trace context
my_function(1, 2, abv_trace_id="1234567890abcdef1234567890abcdef")
```
The observe decorator is capturing the args, kwargs and return value of decorated functions by default. This may lead to performance issues in your application if you have large or deeply nested objects there. To avoid this, explicitly disable function IO capture on the decorated function by passing `capture_input=False` and/or `capture_output=False` parameters.
## Context Managers
You can create spans or generations anywhere in your application. If you need more control than the `@observe` decorator, the primary way to do this is using context managers (with `with` statements), which ensure that observations are properly started and ended.
- `abv.start_as_current_span()`: Creates a new span and sets it as the currently active observation in the OTEL context for its duration. Any new observations created within this block will be its children.
- `abv.start_as_current_generation()`: Similar to the above, but creates a specialized "generation" observation for LLM calls.
```python
from abvdev import ABV, observe
abv = ABV(
api_key="sk-abv-", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
with abv.start_as_current_span(
name="user-request-pipeline",
input={"user_query": "Tell me a joke about OpenTelemetry"},
) as root_span:
# This span is now active in the context.
# Add trace attributes
root_span.update_trace(
user_id="user_123",
session_id="session_abc",
tags=["experimental", "comedy"]
)
# Create a nested generation
with abv.start_as_current_generation(
name="joke-generation",
model="gpt-5-2025-08-07",
input=[{"role": "user", "content": "Tell me a joke about OpenTelemetry"}],
model_parameters={"temperature": 0.7}
) as generation:
# Simulate an LLM call
joke_response = "Why did the OpenTelemetry collector break up with the span? Because it needed more space... for its attributes!"
token_usage = {"input_tokens": 10, "output_tokens": 25}
generation.update(
output=joke_response,
usage_details=token_usage
)
# Generation ends automatically here
root_span.update(output={"final_joke": joke_response})
# Root span ends automatically here
```
## Manual Observations
For scenarios where you need to create an observation (a span or generation) without altering the currently active OpenTelemetry context, you can use `abv.start_span()` or `abv.start_generation()`.
```python
from abvdev import ABV
abv = ABV(
api_key="sk-abv-", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
span = abv.start_span(name="my-span")
span.end() # Important: Manually end the span
```
If you use manual span management with `start_span()` or `start_generation()`, you must remember to call `.end()` on each observation to ensure data is properly sent to ABV. Failing to end observations can lead to incomplete traces.
### **Key Characteristics:**
- **No Context Shift**: Unlike their `start_as_current_...` counterparts, these methods **do not** set the new observation as the active one in the OpenTelemetry context. The previously active span (if any) remains the current context for subsequent operations in the main execution flow.
- **Parenting**: The observation created by `start_span()` or `start_generation()` will still be a child of the span that was active in the context at the moment of its creation.
- **Manual Lifecycle**: These observations are not managed by a `with` block and therefore **must be explicitly ended** by calling their `.end()` method.
- **Nesting Children**:
- Subsequent observations created using the global `abv.start_as_current_span()` (or similar global methods) will *not* be children of these "manual" observations. Instead, they will be parented by the original active span.
- To create children directly under a "manual" observation, you would use methods *on that specific observation object* (e.g., `manual_span.start_as_current_span(...)`).
**When to Use:**
This approach is useful when you need to:
- Record work that is self-contained or happens in parallel to the main execution flow but should still be part of the same overall trace (e.g., a background task initiated by a request).
- Manage the observation's lifecycle explicitly, perhaps because its start and end are determined by non-contiguous events.
- Obtain an observation object reference before it's tied to a specific context block.
**Example with more complex nesting:**
```python
from abvdev import ABV
abv = ABV(
api_key="sk-abv-", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# This outer span establishes an active context.
with abv.start_as_current_span(name="main-operation") as main_operation_span:
# 'main_operation_span' is the current active context.
# 1. Create a "manual" span using abv.start_span().
# - It becomes a child of 'main_operation_span'.
# - Crucially, 'main_operation_span' REMAINS the active context.
# - 'manual_side_task' does NOT become the active context.
manual_side_task = abv.start_span(name="manual-side-task")
manual_side_task.update(input="Data for side task")
# 2. Start another operation that DOES become the active context.
# This will be a child of 'main_operation_span', NOT 'manual_side_task',
# because 'manual_side_task' did not alter the active context.
with abv.start_as_current_span(name="core-step-within-main") as core_step_span:
# 'core_step_span' is now the active context.
# 'manual_side_task' is still open but not active in the global context.
core_step_span.update(input="Data for core step")
# ... perform core step logic ...
core_step_span.update(output="Core step finished")
# 'core_step_span' ends. 'main_operation_span' is the active context again.
# 3. Complete and end the manual side task.
# This could happen at any point after its creation, even after 'core_step_span'.
manual_side_task.update(output="Side task completed")
manual_side_task.end() # Manual end is crucial for 'manual_side_task'
main_operation_span.update(output="Main operation finished")
# 'main_operation_span' ends automatically here.
# Expected trace structure in ABV:
# - main-operation
# |- manual-side-task
# |- core-step-within-main
# (Note: 'core-step-within-main' is a sibling to 'manual-side-task', both children of 'main-operation')
```
## Nesting Observations
### Observe Decorator
The function call hierarchy is automatically captured by the `@observe` decorator reflected in the trace.
```python
from abvdev import ABV, observe
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
@observe
def my_data_processing_function(data, parameter):
# ... processing logic ...
return {"processed_data": data, "status": "ok"}
@observe
def main_function(data, parameter):
return my_data_processing_function(data, parameter)
# call function
main_function("test_data", "test_parameter")
```
### Context Managers
Nesting is handled automatically by OpenTelemetry’s context propagation. When you create a new observation (span or generation) using start\_as\_current\_span or start\_as\_current\_generation, it becomes a child of the observation that was active in the context when it was created.
```python
from abvdev import ABV
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
with abv.start_as_current_span(name="outer-process") as outer_span:
# outer_span is active
with abv.start_as_current_observation(as_type='generation', name="llm-step-1") as gen1:
# gen1 is active, child of outer_span
gen1.update(output="LLM 1 output")
with outer_span.start_as_current_observation(as_type='span', name="intermediate-step") as mid_span:
# mid_span is active, also a child of outer_span
# This demonstrates using the yielded span object to create children
with mid_span.start_as_current_observation(as_type='generation', name="llm-step-2") as gen2:
# gen2 is active, child of mid_span
gen2.update(output="LLM 2 output")
mid_span.update(output="Intermediate processing done")
outer_span.update(output="Outer process finished")
```
### Manual
If you are creating observations manually (not \_as\_current\_), you can use the methods on the parent ABVSpan or ABVGeneration object to create children. These children will not become the current context unless their \_as\_current\_ variants are used.
```python
from abvdev import ABV
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
parent = abv.start_span(name="manual-parent")
child_span = parent.start_span(name="manual-child-span")
# ... work ...
child_span.end()
child_gen = parent.start_observation(as_type='generation', name="manual-child-generation")
# ... work ...
child_gen.end()
parent.end()
```
## Updating Observations
You can update observations with new information as your code executes.
- For spans/generations created via context managers or assigned to variables: use the `.update()` method on the object.
- To update the *currently active* observation in the context (without needing a direct reference to it): use `abv.update_current_span()` or `abv.update_current_generation()`.
`ABVSpan.update()`** / **`ABVGeneration.update()` **parameters****:**
| Parameter | Type | Description | Applies To |
| ----------------------- | ------------------------------- | --------------------------------------------------------------------- | ---------- |
| `input` | `Optional[Any]` | Input data for the operation. | Both |
| `output` | `Optional[Any]` | Output data from the operation. | Both |
| `metadata` | `Optional[Any]` | Additional metadata (JSON-serializable). | Both |
| `version` | `Optional[str]` | Version identifier for the code/component. | Both |
| `level` | `Optional[SpanLevel]` | Severity: `"DEBUG"`, `"DEFAULT"`, `"WARNING"`, `"ERROR"`. | Both |
| `status_message` | `Optional[str]` | A message describing the status, especially for errors. | Both |
| `completion_start_time` | `Optional[datetime]` | Timestamp when the LLM started generating the completion (streaming). | Generation |
| `model` | `Optional[str]` | Name/identifier of the AI model used. | Generation |
| `model_parameters` | `Optional[Dict[str, MapValue]]` | Parameters used for the model call (e.g., temperature). | Generation |
| `usage_details` | `Optional[Dict[str, int]]` | Token usage (e.g., `{"input_tokens": 10, "output_tokens": 20}`). | Generation |
| `cost_details` | `Optional[Dict[str, float]]` | Cost information (e.g., `{"total_cost": 0.0023}`). | Generation |
| `prompt` | `Optional[PromptClient]` | Associated `PromptClient` object from ABV prompt management. | Generation |
```python
from abvdev import ABV
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
with abv.start_as_current_observation(as_type='generation', name="llm-call", model="gpt-5-2025-08-07") as gen:
gen.update(input={"prompt": "Why is the sky blue?"})
# ... make LLM call ...
response_text = "Rayleigh scattering..."
gen.update(
output=response_text,
usage_details={"input_tokens": 5, "output_tokens": 50},
metadata={"confidence": 0.9}
)
# Alternatively, update the current observation in context:
with abv.start_as_current_span(name="data-processing"):
# ... some processing ...
abv.update_current_span(metadata={"step1_complete": True})
# ... more processing ...
abv.update_current_span(output={"result": "final_data"})
```
## Setting Trace Attributes
Trace-level attributes apply to the entire trace, not just a single observation. You can set or update these using:
- The `.update_trace()` method on any `ABVSpan` or `ABVGeneration` object within that trace.
- `abv.update_current_trace()` to update the trace associated with the currently active observation.
**Trace attribute parameters:**
| Parameter | Type | Description |
| ------------ | --------------------- | ---------------------------------------------------------------- |
| `name` | `Optional[str]` | Name for the trace. |
| `user_id` | `Optional[str]` | ID of the user associated with this trace. |
| `session_id` | `Optional[str]` | Session identifier for grouping related traces. |
| `version` | `Optional[str]` | Version of your application/service for this trace. |
| `input` | `Optional[Any]` | Overall input for the entire trace. |
| `output` | `Optional[Any]` | Overall output for the entire trace. |
| `metadata` | `Optional[Any]` | Additional metadata for the trace. |
| `tags` | `Optional[List[str]]` | List of tags to categorize the trace. |
| `public` | `Optional[bool]` | Whether the trace should be publicly accessible (if configured). |
**Example: Setting Multiple Trace Attributes**
```python
from abvdev import ABV
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
with abv.start_as_current_span(name="initial-operation") as span:
# Set trace attributes early
span.update_trace(
user_id="user_xyz",
session_id="session_789",
tags=["beta-feature", "llm-chain"]
)
# ...
# Later, from another span in the same trace:
with span.start_as_current_observation(as_type='generation', name="final-generation") as gen:
# ...
abv.update_current_trace(output={"final_status": "success"}, public=True)
```
## Trace Input/Output Behavior
Trace input and output are automatically set from the **root observation** (first span/generation) by default.
### Default Behavior
```python
from abvdev import ABV
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
with abv.start_as_current_span(
name="user-request",
input={"query": "What is the capital of France?"} # This becomes the trace input
) as root_span:
with abv.start_as_current_observation(
as_type='generation',
name="llm-call",
model="gpt-4o",
input={"messages": [{"role": "user", "content": "What is the capital of France?"}]}
) as gen:
response = "Paris is the capital of France."
gen.update(output=response)
# LLM generation input/output are separate from trace input/output
root_span.update(output={"answer": "Paris"}) # This becomes the trace output
```
### Override Default Behavior
If you need different trace inputs/outputs than the root observation, explicitly set them:
```python
from abvdev import ABV
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
with abv.start_as_current_span(name="complex-pipeline") as root_span:
# Root span has its own input/output
root_span.update(input="Step 1 data", output="Step 1 result")
# But trace should have different input/output (e.g., for LLM-as-a-judge)
root_span.update_trace(
input={"original_query": "User's actual question"},
output={"final_answer": "Complete response", "confidence": 0.95}
)
# Now trace input/output are independent of root span input/output
```
### Critical for LLM-as-a-Judge Features
LLM-as-a-judge and evaluation features typically rely on trace-level inputs and outputs. Make sure to set these appropriately:
```python
from abvdev import ABV, observe
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
@observe()
def process_user_query(user_question: str):
# LLM processing simulation...
answer = f"call_llm: {user_question}"
# Explicitly set trace input/output for evaluation features
abv.update_current_trace(
input={"question": user_question},
output={"answer": answer}
)
return answer
# call function
process_user_query("how are you?")
```
### Trace and Observation IDs
ABV uses W3C Trace Context compliant IDs:
- **Trace IDs**: 32-character lowercase hexadecimal string (16 bytes).
- **Observation IDs (Span IDs)**: 16-character lowercase hexadecimal string (8 bytes).
You can retrieve these IDs:
- `abv.get_current_trace_id()`: Gets the trace ID of the currently active observation.
- `abv.``get_current_observation_id``()`: Gets the ID of the currently active observation.
- `span_obj.trace_id` and `span_obj.id`: Access IDs directly from a `ABVSpan` or `ABVGeneration` object.
For scenarios where you need to generate IDs outside of an active trace (e.g., to link scores to traces/observations that will be created later, or to correlate with external systems), use:
- `ABV.create_trace_id(seed: Optional[str] = None)`(static method): Generates a new trace ID. If a `seed` is provided, the ID is deterministic. Use the same seed to get the same ID. This is useful for correlating external IDs with ABV traces.
```python
from abvdev import ABV
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# Get current IDs
with abv.start_as_current_span(name="my-op") as current_op:
trace_id = abv.get_current_trace_id()
observation_id = abv.get_current_observation_id()
print(f"Current Trace ID: {trace_id}, Current Observation ID: {observation_id}")
print(f"From object: Trace ID: {current_op.trace_id}, Observation ID: {current_op.id}")
# Generate IDs deterministically
external_request_id = "req_12345"
deterministic_trace_id = ABV.create_trace_id(seed=external_request_id)
print(f"Deterministic Trace ID for {external_request_id}: {deterministic_trace_id}")
```
**Linking to Existing Traces (Trace Context)**
If you have a `trace_id` (and optionally a `parent_span_id`) from an external source (e.g., another service, a batch job), you can link new observations to it using the `trace_context` parameter. Note that OpenTelemetry offers native cross-service context propagation, so this is not necessarily required for calls between services that are instrumented with OTEL.
```python
from abvdev import ABV
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
existing_trace_id = "abcdef1234567890abcdef1234567890" # From an upstream service
existing_parent_span_id = "fedcba0987654321" # Optional parent span in that trace
with abv.start_as_current_span(
name="process-downstream-task",
trace_context={
"trace_id": existing_trace_id,
"parent_span_id": existing_parent_span_id # If None, this becomes a root span in the existing trace
}
) as span:
# This span is now part of the trace `existing_trace_id`
# and a child of `existing_parent_span_id` if provided.
print(f"This span's trace_id: {span.trace_id}") # Will be existing_trace_id
pass
```
## Client Management
### **flush()**
Manually triggers the sending of all buffered observations (spans, generations, scores, media metadata) to the ABV API. This is useful in short-lived scripts or before exiting an application to ensure all data is persisted.
```python
from abvdev import ABV
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# ... create traces and observations ...
abv.flush() # Ensures all pending data is sent
```
The `flush()` method blocks until the queued data is processed by the respective background threads.
### shutdown()
Gracefully shuts down the ABV client. This includes:
1. Flushing all buffered data (similar to `flush()`).
2. Waiting for background threads (for data ingestion and media uploads) to finish their current tasks and terminate.
It's crucial to call `shutdown()` before your application exits to prevent data loss and ensure clean resource release. The SDK automatically registers an `atexit` hook to call `shutdown()` on normal program termination, but manual invocation is recommended in scenarios like:
- Long-running daemons or services when they receive a shutdown signal.
- Applications where `atexit` might not reliably trigger (e.g., certain serverless environments or forceful terminations).
```python
from abvdev import ABV
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# ... application logic ...
# Before exiting:
abv.shutdown()
```
# Integrations
## Third-party integrations
The ABV SDK seamlessly integrates with any third-party library that uses OpenTelemetry instrumentation. When these libraries emit spans, they are automatically captured and properly nested within your trace hierarchy. This enables unified tracing across your entire application stack without requiring any additional configuration.
For example, if you're using OpenTelemetry-instrumented databases, HTTP clients, or other services alongside your LLM operations, all these spans will be correctly organized within your traces in ABV.
### Example Anthropic
You can use any third-party, OTEL-based instrumentation library for Anthropic to automatically trace all your Anthropic API calls in ABV.
In this example, we are using the [opentelemetry-instrumentation-anthropic](https://pypi.org/project/opentelemetry-instrumentation-anthropic/).
install packages
```bash
pip install opentelemetry-instrumentation-anthropic, anthropic
```
```python
from anthropic import Anthropic
from opentelemetry.instrumentation.anthropic import AnthropicInstrumentor
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# This will automatically emit OTEL-spans for all Anthropic API calls
AnthropicInstrumentor().instrument()
anthropic_client = Anthropic(api_key="sk-ant-...")
with abv.start_as_current_span(name="myspan"):
# This will be traced as a ABV generation nested under the current span
message = anthropic_client.messages.create(
model="claude-3-7-sonnet-20250219",
max_tokens=1024,
messages=[{"role": "user", "content": "Hello, Claude"}],
)
print(message.content)
# Flush events to ABV in short-lived applications
abv.flush()
```
### Example LlamaIndex
You can use the third-party, OTEL-based instrumentation library for LlamaIndex to automatically trace your LlamaIndex calls in ABV.
In this example, we are using the [openinference-instrumentation-llama-index](https://pypi.org/project/openinference-instrumentation-llama-index/).
install packages
```bash
pip install openinference-instrumentation-llama-index llama-index-llms-openai llama-index -U
```
```python
from abvdev import get_client
from llama_index.llms.openai import OpenAI
from openinference.instrumentation.llama_index import LlamaIndexInstrumentor
import os
os.environ["ABV_API_KEY"] = "sk-abv-..."
os.environ["ABV_HOST"] = "https://app.abv.dev" # "https://eu.app.abv.dev", for EU region
# Your OpenAI key
os.environ["OPENAI_API_KEY"] = "sk-proj-..."
LlamaIndexInstrumentor().instrument()
abv = get_client()
llm = OpenAI(model="gpt-4o")
with abv.start_as_current_span(name="myspan"):
response = llm.complete("Hello, world!")
print(response)
abv.flush()
```
--------------------------------------------------------------------------------
## Python SDK - Advanced Usage
URL: /developer/sdks/python/advanced-usage
The Python SDK provides advanced usage options for your application. This includes data masking, logging, sampling, filtering, and more.
# Masking Sensitive Data
Install package
```bash
pip install abvdev
```
If your trace data (inputs, outputs, metadata) might contain sensitive information (PII, secrets), you can provide a `mask` function during client initialization. This function will be applied to all relevant data before it's sent to ABV.
The `mask` function should accept `data` as a keyword argument and return the masked data. The returned data must be JSON-serializable.
```python
from abvdev import ABV
import re
def pii_masker(data: any, **kwargs) -> any:
# Example: Simple email masking. Implement your more robust logic here.
if isinstance(data, str):
return re.sub(r"[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+", "[EMAIL_REDACTED]", data)
elif isinstance(data, dict):
return {k: pii_masker(data=v) for k, v in data.items()}
elif isinstance(data, list):
return [pii_masker(data=item) for item in data]
return data
abv = ABV(
mask=pii_masker,
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# Now, any input/output/metadata will be passed through pii_masker
with abv.start_as_current_span(name="user-query", input={"email": "test@example.com", "query": "..."}) as span:
# The 'email' field in the input will be masked.
pass
```
# Logging
The ABV SDK uses Python's standard `logging` module. The main logger is named `"abv"`.
To enable detailed debug logging, you can either:
1. Set the `debug=True` parameter when initializing the `ABV` client.
2. Set the `ABV_DEBUG="True"` environment variable.
3. Configure the `"abv"` logger manually:
```python
import logging
abv_logger = logging.getLogger("abv")
abv_logger.setLevel(logging.DEBUG)
```
The default log level for the `abv` logger is `logging.WARNING`.
# Sampling
You can configure the SDK to sample traces by setting the `sample_rate` parameter during client initialization (or via the `ABV_SAMPLE_RATE` environment variable). This value should be a float between `0.0` (sample 0% of traces) and `1.0` (sample 100% of traces).
If a trace is not sampled, none of its observations (spans, generations) or associated scores will be sent to ABV.
```python
from abvdev import ABV
# Sample approximately 20% of traces
abv_sampled = ABV(
sample_rate=0.2,
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
```
# Filtering by Instrumentation Scope
You can configure the SDK to filter out spans from specific instrumentation libraries by using the `blocked_instrumentation_scopes` parameter. This is useful when you want to exclude infrastructure spans while keeping your LLM and application spans.
```python
from abvdev import ABV
# Filter out database spans
abv = ABV(
blocked_instrumentation_scopes=["sqlalchemy", "psycopg"],
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
```
**How it works:**
When third-party libraries create OpenTelemetry spans (through their instrumentation packages), each span has an associated "instrumentation scope" that identifies which library created it. The ABV SDK filters spans at the export level based on these scope names.
You can see the instrumentation scope name for any span in the ABV UI under the span's metadata (`metadata.scope.name`). Use this to identify which scopes you want to filter.
**Cross-Library Span Relationships**
When filtering instrumentation scopes, be aware that blocking certain libraries may break trace tree relationships if spans from blocked and non-blocked libraries are nested together.
For example, if you block parent spans but keep child spans from a separate library, you may see "orphaned" LLM spans whose parent spans were filtered out. This can make traces harder to interpret.
Consider the impact on trace structure when choosing which scopes to filter.
# Isolated TracerProvider
You can configure a separate OpenTelemetry TracerProvider for use with ABV. This creates isolation between ABV tracing and your other observability systems.
**Benefits of isolation:**
- ABV spans won't be sent to your other observability backends (e.g., Datadog, Jaeger, Zipkin)
- Third-party library spans won't be sent to ABV
- Independent configuration and sampling rates
While TracerProviders are isolated, they share the same OpenTelemetry context for tracking active spans. This can cause span relationship issues where:
- A parent span from one TracerProvider might have children from another TracerProvider
- Some spans may appear "orphaned" if their parent spans belong to a different TracerProvider
- Trace hierarchies may be incomplete or confusing
Plan your instrumentation carefully to avoid confusing trace structures.
```python
from opentelemetry.sdk.trace import TracerProvider
from abvdev import ABV
abv_tracer_provider = TracerProvider() # do not set to global tracer provider to keep isolation
abv = ABV(
tracer_provider=abv_tracer_provider,
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
abv.start_span(name="myspan").end() # Span will be isolated from remaining OTEL instrumentation
```
# Using ThreadPoolExecutors or ProcessPoolExecutors
The observe decorator uses Python’s `contextvars` to store the current trace context and to ensure that the observations are correctly associated with the current execution context. However, when using Python’s ThreadPoolExecutors and ProcessPoolExecutors *and* when spawning threads from inside a trace (i.e. the executor is run inside a decorated function) the decorator will not work correctly as the `contextvars `are not correctly copied to the new threads or processes. There is an [existing issue](https://github.com/python/cpython/pull/9688#issuecomment-544304996) in Python’s standard library and a [great explanation](https://github.com/fastapi/fastapi/issues/2776#issuecomment-776659392) in the fastapi repo that discusses this limitation.
The recommended workaround is to pass the parent observation id and the trace ID as a keyword argument to each multithreaded execution, thus re-establishing the link to the parent span or trace:
```python
from concurrent.futures import ThreadPoolExecutor, as_completed
from abvdev import ABV, observe
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
@observe
def execute_task(*args):
return args
@observe
def execute_groups(task_args):
trace_id = abv.get_current_trace_id()
observation_id = abv.get_current_observation_id()
with ThreadPoolExecutor(3) as executor:
futures = [
executor.submit(
execute_task,
*task_arg,
abv_trace_id=trace_id,
abv_parent_observation_id=observation_id,
)
for task_arg in task_args
]
for future in as_completed(futures):
future.result()
return [f.result() for f in futures]
@observe()
def main():
task_args = [["a", "b"], ["c", "d"]]
execute_groups(task_args)
main()
abv.flush()
```
# Distributed tracing
To maintain the trace context across service / process boundaries, please rely on the OpenTelemetry native context propagation across service / process boundaries as much as possible.
Using the `trace_context` argument to ‘force’ the parent child relationship may lead to unexpected trace updates as the resulting span will be treated as a root span server side.
- If you are using multiprocessing, [see here for details on how to propagate the OpenTelemetry context](https://github.com/open-telemetry/opentelemetry-python/issues/2765#issuecomment-1158402076).
- If you are using Pydantic Logfire, please set `distributed_tracing` to `True`.
# Multi-Project Setup (Experimental)
Multi-project setups are experimental
The ABV Python SDK supports routing traces to different projects within the same application by using multiple api keys. This works because the ABV SDK adds a specific span attribute containing the api key to all spans it generates.
**How it works:**
1. **Span Attributes**: The ABV SDK adds a specific span attribute containing the api key to spans it creates
2. **Multiple Processors**: Multiple span processors are registered onto the global tracer provider, each with their respective exporters bound to a specific api key
3. **Filtering**: Within each span processor, spans are filtered based on the presence and value of the api key attribute
**Important Limitation with Third-Party Libraries:**
Third-party libraries that emit OpenTelemetry spans automatically (e.g., HTTP clients, databases, other instrumentation libraries) do **not** have the ABV api key span attribute. As a result:
- These spans cannot be routed to a specific project
- They are processed by all span processors and sent to all projects
- All projects will receive these third-party spans
**Why is this experimental?**
This approach requires that the `api_key` parameter be passed to all ABV SDK executions across all integrations to ensure proper routing, and third-party spans will appear in all projects.
## Initialization
To set up multiple projects, initialize separate ABV clients for each project:
```python
from abvdev import ABV
# Initialize clients for different projects
project_a_client = ABV(
api_key="sk-abv-project-a-...",
host="https://app.abv.dev"
# or https://eu.app.abv.dev
)
project_b_client = ABV(
api_key="sk-abv-project-b-...",
host="https://app.abv.dev"
# or https://eu.app.abv.dev
)
```
## Integration Usage
For all integrations in multi-project setups, you must specify the `api_key` parameter to ensure traces are routed to the correct project.
**Observe Decorator:**
Pass `abv_api_key`as a keyword argument to the *top-most *observed function (not the decorator). Nested decorated functions will automatically pick up the api key from the execution context they are currently into. Also, calls to `get_client` will be also aware of the current `abv_api_key` in the decorated function execution context, so passing the abv\_api\_key here again is not necessary.
```python
from abvdev import observe, get_client
@observe
def nested():
# get_client call is context aware
# if it runs inside another decorated function that has
# abv_api_key passed, it does not need passing here again
get_client().update_current_trace(user_id='myuser')
@observe
def process_data_for_project_a(data):
# passing `abv_api_key` here again is not necessarily
# as it is stored in execution context
nested()
return {"processed": data}
@observe
def process_data_for_project_b(data):
# passing `abv_api_key` here again is not necessarily
# as it is stored in execution context
nested()
return {"enhanced": data}
# Route to Project A
# Top-most decorated function needs `abv_api_key` kwarg
result_a = process_data_for_project_a(
data="input data",
abv_api_key="sk-abv-project-a-..."
)
# Route to Project B
# Top-most decorated function needs `abv_api_key` kwarg
result_b = process_data_for_project_b(
data="input data",
abv_api_key="sk-abv-project-b-..."
)
```
**Important Considerations:**
- Every ABV SDK execution across all integrations must include the appropriate api key parameter
- Missing api key parameters may result in traces being routed to the default project or lost
- Third-party OpenTelemetry spans (from HTTP clients, databases, etc.) will appear in all projects since they lack the ABV api key attribute
# Passing *completion\_start\_time* for TTFT tracking
If you are using the Python SDK to manually create generations, you can pass the
`completion_start_time parameter`. This allows abv to calculate the time to first token (TTFT) for you.
```python
from abvdev import ABV
import datetime
import time
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# Start observation with specific type
with abv.start_as_current_observation(
as_type="generation",
name="TTFT-Generation"
) as generation:
# simulate LLM time to first token
time.sleep(3)
# Update the generation with the time the model started to generate
generation.update(
completion_start_time=datetime.datetime.now(),
output="some response",
)
# Flush events in short-lived applications
abv.flush()
```
# Observation Types
ABV supports multiple observation types to provide context for different components of LLM applications. The full list of the observation types is document here: Observation types.
## Setting observation types with the *@observe* decorator
By setting the as\_type parameter in the @observe decorator, you can specify the observation type for a method:
```python
from abvdev import observe
# Tool calls to external services
@observe(as_type="tool")
def retrieve_context(query):
results = vector_store.get(query)
return results
```
The context manager approach provides automatic resource cleanup:
```python
from abvdev import get_client
abv = get_client()
def process_with_context_managers():
with abv.start_as_current_observation(
as_type="chain",
name="retrieval-pipeline",
) as chain:
# Retrieval step
with abv.start_as_current_observation(
as_type="retriever",
name="vector-search",
) as retriever:
search_results = perform_vector_search("user question")
retriever.update(output={"results": search_results})
```
--------------------------------------------------------------------------------
## Python SDK - Evaluations
URL: /developer/sdks/python/evaluations
The ABV Python SDK provides ways to evaluate your application. You can add custom scores to your traces and observations, or use the SDK to execute Dataset Runs.
This page shows the evaluation methods that are supported by the Python SDK. Please refer to the [Evaluation Overview](/developer/evaluations/overview) for more information on how to evaluate your application in ABV.
# Scoring traces and observations
Install package
```bash
pip install abvdev
```
### ABVSpan / ABVGeneration object methods
- span\_or\_generation\_obj.score(): Scores the specific observation object.
- span\_or\_generation\_obj.score\_trace(): Scores the entire trace to which the object belongs.
```python
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
with abv.start_as_current_observation(as_type='generation', name="summary_generation") as gen:
# ... LLM call ...
gen.update(output="summary text...")
# Score this specific generation
gen.score(name="conciseness", value=0.8, data_type="NUMERIC")
# Score the overall trace
gen.score_trace(name="user_feedback_rating", value="positive", data_type="CATEGORICAL")
```
### Context-aware methods
- abv.score\_current\_span(): Scores the currently active observation in the context.
- abv.score\_current\_trace(): Scores the trace of the currently active observation.
```python
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# imitate successful task completion
task_is_fully_successful = True
with abv.start_as_current_span(name="complex_task") as task_span:
# ... perform task ...
abv.score_current_span(name="task_component_quality", value=True, data_type="BOOLEAN")
# ...
if task_is_fully_successful:
abv.score_current_trace(name="overall_success", value=1.0, data_type="NUMERIC")
```
### Low-level method
- Creates a score for a specified trace\_id and optionally observation\_id.
- Useful when IDs are known, or for scoring after the trace/observation has completed.
```python
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
abv.create_score(
name="fact_check_accuracy",
value=0.95, # Can be float for NUMERIC/BOOLEAN, string for CATEGORICAL
trace_id="your_trace_id",
observation_id="your_observation_id", # Optional: if scoring a specific observation
session_id="your_session_id", # Optional: if scoring a specific session
data_type="NUMERIC", # "NUMERIC", "BOOLEAN", "CATEGORICAL"
comment="Source verified for 95% of claims."
)
```
## **Score Parameters:**
| Parameter | Type | Description |
| ---------------- | ------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| `name` | `str` | Name of the score (e.g., "relevance", "accuracy"). **Required.** |
| `value` | `Union[float, str]` | Score value. Float for `NUMERIC`/`BOOLEAN`, string for `CATEGORICAL`. **Required.** |
| `trace_id` | `str` | ID of the trace to associate with (for `create_score`). **Required.** |
| `session_id` | `Optional[str]` | ID of the specific session to score (for
`create_score`).
|
| `observation_id` | `Optional[str]` | ID of the specific observation to score (for `create_score`). |
| `score_id` | `Optional[str]` | Custom ID for the score (auto-generated if None). |
| `data_type` | `Optional[ScoreDataType]` | `"NUMERIC"`, `"BOOLEAN"`, or `"CATEGORICAL"`. Inferred if not provided based on value type and score config on server. |
| `comment` | `Optional[str]` | Optional comment or explanation for the score. |
| `config_id` | `Optional[str]` | Optional ID of a pre-defined score configuration in ABV. |
See [Scoring](/developer/evaluations/scores-data-model) for more details.
# Datasets
[ABV Datasets](/developer/evaluations/datasets) are essential for evaluating and testing your LLM applications by allowing you to manage collections of inputs and their expected outputs.
## Interacting with Datasets
- **Creating**: You can programmatically create new datasets with `abv.create_dataset(...)` and add items to them using `abv.create_dataset_item(...)`.
- **Fetching**: Retrieve a dataset and its items using `abv.get_dataset(name: str)`. This returns a `DatasetClient` instance, which contains a list of `DatasetItemClient` objects (accessible via `dataset.items`). Each `DatasetItemClient` holds the `input`, `expected_output`, and `metadata` for an individual data point.
```python
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
# Fetch an existing dataset
dataset = abv.get_dataset(name="my-eval-dataset")
for item in dataset.items:
print(f"Input: {item.input}, Expected: {item.expected_output}")
# Briefly: Creating a dataset and an item
new_dataset = abv.create_dataset(name="new-summarization-tasks")
abv.create_dataset_item(
dataset_name="new-summarization-tasks",
input={"text": "Long article..."},
expected_output={"summary": "Short summary."}
)
```
## Linking Traces to Dataset Items for Runs
The most powerful way to use datasets is by linking your application's executions (traces) to specific dataset items when performing an evaluation run. See our [datasets](/developer/evaluations/datasets) for more details. The `DatasetItemClient.run()` method provides a context manager to streamline this process.
**How **`item.run()`** works:**
When you use `with item.run(run_name="your_eval_run_name") as root_span:`:
1. **Trace Creation**: A new ABV trace is initiated specifically for processing this dataset item within the context of the named run.
2. **Trace Naming & Metadata**:
- The trace is automatically named (e.g., "Dataset run: your\_eval\_run\_name").
- Essential metadata is added to this trace, including `dataset_item_id` (the ID of `item`), `run_name`, and `dataset_id`.
3. **DatasetRunItem Linking**: The SDK makes an API call to ABV to create a `DatasetRunItem`. This backend object formally links:
- The `dataset_item_id`
- The `trace_id` of the newly created trace
- The provided `run_name`
- Any `run_metadata` or `run_description` you pass to `item.run()`.
This linkage is what populates the "Runs" tab for your dataset in the ABV UI, allowing you to see all traces associated with a particular evaluation run.
4. **Contextual Span**: The context manager yields `root_span`, which is a `ABVSpan` object representing the root span of this new trace.
5. **Automatic Nesting**: Any ABV observations (spans or generations) created *inside* the `with` block will automatically become children of `root_span` and thus part of the trace linked to this dataset item and run.
**Example:**
```python
from abvdev import ABV
# ABV client initialization
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
dataset_name = "qna-eval"
current_run_name = "qna_model_v3_run_05_20" # Identifies this specific evaluation run
# Assume 'my_qna_app' is your instrumented application function
def my_qna_app(question: str, context: str, item_id: str, run_name: str):
with abv.start_as_current_observation(
as_type='generation',
name="qna-llm-call",
input={"question": question, "context": context},
metadata={"item_id": item_id, "run": run_name}, # Example metadata for the generation
model="gpt-5-2025-08-07"
) as generation:
# Simulate LLM call
answer = f"Answer to '{question}' using context." # Replace with actual LLM call
generation.update(output={"answer": answer})
# Update the trace with the input and output
generation.update_trace(
input={"question": question, "context": context},
output={"answer": answer},
)
return answer
dataset = abv.get_dataset(name=dataset_name) # Fetch your pre-populated dataset
for item in dataset.items:
print(f"Running evaluation for item: {item.id} (Input: {item.input})")
# Use the item.run() context manager
with item.run(
run_name=current_run_name,
run_metadata={"model_provider": "OpenAI", "temperature_setting": 0.7},
run_description="Evaluation run for Q&A model v3 on May 20th"
) as root_span: # root_span is the root span of the new trace for this item and run.
# All subsequent abv operations within this block are part of this trace.
# Call your application logic
generated_answer = my_qna_app(
question=item.input["question"],
context=item.input["context"],
item_id=item.id,
run_name=current_run_name
)
print(f" Item {item.id} processed. Trace ID: {root_span.trace_id}")
# Optionally, score the result against the expected output
if item.expected_output and generated_answer == item.expected_output.get("answer"):
root_span.score_trace(name="exact_match", value=1.0)
else:
root_span.score_trace(name="exact_match", value=0.0)
print(f"\nFinished processing dataset '{dataset_name}' for run '{current_run_name}'.")
```
By using `item.run()`, you ensure each dataset item's processing is neatly encapsulated in its own trace, and these traces are aggregated under the specified `run_name` in the ABV UI. This allows for systematic review of results, comparison across runs, and deep dives into individual processing traces.
--------------------------------------------------------------------------------
## Python SDK - Troubleshooting
URL: /developer/sdks/python/troubleshooting
This page addresses frequently asked questions and common troubleshooting topics for the ABV Python SDK.
For bug reports, please open a ticket on [bug report form](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiJlZmZmN2JhYy0xMzZlLTNkOTUtOGIxYi1hMWQ3NWNkZWM3NGYiLCJpYXQiOjE3NTY4NDU1MTk1NzN9._OjNB4ESzMwKHLbUJPaQOjVQQqB7aCoKiCaFxhBrFoo). For general questions or support, visit our support page.
- **Authentication Issues**:
- Ensure `ABV_API_KEY` and `ABV_HOST` (if not using default host) are correctly set either as environment variables or in the `ABV()` constructor.
- Use `abv.auth_check()` after initialization to verify credentials. Do not use this in production as this method waits for a response from the server.
- **No Traces Appearing**:
- Check if `tracing_enabled` is `True` (default).
- Verify `sample_rate` is not `0.0`.
- Ensure `abv.shutdown()` is called or the program exits cleanly to allow `atexit` hooks to flush data. Manually call `abv.flush()` to force data sending.
- Enable debug logging (`debug=True` or `ABV_DEBUG="True"`) to see SDK activity and potential errors during exporting.
- **Incorrect Nesting or Missing Spans**:
- Ensure you are using context managers (`with abv.start_as_current_span(...)`) for proper context propagation.
- If manually creating spans (`abv.start_span()`), ensure they are correctly ended with `.end()`.
- In async code, ensure context is not lost across `await` boundaries if not using ABV's async-compatible methods.
- **Media Not Appearing**:
- Ensure `ABVMedia` objects are correctly initialized and passed in `input`, `output`, or `metadata`.
- Check debug logs for any media upload errors. Media uploads happen in background threads.
If you encounter persistent issues, please:
1. Enable debug logging to gather more information.
2. Check the ABV status page.
3. Raise an issue to our [support](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiJlZmZmN2JhYy0xMzZlLTNkOTUtOGIxYi1hMWQ3NWNkZWM3NGYiLCJpYXQiOjE3NTY4NDU1MTk1NzN9._OjNB4ESzMwKHLbUJPaQOjVQQqB7aCoKiCaFxhBrFoo) with details about your setup, SDK version, code snippets, and debug logs.
--------------------------------------------------------------------------------
## Python SDK (v3)
URL: /developer/references/python-sdk-v3
# Python SDK API Reference
The complete API reference for the ABV Python SDK v3 is available through the SDK's documentation.
## Getting Started
To get started with the Python SDK, see:
- [Python SDK Overview](/developer/sdks/python/overview)
- [Python SDK Setup](/developer/sdks/python/setup)
- [Python SDK Instrumentation](/developer/sdks/python/instrumentation)
- [Python SDK Advanced Usage](/developer/sdks/python/advanced-usage)
## Installation
```bash
pip install abvdev
```
## Core Classes and Methods
### ABV Client
The main client for interacting with ABV:
```python
from abvdev import ABV, get_client
# Initialize client
abv = ABV(
api_key="sk-abv-...",
host="https://app.abv.dev"
)
# Or use the singleton
client = get_client()
```
### @observe Decorator
Decorator for automatic function tracing:
```python
from abvdev import observe
@observe
def my_function(arg1, arg2):
# Function automatically traced
return result
```
### Context Managers
For manual span management:
```python
from abvdev import get_client
abv = get_client()
with abv.start_as_current_span(name="my-span") as span:
span.update(input="input data", output="output data")
```
## Additional Resources
For comprehensive API documentation, examples, and advanced usage:
- [Python SDK Documentation](/developer/sdks/python/overview)
- [Troubleshooting Guide](/developer/sdks/python/troubleshooting)
- [GitHub Repository](https://github.com/abv-dev)
--------------------------------------------------------------------------------
## JavaScript/TypeScript SDK
URL: /developer/references/js-sdk
# JavaScript/TypeScript SDK API Reference
The complete API reference for the ABV JavaScript/TypeScript SDK is available through the SDK's documentation.
## Getting Started
To get started with the JS/TS SDK, see:
- [TypeScript SDK Overview](/developer/sdks/js-ts/overview)
- [TypeScript SDK Setup](/developer/sdks/js-ts/setup)
- [TypeScript SDK Instrumentation](/developer/sdks/js-ts/instrumentation)
- [TypeScript SDK Advanced Configuration](/developer/sdks/js-ts/advanced-configuration)
## Installation
```bash
npm install @abvdev/tracing @abvdev/otel @opentelemetry/sdk-node
```
## Core Packages
### @abvdev/tracing
Core tracing functions for creating observations:
```typescript
import { startActiveObservation, startObservation } from "@abvdev/tracing";
// Active observation with automatic context management
await startActiveObservation("my-trace", async (span) => {
span.update({
input: "input data",
output: "output data"
});
});
// Manual observation
const observation = startObservation("my-observation", {
input: "data"
}, { asType: "generation" });
observation.update({ output: "result" }).end();
```
### @abvdev/otel
OpenTelemetry integration for exporting traces to ABV:
```typescript
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
const sdk = new NodeSDK({
spanProcessors: [new ABVSpanProcessor()],
});
sdk.start();
```
### @abvdev/client
Client for prompts, datasets, and other ABV features:
```typescript
import { ABVClient } from "@abvdev/client";
const abv = new ABVClient({
apiKey: "sk-abv-...",
baseUrl: "https://app.abv.dev"
});
```
## Additional Resources
For comprehensive API documentation, examples, and advanced usage:
- [TypeScript SDK Documentation](/developer/sdks/js-ts/overview)
- [Cookbook with Examples](/developer/sdks/js-ts/cookbook)
- [Troubleshooting Guide](/developer/sdks/js-ts/troubleshooting-faq)
- [GitHub Repository](https://github.com/abv-dev)
--------------------------------------------------------------------------------
## Security & Compliance Overview
URL: /developer/references/security-compliance-overview
# Security & Compliance Overview
ABV is committed to providing a secure and compliant platform for LLM observability. We implement industry-standard security measures and maintain multiple compliance certifications to protect your data.
## Security Features
Learn about our security infrastructure and practices:
- [**Authentication & Authorization**](/security/security/authentication-authorization) - SSO, RBAC, and access controls
- [**Encryption**](/security/security/encryption) - Data encryption at rest and in transit
- [**Data Regions & Availability**](/security/security/data-regions-availability) - Geographic data residency options
- [**Penetration Testing**](/security/security/penetration-testing) - Regular security assessments
- [**Vulnerability Management**](/security/security/vulnerability-management) - Continuous security monitoring
- [**Incident Response**](/security/security/incident-breach) - Security incident procedures
- [**Responsible Disclosure**](/security/security/responsible-disclosure) - Security vulnerability reporting
## Compliance Certifications
ABV maintains compliance with major industry standards:
- [**HIPAA Compliance**](/security/compliance/hipaa-compliance) - Healthcare data protection
- [**ISO 27001 Compliance**](/security/compliance/iso-27001-compliance) - Information security management
- [**ISO 42001 Compliance**](/security/compliance/iso-42001-compliance) - AI management system
- [**NIS2 Compliance**](/security/compliance/nis2-compliance) - Network and information security
- [**Compliance Policies**](/security/compliance/policies) - Policy documentation
## Privacy & Data Protection
Understand how we handle your data:
- [**GDPR Compliance**](/security/privacy/gdpr-compliance) - EU data protection compliance
- [**Managing Personal Data**](/security/privacy/managing-personal-data) - Data subject rights
- [**Data Processing Agreement**](/security/privacy/data-processing-agreement) - Legal framework for data processing
- [**Privacy FAQ**](/security/privacy/faq) - Common privacy questions
## Platform Security
Platform-level security features:
- [**Role-Based Access Controls**](/developer/platform/administration/role-based-access-controls) - Fine-grained permissions
- [**Audit Logs**](/developer/platform/administration/audit-logs) - Activity monitoring and compliance
- [**Data Retention**](/developer/platform/administration/data-retention) - Configurable retention policies
- [**Data Deletion**](/developer/platform/administration/data-deletion) - Secure data removal
## Additional Resources
For specific security questions or compliance documentation:
- Contact our security team: security@abv.dev
- View our [Security FAQ](/security/security/faq)
- Review our complete [Security Documentation](/security/overview)
--------------------------------------------------------------------------------
## Data Retention
URL: /developer/platform/administration/data-retention
With ABV's Data Retention feature, you can control how long your event data (Traces, Observations, Scores, and Media Assets) is stored in ABV. Data retention is configured on a project level, and we accept a number of days with a minimum of 3 days. Project owners and administrators can change the data retention setting within the Project Settings view.
Dat Retention configuration is available for Enterprise plan:
By default, ABV stores event data (Traces, Observations, Scores, and Media Assets) indefinitely.
# Details
On a nightly basis, ABV selects traces, observations, scores, and media assets that are older than the configured retention period and deletes them.
We use the following properties per entity to decide whether they are outside the retention window:
- **Traces**: `timestamp`
- **Observations**: `start_time`
- **Scores**: `timestamp`
- **Media Assets**: `created_at`
Deleted assets cannot be recovered.
--------------------------------------------------------------------------------
## SCIM & Organization-Key Scoped API Routes
URL: /developer/platform/administration/scim-org-key-scoped-api-routes
Via organization-scoped API keys, you can administer projects, users, and project/organization memberships (see [RBAC docs](/developer/platform/administration/role-based-access-controls)).
You can use these endpoints to automate project and user management on your ABV organization.
This documentation covers organization management APIs, SCIM-compliant user provisioning endpoints, and includes a comprehensive guide for setting up Okta authentication and user provisioning with ABV.
# Authentication
Authenticate with the API using [Basic Auth](https://en.wikipedia.org/wiki/Basic_access_authentication).
Organization scoped API keys can be created via the Organization Management API or in the Organization Settings within the ABV UI.
Example:
```curl
curl -u public-key:api-key https://app.abv.dev/api/public/projects/{projectId}/apiKeys
```
# Organization Management
All applicable endpoints are marked with `(requires organization-scoped API key)`.
Those include the following routes:
- `POST /api/public/projects`
- `PUT /api/public/projects/{projectId}`
- `DELETE /api/public/projects/{projectId}`
- `GET /api/public/projects/{projectId}/apiKeys`
- `POST /api/public/projects/{projectId}/apiKeys`
- `DELETE /api/public/projects/{projectId}/apiKeys/{apiKeyId}`
- `PUT /api/public/organizations/memberships`
- `GET /api/public/organizations/memberships`
- `PUT /api/public/projects/{projectId}/memberships`
- `DELETE /api/public/projects/{projectId}/memberships`
# User Management via SCIM
In addition, we implement the following [SCIM](https://datatracker.ietf.org/doc/html/rfc7642) compliant endpoints.
Use `/api/public/scim` as the base URI for them.
To create a new user within ABV, you can use the SCIM-style endpoints and `POST /Users`.
This will create a new user if the email does not exist yet. Then it will add the user to the organization with role `NONE`.
Afterward, the role can be updated using the membership endpoints either on an organization or a project level (see above).
To remove a user from an organization, call the `DELETE /Users/{id}` endpoint.
This will not delete the user itself, only its membership with the organization.
You can either supply an initial password for users via the API and share it with them, or use Single Sign-On (SSO) to authenticate users. In the latter case, you need to:
- ABV: configure an Enterprise SSO provider ().
The following SCIM endpoints are available:
- `GET /ServiceProviderConfig`
- `GET /ResourceTypes`
- `GET /Schemas`
- `GET /Users`
- `POST /Users`
- `GET /Users/{id}`
- `DELETE /Users/{id}`
# Okta Setup Guide
This guide will cover how to setup Okta authentication and user provisioning for ABV.
Authentication and provisioning are two separate steps in Okta. Follow the one that is applicable to your use case.
## Authentication
For authentication, ABV supports the [OpenID Connect (OIDC)](https://openid.net/connect/) protocol.
This allows you to use Okta as an identity provider for ABV. To set up Okta authentication, follow these steps:
1. **Create an Okta Application**:
- Log in to your Okta admin console.
- Navigate to **Applications** > **Create App Integration**.
- Choose **OIDC - OpenID Connect** as the sign-in method and click **Next**.
- Choose **Web Application** as the application type and click **Next**.
- Fill in the application settings. Use one of the ABV domains.
- **App name**: `ABV`
- **Login redirect URIs**: `https://your-abv-domain.dev/api/auth/callback/okta`
- **Logout redirect URIs**: `https://your-abv-domain.dev/`
- Click **Save**.
2. **Setup ABV**:
- Please reach out to the ABV support team at [support@abv.dev](mailto:support@abv.dev) to configure Okta SSO.
- Once the SSO is configured, you can type in your email on the sign-in page and are automatically redirected to Okta.
3. **Test the Integration**:
- Assign users to your Okta application and test the login flow.
- For any issues, please contact [support](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiJlZmZmN2JhYy0xMzZlLTNkOTUtOGIxYi1hMWQ3NWNkZWM3NGYiLCJpYXQiOjE3NTY4NDU1MTk1NzN9._OjNB4ESzMwKHLbUJPaQOjVQQqB7aCoKiCaFxhBrFoo).
## User Provisioning
For user provisioning, ABV supports the SCIM 2.0 protocol. To setup user provisioning in Okta, follow these steps:
1. **Create a SAML/SCIM Application**:
- Log in to your Okta admin console.
- Navigate to **Applications** > **Create App Integration**.
- Choose **SAML 2.0** as the sign-in method and click **Next**.
- Fill in the application settings. Use one of the ABV domains.
- **App name**: `ABV SCIM`
- **Single sign-on URL**: `https://your-abv-domain.dev/api/auth/callback/okta`
- **Audience URI**: `abv`
- Click **Next** and then **Finish**.
2. **Configure SCIM Settings**:
- In the **General** tab, set `Provisioning` to SCIM.
- In the **Provisioning** tab, edit your **SCIM Connection**.
- Enter your credentials:
- **SCIM connector base URL**: `https://your-abv-domain.dev/api/public/scim`
- **Unique identifier field for users**: `userName`
- **Supported provisioning actions**: `Import new Users and Profile Updates`, `Push New Users`, `Push Profile Updates`
- **Basic Auth - Username**: Use a public key from your Organization settings.
- **Basic Auth - Password**: Use a private key from your Organization settings.
- Test the API credentials and press **Save**.
3. **Configure Provisioning**:
- In the **Provisioning** tab, enable the following options:
- **Create Users**
- **Update User Attributes**
- **Deactivate Users**
- Click **Save**.
4. **Add Default User Permissions** (Optional):
- In the **Provisioning** tab, go to the Profile Editor and add a new `roles` attribute:
- **Data type**: `string array`
- **Display Name**: ABV Roles
- **Variable Name**: `roles`
- **External Name**: `roles`
- **External Namespace**: `urn:ietf:params:scim:schemas:core:2.0:User`
- **Attribute members**: `NONE`, `VIEWER`, `MEMBER`, `ADMIN`
- **Attribute type**: `Personal`
- In the **Provisioning** tab, modify the `roles` attribute to set default permissions for new users.
- You can set it for all users of the application to provide a default. Set it to "NONE", "VIEWER", "MEMBER", or "ADMIN".
5. **Assign Users**:
- Navigate to the **Assignments** tab.
- Click **Assign** > **Assign to People**.
- Select the users you want to assign to the ABV SCIM application. You can overwrite the role here.
- Click **Done** and then **Save**.
- Users should appear as Member within your ABV Organization.
--------------------------------------------------------------------------------
## Usage Alerts
URL: /developer/platform/administration/usage-alerts
ABV allows you to configure usage alerts to receive email notifications when the number of events in your organization exceeds a predefined threshold. This helps you monitor your usage and take action before you are invoiced. Navigate to your organization settings and the **Billing** tab to configure usage alerts.
Usage alerts only consider usage that was added after the alert was configured or last updated. The alerts trigger at most once per billing cycle.
--------------------------------------------------------------------------------
## LLM Connections
URL: /developer/platform/administration/llm-connections
LLM connections are used to call models in the ABV Playground or for LLM-as-a-Judge evaluations.
# Setup
Navigate to your `Project Settings` > `LLM Connections` and **click on** `Add new LLM API key`.
Enter the name of the LLM connection and the API key for the model you want to use.
# Supported providers
The ABV platform is currently supporting the following LLM providers:
- OpenAI
- Azure OpenAI
- Anthropic
- Google AI Studio
- Google Vertex AI
- Amazon Bedrock
### Supported models
Currently the playground supports the following models by default. You may configure additional custom model names when adding your LLM API Key in the ABV project settings, e.g. when using a custom model or proxy.
- **Any model that supports the OpenAI API schema:** The Playground and LLM-as-a-Judge evaluations can be used by any framework that supports the OpenAI API schema such as Groq, OpenRouter, Vercel AI Gateway, LiteLLM, Hugging Face, and more. Just replace the API Base URL with the appropriate endpoint for the model you want to use and add the providers API keys for authentication.
- **OpenAI / Azure OpenAI:** o3, o3-2025-04-16, o4-mini, o4-mini-2025-04-16, gpt-4.1, gpt-4.1-2025-04-14, gpt-4.1-mini-2025-04-14, gpt-4.1-nano-2025-04-14, gpt-4o, gpt-4o-2024-08-06, gpt-4o-2024-05-13, gpt-4o-mini, gpt-4o-mini-2024-07-18, o3-mini, o3-mini-2025-01-31, o1-preview, o1-preview-2024-09-12, gpt-4-turbo-preview, gpt-4-1106-preview, gpt-4-0613, gpt-4-0125-preview, gpt-4, gpt-3.5-turbo-16k-0613, gpt-3.5-turbo-16k, gpt-3.5-turbo-1106, gpt-3.5-turbo-0613, gpt-3.5-turbo-0301, gpt-3.5-turbo-0125, gpt-3.5-turbo
- **Anthropic:** claude-3-7-sonnet-20250219, claude-3-5-sonnet-20241022, claude-3-5-sonnet-20240620, claude-3-opus-20240229, claude-3-sonnet-20240229, claude-3-5-haiku-20241022, claude-3-haiku-20240307, claude-2.1, claude-2.0, claude-instant-1.2
- **Google Vertex AI:** gemini-2.5-pro-exp-03-25, gemini-2.0-pro-exp-02-05, gemini-2.0-flash-001, gemini-2.0-flash-lite-preview-02-05, gemini-2.0-flash-exp, gemini-1.5-pro, gemini-1.5-flash, gemini-1.0-pro. You may also add additional model names supported by Google Vertex AI platform and enabled in your GCP account through the \`Custom model names\` section in the LLM API Key creation form.
- **Google AI Studio:** gemini-2.5-pro-exp-03-25, gemini-2.0-pro-exp-02-05, gemini-2.0-flash-001, gemini-2.0-flash-lite-preview-02-05, gemini-2.0-flash-exp, gemini-1.5-pro, gemini-1.5-flash, gemini-1.0-pro
- **Amazon Bedrock:** All Amazon Bedrock models are supported. The required permission on AWS is \`bedrock\:InvokeModel\`.
You may connect to third party LLM providers if their API schema implements the schema of one of our supported provider adapters. For example, you may connect to Mistral by using the OpenAI adapter in ABV to connect to Mistral's OpenAI compliant API.
# Advanced configurations
## Additional provider options
Provider options are [Playground](/developer/basic-features/prompt-playground) or during [LLM-as-a-Judge](/developer/evaluations/llm-as-a-judge) Evaluator setup.
LLM calls from a created LLM connection can be configured with a specific set of parameters, such as `temperature`, `top_p`, and `max_tokens`.
However, many LLM providers allow for additional parameters, including `reasoning_effort`, `service_tier`, and others when invoking a model. These parameters often differ between providers.
You can provide additional configurations as a JSON object for all LLM invocations. In the model parameters settings, you will find a "provider options" field at the bottom. This field allows you to enter specific key-value pairs accepted by your LLM provider's API endpoint.
Please see your providers API reference for what additional fields are supported:
- [Anthropic Messages API Reference](https://docs.anthropic.com/en/api/messages)
- [OpenAI Chat Completions API Reference](https://platform.openai.com/docs/api-reference/chat/create)
This feature is currently available for the adapters for:
- Anthropic
- OpenAI
- AWS (Amazon Bedrock)
Example for forcing reasoning effort `minimal` on a OpenAI gpt-5 invocation:
## Connecting via proxies
You can use an LLM proxy to power LLM-as-a-judge or the Playground in ABV. Please create an LLM API Key in the project settings and set the base URL to resolve to your proxy's host. The proxy must accept the API format of one of our adapters and support tool calling.
For OpenAI compatible proxies, here is an example tool calling request that must be handled by the proxy in OpenAI format to support LLM-as-a-judge in ABV:
```bash
curl -X POST 'https:///chat/completions' \
-H 'accept: application/json' \
-H 'content-type: application/json' \
-H 'authorization: Bearer ' \
-H 'x-test-header-1: ' \
-H 'x-test-header-2: ' \
-d '{
"model": "",
"temperature": 0,
"top_p": 1,
"frequency_penalty": 0,
"presence_penalty": 0,
"max_tokens": 256,
"n": 1,
"stream": false,
"tools": [
{
"type": "function",
"function": {
"name": "extract",
"parameters": {
"type": "object",
"properties": {
"score": {
"type": "string"
},
"reasoning": {
"type": "string"
}
},
"required": [
"score",
"reasoning"
],
"additionalProperties": false,
"$schema": "http://json-schema.org/draft-07/schema#"
}
}
}
],
"tool_choice": {
"type": "function",
"function": {
"name": "extract"
}
},
"messages": [
{
"role": "user",
"content": "Evaluate the correctness of the generation on a continuous scale from 0 to 1. A generation can be considered correct (Score: 1) if it includes all the key facts from the ground truth and if every fact presented in the generation is factually supported by the ground truth or common sense.\n\nExample:\nQuery: Can eating carrots improve your vision?\nGeneration: Yes, eating carrots significantly improves your vision, especially at night. This is why people who eat lots of carrots never need glasses. Anyone who tells you otherwise is probably trying to sell you expensive eyewear or does not want you to benefit from this simple, natural remedy. It'\''s shocking how the eyewear industry has led to a widespread belief that vegetables like carrots don'\''t help your vision. People are so gullible to fall for these money-making schemes.\nGround truth: Well, yes and no. Carrots won'\''t improve your visual acuity if you have less than perfect vision. A diet of carrots won'\''t give a blind person 20/20 vision. But, the vitamins found in the vegetable can help promote overall eye health. Carrots contain beta-carotene, a substance that the body converts to vitamin A, an important nutrient for eye health. An extreme lack of vitamin A can cause blindness. Vitamin A can prevent the formation of cataracts and macular degeneration, the world'\''s leading cause of blindness. However, if your vision problems aren'\''t related to vitamin A, your vision won'\''t change no matter how many carrots you eat.\nScore: 0.1\nReasoning: While the generation mentions that carrots can improve vision, it fails to outline the reason for this phenomenon and the circumstances under which this is the case. The rest of the response contains misinformation and exaggerations regarding the benefits of eating carrots for vision improvement. It deviates significantly from the more accurate and nuanced explanation provided in the ground truth.\n\n\n\nInput:\nQuery: {{query}}\nGeneration: {{generation}}\nGround truth: {{ground_truth}}\n\n\nThink step by step."
}
]
}'
```
--------------------------------------------------------------------------------
## Role-Based Access Controls in ABV
URL: /developer/platform/administration/role-based-access-controls
The role-based access control (RBAC) in ABV is based on organizations, projects, and roles:
- `Organizations` are the top-level entities that contain projects.
- `Projects` group all ABV data to allow for fine-grained role-based access control (RBAC).
- `Roles` define the permissions of users within an organization and project:
- By default, users get assigned a role on the organizational level.
- For more fine-grained control, users can be assigned project-roles. This is useful when you want to differentiate permissions for different projects within the same organization.
`API Keys` are used to authenticate with the ABV API. They are associated with a project and can be used to access the project's data programmatically. API keys are not tied to a user.
# Access Organizations and Projects
You can easily switch between organizations and projects using the dropdowns in the top navigation bar.
# Roles and Scopes
- `Owner`: has all permissions
- `Admin`: can edit the project settings and grant access to other users
- `Member`: can view all metrics & create scores, but cannot configure the project
- `Viewer`: view-only access to the project and organization, most of the configuration is hidden
- `None`: no default access to the organization, to be used when user should have access to a single project only
Organization-level scopes
| OWNER | ADMIN | MEMBER | VIEWER | NONE |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- | ------ | ---- |
| abvCloudBilling\:CRUD
organization\:CRUD\_apiKeys
organization\:delete
organization\:update
organizationMembers\:CUD
organizationMembers\:read
projects\:create
projects\:transfer\_org | organization\:CRUD\_apiKeys
organization\:update
organizationMembers\:CUD
organizationMembers\:read
projects\:create
projects\:transfer\_org | organizationMembers\:read
| | |
Project-level scopes
| OWNER | ADMIN | MEMBER | VIEWER | NONE |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---- |
| annotationQueueAssignments\:CUD
annotationQueueAssignments\:read
annotationQueues\:CUD
annotationQueues\:read
apiKeys\:CUD
apiKeys\:read
auditLogs\:read
automations\:CUD
automations\:read
batchExports\:create
batchExports\:read
comments\:CUD
comments\:read
dashboards\:CUD
dashboards\:read
datasets\:CUD
evalDefaultModel\:CUD
evalDefaultModel\:read
evalJob\:CUD
evalJob\:read
evalJobExecution\:read
evalTemplate\:CUD
evalTemplate\:read
integrations\:CRUD
llmApiKeys\:create
llmApiKeys\:delete
llmApiKeys\:read
llmApiKeys\:update
llmSchemas\:CUD
llmSchemas\:read
llmTools\:CUD
llmTools\:read
models\:CUD
objects\:bookmark
objects\:publish
objects\:tag
project\:delete
project\:read
project\:update
projectMembers\:CUD
projectMembers\:read
promptExperiments\:CUD
promptExperiments\:read
promptProtectedLabels\:CUD
prompts\:CUD
prompts\:read
scoreConfigs\:CUD
scoreConfigs\:read
scores\:CUD
TableViewPresets\:CUD
TableViewPresets\:read
traces\:delete | annotationQueueAssignments\:CUD
annotationQueueAssignments\:read
annotationQueues\:CUD
annotationQueues\:read
apiKeys\:CUD
apiKeys\:read
auditLogs\:read
automations\:CUD
automations\:read
batchExports\:create
batchExports\:read
comments\:CUD
comments\:read
dashboards\:CUD
dashboards\:read
datasets\:CUD
evalDefaultModel\:CUD
evalDefaultModel\:read
evalJob\:CUD
evalJob\:read
evalJobExecution\:read
evalTemplate\:CUD
evalTemplate\:read
integrations\:CRUD
llmApiKeys\:create
llmApiKeys\:delete
llmApiKeys\:read
llmApiKeys\:update
llmSchemas\:CUD
llmSchemas\:read
llmTools\:CUD
llmTools\:read
models\:CUD
objects\:bookmark
objects\:publish
objects\:tag
project\:read
project\:update
projectMembers\:CUD
projectMembers\:read
promptExperiments\:CUD
promptExperiments\:read
promptProtectedLabels\:CUD
prompts\:CUD
prompts\:read
scoreConfigs\:CUD
scoreConfigs\:read
scores\:CUD
TableViewPresets\:CUD
TableViewPresets\:read
traces\:delete | annotationQueueAssignments\:read
annotationQueues\:CUD
annotationQueues\:read
apiKeys\:read
automations\:read
batchExports\:create
batchExports\:read
comments\:CUD
comments\:read
dashboards\:CUD
dashboards\:read
datasets\:CUD
evalDefaultModel\:CUD
evalDefaultModel\:read
evalJob\:CUD
evalJob\:read
evalJobExecution\:read
evalTemplate\:CUD
evalTemplate\:read
llmApiKeys\:read
llmSchemas\:read
llmTools\:read
objects\:bookmark
objects\:publish
objects\:tag
project\:read
projectMembers\:read
promptExperiments\:CUD
promptExperiments\:read
prompts\:CUD
prompts\:read
scoreConfigs\:CUD
scoreConfigs\:read
scores\:CUD
TableViewPresets\:CUD
TableViewPresets\:read | annotationQueues\:read
automations\:read
comments\:read
dashboards\:read
evalDefaultModel\:read
evalJob\:read
evalJobExecution\:read
evalTemplate\:read
llmApiKeys\:read
llmSchemas\:read
llmTools\:read
project\:read
promptExperiments\:read
prompts\:read
scoreConfigs\:read
TableViewPresets\:read | |
# Managing users
## Add a new user to an organization
In the organization settings, you can add users via their email address and assign them a role. They will receive an email notification and will be able to access the organization once they log in. Users who do not have a ABV account yet, will be listed as pending invites until they sign up.
## Changing user roles
Any user with the `members:CUD` permission can change the role of a user in the organization settings. This will affect the user's permissions across all projects in the organization. Users can only assign roles that are lower or equal to their own role.
# Managing Projects
## Add a new project
Any user with the `projects:create` permission can create a new project within a ABV organization.
## Transfer a project to another organization
Only users with the `projects:transfer_organization` permission can transfer a project to another organization. This will remove the project from the current organization and add it to the new one. Access to the project will depend on the roles configured in the new organization.
During this process, no data will be lost, all project settings, data, and configurations will be transferred to the new organization. The project remains fully operational as API keys, settings (except for access management), and data will remain unchanged and associated with the project. All features (e.g. tracing, prompt management) will continue to work without any interruption.
# Project-level roles
Where is this feature available?
Users by default inherit the role of the organization they are part of. For more fine-grained control, you can assign a user a role on the project level. This is useful when you want to differentiate permissions for different projects within the same organization.
If a project-level role is assigned, it will override the organization-level role for that project.
If you want to give a user access to only certain projects within an organization, you can set their role to `None` on the organization level and then assign them a role on the project level.
--------------------------------------------------------------------------------
## Audit Logs
URL: /developer/platform/administration/audit-logs
ABV's audit logging system provides comprehensive tracking of all system activities, capturing detailed information about who performed what actions, when they occurred, and what changes were made. This feature is essential for enterprise security, compliance requirements, and incident investigation.
# What are Audit Logs?
Audit logs are immutable records of all significant activities within your ABV organization and projects.
They capture:
- **Who**: User or API key that performed the action
- **What**: The specific action taken (create, update, delete)
- **When**: Precise timestamp of the action
- **Where**: Organization and project context
- **Details**: Complete before/after state for modifications
These logs provide a complete audit trail for security monitoring, compliance reporting, and forensic analysis.
# Viewing Audit Logs
The audit log viewer is available in the Enterprise plan and provides:
## Access Control
- **Available **to users with `auditLogs:read` permission
- Typically accessible to project OWNERs and ADMINs
- Controlled through ABV's role-based access control system
## Filtering and Navigation
- **Time-based filtering**: View logs for specific time periods
- **Project filtering**: Focus on specific project activities
- **Pagination**: Efficient browsing of large audit trails
# Exporting Audit Logs
You can export audit logs directly from the UI. Thereby you can analyze your data externally or create backups of important information.
**How to use:**
- Navigate to your audit logs table
- Click the export button on the top right
# Event Types and Data Capture
## Auditable Resources
ABV tracks specific actions across all system resources. The following table shows the exact resource types and actions that are logged:
| Resource | Actions |
| ---------------------------- | ----------------------------------------------------- |
| **Annotation Queue** | create, delete, update |
| **Annotation Queue Item** | complete, create, delete |
| **API Key** | create, delete, update |
| **Batch Action** | create, delete |
| **Batch Export** | create |
| **Blob Storage Integration** | update |
| **Comment** | create, delete |
| **Dataset** | create, delete, update |
| **Dataset Item** | create, delete, update |
| **Dataset Run** | delete |
| **Evaluation Template** | create |
| **Job** | create, delete, update |
| **LLM API Key** | create, delete |
| **Membership** | create, delete |
| **Membership Invitation** | create, delete |
| **Model** | create, delete, update |
| **Organization** | create, delete, update |
| **Organization Membership** | create, delete, update |
| **PostHog Integration** | delete, update |
| **Project** | create, delete, transfer, update |
| **Project Membership** | create, delete, update |
| **Prompt** | create, delete, promote, setLabel, update, updateTags |
| **Prompt Protected Label** | create |
| **Score** | create, delete, update |
| **Score Config** | create, update |
| **Session** | bookmark, publish |
| **Stripe Checkout Session** | create |
| **Trace** | bookmark, delete, publish, updateTags |
## State Capture
For update operations, ABV captures:
- **Before State**: Complete resource state prior to modification
- **After State**: Complete resource state after modification
States are stored as JSON, providing full context for any modifications made to your data.
## User Attribution
### Identity Sources
Audit logs distinguish between different types of actors:
- **User Actions** (`USER` type): Actions performed by authenticated users through the web interface
- **API Key Actions** (`API_KEY` type): Programmatic actions via API keys
### Context Information
Each entry includes:
- **User ID**: For user-initiated actions
- **API Key ID**: For programmatic actions
- **Organization ID**: Organizational context
- **Role Context**: User's organizational and project roles at the time of action
--------------------------------------------------------------------------------
## Data Deletion
URL: /developer/platform/administration/data-deletion
There may be use-cases where you want to remove selected data from ABV, like erroneously created traces in a development flow, user data for PII, or your whole project.
In case you want to retain only recent data, you can use our [Data Retention](/developer/platform/administration/data-retention) feature.
You can delete unwanted data from ABV by:
- Deleting a single trace;
- Deleting a batch of traces;
- Deleting all traces that match a query filter;
- Deleting a project;
- Deleting an organization; or
- Deleting a user account.
Below, we will walk through each of the options and their guarantees.
# Deleting Traces
Note that all trace deletions will delete related entities like scores and observations across all data storages.
## Single Trace
### **UI**
To delete a single trace, in traces view, scroll right and in `Action` column press on three dots button and hit the `Delete` button. Confirm that you want to delete the given trace.
### **API**
```text
DELETE /api/public/traces/{traceId}
```
See .
## Batch of Traces
### UI
To delete a batch of traces, select them in the trace list and select `Delete` in the `Actions` dropdown.
### API
```text
DELETE /api/public/traces
```
See .
## Delete by Query
To delete all traces that match a query filter, configure your desired filter in the traces list. Select all items on the current page and change that to all items in the top bar.
Then select `Delete` in the `Actions` dropdown.
## Limitations
Most deletions in ABV happen instantly, but the deletion of tracing data does not.
Removing those records from our data warehouse is a resource intensive operation and, therefore, we rate limit how many deletions we process at any point in time. Usually, trace data is deleted from our system within 15 minutes of the delete call and there is no confirmation, i.e. To verify that your data got deleted, you will have to query it again.
# Deleting a Project
To delete a project, navigate to the project settings and scroll to the `Danger Zone` within the `General` section. Confirm that you want to delete your project. This action immediately revokes all API keys and schedules the project for deletion. Within the next minutes, all related data is irreversibly removed from our system.
Deleting a project is irreversible and all data will be removed. Be cautious when executing this action.
# Deleting an Organization
If there are no projects left in an organization, you can delete the organization in the organization settings. Navigate to the organization settings and scroll to the `Danger Zone` within the `General` section. Confirm that you want to delete your organization. The organization and all associated user information will be removed from our system.
# Deleting a User Account
To delete your user account, please message us at [support@abv.dev](mailto:support@abv.dev).
--------------------------------------------------------------------------------
## Metrics overview
URL: /developer/platform/metrics/overview
ABV metrics derive actionable insights from [observability ](/developer/basic-features/observability-tracing) and [evaluation](/developer/evaluations/overview) traces.
Metrics can be sliced and diced via the [customizable dashboards](/developer/platform/metrics/custom-dashboards) and the [metrics API](/developer/platform/metrics/metrics-api).
# Features
[Custom Dashboards](/developer/platform/metrics/custom-dashboards)
[Metrics API](/developer/platform/metrics/metrics-api)
# Metrics & Dimensions
Metrics:
- **Quality** is measured through user feedback, model-based scoring, human-in-the-loop scored samples or custom scores via SDKs/API (see [scores](/developer/evaluations/overview)). Quality is assessed over time as well as across prompt versions, LLMs and users.
- **Cost and Latency** are accurately measured and broken down by user, session, geography, feature, model and prompt version.
- **Volume** based on the ingested traces and tokens used.
Dimensions:
- Trace name: differentiate between different use cases, features, etc. by adding a `name` field to your traces.
- User: track usage and cost by user. Just add a `userId` to your traces ([docs](/developer/basic-features/user-tracking)).
- Tags: filter different use cases, features, etc. by adding [tags](/developer/basic-features/tags) to your traces.
- Release and version numbers: track how changes to the LLM application affected your metrics.
For an exact definition, please refer to the [metrics API docs](/developer/platform/metrics/metrics-api).
--------------------------------------------------------------------------------
## Custom Dashboards
URL: /developer/platform/metrics/custom-dashboards
Transform your LLM application data into actionable insights with ABV custom dashboards.
Create personalized views that track the metrics that matter most to your team - from latency and cost optimization to quality monitoring and user behavior analysis.
Custom dashboards provide a flexible, self-service analytics solution built on a powerful query engine that supports multi-level aggregations across your tracing data.
Whether you're monitoring production performance, analyzing user feedback trends, or correlating costs with quality metrics, dashboards give you the visualization tools to make data-driven decisions.
# Key Capabilities
- **Flexible Query Engine**: Built on the ABV data model with support for complex aggregations across traces, observations, users, sessions, and scores
- **Rich Visualization Options**: Multiple chart types including line charts, bar charts, and time series with customizable layouts
- **Advanced Filtering**: Filter by metadata, timestamps, user properties, model parameters, and more
- **Multi-Level Aggregations**: Aggregate data at trace, user, or session levels to answer complex analytical questions
- **Real-Time Updates**: Dashboards reflect live data from your LLM applications
- **Team Collaboration**: Share dashboards across your project for unified monitoring and insights
- **ABV Curated Dashboards**: A set of pre-built dashboards focused on Latency, Cost, and ABV usage to quickly get started
# Quick Start
Get started with custom dashboards in two simple steps or use ABV's curated dashboards right away.
## 1) Create Your First Widget
Widgets are individual visualization components that display specific metrics from your LLM application data.
1. Navigate to the **Dashboards** section in your ABV project
2. Select the **Widgets** tab
3. Click **New Widget**
4. Configure your widget:
- **Data Source**: Choose from traces, observations, or evaluation scores
- **Metrics**: Select what to measure (count, latency, cost, scores, etc.)
- **Dimensions**: Group by user, model, time, trace name, etc.
- **Filters**: Narrow down to specific data subsets
- **Chart Type**: Pick the best visualization for your data
5. Click **Save** to store your widget
## 2) Build Your Dashboard
Combine multiple widgets into comprehensive dashboards that tell the story of your LLM application performance.
1. Navigate to the **Dashboards** tab
2. Click **New Dashboard**
3. Give your dashboard a descriptive name (e.g., "Production Monitoring", "Cost Analysis", "Quality Metrics")
4. Add widgets by selecting from your existing widgets or creating new ones
5. Arrange widgets using the drag-and-drop interface
6. Resize widgets to emphasize important metrics
### Leverage Curated Dashboards
Jump-start your analytics with ABV-curated dashboards that focus on common LLM application monitoring needs:
- **Latency Dashboard**: Monitor response times across models and user segments
- **Cost Dashboard**: Track token usage and associated costs over time
- **Usage Dashboard**: Understand your ABV platform utilization
These pre-built dashboards can be used as-is or cloned and customized to match your specific requirements.
## Advanced Features
## Advanced Filtering and Grouping
Create precise data views using ABV's powerful filtering capabilities:
- **Metadata Filters**: Filter by custom metadata attached to traces and observations
- **Time-Based Filters**: Analyze specific time periods or compare time ranges
- **User Properties**: Segment by user characteristics and behavior patterns
- **Model Parameters**: Filter by specific model configurations or versions
- **Tags and Labels**: Use [trace tags](/developer/basic-features/tags) for categorical filtering
- **Score Thresholds**: Filter by quality score ranges or feedback ratings
## Chart Types and Visualization
Choose the right visualization for your data:
- **Line Charts**: Perfect for tracking trends over time (latency, cost, usage)
- **Bar Charts**: Compare values across categories (models, users, features)
- **Time Series**: Monitor real-time metrics with temporal granularity
- **Pie Charts**: Display proportions of categorical data (e.g., feedback ratings)
## Dynamic Layout and Responsiveness
- **Drag-and-Drop Interface**: Easily rearrange widgets to create logical groupings
- **Responsive Design**: Dashboards adapt to different screen sizes and devices
- **Widget Resizing**: Emphasize important metrics with larger visualizations
- **Grid System**: Maintain clean, organized layouts automatically
## Data Export and Integration
Export your dashboard data for further analysis or integration with external tools. See the [Exporte Data](/developer/platform/api-data-platform/overview) guide for comprehensive export options including:
- CSV export of dashboard data
- Integration with external analytics tools
- Programmatic access via the [Metrics API](/developer/platform/metrics/metrics-api)
# Use Cases and Examples
## Production Monitoring Dashboard
Monitor the health and performance of your LLM application in real-time:
- **Error Rate Tracking**: Monitor failed requests and error patterns
- **Latency Analysis**: Track P95 and P99 response times across different endpoints
- **Throughput Monitoring**: Visualize request volume and capacity utilization
- **Model Performance**: Compare accuracy and quality metrics across model versions
- **Tool Usage & Latency**: Track how often each external tool (e.g. API calls, database queries) is invoked and its latency
## Cost **Optimization **Dashboard
Understand and optimize your LLM usage costs:
- **Token Usage Trends**: Track input/output token consumption over time
- **Cost per User**: Identify high-usage users and optimize pricing strategies
- **Model Cost Comparison**: Compare costs across different LLM providers and models
- **Feature Cost Analysis**: Understand which application features drive the highest costs
## Quality and User Experience Dashboard
Monitor the quality and user satisfaction of your LLM application:
- **User Feedback Trends**: Track thumbs up/down ratings and detailed feedback
- **Score Distribution**: Visualize the distribution of quality scores over time
- **User Behavior Analysis**: Understand how users interact with different features
- **A/B Test Results**: Compare quality metrics between different model versions or prompts
--------------------------------------------------------------------------------
## Metrics API
URL: /developer/platform/metrics/metrics-api
```bash
GET /api/public/metrics
```
The **Metrics API** enables you to retrieve customized analytics from your ABV data.
This endpoint allows you to specify dimensions, metrics, filters, and time granularity to build powerful custom reports and dashboards for your LLM applications.
# Overview
The Metrics API supports querying across different views (traces, observations, scores) and allows you to:
- Select specific dimensions to group your data
- Apply multiple metrics with different aggregation methods
- Filter data based on metadata, timestamps, and other properties
- Analyze data across time with customizable granularity
- Order results according to your needs
# Query Parameters
The API accepts a JSON query object passed as a URL-encoded parameter:
| Parameter | Type | Description |
| --------- | ----------- | ---------------------------------------------------------- |
| `query` | JSON string | The encoded query object defining what metrics to retrieve |
## Query Object Structure
| Field | Type | Required | Description |
| --------------- | ------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `view` | string | Yes | The data view to query: `"traces"`, `"observations"`, `"scores-numeric"`, or `"scores-categorical"` |
| `dimensions` | array | No | Array of dimension objects to group by, e.g. `[{ "field": "name" }]` |
| `metrics` | array | Yes | Array of metric objects to calculate, e.g. `[{ "measure": "latency", "aggregation": "p95" }]` |
| `filters` | array | No | Array of filter objects to narrow results, e.g. `[{ "column": "metadata", "operator": "contains", "key": "customKey", "value": "customValue", "type": "stringObject" }]` |
| `timeDimension` | object | No | Configuration for time-based analysis, e.g. `{ "granularity": "day" }` |
| `fromTimestamp` | string | Yes | ISO timestamp for the start of the query period |
| `toTimestamp` | string | Yes | ISO timestamp for the end of the query period |
| `orderBy` | array | No | Specification for result ordering, e.g. `[{ "field": "name", "direction": "asc" }]` |
## Dimension Object Structure
```json
{ "field": "name" }
```
## Metric Object Structure
```json
{ "measure": "count", "aggregation": "count" }
```
Common measure types include:
- `count` - Count of records
- `latency` - Duration/latency metrics
Aggregation types include:
- `sum` - Sum of values
- `avg` - Average of values
- `count` - Count of records
- `max` - Maximum value
- `min` - Minimum value
- `p50` - 50th percentile
- `p75` - 75th percentile
- `p90` - 90th percentile
- `p95` - 95th percentile
- `p99` - 99th percentile
## Filter Object Structure
```json
{
"column": "metadata",
"operator": "contains",
"key": "customKey",
"value": "customValue",
"type": "stringObject"
}
```
## Time Dimension Object
```json
{
"granularity": "day"
}
```
Supported granularities include: `hour`, `day`, `week`, `month`, and `auto`.
# Example
Here's an example of querying the number of traces grouped by name:
### API
```bash
curl \
-H "Authorization: Basic " \
-G \
--data-urlencode 'query={
"view": "traces",
"metrics": [{"measure": "count", "aggregation": "count"}],
"dimensions": [{"field": "name"}],
"filters": [],
"fromTimestamp": "2025-05-01T00:00:00Z",
"toTimestamp": "2025-05-13T00:00:00Z"
}' \
https://app.abv.dev/api/public/metrics
```
Response:
```json
{"data":[{"name":"trace-test-2","count_count":"10"},{"name":"trace-test-3","count_count":"5"},{"name":"trace-test-1","count_count":"3"}]}
```
### Python SDK
Install package
```bash
pip install abvdev
```
```python
query = """
{
"view": "traces",
"metrics": [{"measure": "count", "aggregation": "count"}],
"dimensions": [{"field": "name"}],
"filters": [],
"fromTimestamp": "2025-05-01T00:00:00Z",
"toTimestamp": "2025-05-13T00:00:00Z"
}
"""
abv.api.metrics.metrics(query = query)
```
# Data Model
The Metrics API provides access to several data views, each with its own set of dimensions and metrics you can query. This section outlines the available options for each view.
## Available Views
| View | Description |
| -------------------- | ----------------------------------- |
| `traces` | Query data at the trace level |
| `observations` | Query data at the observation level |
| `scores-numeric` | Query numeric and boolean scores |
| `scores-categorical` | Query categorical (string) scores |
## Trace Dimensions
| Dimension | Type | Description |
| ----------------- | --------- | --------------------------------------- |
| `id` | string | Trace ID |
| `name` | string | Trace name |
| `tags` | string\[] | Trace tags |
| `userId` | string | User ID associated with the trace |
| `sessionId` | string | Session ID associated with the trace |
| `release` | string | Release tag |
| `version` | string | Version tag |
| `environment` | string | Environment (e.g., production, staging) |
| `observationName` | string | Name of related observations |
| `scoreName` | string | Name of related scores |
## Trace Metrics
| Metric | Description |
| ------------------- | ----------------------------------- |
| `count` | Count of traces |
| `observationsCount` | Count of observations within traces |
| `scoresCount` | Count of scores within traces |
| `latency` | Trace duration in milliseconds |
| `totalTokens` | Total tokens used in the trace |
| `totalCost` | Total cost of the trace |
## Observation Dimensions
| Dimension | Type | Description |
| --------------------- | ------ | --------------------------------------- |
| `id` | string | Observation ID |
| `traceId` | string | Associated trace ID |
| `traceName` | string | Name of the parent trace |
| `environment` | string | Environment (e.g., production, staging) |
| `parentObservationId` | string | ID of parent observation |
| `type` | string | Observation type |
| `name` | string | Observation name |
| `level` | string | Log level |
| `version` | string | Version |
| `providedModelName` | string | Model name |
| `promptName` | string | Prompt name |
| `promptVersion` | string | Prompt version |
| `userId` | string | User ID from parent trace |
| `sessionId` | string | Session ID from parent trace |
| `traceRelease` | string | Release from parent trace |
| `traceVersion` | string | Version from parent trace |
| `scoreName` | string | Related score name |
## Observation Metrics
| Metric | Description |
| ------------------ | ------------------------------------ |
| `count` | Count of observations |
| `latency` | Observation duration in milliseconds |
| `totalTokens` | Total tokens used |
| `totalCost` | Total cost |
| `timeToFirstToken` | Time to first token in milliseconds |
| `countScores` | Count of related scores |
## Score Dimensions (Common)
| Dimension | Type | Description |
| -------------------------- | ------ | ------------------------------------------ |
| `id` | string | Score ID |
| `name` | string | Score name |
| `environment` | string | Environment |
| `source` | string | Score source |
| `dataType` | string | Data type |
| `traceId` | string | Related trace ID |
| `traceName` | string | Related trace name |
| `userId` | string | User ID from trace |
| `sessionId` | string | Session ID from trace |
| `observationId` | string | Related observation ID |
| `observationName` | string | Related observation name |
| `observationModelName` | string | Model used in related observation |
| `observationPromptName` | string | Prompt name used in related observation |
| `observationPromptVersion` | string | Prompt version used in related observation |
| `configId` | string | Configuration ID |
## Score Metrics
### Numeric Scores
| Metric | Description |
| ------- | ------------------- |
| `count` | Count of scores |
| `value` | Numeric score value |
### Categorical Scores
| Metric | Description |
| ------- | --------------- |
| `count` | Count of scores |
Categorical scores have an additional dimension:
| Dimension | Type | Description |
| ------------- | ------ | ------------------------------------- |
| `stringValue` | string | String value of the categorical score |
--------------------------------------------------------------------------------
## API & Data Platform overview
URL: /developer/platform/api-data-platform/overview
**ABV is designed to be open, extensible and flexible**. People using ABV are building all kinds of workflows and customizations on top of it. This is powered by our open data platform.
Example use cases:
- Billing based on LLM costs tracked in ABV
- Reporting of online evaluations in external dashboards
- Fine-tuning based on raw exports of traces
- Correlation of LLM Evals with observed user behavior in Data Warehouse
# Features
[Public API](/developer/platform/api-data-platform/public-api)
[Export Data from UI](/developer/platform/api-data-platform/export-data-from-ui)
[Export for Fine-Tuning](/developer/platform/api-data-platform/export-for-fine-tuning)
[Query Data via SDKs](/developer/platform/api-data-platform/query-data-via-sdks)
[Export to Blob Storage](/developer/platform/api-data-platform/export-via-blob-storage)
--------------------------------------------------------------------------------
## Export Data from UI
URL: /developer/platform/api-data-platform/export-data-from-ui
Export your observability data for analysis, fine-tuning, model training, or integration with external tools.
Most tables in ABV support batch-exports. All filters applied to the table will be applied to the export.
Available export formats:
- CSV
- JSON
# Alternatives
You can also export data via:
- [Blob Storage](/developer/platform/api-data-platform/export-via-blob-storage) - Scheduled automated exports to cloud storage
- [SDKs/API](/developer/platform/api-data-platform/public-api) - Programmatic access using ABV SDKs or API
--------------------------------------------------------------------------------
## Export via Blob Storage Integration
URL: /developer/platform/api-data-platform/export-via-blob-storage
Where is this feature available?
You can create schedule exports to a Blob Storage, e.g. S3, GCS, or Azure Blob Storage, for `traces`, `observations`, and `scores`.
Those exports can run on an `hourly`, `daily`, or `weekly` schedule. Navigate to your project settings and select `Integrations > Blob Storage` to set up a new export. Select whether you want to use S3, a S3 compatible storage, Google Cloud Storage, or Azure Blob Storage.
# Start exporting via Blob Storage
To set up the export navigate to `Your Project` > `Settings` > `Integrations` > `Blob Storage`.
Fill in the settings to authenticate with your vendor, enable the integration, and press save.
Within an hour an initial export should start and continue based on the schedule you have selected. The export supports CSV, JSON, and JSONL file formats. for more information on how to get credentials for your specific vendor.
# Alternatives
You can also export data via:
- [UI](/developer/platform/api-data-platform/export-data-from-ui) - Manual batch-exports from the ABV UI
- [SDKs/API](/developer/platform/api-data-platform/public-api) - Programmatic access using ABV SDKs or API
--------------------------------------------------------------------------------
## Query Data via SDKs
URL: /developer/platform/api-data-platform/query-data-via-sdks
You can query data via: [SDKs](./#sdks) and [API](/developer/platform/api-data-platform/public-api). For export functionality, see [Export Data](/developer/platform/api-data-platform/overview).
Common use cases:
- Train or [fine-tune](/developer/platform/api-data-platform/export-for-fine-tuning) models on the production traces in ABV. E.g. to create a small model after having used a large model in production for a specific use case.
- Collect few-shot examples to improve quality of output.
- Programmatically create [datasets](/developer/evaluations/datasets).
New data is typically available for querying within 15-30 seconds of ingestion, though processing times may vary at times. Please visit [status page](https://status.abv.dev/).If you encounter any issues.
# SDKs
Via the [SDKs](/developer/sdks/overview) for Python and JS/TS you can easily query the API without having to write the HTTP requests yourself.
## Python SDK
```bash
pip install abvdev
```
```python
from abvdev import ABV
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
```
The `api` namespace is auto-generated from the Public API (OpenAPI). Method names mirror REST resources and support filters and pagination.
### **Traces**
```python
traces = abv.api.trace.list(limit=100, user_id="user_123", tags=["production"]) # pagination via cursor
trace = abv.api.trace.get("traceId")
```
### ** Observations**
```python
observations = abv.api.observations.get_many(trace_id="abcdef1234", type="GENERATION", limit=100)
observation = abv.api.observations.get("observationId")
```
### Sessions
```python
sessions = abv.api.sessions.list(limit=50)
```
### Scores
```python
abv.api.score_v_2.get(score_ids = "ScoreId")
```
### Prompts
Please refer to the [prompt management documentation](/developer/prompt-management/get-started) on fetching prompts.
### Datasets
```python
# Namespaces:
# - abv.api.datasets.*
# - abv.api.dataset_items.*
# - abv.api.dataset_run_items.*py
```
### Metrics
```python
query = """
{
"view": "traces",
"metrics": [{"measure": "count", "aggregation": "count"}],
"dimensions": [{"field": "name"}],
"filters": [],
"fromTimestamp": "2025-05-01T00:00:00Z",
"toTimestamp": "2025-05-13T00:00:00Z"
}
"""
abv.api.metrics.metrics(query = query)
```
### Async equivalents
```python
# All endpoints are also available as async under `async_api`:
trace = await abv.async_api.trace.get("traceId")
traces = await abv.async_api.trace.list(limit=100)
```
### Common filtering & pagination
- limit, cursor (pagination)
- time range filters (e.g., start\_time, end\_time)
- entity filters: user\_id, session\_id, trace\_id, type, name, tags, level, etc.
See the Public API for the exact parameters per resource.
## JS/TS SDK
The dedicated `fetch*` methods for core entities are covered by tests and semantic versioning. The methods on the `abv.api` are auto-generated from the API reference and cover all entities.
```bash
npm install @abvdev/client
```
**Environment variables**
Add your ABV credentials as environment variables, e.g. use .env file and dotenv package to load variable values.
```bash
npm install dotenv
```
```bash title=".env"
ABV_API_KEY = "sk-abv-...";
ABV_BASEURL = "https://app.abv.dev"; US region
# ABV_BASEURL = "https://eu.app.abv.dev"; EU region
```
```typescript
import { ABVClient } from "@abvdev/client";
const abv = new ABVClient();
```
**alternatively use Constructor parameters**
```typescript
import { ABVClient } from "@abvdev/client";
const abv = new ABVClient({
apiKey: "sk-abv-...",
baseUrl: "https://app.abv.dev", // US region
// baseUrl: "https://eu.app.abv.dev", // EU region
});
```
**Use api**
```typescript
import { ABVClient } from "@abvdev/client";
import dotenv from "dotenv";
dotenv.config();
const abv = new ABVClient();
async function main() {
// Fetch list of traces, supports filters and pagination
const traces = await abv.api.trace.list();
// Fetch a single trace by ID
const trace = await abv.api.trace.get("traceId");
// Fetch list of observations, supports filters and pagination
const observations = await abv.api.observations.getMany();
// Fetch a single observation by ID
const observation = await abv.api.observations.get("observationId");
// Fetch list of sessions
const sessions = await abv.api.sessions.list();
// Fetch a single session by ID
const session = await abv.api.sessions.get("sessionId");
// Fetch list of scores
const scores = await abv.api.scoreV2.get();
// Fetch a single score by ID
const score = await abv.api.scoreV2.getById("scoreId");
// Explore more entities via Intellisense
}
```
JS/TS SDK reference including all available filters:
- `fetchTraces()`
- `fetchTrace()`
- `fetchObservations()`
- `fetchObservation()`
- `fetchSessions()`
--------------------------------------------------------------------------------
## Export for Fine-Tuning
URL: /developer/platform/api-data-platform/export-for-fine-tuning
You can easily [trace](/developer/basic-features/observability-tracing) your application, collect user feedback, and then use the data to fine-tune a model for your specific use case.
# Export Data for Fine-Tuning
To export generations and training data for fine-tuning, see the comprehensive [Export Data](/developer/platform/api-data-platform/overview) guide which includes:
- Export generations in OpenAI JSONL format for fine-tuning
- Filter by quality scores to export high-performing examples
- Multiple export formats (CSV, JSON, JSONL)
- Programmatic access via SDKs and APIs
# SDKs/API
All data collected in ABV is also available programmatically via the API and SDKs (Python, JS/TS). Refer to the API reference and [Export Data](/developer/platform/api-data-platform/overview)for more information.
--------------------------------------------------------------------------------
## Public API
URL: /developer/platform/api-data-platform/public-api
All ABV data and features are available via the API.
**Path**
```text
/api/public
```
**US**
```text
https://app.abv.devapi/public
```
**EU**
```text
https://eu.app.abv.dev/api/public
```
References:
- API Reference:
- OpenAPI spec:
- Postman collection:
# Authentication
Authenticate with the API using [Basic Auth](https://en.wikipedia.org/wiki/Basic_access_authentication). The API key is available in the ABV project settings.
- Username: ABV Public Key
- Password: ABV Secret Key
Example:
```bash
curl -u public-key:api-key https://app.abv.dev/api/public/projects
```
# Access via SDKs
Both the ABV [Python SDK](/developer/sdks/python/overview) and the [JS/TS SDK](/developer/sdks/js-ts/overview) provide a strongly-typed wrapper around our public REST API for your convenience. The API methods are accessible via the `api` property on the ABV client instance in both SDKs.
You can use your editor's Intellisense to explore the API methods and their parameters.
When fetching [prompts](/developer/prompt-management/get-started), please use the `get_prompt` (Python) / `getPrompt` (JS/TS) methods on the ABV client to benefit from client-side caching, automatic retries, and fallbacks.
### Python SDK
Install package
```bash
pip install abvdev
```
```python
from abvdev import ABV
abv = ABV(
api_key="sk-abv-...", # your api key here
host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)
...
# fetch a trace
abv.api.trace.get(trace_id)
# async client via asyncio
await abv.async_api.trace(trace_id)
# explore more endpoints via Intellisense
abv.api.*
await abv.async_api.*
```
### JS/TS SDK
```bash
npm i @abvdev/client
```
**Environment variables**
Add your ABV credentials as environment variables, e.g. use .env file and dotenv package to load variable values.
```bash
npm install dotenv
```
```bash title=".env"
ABV_API_KEY = "sk-abv-...";
ABV_BASEURL = "https://app.abv.dev"; US region
# ABV_BASEURL = "https://eu.app.abv.dev"; EU region
```
```typescript
import { ABVClient } from "@abvdev/client";
const abv = new ABVClient();
```
**alternatively use Constructor parameters**
```typescript
import { ABVClient } from "@abvdev/client";
const abv = new ABVClient({
apiKey: "sk-abv-...",
baseUrl: "https://app.abv.dev", // US region
// baseUrl: "https://eu.app.abv.dev", // EU region
});
```
**Explore api**
```typescript
import { ABVClient } from "@abvdev/client";
import dotenv from "dotenv";
dotenv.config();
const abv = new ABVClient();
// ...
// fetch a trace
const traceId = "target_trace_id";
async function main() {
console.log(await abv.api.trace.get(traceId));
}
main();
// explore more endpoints via Intellisense
// abv.api.*
```
# Ingest Traces via the API
It is recommended to use the OpenTelemetry Endpoint to ingest traces. Please refer to the for more information.
# Alternatives
You can also export data via:
- [UI](/developer/platform/api-data-platform/export-data-from-ui) - Manual batch-exports from the ABV UI
- [Blob Storage](/developer/platform/api-data-platform/export-via-blob-storage) - Scheduled automated exports to cloud storage
# FAQ
- Are there any limits to the ABV API?
--------------------------------------------------------------------------------
## Troubleshooting and FAQ
URL: /developer/troubleshooting-faq
# FAQ
In short-lived serverless environments, you must explicitly flush traces before the process exits or the runtime environment is frozen.
**For JS/TS:**
Export the processor from your `instrumentation.ts` file:
```typescript title="instrumentation.ts"
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";
// Export the processor to be able to flush it
export const abvSpanProcessor = new ABVSpanProcessor();
const sdk = new NodeSDK({
spanProcessors: [abvSpanProcessor],
});
sdk.start();
```
Then call `forceFlush()` before the function exits:
```typescript title="handler.ts"
import { abvSpanProcessor } from "./instrumentation";
export async function handler(event, context) {
// ... your application logic ...
// Flush before exiting
await abvSpanProcessor.forceFlush();
}
```
**For Vercel Cloud Functions**, use the `after` utility:
```typescript
import { after } from "next/server";
import { abvSpanProcessor } from "./instrumentation";
export async function POST() {
// ... existing request logic ...
// Schedule flush after request has completed
after(async () => {
await abvSpanProcessor.forceFlush();
});
// ... send response ...
}
```
**For Python:**
```python
from abvdev import get_client
abv = get_client()
# Your application logic here
# Flush all pending observations before function exits
abv.flush()
```
For complete shutdown (no more events will be sent):
```python
abv.shutdown()
```
ABV tracks usage and costs of your LLM generations with breakdowns by usage types (input, output, cached tokens, audio tokens, etc.).
**Option 1: Ingest usage and cost (most accurate)**
Many ABV integrations automatically capture usage from LLM responses. You can also manually ingest them:
**Python SDK:**
```python
from abvdev import ABV
abv = ABV(api_key="sk-abv-...", host="https://app.abv.dev")
with abv.start_as_current_observation(
as_type='generation',
name="llm-call",
model="gpt-4o"
) as generation:
response = openai_client.chat.completions.create(...)
generation.update(
output=response.choices[0].message.content,
usage_details={
"input": response.usage.input_tokens,
"output": response.usage.output_tokens,
},
cost_details={
"input": 0.01, # USD cost
"output": 0.03,
}
)
```
**JS/TS SDK:**
```typescript
generation.update({
usageDetails: {
prompt_tokens: response.usage.prompt_tokens,
completion_tokens: response.usage.completion_tokens,
total_tokens: response.usage.total_tokens,
},
output: { content: llmOutput },
});
```
**Option 2: Infer usage and cost automatically**
If you don't ingest usage/cost, ABV will automatically infer them based on the `model` parameter. ABV includes predefined models and tokenizers for OpenAI, Anthropic, and Google models.
You can also add custom model definitions via the ABV UI or API for your own models.
Empty inputs and outputs typically occur when:
1. **You didn't set them**: Make sure to call `.update()` with `input` and `output` parameters:
```python
# Python
with abv.start_as_current_span(name="my-operation") as span:
span.update(input={"query": "user question"})
# ... your logic ...
span.update(output="response")
```
```typescript
// JS/TS
await startActiveObservation("my-operation", async (span) => {
span.update({ input: { query: "user question" } });
// ... your logic ...
span.update({ output: "response" });
});
```
2. **Timing issues in serverless**: If the function exits before data is flushed, use `abv.flush()` (Python) or `await abvSpanProcessor.forceFlush()` (JS/TS).
3. **Data was masked**: Check if you have masking rules that might be removing sensitive data.
**To disable tracing entirely:**
**Python SDK:**
Simply don't initialize the ABV client or don't use the `@observe` decorator.
**JS/TS SDK:**
Don't import the `instrumentation.ts` file, or conditionally initialize it:
```typescript
if (process.env.ABV_ENABLED === "true") {
await import("./instrumentation");
}
```
**To use sampling (partial tracing):**
Configure sampling rate via environment variable or in code:
**Python:**
```bash
ABV_SAMPLE_RATE=0.1 # Sample 10% of traces
```
**JS/TS:**
```typescript
import { TraceIdRatioBasedSampler } from "@opentelemetry/sdk-trace-base";
const sdk = new NodeSDK({
sampler: new TraceIdRatioBasedSampler(0.1), // Sample 10% of traces
spanProcessors: [new ABVSpanProcessor()],
});
```
Environments help you organize traces from different contexts (production, staging, development).
**Set via environment variable (recommended):**
```bash
ABV_TRACING_ENVIRONMENT="production"
```
**Python SDK:**
```python
from abvdev import get_client
# Will use environment variable
abv = get_client()
# Or set explicitly
abv = ABV(
api_key="sk-abv-...",
host="https://app.abv.dev",
environment="staging"
)
```
**JS/TS SDK:**
```bash title=".env"
ABV_TRACING_ENVIRONMENT="production"
```
The environment is automatically attached to all traces, observations, scores, and sessions. You can filter by environment in the ABV UI.
**Environment naming rules:**
- Cannot start with "abv"
- Only lowercase letters, numbers, hyphens, and underscores
- Maximum 40 characters
**Common causes and solutions:**
1. **Missing flush in serverless/short-lived applications:**
- Python: Call `abv.flush()` before exit
- JS/TS: Call `await abvSpanProcessor.forceFlush()` before exit
2. **Incorrect API credentials:**
- Verify your API key is correct
- Check if you're using the right region (US: `https://app.abv.dev`, EU: `https://eu.app.abv.dev`)
- Python: Use `abv.auth_check()` to verify credentials (don't use in production)
3. **Instrumentation not loaded:**
- JS/TS: Ensure `import "./instrumentation"` is the FIRST import in your application
- Python: Ensure you've initialized the client with `get_client()` or `ABV()`
4. **Network/firewall issues:**
- Check if your application can reach the ABV API
- Verify no proxy/firewall is blocking requests
5. **Sampling is too aggressive:**
- Check if you have sampling enabled that might be filtering out traces
- Temporarily set sample rate to 1.0 (100%) to test
6. **Wrong project:**
- Verify you're looking at the correct project in the ABV UI
- Check if the API key belongs to the project you're viewing
7. **For JS/TS with @vercel/otel:**
- Use manual OpenTelemetry setup via `NodeTracerProvider` instead of `registerOTel` from `@vercel/otel`
- The @vercel/otel package doesn't support OpenTelemetry JS SDK v2 yet
8. **Check the logs:**
- Enable debug logging to see what's happening
- Python: Set log level in code
- JS/TS: Set `ABV_LOG_LEVEL="DEBUG"` in environment variables
1. Sign in to your ABV account at [https://app.abv.dev](https://app.abv.dev)
2. Navigate to **Project Settings**
3. Go to the **API Keys** section
4. Click **Create new API credentials**
API keys are project-specific and should be stored securely as environment variables:
```bash title=".env"
ABV_API_KEY="sk-abv-..."
ABV_BASE_URL="https://app.abv.dev" # or https://eu.app.abv.dev for EU
```
**Security best practices:**
- Never commit API keys to version control
- Use environment variables or secrets management
- Rotate keys regularly
- Use different keys for different environments
ABV SDKs batch and queue events for efficiency. In short-lived applications (serverless functions, scripts, CLI tools), you must flush events before the process exits.
**Python SDK:**
```python
from abvdev import get_client
abv = get_client()
# Your application logic here
# Flush all pending observations
abv.flush()
# Or use shutdown for complete cleanup (no more events will be sent)
abv.shutdown()
```
**JS/TS SDK:**
Export the processor in your `instrumentation.ts`:
```typescript
export const abvSpanProcessor = new ABVSpanProcessor();
```
Then flush before exit:
```typescript
await abvSpanProcessor.forceFlush();
```
**Configuration options:**
You can also adjust batching behavior:
**Python:**
```python
abv = ABV(
flush_at=1, # Send after 1 event
flush_interval=1, # Send after 1 second
)
```
**JS/TS:**
```typescript
new ABVSpanProcessor({
flushAt: 1, // Send after 1 event
flushInterval: 1, // Send after 1ms
exportMode: "immediate", // Send immediately
})
```
ABV uses OpenTelemetry concepts with LLM-specific enhancements:
**Spans (Observations):**
- Generic units of work in your application
- Can be nested to form a tree structure
- Examples: API calls, database queries, function executions
- Created with `start_as_current_span()` or `startActiveObservation()`
**Generations:**
- Special type of span specifically for LLM calls
- Include additional fields: `model`, `usage`, `cost`
- Automatically tracked for metrics and costs
- Created with `as_type="generation"` parameter
- Examples: OpenAI completion, Anthropic message, embeddings
**Events:**
- Point-in-time occurrences with no duration
- Lightweight, don't have start/end times
- Examples: logging, status updates, warnings
- Created with `add_event()` method
**Traces:**
- Collection of related spans/observations
- Represent a complete workflow or request
- All spans in a trace share the same `trace_id`
**Example hierarchy:**
```
Trace: "User request"
Span: "Process query"
Generation: "LLM call to GPT-4"
Span: "Database lookup"
Event: "Cache miss"
Span: "Format response"
```
Use **generations** for LLM calls, **spans** for other operations, and **events** for point-in-time logs.
Metadata and tags help you categorize, filter, and analyze traces.
**Metadata** (arbitrary JSON object):
**Python SDK:**
```python
from abvdev import ABV, observe
abv = ABV(api_key="sk-abv-...", host="https://app.abv.dev")
# With decorator
@observe()
def my_function():
abv.update_current_trace(
metadata={"user_id": "123", "version": "1.2.3"}
)
abv.update_current_span(
metadata={"stage": "processing"}
)
# With context manager
with abv.start_as_current_span(name="operation") as span:
span.update_trace(metadata={"request_id": "req_12345"})
span.update(metadata={"stage": "parsing"})
```
**JS/TS SDK:**
```typescript
import { startActiveObservation, updateActiveTrace } from "@abvdev/tracing";
await startActiveObservation("operation", async (span) => {
// Update trace metadata
updateActiveTrace({
metadata: { user_id: "123", version: "1.2.3" }
});
// Update span metadata
span.update({
metadata: { stage: "processing" }
});
});
```
**Tags** (list of strings):
**Python SDK:**
```python
# With decorator
@observe()
def my_function():
abv.update_current_trace(tags=["production", "v2", "feature-x"])
# With context manager
with abv.start_as_current_span(name="operation") as span:
span.update_trace(tags=["experiment-a", "beta"])
```
**JS/TS SDK:**
```typescript
await startActiveObservation("operation", async (span) => {
updateActiveTrace({
tags: ["production", "v2", "feature-x"]
});
});
```
**Best practices:**
- Use metadata for structured data (objects, nested values)
- Use tags for simple labels and filtering
- Common tag examples: environments, versions, features, experiments
- Metadata updates are merged by top-level key
- Tags are additive (cannot be removed once added)
--------------------------------------------------------------------------------
## Security & Compliance Overview
URL: /security/overview
**At ABV, we prioritize data privacy and security.** We understand that the data you entrust to us is a vital asset to your business, and we treat it with the utmost care.
We take active steps to demonstrate our commitment to data security and privacy such as annual ISO27001 audit as well as External Penetration Tests. You can request access to the reports here.
ABV is built with enterprise needs in mind, focusing on:
- **Security Measures:** Robust [Encryption](./security/encryption), access controls, and regular [Penetration Testing](./security/penetration-testing).
- **Privacy Measures:** Protecting user data according to regulations like [GDPR](./privacy/gdpr-compliance). We offer a [DPA](/security/privacy/data-processing-agreement) and adhere to our [Privacy Policy](https://abv.dev/privacy).
- **Transparency:** Open-source core and clear information on software dependencies.
- **Reporting:** Clear channels for [Responsible Disclosure](./security/responsible-disclosure) and [Whistleblowing](./security/whistleblowing).
# Compliance
We maintain [internal policies](./compliance/policies) and adhere to several industry-standard compliance frameworks.
[ISO 27001](./compliance/iso-27001-compliance)
[ISO 42001 Compliance](./compliance/iso-42001-compliance)
[HIPAA](./compliance/hipaa-compliance)
# Privacy
ABV is [GDPR](./privacy/gdpr-compliance) compliant, and offers data retention, data masking and data deletion capabilities to [manage the processing of personal data](./privacy/managing-personal-data). You can enter into a [DPA](/security/privacy/data-processing-agreement) with ABV.
# Deployment
### What deployment models are available?
- Fully‑managed SaaS (multi‑tenant) with US, EU and HIPAA data regions
- Managed single tenant environment available for enterprise customers.
### Which cloud provider and regions do you use?
ABV Cloud mainly runs on **AWS and Clickhouse via AWS**:
- **US region**: us-east-1 (Virginia)
- **EU region**: eu-west-1 (Ireland)
# Contact
- For inquiries: [*security@abv.dev*](mailto:security@abv.dev)
--------------------------------------------------------------------------------
## Authentication & Authorization
URL: /security/security/authentication-authorization
ABV provides robust mechanisms for both authenticating users and authorizing their access to specific resources within the platform.
# Authentication
Authentication verifies the identity of a user attempting to access ABV.
### Email/Password authentication
By default, ABV uses email and password authentication. ABV enforces standard password complexity requirements.
If you signed up with a social login, you can add a password via the “reset password” link in the login page.
### Social Logins
For simplified access, users can sign in with Google.
By default, ABV does not support switching between social logins or signing up with a social login after signing up with email/password. Please reach out to support if you need help.
### Enterprise SSO & SSO Enforcement
**Where is this feature available?**
ABV supports **Enterprise SSO** (e.g. Okta, Azure AD, Keycloak etc.) via OIDC. Please [contact us](https://webforms.pipedrive.com/f/bYZXNyRLrqiEcMydUyDJpJDXQ3XChKLKmaft0Khn2t7AmItimt6piRq2Y2IzBOCCjN) to enable this feature.
To sign in with an Enterprise SSO provider, please (1) enter your email address, and (2) press “Continue”. You will be redirected to the Enterprise SSO provider to authenticate.
Optionally, you can **enforce the use of SSO / social logins for your domain**. Please [contact us](https://webforms.pipedrive.com/f/bYZXNyRLrqiEcMydUyDJpJDXQ3XChKLKmaft0Khn2t7AmItimt6piRq2Y2IzBOCCjN) to enable this feature.
# Authorization (RBAC)
ABV supports Role-based Access Control (RBAC). Please refer to the dedicated [RBAC documentation](/developer/platform/administration/role-based-access-controls) for a detailed explanation of roles, permissions, and how to manage user access within organizations and projects.
--------------------------------------------------------------------------------
## Encryption
URL: /security/security/encryption
ABV employs robust encryption methods to protect your data both while it’s being transferred and while it’s stored.
# Encryption in Transit
All data transferred between your applications, the ABV SDKs, and the ABV server is encrypted using **TLS (Transport Layer Security)**. This ensures that data is protected from eavesdropping or tampering during transmission.
# Encryption at Rest
Data stored within the ABV infrastructure is encrypted at rest using **AES-256**, a strong industry-standard encryption algorithm.
This applies to data stored in:
| Service | Encryption Standard |
| ------------------- | ------------------- |
| Elasticache (Redis) | AES-256 |
| Aurora (Postgres) | AES-256 |
| Clickhouse | AES-256 |
| S3 / Blob Storage | AES-256 |
# Contact
For questions regarding encryption practices, please contact [*security@abv.dev*](mailto:security@abv.dev).
--------------------------------------------------------------------------------
## Data Regions & Availability
URL: /security/security/data-regions-availability
ABV Cloud is designed for high availability and offers multiple data regions to meet your needs.
Our database and application run on AWS infrastructure, partly managed by Clickhouse.
# ABV Cloud Regions
All data, user accounts, and infrastructure are completely separated between the regions. You can have accounts in each regions.
| Region | URL | Location |
| ------ | ------------------------------------------------------------------------------------------------- | ------------------------ |
| US | [https://app.abv.dev](https://app.abv.dev/) | Virginia (AWS us-east-1) |
| EU | https://app.abv.dev | Ireland (AWS eu-west-1) |
# Connecting to a Region
To connect to a specific data region using the ABV SDKs, you need to set the base URL environment variable or pass it during initialization:
- **Python:** Set ABV\_HOST environment variable or use the host parameter.
- **JS/TS:** Set ABV\_BASEURL environment variable or use the baseUrl parameter.
Example base URLs:
- US: [https://app.abv.dev](https://app.abv.dev)
- EU: https://app.abv.dev
Refer to the [SDK documentation](/developer/sdks/overview) for detailed configuration instructions.
# Choosing a Region
When selecting a data region, consider the following factors:
- **Compliance and data privacy requirements:** Choose the region that aligns with your organization’s data residency needs (e.g., GDPR often favors the EU region).
- **Latency for Prompt Management:** If using ABV Prompt Management, select the region closer to your application servers for lower latency when fetching prompts. This is hardly noticeable as the prompts are cached by the ABV SDK.
- **Latency for UI access:** Choose the region closer to your team’s location for a faster experience when using the ABV web interface.
Less critical factor:
- **Tracing ingestion latency:** Trace data is sent asynchronously in batches, making ingestion latency less of a direct concern for application performance.
# Business Continuity & Availability
| Control | Details |
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| High availability | Multi‑AZ databases & load‑balanced stateless application layer on AWS. |
| Disaster recovery | Encrypted backups stored cross‑region; tested at least annually |
| Status page | [https://status.abv.dev](https://status.abv.dev) with historical uptime and incidents. |
# Contact
For questions about data regions or availability, contact [*security@abv.dev*](mailto:security@abv.dev).
--------------------------------------------------------------------------------
## Penetration Testing
URL: /security/security/penetration-testing
ABV is committed to maintaining a high level of security. As part of this commitment, we undergo **annual external penetration tests**.
You can request access to the reports by contacting [security@abv.dev](mailto:security@abv.dev).
# Purpose
These tests are conducted by independent, third-party security experts to:
- Proactively identify potential security vulnerabilities in our infrastructure and application.
- Simulate real-world attack scenarios.
- Validate the effectiveness of our existing security controls.
- Ensure ongoing resilience against emerging threats.
# Reporting
The findings from these penetration tests help us continuously improve our security posture. Summaries of the results and remediation actions taken are documented.
- **Report Availability:** A summary of the latest penetration test report is available upon request for customers on the Pro or Enterprise plans.
- **Request Access:** You can request access to the report summary at [security@abv.dev](mailto:security@abv.dev).
# Contact
For questions regarding our penetration testing program, please contact [security@abv.dev](mailto:security@abv.dev).
--------------------------------------------------------------------------------
## Vulnerability Management
URL: /security/security/vulnerability-management
ABV is committed to maintaining the security of its systems and protecting customer data. Our vulnerability management program is a key component of this commitment, designed to identify, assess, prioritize, and remediate security vulnerabilities in a timely manner.
# Identification
We employ a multi-layered approach to identify potential vulnerabilities:
- **Automated Scanning:** We utilize industry-standard tools, including **GitHub code scanning** and **Snyk**, to continuously scan our codebase, dependencies, and infrastructure for known vulnerabilities.
- **External Penetration Testing:** ABV undergoes regular [penetration tests](/security/security/penetration-testing) conducted by independent third-party security experts. Findings from these tests are integrated into our remediation process.
- **Responsible Disclosure Program:** We encourage security researchers to report potential vulnerabilities through our [responsible disclosure program](./responsible-disclosure).
- **Internal Reviews:** Our engineering teams conduct regular security reviews of code and infrastructure configurations.
# Triage and Remediation
Identified vulnerabilities are triaged based on severity and potential impact. High-priority vulnerabilities are addressed promptly according to predefined Service Level Agreements (SLAs). Our remediation process involves:
1. **Assessment:** Understanding the vulnerability’s impact and exploitability.
2. **Prioritization:** Ranking vulnerabilities based on risk.
3. **Remediation:** Applying patches, configuration changes, or code fixes.
4. **Verification:** Confirming the vulnerability has been successfully addressed.
# Compliance
Our vulnerability management processes are designed to meet the requirements of our [ISO 27001](/security/compliance/iso-27001-compliance) and [ISO 42001](/security/compliance/iso-42001-compliance) certifications. This includes maintaining a formal Vulnerability Management Policy, regular scanning, timely remediation, and detailed record-keeping.
--------------------------------------------------------------------------------
## Incident & Breach
URL: /security/security/incident-breach
# Incident Response
We follow our internal incident identification and response process, which is available on request. We post status updates on our [status page](https://status.abv.dev/).
# Notification
We will notify customers of any breach as soon as reasonably possible.
--------------------------------------------------------------------------------
## Responsible Disclosure
URL: /security/security/responsible-disclosure
We value the security community and prioritize the security of our systems. We encourage the responsible disclosure of security vulnerabilities to help us protect the security and privacy of our users and customers.
# Reporting a Vulnerability
If you believe you have found a security vulnerability in ABV, please send an actionable vulnerability report to [security@abv.dev](mailto:security@abv.dev).
Please include the following details in your report:
- A clear description of the vulnerability, including its potential impact.
- Steps to reproduce the vulnerability, including any specific configurations or conditions required.
- Any proof-of-concept code, scripts, or screenshots that demonstrate the vulnerability.
We will acknowledge receipt of your report, typically within 2 business days, and will work with you to understand and resolve the issue.
# Our Commitment
- We will investigate reported vulnerabilities promptly.
- We will keep you informed of our progress.
- We will take appropriate steps to remediate confirmed vulnerabilities.
- We will publicly acknowledge your contribution if you wish, once the vulnerability is fixed.
# Bug Bounty Program
Please note that we **currently do not operate a formal bug bounty program** with monetary rewards.
# Contact
For all security-related inquiries, including vulnerability disclosures, please contact [security@abv.dev](mailto:security@abv.dev).
--------------------------------------------------------------------------------
## Whistleblowing
URL: /security/security/whistleblowing
ABV encourages employees and third parties to report potential breaches of law, regulations, or company policies.
# How to Report
You can report concerns via:
- **Email:** [legal@abv.dev](mailto:legal@abv.dev)
# Confidentiality & Anonymity
You can choose to report anonymously or request that we protect your privacy during the investigation process.
# Internal Policy
For more detailed information, ABV employees can refer to the company’s internal Responsible Disclosure Policy.
# Contact
For inquiries specifically related to whistleblowing, please use the channels mentioned above.
--------------------------------------------------------------------------------
## Security FAQ
URL: /security/security/faq
This page addresses frequently asked questions and common security topics for ABV.
If you don’t find a solution to your issue here, reach out to [security@abv.dev](mailto:security@abv.dev).
### How is data encrypted in transit and at rest?
TLS 1.2+ protects traffic; all stored data uses AES‑256 encryption. See [encryption documentation](./encryption) for more details.
### Do you ever use customer data to train models or analytics?
No—customer traces and prompts are processed only to provide the ABV service and are never used to train internal or third‑party ML models. See [security overview](/developer/guardrails/llm-security-guardrails-overview) for more details.
### What retention, deletion and export controls exist?
Each project can set its own retention window; data older than that is purged nightly, and users/API can trigger immediate deletion or export. See [data retention](/developer/platform/administration/data-retention) and [data deletion](/developer/platform/administration/data-deletion) documentation.
### Can I deploy ABV in a single-tenant environment?
Yes, please contact us for pricing and setup.
### How is tenant isolation enforced?
Cloud tenants are logically isolated by project‑level RBAC. Optional physical separation is available for enterprise clients. See [RBAC documentation](/developer/platform/administration/role-based-access-controls) for more details.
### Can customers pin data to specific regions?
Yes—EU, US and HIPAA‑ready US zones are available.
# Compliance & Certifications
### Which audits and attestations are in place?
ABV is ISO 27001 and ISO 42001 compliant; GDPR & HIPAA controls are implemented. See [see compliance overview](/security/compliance/iso-27001-compliance) for more details.
### How often are third‑party pen tests performed, and are results shareable?
Independent penetration tests occur annually, plus continuous vulnerability scans. See penetration testing for more details and past results.
# Identity & Access Management
### Which authentication options are supported?
OIDC SSO, email/password, and SCIM provisioning; MFA or passkeys can be enforced via your IdP. See auth documentation for more details.
### How is least‑privilege enforced?
RBAC lets you scope roles to organisation or project. See [RBAC documentation](/developer/platform/administration/role-based-access-controls) for more details.
# Infrastructure & Network Security
### Where is ABV hosted and how is the perimeter protected?
ABV runs on AWS in isolated VPCs with WAF and AWS Shield for DDoS mitigation.
# Application Security & SDLC
### What secure‑coding and testing practices are in place?
Every commit passes our CI pipeline of end-to-end, unit, and security tests.
# Incident Response & Business Continuity
### What is the incident‑response process?
24 × 7 monitoring triggers an on‑call engineer; affected customers are notified and post‑mortems are published for larger incidents. See [Incident & Breach](/security/security/incident-breach) documentation for more details.
# Vulnerability & Pen‑Testing
### How is the disclosure program run?
ABV maintains a public responsible‑disclosure policy; CVSS drives remediation SLAs. See [penetration testing](/security/security/penetration-testing) for more details.
### Can customers run their own pen‑tests?
No.
# Sub‑processors & Third‑Party Risk
### Which sub‑processors have access to customer data?
Please contact [privacy@abv.dev](mailto:privacy@abv.dev) for the current list of sub-processors.
# AI / LLM‑Specific Concerns
### Does ABV store PII or trade secrets from prompts?
ABV stores the data as-is. You can redact sensitive data via [data masking](/developer/basic-features/masking-sensitive-data).
### Can long‑term retention be disabled?
Yes—you can configure custom [data retention](/developer/platform/administration/data-retention) policies.
### Is prompt/trace data ever used for benchmarking or training?
No. ABV does not repurpose customer data for external benchmarks or model training. See [security overview](/developer/guardrails/llm-security-guardrails-overview) for more details.
# Governance, People & Culture
### How are employees vetted and trained?
All staff pass background checks, sign NDAs and complete security training. See [ISO 27001 compliance](/security/compliance/iso-27001-compliance) and [ISO 42001 compliance](/security/compliance/iso-42001-compliance) for more details.
### Who owns security inside ABV?
ABV CISO leads security efforts at ABV and reviews risk quarterly and drives continuous improvement. See [ISO 27001 compliance](/security/compliance/iso-27001-compliance) and [ISO 42001 compliance](/security/compliance/iso-42001-compliance) for more details.
--------------------------------------------------------------------------------
## Policies
URL: /security/compliance/policies
Our compliance program is based on the following internal policies:
- AI Data Handling SOP
- AI Development and Evaluation Policy
- AI Ethics Policy
- AI Risk Management Policy
- AI Security SOP
- AI Systems Lifecycle Policy
- AI Policy
- Background Check SOP
- BCP Tabletop Exercise SOP
- Business Continuity Plan
- BYOD Policy
- Code of Conduct
- Corporate Security Policy
- Data Classification Policy
- Data Protection Policy
- Data Retention Policy
- Data Retention Schedule
- Disaster Recovery Policy
- Incident Response Plan
- IR Tabletop Exercise SOP
- ISMS & AIMS Plan
- Organizational Controls Policy
- Password Management Policy
- People Controls Policy
- Physical Controls Policy
- Responsible AI Governance Policy
- Responsible Disclosure Policy
- Risk Assessment Policy
- Secure SDLC SOP
- Software Development Lifecycle Policy
- Technological Controls Policy
- Use of Cloud Services SOP
--------------------------------------------------------------------------------
## HIPAA Compliance
URL: /security/compliance/hipaa-compliance
ABV is aligned with **HIPAA**, enabling healthcare organizations and partners to use ABV while adhering to the requirements of the Health Insurance Portability and Accountability Act (HIPAA). ABV offers a Business Associate Agreement (BAA) to cover the safeguarding of Protected Health Information (PHI).
For questions regarding HIPAA compliance, please contact [security@abv.dev](mailto:security@abv.dev).
--------------------------------------------------------------------------------
## ISO 27001 Compliance
URL: /security/compliance/iso-27001-compliance
ABV Cloud is **ISO 27001 certified**. This international standard specifies the requirements for establishing, implementing, maintaining, and continually improving an information security management system (ISMS).
- **Status:** Certified
- **Certificate Availability:** The ISO 27001 certificate is available upon request for customers on the Pro or Enterprise plans.
- **Request Access:** You can request access to the certificate [here](https://webforms.pipedrive.com/f/bYZXNyRLrqiEcMydUyDJpJDXQ3XChKLKmaft0Khn2t7AmItimt6piRq2Y2IzBOCCjN).
## Contact
For questions regarding ISO 27001 compliance, please contact [security@abv.dev](mailto:security@abv.dev).
--------------------------------------------------------------------------------
## ISO 42001 Compliance
URL: /security/compliance/iso-42001-compliance
ABV Cloud is **ISO 42001 certified**. This international standard specifies the requirements for establishing, implementing, maintaining, and continually improving an information security management system (ISMS).
- **Status:** Certified
- **Certificate Availability:** The ISO 42001 certificate is available upon request for customers on the Pro or Enterprise plans.
- **Request Access:** You can request access to the certificate [here](https://webforms.pipedrive.com/f/bYZXNyRLrqiEcMydUyDJpJDXQ3XChKLKmaft0Khn2t7AmItimt6piRq2Y2IzBOCCjN).
## Contact
For questions regarding ISO 42001 compliance, please contact [security@abv.dev](mailto:security@abv.dev).
--------------------------------------------------------------------------------
## NIS 2 Compliance
URL: /security/compliance/nis2-compliance
ABV complies with the **EU NIS2 Directive**, which sets cybersecurity risk management and incident reporting requirements for essential and important entities across the European Union. Our compliance with **ISO 27001** serves as a strong foundation for meeting NIS2 obligations, as the international standard specifies requirements for establishing, implementing, maintaining, and continually improving an information security management system (ISMS).
### Status: **Aligned with NIS2**
ABV’s ISO 27001-certified ISMS provides the necessary controls, governance, and security practices that map closely to NIS2 requirements. We continue to enhance our security and compliance measures in line with the directive.
### Documentation Availability
While ISO 27001 certification is available upon request, NIS2 does not issue certificates in the same way. Instead, ABV can provide:
- A mapping of ISO 27001 controls to NIS2 requirements
- Details on our security governance and incident response practices
- Assurance statements regarding our ongoing compliance efforts
### Request Access
To request documentation related to NIS2 compliance, please contact us at [security@abv.dev](mailto:security@abv.dev).
--------------------------------------------------------------------------------
## Managing Personal Data
URL: /security/privacy/managing-personal-data
It is up to you to decide which kind of personal data you want to process with ABV. You can manage how personal data is processed and retained via:
- [Data masking](/developer/basic-features/masking-sensitive-data): Mask PII data in traces, observations, and scores.
- [Data deletion](/developer/platform/administration/data-deletion): Delete personal data upon request; add [userIds](/developer/basic-features/user-tracking) to tracing data to facilitate efficient deletion.
- [Data retention](/developer/platform/administration/data-retention): Control how long data is stored in ABV.
--------------------------------------------------------------------------------
## GDPR Compliance
URL: /security/privacy/gdpr-compliance
ABV is committed to complying with the General Data Protection Regulation (GDPR). We ensure that personal data is processed lawfully, fairly, and transparently.
More information:
- [ABV DPA](/security/privacy/data-processing-agreement)
- [Managing processing of personal data (within ABV)](./managing-personal-data)
- [Privacy Policy](https://abv.dev/privacy) for details on how we handle personal data
# Contact
For questions regarding GDPR compliance or data privacy, please contact [privacy@abv.dev](mailto:privacy@abv.dev).
# Data Subject Access Request (DSAR)
Under applicable data protection laws, you may have the right to request access to and receive information about the personal data we maintain about you, to update and correct inaccuracies in your personal data, to restrict or object to the processing of your personal data, to have the information anonymized or deleted, as appropriate, or to exercise your right to data portability.
To submit a Data Subject Access Request, please email us at [privacy@abv.dev](mailto:privacy@abv.dev). We will respond to your request within the time limits established by applicable law.
--------------------------------------------------------------------------------
## Data Processing Agreement (DPA)
URL: /security/privacy/data-processing-agreement
We offer a Data Processing Agreement (DPA) to govern the processing of personal data on behalf of our customers.
- **Availability:** Available upon request for customers on the Pro or Enterprise plans.
- **Template:** You can review our standard DPA template by contacting [privacy@abv.dev](mailto:privacy@abv.dev).
- **Request Process:** To execute a DPA, please email [privacy@abv.dev](mailto:privacy@abv.dev) with a signed copy of the template. We will then countersign and return it.
--------------------------------------------------------------------------------
## Privacy FAQ
URL: /security/privacy/faq
This page addresses frequently asked questions and common privacy topics for ABV.
If you don’t find a solution to your issue here, reach out to [privacy@abv.dev](mailto:privacy@abv.dev).
### Do you use customer data to train AI models?
No. ABV does not train or fine‑tunes ML/LLM models on Client Data. See [terms of service](https://abv.dev/tos) for more details.
### **Can we enter into a Data Processing Agreement (DPA)?**
Yes. You can enter into a [DPA](/security/privacy/data-processing-agreement) with ABV.
### **How long is data retained?**
[Data retention](/developer/platform/administration/data-retention) can be defined via project‑level policies: ≥ 3 days up to unlimited. Data is purged nightly.
### **How can we delete data?**
Customers can delete single traces, batches, projects, organisations, or user accounts via the UI or the public API. See [data deletion documentation](/developer/platform/administration/data-deletion) for more details.
### **How do you delete data after the End of the Contract?**
Upon termination, ABV will return or destroy Client Data within 30 days. If return or destruction is required or infeasible, ABV will continue to protect Client Data and limit further uses and disclosures.
--------------------------------------------------------------------------------