Skip to main content
You can track the effect of changes to your LLM app on metrics in ABV. This allows you to:
  • Run experiments (A/B tests) in production and measure the impact on costs, latencies and quality.
    • Example: “What is the impact of switching to a new model?”
  • Explain changes to metrics over time.
    • Example: “Why did latency in this chain increase?”

​
Releases

Releases in ABV
A release tracks the overall version of your application. Commonly it is set to the semantic version or git commit hash of your application. The SDKs look for a release in the following order:
  1. SDK initialization
  2. Environment variable
  3. Automatically on popular platforms

​
SDK initialization

​
Environment variable

The SDKs will look for a ABV_RELEASE environment variable. Use it to configure the release e.g. in your CI/CD pipeline. 
ABV_RELEASE = "<release_tag>" # <- github sha or other identifier

​
Python SDK

Install packages 
pip install abvdev, python-dotenv
Add credentials and ABV_RELEASE environment variable: Add your ABV credentials to your environment variables. Make sure that you have a .env file in your project root.
.env
ABV_API_KEY = "sk-abv-..."
ABV_HOST = "https://app.abv.dev" # US region
# ABV_HOST = "https://eu.app.abv.dev" # EU region
ABV_RELEASE = "3.2.25"
load environment variables and initialize client with get_client: 
# Import the function to load the .env file
from dotenv import load_dotenv 
from abvdev import observe, get_client

# Load the environment variables from the .env file
load_dotenv() 

@observe
def my_function():
    return "Hello, world!" # Input/output and timings are automatically captured
 
my_function()
 
# Flush events in short-lived applications
abv = get_client()
abv.flush()
The Python SDK also allows you to set the release when initializing the client with client parameters: 
from abvdev import ABV, observe

# Set the release when initializing the client
abv = ABV(
    release="v2.1.25",
    api_key="sk-abv-...", # your api key here
    host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)

@observe
def my_function():
    return "Hello, world!" # Input/output and timings are automatically captured
 
my_function()
 
# Flush events in short-lived applications
abv.flush()

​
JS/TS SDK

Install packages 
npm install @abvdev/tracing @abvdev/otel @opentelemetry/sdk-node
**Add credentials and **ABV_RELEASE environment variable Add your ABV credentials to your environment variables. Make sure that you have a .env file in your project root and a package like dotenv to load the variables.
.env
ABV_API_KEY = "sk-abv-..."
ABV_BASE_URL = "https://app.abv.dev" # US region
# ABV_BASE_URL = "https://eu.app.abv.dev" # EU region
ABV_RELEASE = "1.0.0"
Create a instrumentation.ts file that initializes the OpenTelemetry NodeSDK and registers the ABVSpanProcessor. 
npm install dotenv
instrumentation.ts
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";

const sdk = new NodeSDK({
  spanProcessors: [new ABVSpanProcessor()],
});

sdk.start();
Modify instrumentation.ts file to use dotenv package to load the variables. Additional parameters are provided to get trace visible in the UI immediately.
instrumentation.ts
import dotenv from "dotenv";
dotenv.config();

import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";

const sdk = new NodeSDK({
  spanProcessors: [
    new ABVSpanProcessor({
      apiKey: process.env.ABV_API_KEY,
      baseUrl: process.env.ABV_BASE_URL,
      exportMode: "immediate",
      flushAt: 1,
      flushInterval: 1,
      additionalHeaders: {
        "Content-Type": "application/json",
        "Accept": "application/json"
      }
    })
  ],
});

sdk.start();
Import the instrumentation.ts file at the top of your application.
index.ts
import "./instrumentation"; // Must be the first import
Instrument application The JS/TS SDK will look for a ABV_RELEASE environment variable. Use it also to configure the release e.g. in your CI/CD pipeline. If no other release is set, the ABV SDKs default to a set of known release environment variables. Supported platforms include: Vercel, Heroku, Netlify. See the full list of support environment variables for JS/TS and .

​
Versions

Versions in ABV
The version parameter can be added to traces and all observation types (span, generation, event). Thereby, you can track the effect of a new version on the metrics of an object with a specific name using ABV analytics.

​
With Python SDK

When using the @observe() decorator, version could be specified at both trace or observation levels: 
from abvdev import ABV, observe

# ABV client initialization          
abv = ABV(
    api_key="sk-abv-...", # your api key here
    host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)

@observe()
def process_data():
    # Set version at trace level
    abv.update_current_trace(version="1.0")

    # Set version at observation level
    abv.update_current_span(version="1.0")
    
process_data()
When creating spans directly: 
from abvdev import ABV
from openai import OpenAI

# ABV client initialization          
abv = ABV(
    api_key="sk-abv-...", # your api key here
    host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)

openai_client = OpenAI(api_key = "sk-proj-...")

# Set version when creating a span
with abv.start_as_current_span(
    name="process-data",
    version="1.0"
) as span:
    # Processing...
    span.update(output="Processing complete")

    # Create a generation with version
    with span.start_as_current_observation(
        as_type='generation',
        name="guess-countries",
        model="gpt-4o",
        version="1.1"
    ) as generation:
        # Generation code...
        response = openai_client.chat.completions.create(
            messages=[{"role": "user", "content": "Hi openai, how is that going?"}],
            model="gpt-4o",
        )
            
        generation.update(output=response.choices[0].message.content)
        
# Flush events in short-lived applications
abv.flush()
The version parameter can be included in both spans and generations, and can be set either during creation or updated later.

​
With JS/TS SDK

import "./instrumentation";
import { startActiveObservation } from "@abvdev/tracing";

async function main() {
  await startActiveObservation("my-first-trace-version1", async (span) => {
    span.update({
      input: "Hello, ABV!",
      output: "This is my first trace!",
      version: "1.0"
    });
  });
}

main();
abv.trace(), abv.span() and abv.event() also take an optional version parameter.

​
In UI

Version parameter in ABV interface