Event Queuing/Batching

ABV’s client SDKs and integrations are all designed to queue and batch requests in the background to optimize API calls and network time. Batches are determined by a combination of time and size (number of events and size of batch).

Configuration

All integrations have a sensible default configuration, but you can customize the batching behaviour to suit your needs.

Option (Python) [SDK constructor, Environment]	Option (JS)	Description
`flush_at`, `ABV_FLUSH_AT`	`flushAt`	The maximum number of events to batch up before sending.
`flush_interval`, `ABV_FLUSH_INTERVAL` (s)	`flushInterval` (ms)	The maximum time to wait before sending a batch.

You can e.g. set flushAt=1 to send every event immediately, or flushInterval=1000 to send every second.

Manual flushing

This is especially relevant for short-lived applications like serverless functions. If you do not flush the client, you may lose events.

If you want to send a batch immediately, you can call the flush method on the client. In case of network issues, flush will log an error and retry the batch, it will never throw an exception.

Python SDK

Install package

pip install abvdev

from abvdev import ABV

# ABV client initialization          
abv = ABV(
    api_key="sk-abv-...", # your api key here
    host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)

# Flush all pending observations
abv.flush()

If you exit the application, use shutdown method to make sure all requests are flushed and pending requests are awaited before the process exits. On success of this function, no more events will be sent to ABV API.

from abvdev import ABV

# ABV client initialization          
abv = ABV(
    api_key="sk-abv-...", # your api key here
    host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)

abv.shutdown()

JS/TS SDK

Install packages

npm install @abvdev/tracing @abvdev/otel @opentelemetry/sdk-node dotenv

Add credentials Add your ABV credentials to your environment variables. Make sure that you have a .env file in your project root and a package like dotenv to load the variables.

.env

ABV_API_KEY="sk-abv-..."
ABV_BASE_URL="https://app.abv.dev" # US region
# ABV_BASE_URL="https://eu.app.abv.dev" # EU region

Createinstrumentation.ts file and use dotenv package to load the variables. Additional parameters are provided to get trace visible in the UI immediately.

instrumentation.ts

import dotenv from "dotenv";
dotenv.config();

import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";

const sdk = new NodeSDK({
  spanProcessors: [
    new ABVSpanProcessor({
      apiKey: process.env.ABV_API_KEY,
      baseUrl: process.env.ABV_BASE_URL,
      exportMode: "immediate",
      flushAt: 1,
      flushInterval: 1,
      additionalHeaders: {
        "Content-Type": "application/json",
        "Accept": "application/json"
      }
    })
  ],
});

sdk.start();

Import the instrumentation.ts file at the top of your application. The ABVSpanProcessor buffers events and sends them in batches, so a final flush ensures no data is lost. You can export the processor from your OTEL SDK setup file.

instrumentation.ts

import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";

// Export the processor to be able to flush it
export const abvSpanProcessor = new ABVSpanProcessor();

const sdk = new NodeSDK({
  spanProcessors: [abvSpanProcessor],
});

sdk.start();

Then, in your serverless function handler, call forceFlush() before the function exits.

handler.ts

import { abvSpanProcessor } from "./instrumentation";

export async function handler(event, context) {
  // ... your application logic ...

  // Flush before exiting
  await abvSpanProcessor.forceFlush();
}

Getting Started

Basic Features

LLM Gateway

Guardrails

Evaluations

Prompt Management

Cookbook

SDKs

Platform

Support

Configuration

Manual flushing

Python SDK

JS/TS SDK

​Configuration

​Manual flushing

​Python SDK

​JS/TS SDK

Configuration

Manual flushing

Python SDK

JS/TS SDK