ABV supports multi-modal traces including text, images, audio, and other attachments. By default, base64 encoded data URIs are handled automatically by the ABV SDKs. They are extracted from the payloads commonly used in multi-modal LLMs, uploaded to ABV’s object storage, and linked to the trace. This also works if you:

Reference media files via external URLs.
Customize the handling of media files in the SDKs via the ABVMedia class.
Integrate via the ABV API directly.

Learn more on how to get started and how this works under the hood below. Examples:

Supported media formats

ABV supports:

Images: .png, .jpg, .webp
Audio files: .mpeg, .mp3, .wav
Other attachments: .pdf, plain text

Get Started

Base64 data URI encoded media

If you use base64 encoded images, audio, or other files in your LLM applications, upgrade to the latest version of the ABV SDKs. The ABV SDKs automatically detect and handle base64 encoded media by extracting it, uploading it separately as a ABV Media file, and including a reference in the trace. This works with standard Data URI (MDN) formatted media (like those used by OpenAI and other LLMs). For examples using the OpenAI SDK and LangChain, refer to the Python SDK instrumentation guide or JavaScript/TypeScript SDK instrumentation guide.

External media (URLs)

ABV supports in-line rendering of media files via URLs if they follow common formats. In this case, the media file is not uploaded to ABV’s object storage but simply rendered in the UI directly from the source.

Supported formats:

Markdown images

![Alt text](https://example.com/image.jpg)

OpenAI content parts

{
  "content": [
    {
      "role": "system",
      "content": "You are an AI trained to describe and interpret images. Describe the main objects and actions in the image."
    },
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "What's happening in this image?"
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "https://example.com/image.jpg"
          }
        }
      ]
    }
  ]
}

Custom attachments

If you want to have more control or your media is not base64 encoded, you can upload arbitrary media attachments to ABV via the SDKs using the new ABVMedia class. Wrap media with ABVMedia before including it in trace inputs, outputs, or metadata. See the multi-modal documentation for examples.

Python SDK

Install package

pip install abvdev

from abvdev import ABV, observe
from abvdev.media import ABVMedia

# ABV client initialization          
abv = ABV(
    api_key="sk-abv-...", # your api key here
    host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)

# Create a ABVMedia object from a file
with open("static/bitcoin.pdf", "rb") as pdf_file:
    pdf_bytes = pdf_file.read()

# Wrap media in ABVMedia class

pdf_media = ABVMedia(content_bytes=pdf_bytes, content_type="application/pdf")

# Using with the decorator

@observe()
def process_document():
    # Update the current trace with the media file
    abv.update_current_trace(
        metadata={"document": pdf_media}
    )

    # Or update the current span
    abv.update_current_span(
        input={"document": pdf_media}
    )

# Using with context managers
with abv.start_as_current_span(name="analyze-document") as span: # Include media in the span input, output, or metadata
    span.update(
        input={"document": pdf_media},
        metadata={"file_size": len(pdf_bytes)}
    )

    # Process document...

    # Add results with media to the output
    span.update(output={
        "summary": "This document explains Bitcoin...",
        "original": pdf_media
    })

JS/TS SDK

Install packages

npm install @abvdev/tracing @abvdev/otel @opentelemetry/sdk-node dotenv

npm install @abvdev/core

Add credentials Add your ABV credentials to your environment variables. Make sure that you have a .env file in your project root and a package like dotenv to load the variables. Createinstrumentation.ts file and use dotenv package to load the variables. Additional parameters are provided to get trace visible in the UI immediately. Import the instrumentation.ts file at the top of your application.

import fs from "fs";
import { ABVMedia } from "@abvdev/core";
import { startObservation } from "@abvdev/tracing";
 
// Wrap media in ABVMedia class
const wrappedMedia = new ABVMedia({
  source: "bytes",
  contentBytes: fs.readFileSync(new URL("./bitcoin.pdf", import.meta.url)),
  contentType: "application/pdf",
});
 
// Optionally, access media via wrappedMedia.obj
console.log(wrappedMedia.obj);
 
// Include media in any trace or observation
const span3 = startObservation("media-pdf-generation");
 
const generation3 = span3.startObservation('llm-call', {
  model: 'gpt-4',
  input: wrappedMedia,
}, {asType: "generation"});
 
generation3.end();
 
span3.end();

API

If you use the API directly to log traces to ABV, you need to follow these steps:

1) Upload media to ABV

If you use base64 encoded media: you need to extract it from the trace payloads similar to how the ABV SDKs do it.
Initialize the upload and get a mediaId and presignedURL: POST /api/public/media.
Upload media file: PUT [presignedURL].

See the Public API documentation for details on using the API directly to upload media files.

2) Add reference to mediaId in trace/observation

Use the ABV Media Token to reference the mediaId in the trace or observation input, output, or metadata.

How does it work?

When using media files (that are not referenced via external URLs), ABV handles them in the following way:

1. Media Upload Process

Detection and Extraction

ABV supports media files in traces and observations on input, output, and metadata fields
SDKs separate media from tracing data client-side for performance optimization
Media files are uploaded directly to object storage (AWS S3 or compatible)
Original media content is replaced with a reference string

Security and Optimization

Uploads use presigned URLs with content validation (content length, content type, content SHA256 hash)
Deduplication: Files are simply replaced by their mediaId reference string if already uploaded
File uniqueness determined by project, content type, and content SHA256 hash

Implementation Details

Python SDK: Background thread handling for non-blocking execution
JS/TS SDKs: Asynchronous, non-blocking implementation
API support for direct uploads (see Public API documentation)

2. Media Reference System

The base64 data URIs and the wrapped ABVMedia objects in ABV traces are replaced by references to the mediaId in the following standardized token format, which helps reconstruct the original payload if needed:

@@@abvdevMedia:type={MIME_TYPE}|id={ABV_MEDIA_ID}|source={SOURCE_TYPE}@@@

MIME_TYPE: MIME type of the media file, e.g., image/jpeg
ABV_MEDIA_ID: ID of the media file in ABV’s object storage
SOURCE_TYPE: Source type of the media file, can be base64_data_uri, bytes, or file

Based on this token, the ABV UI can automatically detect the mediaId and render the media file inline. The ABVMedia class provides utility functions to extract the mediaId from the reference string.

3. Resolving Media References

When dealing with traces, observations, ~~or ~~dataset items that include media references, you can convert them back to their base64 data URI format using the resolve_media_references utility method provided by the ABV client. This is particularly useful for reinserting the original content during fine-tuning, dataset runs, or replaying a generation. The utility method traverses the parsed object and returns a deep copy with all media reference strings replaced by the corresponding base64 data URI representations.

Python SDK

from abvdev import ABV

# ABV client initialization          
abv = ABV(
    api_key="sk-abv-...", # your api key here
    host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)

# Example object with media references
obj = {
    "summary": "This document explains Bitcoin...",
    "original": "@@@abvMedia:type=application/pdf|id=your_attached_document_id|source=bytes@@@"
}

# Resolve media references to base64 data URIs
resolved_obj = abv.resolve_media_references(
    obj=obj,
    resolve_with="base64_data_uri"
)

print(resolved_obj)

# Result:
# {
#     "summary": "This document explains Bitcoin...",
#     "original": "data:application/pdf;base64,JVBERi0xLjQK..."
# }

JS/TS SDK

npm install @abvdev/client

import "./instrumentation"; // Must be the first import
import { ABVClient } from "@abvdev/client";
 
const abv = new ABVClient()
 
// Example object with media references
const obj = {
  summary: "This document explains Bitcoin...",
  pdf: "@@@abvdevMedia:type=application/pdf|id=sWdBkaiOxc3XM-QkCoGAMQ|source=base64_data_uri@@@",
};
 
async function main() {
    // Resolve media references to base64 data URIs
    const resolvedTrace = await abv.media.resolveReferences({
    obj: obj,
    resolveWith: "base64DataUri",
    });

    console.log(resolvedTrace);
    
    // Result:
    // {
    // summary: 'This document explains Bitcoin...',
    // pdf: 'data:application/pdf;base64,JVBERi0x...'
    // }
}

main();

Getting Started

Basic Features

LLM Gateway

Guardrails

Evaluations

Prompt Management

Cookbook

SDKs

Platform

Support

Multi-Modality and Attachments

Supported media formats

Get Started

Base64 data URI encoded media

External media (URLs)

Supported formats:

Custom attachments

Python SDK

JS/TS SDK

API

1) Upload media to ABV

2) Add reference to mediaId in trace/observation

How does it work?

1. Media Upload Process

Detection and Extraction

Security and Optimization

Implementation Details

2. Media Reference System

3. Resolving Media References

Python SDK

JS/TS SDK

​Supported media formats

​Get Started

​Base64 data URI encoded media

​External media (URLs)

​Supported formats:

​Custom attachments

​Python SDK

​JS/TS SDK

​API

​1) Upload media to ABV

​2) Add reference to mediaId in trace/observation

​How does it work?

​1. Media Upload Process

​Detection and Extraction

​Security and Optimization

​Implementation Details

​2. Media Reference System

​3. Resolving Media References

​Python SDK

​JS/TS SDK

Supported media formats

Get Started

Base64 data URI encoded media

External media (URLs)

Supported formats:

Custom attachments

Python SDK

JS/TS SDK

API

1) Upload media to ABV

2) Add reference to mediaId in trace/observation

How does it work?

1. Media Upload Process

Detection and Extraction

Security and Optimization

Implementation Details

2. Media Reference System

3. Resolving Media References

Python SDK

JS/TS SDK