> ## Documentation Index
> Fetch the complete documentation index at: https://docs.abv.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Multi-Modality and Attachments

ABV supports multi-modal traces including text, images, audio, and other attachments.

By default, [base64 encoded data URIs](https://developer.mozilla.org/en-US/docs/Web/URI/Schemes/data#syntax) are handled automatically by the ABV SDKs. They are extracted from the payloads commonly used in multi-modal LLMs, uploaded to ABV's object storage, and linked to the trace.

This also works if you:

1. Reference media files via external URLs.
2. Customize the handling of media files in the SDKs via the `ABVMedia` class.
3. Integrate via the ABV API directly.

Learn more on how to get started and how this works under the hood below.

*Examples:*

<img src="https://mintlify.s3.us-west-1.amazonaws.com/abv-2be93c70/images/rJw1gdzAXF0t0odR3JQVw_image.png" alt="" />

<img src="https://mintlify.s3.us-west-1.amazonaws.com/abv-2be93c70/images/55CYeDGMbH686V44gx6Lf_image.png" alt="" />

<img src="https://mintlify.s3.us-west-1.amazonaws.com/abv-2be93c70/images/cMyL70KRH2Y-UwUr2F7UB_image.png" alt="" />

# Supported media formats

ABV supports:

* **Images**: .png, .jpg, .webp
* **Audio files**: .mpeg, .mp3, .wav
* **Other attachments**: .pdf, plain text

# Get Started

## Base64 data URI encoded media

If you use base64 encoded images, audio, or other files in your LLM applications, upgrade to the latest version of the ABV SDKs. The ABV SDKs automatically detect and handle base64 encoded media by extracting it, uploading it separately as a ABV Media file, and including a reference in the trace.

This works with standard Data URI ([MDN](https://developer.mozilla.org/en-US/docs/Web/URI/Schemes/data#syntax)) formatted media (like those used by OpenAI and other LLMs).

For examples using the OpenAI SDK and LangChain, refer to the [Python SDK instrumentation guide](/developer/sdks/python/instrumentation) or [JavaScript/TypeScript SDK instrumentation guide](/developer/sdks/js-ts/instrumentation).

## External media (URLs)

ABV supports in-line rendering of media files via URLs if they follow common formats. In this case, the media file is not uploaded to ABV's object storage but simply rendered in the UI directly from the source.

### Supported formats:

**Markdown images**

```md theme={null}
![Alt text](https://example.com/image.jpg)
```

**OpenAI content parts**

```json theme={null}
{
  "content": [
    {
      "role": "system",
      "content": "You are an AI trained to describe and interpret images. Describe the main objects and actions in the image."
    },
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "What's happening in this image?"
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "https://example.com/image.jpg"
          }
        }
      ]
    }
  ]
}
```

## Custom attachments

If you want to have more control or your media is not base64 encoded, you can upload arbitrary media attachments to ABV via the SDKs using the new `ABVMedia` class. Wrap media with ABVMedia before including it in trace inputs, outputs, or metadata. See the multi-modal documentation for examples.

### Python SDK

Install package

```bash theme={null}
pip install abvdev
```

```python theme={null}
from abvdev import ABV, observe
from abvdev.media import ABVMedia

# ABV client initialization          
abv = ABV(
    api_key="sk-abv-...", # your api key here
    host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)

# Create a ABVMedia object from a file
with open("static/bitcoin.pdf", "rb") as pdf_file:
    pdf_bytes = pdf_file.read()

# Wrap media in ABVMedia class

pdf_media = ABVMedia(content_bytes=pdf_bytes, content_type="application/pdf")

# Using with the decorator

@observe()
def process_document():
    # Update the current trace with the media file
    abv.update_current_trace(
        metadata={"document": pdf_media}
    )

    # Or update the current span
    abv.update_current_span(
        input={"document": pdf_media}
    )

# Using with context managers
with abv.start_as_current_span(name="analyze-document") as span: # Include media in the span input, output, or metadata
    span.update(
        input={"document": pdf_media},
        metadata={"file_size": len(pdf_bytes)}
    )

    # Process document...

    # Add results with media to the output
    span.update(output={
        "summary": "This document explains Bitcoin...",
        "original": pdf_media
    })
```

### JS/TS SDK

**Install packages**

```bash theme={null}
npm install @abvdev/tracing @abvdev/otel @opentelemetry/sdk-node dotenv
```

```bash theme={null}
npm install @abvdev/core
```

**Add credentials**

Add your ABV credentials to your environment variables. Make sure that you have a `.env` file in your project root and a package like dotenv to load the variables.

<CodeblockTabs>
  .env

  ```typescript theme={null}
  ABV_API_KEY = "sk-abv-..."
  ABV_BASE_URL = "https://app.abv.dev" # US region
  //ABV_BASE_URL = "https://eu.app.abv.dev" # EU region
  ```
</CodeblockTabs>

Create`instrumentation.ts` file and use `dotenv` package to load the variables.

Additional parameters are provided to get trace visible in the UI immediately.

<CodeblockTabs>
  instrumentation.ts

  ```typescript theme={null}
  import dotenv from "dotenv";
  dotenv.config();

  import { NodeSDK } from "@opentelemetry/sdk-node";
  import { ABVSpanProcessor } from "@abvdev/otel";

  const sdk = new NodeSDK({
    spanProcessors: [
      new ABVSpanProcessor({
        apiKey: process.env.ABV_API_KEY,
        baseUrl: process.env.ABV_BASE_URL,
        exportMode: "immediate",
        flushAt: 1,
        flushInterval: 1,
        additionalHeaders: {
          "Content-Type": "application/json",
          "Accept": "application/json"
        }
      })
    ],
  });

  sdk.start();
  ```
</CodeblockTabs>

Import the `instrumentation.ts` file at the top of your application.

```typescript theme={null}
import fs from "fs";
import { ABVMedia } from "@abvdev/core";
import { startObservation } from "@abvdev/tracing";
 
// Wrap media in ABVMedia class
const wrappedMedia = new ABVMedia({
  source: "bytes",
  contentBytes: fs.readFileSync(new URL("./bitcoin.pdf", import.meta.url)),
  contentType: "application/pdf",
});
 
// Optionally, access media via wrappedMedia.obj
console.log(wrappedMedia.obj);
 
// Include media in any trace or observation
const span3 = startObservation("media-pdf-generation");
 
const generation3 = span3.startObservation('llm-call', {
  model: 'gpt-4',
  input: wrappedMedia,
}, {asType: "generation"});
 
generation3.end();
 
span3.end();
```

## API

If you use the API directly to log traces to ABV, you need to follow these steps:

### 1) Upload media to ABV

1. If you use base64 encoded media: you need to extract it from the trace payloads similar to how the ABV SDKs do it.
2. Initialize the upload and get a `mediaId` and `presignedURL`: `POST /api/public/media`.
3. Upload media file: `PUT [presignedURL]`.

See the [Public API documentation](/developer/platform/api-data-platform/public-api) for details on using the API directly to upload media files.

### 2) Add reference to mediaId in trace/observation

Use the *ABV Media Token* to reference the `mediaId` in the trace or observation `input`, `output`, or `metadata`.

# How does it work?

When using media files (that are not referenced via external URLs), ABV handles them in the following way:

## 1. Media Upload Process

### Detection and Extraction

* ABV supports media files in traces and observations on `input`, `output`, and `metadata` fields
* SDKs separate media from tracing data client-side for performance optimization
* Media files are uploaded directly to object storage (AWS S3 or compatible)
* Original media content is replaced with a reference string

### Security and Optimization

* Uploads use presigned URLs with content validation (content length, content type, content SHA256 hash)
* Deduplication: Files are simply replaced by their `mediaId` reference string if already uploaded
* File uniqueness determined by project, content type, and content SHA256 hash

### Implementation Details

* Python SDK: Background thread handling for non-blocking execution
* JS/TS SDKs: Asynchronous, non-blocking implementation
* API support for direct uploads (see [Public API documentation](/developer/platform/api-data-platform/public-api))

## 2. Media Reference System

The base64 data URIs and the wrapped `ABVMedia` objects in ABV traces are replaced by references to the `mediaId` in the following standardized token format, which helps reconstruct the original payload if needed:

```text theme={null}
@@@abvdevMedia:type={MIME_TYPE}|id={ABV_MEDIA_ID}|source={SOURCE_TYPE}@@@
```

* `MIME_TYPE`: MIME type of the media file, e.g., `image/jpeg`
* `ABV_MEDIA_ID`: ID of the media file in ABV's object storage
* `SOURCE_TYPE`: Source type of the media file, can be `base64_data_uri`, `bytes`, or `file`

Based on this token, the ABV UI can automatically detect the `mediaId` and render the media file inline. The `ABVMedia` class provides utility functions to extract the `mediaId` from the reference string.

## 3. Resolving Media References

When dealing with traces, observations, \~\~or \~\~dataset items that include media references, you can convert them back to their base64 data URI format using the `resolve_media_references` utility method provided by the ABV client. This is particularly useful for reinserting the original content during fine-tuning, dataset runs, or replaying a generation. The utility method traverses the parsed object and returns a deep copy with all media reference strings replaced by the corresponding base64 data URI representations.

### Python SDK

```python theme={null}
from abvdev import ABV

# ABV client initialization          
abv = ABV(
    api_key="sk-abv-...", # your api key here
    host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)

# Example object with media references
obj = {
    "summary": "This document explains Bitcoin...",
    "original": "@@@abvMedia:type=application/pdf|id=your_attached_document_id|source=bytes@@@"
}

# Resolve media references to base64 data URIs
resolved_obj = abv.resolve_media_references(
    obj=obj,
    resolve_with="base64_data_uri"
)

print(resolved_obj)

# Result:
# {
#     "summary": "This document explains Bitcoin...",
#     "original": "data:application/pdf;base64,JVBERi0xLjQK..."
# }
```

### JS/TS SDK

```bash theme={null}
npm install @abvdev/client
```

```typescript theme={null}
import "./instrumentation"; // Must be the first import
import { ABVClient } from "@abvdev/client";
 
const abv = new ABVClient()
 
// Example object with media references
const obj = {
  summary: "This document explains Bitcoin...",
  pdf: "@@@abvdevMedia:type=application/pdf|id=sWdBkaiOxc3XM-QkCoGAMQ|source=base64_data_uri@@@",
};
 
async function main() {
    // Resolve media references to base64 data URIs
    const resolvedTrace = await abv.media.resolveReferences({
    obj: obj,
    resolveWith: "base64DataUri",
    });

    console.log(resolvedTrace);
    
    // Result:
    // {
    // summary: 'This document explains Bitcoin...',
    // pdf: 'data:application/pdf;base64,JVBERi0x...'
    // }
}

main();
```