Multi-Modality and Attachments

abv supports multi modal traces including text, images, audio, and other attachments by default, base64 encoded data uris https //developer mozilla org/en us/docs/web/uri/schemes/data#syntax are handled automatically by the abv sdks they are extracted from the payloads commonly used in multi modal llms, uploaded to abv's object storage, and linked to the trace this also works if you reference media files via external urls customize the handling of media files in the sdks via the abvmedia class integrate via the abv api directly learn more on how to get started and how this works under the hood below examples supported media formats abv supports images png, jpg, webp audio files mpeg, mp3, wav other attachments pdf, plain text get started base64 data uri encoded media if you use base64 encoded images, audio, or other files in your llm applications, upgrade to the latest version of the abv sdks the abv sdks automatically detect and handle base64 encoded media by extracting it, uploading it separately as a abv media file, and including a reference in the trace this works with standard data uri ( mdn https //developer mozilla org/en us/docs/web/uri/schemes/data#syntax ) formatted media (like those used by openai and other llms) this notebook includes a couple of examples using the openai sdk and langchain external media (urls) abv supports in line rendering of media files via urls if they follow common formats in this case, the media file is not uploaded to abv's object storage but simply rendered in the ui directly from the source supported formats markdown images !\[alt text]\(https //example com/image jpg) openai content parts { "content" \[ { "role" "system", "content" "you are an ai trained to describe and interpret images describe the main objects and actions in the image " }, { "role" "user", "content" \[ { "type" "text", "text" "what's happening in this image?" }, { "type" "image url", "image url" { "url" "https //example com/image jpg" } } ] } ] } custom attachments if you want to have more control or your media is not base64 encoded, you can upload arbitrary media attachments to abv via the sdks using the new abvemedia class wrap media with abvmedia before including it in trace inputs, outputs, or metadata see the multi modal documentation for examples python sdk from abvdev import get client, observe from abvdev media import abvmedia \# create a abvmedia object from a file with open("static/bitcoin pdf", "rb") as pdf file pdf bytes = pdf file read() \# wrap media in abvmedia class pdf media = abvmedia(content bytes=pdf bytes, content type="application/pdf") \# using with the decorator @observe() def process document() abv = get client() \# update the current trace with the media file abv update current trace( metadata={"document" pdf media} ) \# or update the current span abv update current span( input={"document" pdf media} ) \# using with context managers abv = get client() with abv start as current span(name="analyze document") as span # include media in the span input, output, or metadata span update( input={"document" pdf media}, metadata={"file size" len(pdf bytes)} ) \# process document \# add results with media to the output span update(output={ "summary" "this document explains bitcoin ", "original" pdf media }) js/ts sdk import fs from "fs"; import { abvmedia } from "@abvdev/core"; import { startobservation } from "@abvdev/tracing"; // wrap media in abvmedia class const wrappedmedia = new abvmedia({ source "bytes", contentbytes fs readfilesync(new url(" /bitcoin pdf", import meta url)), contenttype "application/pdf", }); // optionally, access media via wrappedmedia obj console log(wrappedmedia obj); // include media in any trace or observation const span3 = startobservation("media pdf generation"); const generation3 = span3 startobservation('llm call', { model 'gpt 4', input wrappedmedia, }, {astype "generation"}); generation3 end(); span3 end(); api if you use the api directly to log traces to abv, you need to follow these steps 1\) upload media to abv if you use base64 encoded media you need to extract it from the trace payloads similar to how the abv sdks do it initialize the upload and get a mediaid and presignedurl post /api/public/media upload media file put \[presignedurl] see this end to end example (python) on how to use the api directly to upload media files 2\) add reference to mediaid in trace/observation use the abv media token to reference the mediaid in the trace or observation input , output , or metadata how does it work? when using media files (that are not referenced via external urls), abv handles them in the following way 1\ media upload process detection and extraction abv supports media files in traces and observations on input , output , and metadata fields sdks separate media from tracing data client side for performance optimization media files are uploaded directly to object storage (aws s3 or compatible) original media content is replaced with a reference string security and optimization uploads use presigned urls with content validation (content length, content type, content sha256 hash) deduplication files are simply replaced by their mediaid reference string if already uploaded file uniqueness determined by project, content type, and content sha256 hash implementation details python sdk background thread handling for non blocking execution js/ts sdks asynchronous, non blocking implementation api support for direct uploads (see guide ) 2\ media reference system the base64 data uris and the wrapped abvmedia objects in abv traces are replaced by references to the mediaid in the following standardized token format, which helps reconstruct the original payload if needed @@@abvdevmedia\ type={mime type}|id={abv media id}|source={source type}@@@ mime type mime type of the media file, e g , image/jpeg abv media id id of the media file in abv's object storage source type source type of the media file, can be base64 data uri , bytes , or file based on this token, the abv ui can automatically detect the mediaid and render the media file inline the abvmedia class provides utility functions to extract the mediaid from the reference string 3\ resolving media references when dealing with traces, observations, or dataset items that include media references, you can convert them back to their base64 data uri format using the resolve media references utility method provided by the abv client this is particularly useful for reinserting the original content during fine tuning, dataset runs, or replaying a generation the utility method traverses the parsed object and returns a deep copy with all media reference strings replaced by the corresponding base64 data uri representations python sdk from abvdev import get client \# initialize abv client abv = get client() \# example object with media references obj = { "image" "@@@abvdevmedia\ type=image/jpeg|id=some uuid|source=bytes@@@", "nested" { "pdf" "@@@abvdevmedia\ type=application/pdf|id=some other uuid|source=bytes@@@" } } \# resolve media references to base64 data uris resolved obj = abv resolve media references( obj=obj, resolve with="base64 data uri" ) \# result \# { \# "image" "data\ image/jpeg;base64,/9j/4aaqskzjrg ", \# "nested" { \# "pdf" "data\ application/pdf;base64,jvberi0xljck " \# } \# } js/ts sdk import { abvclient } from "@abvdev/client"; const abv = new abvclient() // example object with media references const obj = { image "@@@abvdevmedia\ type=image/jpeg|id=some uuid|source=bytes@@@", nested { pdf "@@@abvdevmedia\ type=application/pdf|id=some other uuid|source=bytes@@@", }, }; // resolve media references to base64 data uris const resolvedtrace = await abv media resolvereferences({ obj obj, resolvewith "base64datauri", }); // result // { // image "data\ image/jpeg;base64,/9j/4aaqskzjrg ", // nested { // pdf "data\ application/pdf;base64,jvberi0xljck " // } // }