> ## Documentation Index
> Fetch the complete documentation index at: https://docs.abv.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Remote Dataset Runs

Once you [created a dataset](/developer/evaluations/datasets), you can use the dataset to test how your application performs on different inputs.
Remote Dataset Runs are used to programmatically loop your applications or prompts through a dataset and optionally apply Evaluation Methods to the results.

They are called "Remote Dataset Runs" because they can make use of "remote" or external logic and code.

Optionally, you can also trigger Remote Dataset Runs via the ABV UI which will call them via a webhook.

# Why use Remote Dataset Runs?

* Full flexibility to use your own application logic
* Use custom scoring functions to evaluate the outputs
* Run multiple experiments on the same dataset in parallel
* Easy to integrate with your existing evaluation infrastructure

# Setup & Run via SDK

<Expandable title="Sequence Diagram">
  ```mermaid theme={null}
  sequenceDiagram
      actor Person as User
      participant User as Experiment Runner
      participant App as LLM Application
      participant LF as ABV Server

      Note over Person, LF: Setup & Execute Remote Dataset Run

      %% Trigger dataset run
      Person->>User: Trigger dataset run

      %% Load Dataset
      User->>LF: get_dataset("dataset_name")
      LF-->>User: Return dataset with items

      %% Loop through dataset items
      loop For each dataset item
          Note over User, LF: Process dataset item

          %% Start dataset run context
          User->>LF: item.run(run_name, description, metadata)
          LF-->>User: Return run context (root_span)

          %% Execute LLM application
          User->>App: Execute LLM app with item.input
          Note over App: Create ABV trace<br/>for execution
          App->>LF: App is instrumented with ABV and reports traces
          App-->>User: Return application output

          %% Optional: Add evaluation scores from experiment runner
          opt Add Custom Scores
              Note over User: Run evaluation function locally
              User->>LF: root_span.score_trace(name, value, comment)
          end

          %% Link execution to dataset item
          Note over User, LF: Trace automatically linked<br/>to dataset run
      end

      %% Flush data to server
      User->>LF: flush() - Send all pending data
      LF-->>User: Confirm data received
      User-->>Person: Dataset run complete

      %% Optional: ABV server-side evaluations
      opt Server-side Evaluations
          Note over LF: Run configured evaluations<br/>(e.g., LLM-as-a-judge)
          LF->>LF: Add evaluation scores to dataset run
      end

      %% View results in UI
      Note over Person, LF: Analyze dataset run results
      Person->>LF: Access dataset runs UI
      LF-->>Person: Display aggregated scores and comparisons
  ```
</Expandable>

## **1) Instrument your application**

First we create our application runner helper function. This function will be called for every dataset item in the next step. If you use ABV for production observability, you do not need to change your application code.

<Info>
  For a dataset run, it is important that your application creates ABV traces for each execution so they can be linked to the dataset item. Please refer to the integrations page for details on how to instrument the framework you are using.
</Info>

### Python SDK

Install packages

```bash theme={null}
pip install abvdev, openai
```

Assume you already have a ABV-instrumented LLM-app:

```python title="app.py" theme={null}
from abvdev import ABV, observe
from openai import OpenAI

abv = ABV(
    api_key="sk-abv-...", # your api key here
    host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)

openai_client = OpenAI(api_key="sk-proj-...")

@observe
def my_llm_function(question: str):
  with abv.start_as_current_observation(as_type="generation", name="OpenAI-gen"):
    response = openai_client.chat.completions.create(
        model="gpt-4o", messages=[{"role": "user", "content": question}]
    )

    output = response.choices[0].message.content

    # Update trace input / output
    abv.update_current_trace(input=question, output=output)

    return output
```

*See *[Python SDK](/developer/sdks/python/overview)* docs for more details.*

### JS/TS SDK

Please make sure you have the [JS/TS SDK set up](/developer/sdks/js-ts/setup) for tracing of your application. If you use ABV for [observability](/developer/basic-features/observability-tracing), this is the same setup.

**Install packages**

```bash theme={null}
npm install @abvdev/tracing @abvdev/otel @abvdev/client openai
```

**Add credentials**

Add your ABV credentials to your environment variables. Make sure that you have a `.env` file in your project root and a package like dotenv to load the variables.

```bash title=".env" theme={null}
ABV_API_KEY="sk-abv-..."
ABV_BASE_URL="https://app.abv.dev" # US region
# ABV_BASE_URL="https://eu.app.abv.dev" # EU region
OPENAI_API_KEY="sk-proj-..." # added to use openai llm
```

**Initialize OpenTelemetry**

Install the OpenTelemetry Node SDK package:

```bash theme={null}
npm install @opentelemetry/sdk-node
```

Create a `instrumentation.ts` file that initializes the OpenTelemetry `NodeSDK` and registers the `ABVSpanProcessor`.

```typescript title="instrumentation.ts" theme={null}
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";

const sdk = new NodeSDK({
  spanProcessors: [new ABVSpanProcessor()],
});

sdk.start();
```

Modify `instrumentation.ts` file to use `dotenv` package to load the variables.

Additional parameters are provided to get trace visible in the UI immediately.

```bash theme={null}
npm install dotenv
```

```typescript title="instrumentation.ts" theme={null}
import dotenv from "dotenv";
dotenv.config();

import { NodeSDK } from "@opentelemetry/sdk-node";
import { ABVSpanProcessor } from "@abvdev/otel";

const sdk = new NodeSDK({
  spanProcessors: [
    new ABVSpanProcessor({
      apiKey: process.env.ABV_API_KEY,
      baseUrl: process.env.ABV_BASE_URL,
      exportMode: "immediate",
      flushAt: 1,
      flushInterval: 1,
      additionalHeaders: {
        "Content-Type": "application/json",
        "Accept": "application/json"
      }
    })
  ],
});

sdk.start();
```

Import the `instrumentation.ts` file at the top of your application.

```typescript title="index.ts" theme={null}
import "./instrumentation"; // Must be the first import
```

Instrumentation:

```typescript title="app.ts" theme={null}
import { ABVClient } from "@abvdev/client";
import { startActiveObservation } from "@abvdev/tracing";
import OpenAI from "openai";
import dotenv from "dotenv";
dotenv.config();

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});


const myLLMApplication = async (input: string) => {
  return startActiveObservation("my-llm-application", async (span) => {
    const output = await openai.chat.completions.create({
      model: "gpt-4o",
      messages: [{ role: "user", content: input }],
    });

    span.update({ input, output: output.choices[0].message.content });

    // return reference to span and output
    // will be simplified in a future version of the SDK
    return [span, output] as const;
  })
};
```

## 2) Run experiment on dataset

When running an experiment on a dataset, the application that shall be tested is executed for each item in the dataset. The execution trace is then linked to the dataset item. This allows you to compare different runs of the same application on the same dataset. Each experiment is identified by a `run_name`.

### Python SDK

You may then execute that LLM-app for each dataset item to create a dataset run:

execute\_dataset.py

```python theme={null}
from abvdev import get_client
from .app import my_llm_application

# Load the dataset
dataset = get_client().get_dataset("<dataset_name>")

# Loop over the dataset items
for item in dataset.items:
    # Use the item.run() context manager for automatic trace linking
    with item.run(
        run_name="<run_name>",
        run_description="My first run",
        run_metadata={"model": "llama3"},
    ) as root_span:
        # Execute your LLM-app against the dataset item input
        output = my_llm_application.run(item.input)

        # Optionally: Add scores computed in your experiment runner, e.g. json equality check
        root_span.score_trace(
            name="<example_eval>",
            value=my_eval_fn(item.input, output, item.expected_output),
            comment="This is a comment",  # optional, useful to add reasoning
        )

# Flush the abv client to ensure all data is sent to the server at the end of the experiment run
get_client().flush()
```

*See *[Python SDK](/developer/sdks/python/overview)* docs for details on the new OpenTelemetry-based SDK.*

### JS/TS SDK

```typescript theme={null}
import { ABVClient } from "@abvdev/client";
 
const abv = new ABVClient();
 
for (const item of dataset.items) {
  // execute application function and get abvObject (trace/span/generation/event)
  // output also returned as it is used to evaluate the run
  // you can also link using ids, see sdk reference for details
  const [span, output] = await myLlmApplication.run(item.input);
 
  // link the execution trace to the dataset item and give it a run_name
  await item.link(span, "<run_name>", {
    description: "My first run", // optional run description
    metadata: { model: "llama3" }, // optional run metadata
  });
 
  // Optionally: Add scores
  abv.score.trace(span, {
    name: "<score_name>",
    value: myEvalFunction(item.input, output, item.expectedOutput),
    comment: "This is a comment", // optional, useful to add reasoning
  });
}
 
// Flush the abv client to ensure all score data is sent to the server at the end of the experiment run
await abv.flush();
```

If you want to learn more about how adding evaluation scores from the code works, please refer to the docs:

[Add custom scores](./custom-scores)

## 3) Optionally: Run Evals in ABV

In the code above, we show how to add scores to the dataset run from your experiment code.

Alternatively, you can run evals in ABV. This is useful if you want to use the [LLM-as-a-Judge](./llm-as-a-judge) feature to evaluate the outputs of the dataset runs.

[Set up LLM-as-a-judge](./llm-as-a-judge)

## 4) Compare dataset runs

After each experiment run on a dataset, you can check the aggregated score in the dataset runs table and compare results side-by-side.

# Optional: Trigger Remote Dataset Runs via UI

When setting up Remote Dataset Runs via the SDK, it can be useful to expose a trigger in the ABV UI that helps you trigger the experiment runs.

You need to set up a webhook to receive the trigger request from ABV.

## 1) Navigate to the dataset

* **Navigate to** `Your Project` > `Datasets`
* **Click on** the dataset you want to set up a remote experiment trigger for

<img src="https://mintlify.s3.us-west-1.amazonaws.com/abv-2be93c70/images/evaluations-remote-dataset-runs-55000efe.png" alt="" />

## 2) Open the setup page

**Click on** `Start Experiment` to open the setup page

**Click on** `⚡` below `Custom Experiment`

## 3) Configure the webhook

**Enter** the URL of your external evaluation service that will receive the webhook when experiments are triggered.
**Specify** a default config that will be sent to your webhook. Users can modify this when triggering experiments.

## 4) Trigger experiments

Once configured, team members can trigger remote experiments via the `Run` button under the **Custom Experiment** option. ABV will send the dataset metadata (ID and name) along with any custom configuration to your webhook.

**Typical workflow**: Your webhook receives the request, fetches the dataset from ABV, runs your application against the dataset items, evaluates the results, and ingests the scores back into ABV as a new Dataset Run.