> ## Documentation Index
> Fetch the complete documentation index at: https://docs.abv.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Prompt Experiments

You can execute Prompt Experiments in the ABV UI to test different prompt versions from [Prompt Management](/developer/prompt-management/overview) or language models and compare the results side-by-side.

Optionally, you can use [LLM-as-a-Judge Evaluators](./llm-as-a-judge) to automatically score the responses based on the expected outputs to further analyze the results on an aggregate level.

# Why use Prompt Experiments?

* Quickly test different prompt versions or models
* Structure your prompt testing by using a dataset to test different prompt versions and models
* Quickly iterate on prompts through Dataset Runs
* Optionally use LLM-as-a-Judge Evaluators to score the responses based on the expected outputs from the dataset
* Prevent regressions by running tests when making prompt changes

# Prerequisites

## 1) Create a usable prompt

Create a prompt that you want to test and evaluate. [How to create a prompt?](/developer/prompt-management/get-started)

<Info>
  A prompt is usable when:
</Info>

<Expandable title="Example: Prompt Variables & Dataset Item Keys Mapping">
  **Prompt:**

  ```none theme={null}
  You are an ABV expert. Answer based on:
  {{documentation}}

  Question: {{question}}
  ```

  **Dataset Item:**

  ```json theme={null}
  {
    "documentation": "ABV is an LLM Engineering Platform",
    "question": "What is ABV?"
  }
  ```

  In this example:

  * The prompt variable `{{documentation}}` maps to the JSON key `"documentation"`
  * The prompt variable `{{question}}` maps to the JSON key `"question"`
  * Both keys must exist in the dataset item's input JSON for the experiment to run successfully
</Expandable>

<Expandable title="Example: Chat Message Placeholder Mapping">
  In addition to variables, you can also map placeholders in chat message prompts to dataset item keys.
  This is useful when the dataset item also contains for example a chat message history to use.
  Your chat prompt needs to contain a placeholder with a name. Variables within placeholders are not resolved.

  **Chat Prompt:**
  Placeholder named: `message_history`

  **Dataset Item:**

  ```json theme={null}
  {
    "message_history": [
      {
        "role": "user",
        "content": "What is ABV?"
      },
      {
        "role": "assistant",
        "content": "ABV is a tool for tracking and analyzing the performance of language models."
      }
    ],
    "question": "What is ABV?"
  }
  ```

  In this example:

  * The chat prompt placeholder `message_history` maps to the JSON key `"message_history"`.
  * The prompt variable `{{question}}` maps to the JSON key `"question"` in a variable not within a placeholder message.
  * Both keys must exist in the dataset item's input JSON for the experiment to run successfully
</Expandable>

## 2) Create a usable dataset

Create a dataset with the inputs and expected outputs you want to use for your prompt experiments. [How to create a dataset?](/developer/evaluations/datasets)

<Info>
  A dataset is usable when: (1) the dataset items have JSON objects as input and (2) these objects have JSON keys that match the prompt variables of the prompt(s) you will use. See the example below.
</Info>

<Expandable title="Example: Prompt Variables & Dataset Item Keys Mapping">
  **Prompt:**

  ```none theme={null}
  You are an ABV expert. Answer based on:
  {{documentation}}

  Question: {{question}}
  ```

  **Dataset Item:**

  ```json theme={null}
  {
    "documentation": "ABV is an LLM Engineering Platform",
    "question": "What is ABV?"
  }
  ```

  In this example:

  * The prompt variable `{{documentation}}` maps to the JSON key `"documentation"`
  * The prompt variable `{{question}}` maps to the JSON key `"question"`
  * Both keys must exist in the dataset item's input JSON for the experiment to run successfully
</Expandable>

## 3) Configure LLM connection

As your prompt will be executed for each dataset item, you need to configure an LLM connection in the project settings. [How to configure an LLM connection?](/developer/platform/administration/llm-connections)

## 4) Optional: Set up LLM-as-a-judge

You can set up an LLM-as-a-judge evaluator to score the responses based on the expected outputs. Make sure to set the target of the LLM-as-a-Judge to "Experiment runs" and filter for the dataset you want to use. [How to set up LLM-as-a-judge?](./llm-as-a-judge)

# Trigger a Prompt Experiment

## 1) Navigate to the dataset

`Dataset Runs are currently started from the detail page of a dataset.`

* **Navigate to** `Your Project` > `Datasets`
* **Click on** the dataset you want to start a Dataset Run for

<img src="https://mintlify.s3.us-west-1.amazonaws.com/abv-2be93c70/images/evaluations-prompt-experiments-d1e9c3f1.png" alt="" />

## 2) Open the setup page

**Click on** `Start Experiment` to open the setup page

**Click on** `Create` below `Prompt Experiment`

## 3) Configure the Dataset Run

1. **Set** a Dataset Run name
2. **Select** the prompt you want to use
3. **Set up or select** the LLM connection you want to use
4. **Select** the dataset you want to use
5. **Optionally select** the evaluator you want to use
6. **Click on** `Create` to trigger the Dataset Run

This will trigger the Dataset Run and you will be redirected to the Dataset Runs page. The run might take a few seconds or minutes to complete depending on the prompt complexity and dataset size.

## 4) Compare runs

After each experiment run, you can check the aggregated score in the Dataset Runs table and compare results side-by-side.