> ## Documentation Index
> Fetch the complete documentation index at: https://docs.abv.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Python SDK - Evaluations

The ABV Python SDK provides ways to evaluate your application. You can add custom scores to your traces and observations, or use the SDK to execute Dataset Runs.

<Info>
  This page shows the evaluation methods that are supported by the Python SDK. Please refer to the [Evaluation Overview](/developer/evaluations/overview) for more information on how to evaluate your application in ABV.
</Info>

# Scoring traces and observations

Install package

```bash theme={null}
pip install abvdev
```

### ABVSpan / ABVGeneration object methods

* span\_or\_generation\_obj.score(): Scores the specific observation object.
* span\_or\_generation\_obj.score\_trace(): Scores the entire trace to which the object belongs.

```python theme={null}
from abvdev import ABV

# ABV client initialization          
abv = ABV(
    api_key="sk-abv-...", # your api key here
    host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)

with abv.start_as_current_observation(as_type='generation', name="summary_generation") as gen:
    # ... LLM call ...
    gen.update(output="summary text...")
    # Score this specific generation
    gen.score(name="conciseness", value=0.8, data_type="NUMERIC")
    # Score the overall trace
    gen.score_trace(name="user_feedback_rating", value="positive", data_type="CATEGORICAL")
```

### Context-aware methods

* abv.score\_current\_span(): Scores the currently active observation in the context.
* abv.score\_current\_trace(): Scores the trace of the currently active observation.

```python theme={null}
from abvdev import ABV

# ABV client initialization          
abv = ABV(
    api_key="sk-abv-...", # your api key here
    host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)

# imitate successful task completion
task_is_fully_successful = True

with abv.start_as_current_span(name="complex_task") as task_span:
    # ... perform task ...
    abv.score_current_span(name="task_component_quality", value=True, data_type="BOOLEAN")
    # ...
    if task_is_fully_successful:
         abv.score_current_trace(name="overall_success", value=1.0, data_type="NUMERIC")
```

### Low-level method

* Creates a score for a specified trace\_id and optionally observation\_id.
* Useful when IDs are known, or for scoring after the trace/observation has completed.

```python theme={null}
from abvdev import ABV

# ABV client initialization          
abv = ABV(
    api_key="sk-abv-...", # your api key here
    host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)

abv.create_score(
    name="fact_check_accuracy",
    value=0.95, # Can be float for NUMERIC/BOOLEAN, string for CATEGORICAL
    trace_id="your_trace_id",
    observation_id="your_observation_id", # Optional: if scoring a specific observation
    session_id="your_session_id", # Optional: if scoring a specific session   
    data_type="NUMERIC", # "NUMERIC", "BOOLEAN", "CATEGORICAL"
    comment="Source verified for 95% of claims."
)
```

## **Score Parameters:**

| Parameter        | Type                      | Description                                                                                                            |
| ---------------- | ------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| `name`           | `str`                     | Name of the score (e.g., "relevance", "accuracy"). **Required.**                                                       |
| `value`          | `Union[float, str]`       | Score value. Float for `NUMERIC`/`BOOLEAN`, string for `CATEGORICAL`. **Required.**                                    |
| `trace_id`       | `str`                     | ID of the trace to associate with (for `create_score`). **Required.**                                                  |
| `session_id`     | `Optional[str]`           | ID of the specific session to score (for<br /> `create_score`).<br />                                                  |
| `observation_id` | `Optional[str]`           | ID of the specific observation to score (for `create_score`).                                                          |
| `score_id`       | `Optional[str]`           | Custom ID for the score (auto-generated if None).                                                                      |
| `data_type`      | `Optional[ScoreDataType]` | `"NUMERIC"`, `"BOOLEAN"`, or `"CATEGORICAL"`. Inferred if not provided based on value type and score config on server. |
| `comment`        | `Optional[str]`           | Optional comment or explanation for the score.                                                                         |
| `config_id`      | `Optional[str]`           | Optional ID of a pre-defined score configuration in ABV.                                                               |

See [Scoring](/developer/evaluations/scores-data-model) for more details.

# Datasets

[ABV Datasets](/developer/evaluations/datasets) are essential for evaluating and testing your LLM applications by allowing you to manage collections of inputs and their expected outputs.

## Interacting with Datasets

* **Creating**: You can programmatically create new datasets with `abv.create_dataset(...)` and add items to them using `abv.create_dataset_item(...)`.
* **Fetching**: Retrieve a dataset and its items using `abv.get_dataset(name: str)`. This returns a `DatasetClient` instance, which contains a list of `DatasetItemClient` objects (accessible via `dataset.items`). Each `DatasetItemClient` holds the `input`, `expected_output`, and `metadata` for an individual data point.

```python theme={null}
from abvdev import ABV

# ABV client initialization          
abv = ABV(
    api_key="sk-abv-...", # your api key here
    host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)

# Fetch an existing dataset
dataset = abv.get_dataset(name="my-eval-dataset")
for item in dataset.items:
    print(f"Input: {item.input}, Expected: {item.expected_output}")

# Briefly: Creating a dataset and an item
new_dataset = abv.create_dataset(name="new-summarization-tasks")
abv.create_dataset_item(
    dataset_name="new-summarization-tasks",
    input={"text": "Long article..."},
    expected_output={"summary": "Short summary."}
)
```

## Linking Traces to Dataset Items for Runs

The most powerful way to use datasets is by linking your application's executions (traces) to specific dataset items when performing an evaluation run. See our [datasets](/developer/evaluations/datasets) for more details. The `DatasetItemClient.run()` method provides a context manager to streamline this process.

**How **`item.run()`** works:**

When you use `with item.run(run_name="your_eval_run_name") as root_span:`:

1. **Trace Creation**: A new ABV trace is initiated specifically for processing this dataset item within the context of the named run.
2. **Trace Naming & Metadata**:
   * The trace is automatically named (e.g., "Dataset run: your\_eval\_run\_name").
   * Essential metadata is added to this trace, including `dataset_item_id` (the ID of `item`), `run_name`, and `dataset_id`.
3. **DatasetRunItem Linking**: The SDK makes an API call to ABV to create a `DatasetRunItem`. This backend object formally links:
   * The `dataset_item_id`
   * The `trace_id` of the newly created trace
   * The provided `run_name`
   * Any `run_metadata` or `run_description` you pass to `item.run()`.
     This linkage is what populates the "Runs" tab for your dataset in the ABV UI, allowing you to see all traces associated with a particular evaluation run.
4. **Contextual Span**: The context manager yields `root_span`, which is a `ABVSpan` object representing the root span of this new trace.
5. **Automatic Nesting**: Any ABV observations (spans or generations) created *inside* the `with` block will automatically become children of `root_span` and thus part of the trace linked to this dataset item and run.

**Example:**

```python theme={null}
from abvdev import ABV

# ABV client initialization          
abv = ABV(
    api_key="sk-abv-...", # your api key here
    host="https://app.abv.dev", # host="https://eu.app.abv.dev", for EU region
)

dataset_name = "qna-eval"
current_run_name = "qna_model_v3_run_05_20" # Identifies this specific evaluation run

# Assume 'my_qna_app' is your instrumented application function
def my_qna_app(question: str, context: str, item_id: str, run_name: str):
    with abv.start_as_current_observation(
        as_type='generation',
        name="qna-llm-call",
        input={"question": question, "context": context},
        metadata={"item_id": item_id, "run": run_name}, # Example metadata for the generation
        model="gpt-5-2025-08-07"
    ) as generation:
        # Simulate LLM call
        answer = f"Answer to '{question}' using context." # Replace with actual LLM call
        generation.update(output={"answer": answer})

        # Update the trace with the input and output
        generation.update_trace(
            input={"question": question, "context": context},
            output={"answer": answer},
        )

        return answer

dataset = abv.get_dataset(name=dataset_name) # Fetch your pre-populated dataset

for item in dataset.items:
    print(f"Running evaluation for item: {item.id} (Input: {item.input})")

    # Use the item.run() context manager
    with item.run(
        run_name=current_run_name,
        run_metadata={"model_provider": "OpenAI", "temperature_setting": 0.7},
        run_description="Evaluation run for Q&A model v3 on May 20th"
    ) as root_span: # root_span is the root span of the new trace for this item and run.
        # All subsequent abv operations within this block are part of this trace.

        # Call your application logic
        generated_answer = my_qna_app(
            question=item.input["question"],
            context=item.input["context"],
            item_id=item.id,
            run_name=current_run_name
        )

        print(f"  Item {item.id} processed. Trace ID: {root_span.trace_id}")

        # Optionally, score the result against the expected output
        if item.expected_output and generated_answer == item.expected_output.get("answer"):
            root_span.score_trace(name="exact_match", value=1.0)
        else:
            root_span.score_trace(name="exact_match", value=0.0)

print(f"\nFinished processing dataset '{dataset_name}' for run '{current_run_name}'.")
```

By using `item.run()`, you ensure each dataset item's processing is neatly encapsulated in its own trace, and these traces are aggregated under the specified `run_name` in the ABV UI. This allows for systematic review of results, comparison across runs, and deep dives into individual processing traces.