> ## Documentation Index
> Fetch the complete documentation index at: https://docs.abv.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Quickstart

This tutorial walks you through a complete evaluation workflow with a practical example. You'll create a simple dataset, run evaluations, and view results—giving you hands-on experience before diving into advanced concepts.

<Note>
  **Time:** 20-30 minutes

  **Prerequisites:**

  * ABV SDK installed ([Python](/developer/sdks/python/setup) or [JS/TS](/developer/sdks/js-ts/setup))
  * API key configured
  * Basic understanding of [traces](/developer/basic-features/observability-data-model)
</Note>

***

## The Example: Email Response Quality

We'll build an evaluation system for an AI that generates customer service email responses. Our goal is to measure:

* **Politeness:** Is the tone professional and courteous?
* **Completeness:** Does it address all customer questions?
* **Accuracy:** Is the information correct?

This is a common real-world use case that demonstrates key evaluation concepts.

***

## Step 1: Create a Simple Dataset

First, let's create a small dataset with 3 test cases. Each case has an input (customer email) and expected qualities.

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    from abvdev import ABV

    abv = ABV()

    # Create a dataset for email response evaluation
    dataset = abv.datasets.create(
        name="email-responses-v1",
        description="Test cases for customer service email quality"
    )

    # Add test cases
    test_cases = [
        {
            "input": {
                "customer_email": "My order #12345 arrived damaged. I need a refund.",
                "context": "Product: Coffee Maker, Price: $89.99"
            },
            "expected_output": {
                "polite": True,
                "addresses_refund": True,
                "mentions_order_number": True
            }
        },
        {
            "input": {
                "customer_email": "How do I reset my password? I've tried 3 times.",
                "context": "Customer tier: Premium"
            },
            "expected_output": {
                "polite": True,
                "provides_steps": True,
                "offers_additional_help": True
            }
        },
        {
            "input": {
                "customer_email": "Your website is terrible! Nothing works!",
                "context": "Customer tier: Free"
            },
            "expected_output": {
                "polite": True,  # Must stay professional even when customer isn't
                "acknowledges_frustration": True,
                "offers_specific_help": True
            }
        }
    ]

    # Add each test case to the dataset
    for i, case in enumerate(test_cases):
        dataset.create_item(
            input=case["input"],
            expected_output=case["expected_output"],
            metadata={"case_number": i + 1}
        )

    print(f"✅ Created dataset with {len(test_cases)} test cases")
    print(f"Dataset ID: {dataset.id}")
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    import { ABV } from "@abvdev/client";

    const abv = new ABV();

    // Create a dataset for email response evaluation
    const dataset = await abv.datasets.create({
      name: "email-responses-v1",
      description: "Test cases for customer service email quality"
    });

    // Add test cases
    const testCases = [
      {
        input: {
          customer_email: "My order #12345 arrived damaged. I need a refund.",
          context: "Product: Coffee Maker, Price: $89.99"
        },
        expected_output: {
          polite: true,
          addresses_refund: true,
          mentions_order_number: true
        }
      },
      {
        input: {
          customer_email: "How do I reset my password? I've tried 3 times.",
          context: "Customer tier: Premium"
        },
        expected_output: {
          polite: true,
          provides_steps: true,
          offers_additional_help: true
        }
      },
      {
        input: {
          customer_email: "Your website is terrible! Nothing works!",
          context: "Customer tier: Free"
        },
        expected_output: {
          polite: true,  // Must stay professional even when customer isn't
          acknowledges_frustration: true,
          offers_specific_help: true
        }
      }
    ];

    // Add each test case to the dataset
    for (let i = 0; i < testCases.length; i++) {
      await dataset.createItem({
        input: testCases[i].input,
        expected_output: testCases[i].expected_output,
        metadata: { case_number: i + 1 }
      });
    }

    console.log(`✅ Created dataset with ${testCases.length} test cases`);
    console.log(`Dataset ID: ${dataset.id}`);
    ```
  </Tab>
</Tabs>

**What we did:**

* Created a dataset named `email-responses-v1`
* Added 3 test cases covering different scenarios (damaged product, password help, frustrated customer)
* Each case has input data and expected qualities we want to verify

***

## Step 2: Run Your First Evaluation

Now let's run our AI model on each test case and evaluate the results.

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    # Your AI function that generates email responses
    def generate_email_response(customer_email: str, context: str) -> str:
        """
        This is your actual AI function. For this tutorial, we'll use a simple mock.
        In production, this would call your LLM (OpenAI, Anthropic, etc.)
        """
        # Mock response for demonstration
        # Replace this with your actual LLM call
        return f"Dear valued customer, thank you for contacting us about: {customer_email[:50]}..."

    # Run the dataset through your AI
    dataset_run = dataset.run(
        name="Initial evaluation",
        description="Testing baseline email response quality",
        metadata={"model": "gpt-4", "temperature": 0.7}
    )

    # Process each test case
    for item in dataset.items:
        # Generate AI response
        input_data = item.input
        ai_response = generate_email_response(
            customer_email=input_data["customer_email"],
            context=input_data["context"]
        )

        # Record the output
        dataset_run.create_observation(
            dataset_item_id=item.id,
            output=ai_response,
            metadata={"input_case": item.metadata["case_number"]}
        )

    print(f"✅ Completed evaluation run: {dataset_run.id}")
    print("View results in the ABV platform!")
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    // Your AI function that generates email responses
    function generateEmailResponse(customerEmail: string, context: string): string {
      /**
       * This is your actual AI function. For this tutorial, we'll use a simple mock.
       * In production, this would call your LLM (OpenAI, Anthropic, etc.)
       */
      // Mock response for demonstration
      // Replace this with your actual LLM call
      return `Dear valued customer, thank you for contacting us about: ${customerEmail.slice(0, 50)}...`;
    }

    // Run the dataset through your AI
    const datasetRun = await dataset.run({
      name: "Initial evaluation",
      description: "Testing baseline email response quality",
      metadata: { model: "gpt-4", temperature: 0.7 }
    });

    // Process each test case
    const items = await dataset.items();
    for (const item of items) {
      // Generate AI response
      const inputData = item.input;
      const aiResponse = generateEmailResponse(
        inputData.customer_email,
        inputData.context
      );

      // Record the output
      await datasetRun.createObservation({
        dataset_item_id: item.id,
        output: aiResponse,
        metadata: { input_case: item.metadata.case_number }
      });
    }

    console.log(`✅ Completed evaluation run: ${datasetRun.id}`);
    console.log("View results in the ABV platform!");
    ```
  </Tab>
</Tabs>

**What we did:**

* Created a dataset run to track this specific evaluation
* Generated AI responses for each test case
* Recorded outputs so we can score them

***

## Step 3: Score the Results

Now we'll add scores to evaluate quality. We'll use both automated and manual scoring.

### Option A: LLM-as-a-Judge (Automated)

Use another LLM to evaluate the responses:

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    # Score politeness using LLM-as-a-Judge
    def score_politeness_llm(response: str) -> dict:
        """
        Use an LLM to judge if the response is polite
        """
        # In production, call your LLM API
        # For demo, we'll return a mock score
        score_value = 0.85  # 0-1 scale

        return {
            "value": score_value,
            "comment": "Response uses professional language and courteous tone"
        }

    # Apply scores to each observation
    for observation in dataset_run.observations:
        # Get politeness score
        politeness = score_politeness_llm(observation.output)

        # Record the score
        observation.score(
            name="politeness",
            value=politeness["value"],
            comment=politeness["comment"]
        )

    print("✅ Scored all responses for politeness")
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    // Score politeness using LLM-as-a-Judge
    function scorePolitenessLLM(response: string): { value: number; comment: string } {
      /**
       * Use an LLM to judge if the response is polite
       */
      // In production, call your LLM API
      // For demo, we'll return a mock score
      const scoreValue = 0.85;  // 0-1 scale

      return {
        value: scoreValue,
        comment: "Response uses professional language and courteous tone"
      };
    }

    // Apply scores to each observation
    const observations = await datasetRun.observations();
    for (const observation of observations) {
      // Get politeness score
      const politeness = scorePolitenesLLM(observation.output);

      // Record the score
      await observation.score({
        name: "politeness",
        value: politeness.value,
        comment: politeness.comment
      });
    }

    console.log("✅ Scored all responses for politeness");
    ```
  </Tab>
</Tabs>

### Option B: Custom Scoring (Rule-Based)

Create simple rules to check specific criteria:

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    def score_completeness(response: str, expected: dict) -> float:
        """
        Check if response addresses expected points
        Simple keyword matching for demonstration
        """
        score = 0
        checks = 0

        # Check each expected criterion
        for key, should_have in expected.items():
            checks += 1
            if key == "addresses_refund" and "refund" in response.lower():
                score += 1
            elif key == "mentions_order_number" and "#" in response:
                score += 1
            elif key == "provides_steps" and any(word in response.lower() for word in ["step", "first", "then"]):
                score += 1

        return score / checks if checks > 0 else 0

    # Apply custom scores
    for observation in dataset_run.observations:
        # Get the expected output for this case
        item = dataset.get_item(observation.dataset_item_id)
        expected = item.expected_output

        # Calculate completeness score
        completeness = score_completeness(observation.output, expected)

        # Record the score
        observation.score(
            name="completeness",
            value=completeness,
            comment=f"Addressed {int(completeness * 100)}% of expected points"
        )

    print("✅ Scored all responses for completeness")
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    function scoreCompleteness(response: string, expected: Record<string, boolean>): number {
      /**
       * Check if response addresses expected points
       * Simple keyword matching for demonstration
       */
      let score = 0;
      let checks = 0;

      // Check each expected criterion
      for (const [key, shouldHave] of Object.entries(expected)) {
        checks += 1;
        if (key === "addresses_refund" && response.toLowerCase().includes("refund")) {
          score += 1;
        } else if (key === "mentions_order_number" && response.includes("#")) {
          score += 1;
        } else if (key === "provides_steps" && ["step", "first", "then"].some(word => response.toLowerCase().includes(word))) {
          score += 1;
        }
      }

      return checks > 0 ? score / checks : 0;
    }

    // Apply custom scores
    for (const observation of observations) {
      // Get the expected output for this case
      const item = await dataset.getItem(observation.dataset_item_id);
      const expected = item.expected_output;

      // Calculate completeness score
      const completeness = scoreCompleteness(observation.output, expected);

      // Record the score
      await observation.score({
        name: "completeness",
        value: completeness,
        comment: `Addressed ${Math.round(completeness * 100)}% of expected points`
      });
    }

    console.log("✅ Scored all responses for completeness");
    ```
  </Tab>
</Tabs>

***

## Step 4: View Results in the Platform

Now go to the ABV platform to see your results:

1. Navigate to **Evaluations** → **Datasets**
2. Find your dataset: `email-responses-v1`
3. Click on the run: "Initial evaluation"
4. Review scores for each test case

You'll see:

* Overall score statistics (average, min, max)
* Individual test case results
* Scores breakdown by dimension (politeness, completeness)
* Comments explaining each score

<Tip>
  **What to look for:**

  * Which test cases scored lowest? Those need attention.
  * Are scores consistent across dimensions?
  * Do comments explain the reasoning clearly?
</Tip>

***

## Step 5: Iterate and Improve

Based on the results, let's improve our AI and re-evaluate:

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    # Improved version with better prompting
    def generate_email_response_v2(customer_email: str, context: str) -> str:
        """
        Version 2: Improved prompt engineering
        """
        # In production, use a better prompt like:
        prompt = f"""Generate a professional customer service email response.

    Customer email: {customer_email}
    Context: {context}

    Requirements:
    - Be polite and professional
    - Address all customer concerns
    - Offer specific next steps
    - Show empathy when appropriate

    Response:"""

        # Mock improved response
        return "Dear Customer, I sincerely apologize for the issue with your order..."

    # Run evaluation again with v2
    dataset_run_v2 = dataset.run(
        name="Improved model (v2)",
        description="Testing with better prompting",
        metadata={"model": "gpt-4", "temperature": 0.7, "version": "v2"}
    )

    for item in dataset.items:
        input_data = item.input
        ai_response = generate_email_response_v2(
            customer_email=input_data["customer_email"],
            context=input_data["context"]
        )

        dataset_run_v2.create_observation(
            dataset_item_id=item.id,
            output=ai_response
        )

    # Score the new version
    # ... (same scoring code as before)

    print("✅ Completed v2 evaluation - compare results in the platform!")
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    // Improved version with better prompting
    function generateEmailResponseV2(customerEmail: string, context: string): string {
      /**
       * Version 2: Improved prompt engineering
       */
      // In production, use a better prompt like:
      const prompt = `Generate a professional customer service email response.

    Customer email: ${customerEmail}
    Context: ${context}

    Requirements:
    - Be polite and professional
    - Address all customer concerns
    - Offer specific next steps
    - Show empathy when appropriate

    Response:`;

      // Mock improved response
      return "Dear Customer, I sincerely apologize for the issue with your order...";
    }

    // Run evaluation again with v2
    const datasetRunV2 = await dataset.run({
      name: "Improved model (v2)",
      description: "Testing with better prompting",
      metadata: { model: "gpt-4", temperature: 0.7, version: "v2" }
    });

    for (const item of await dataset.items()) {
      const inputData = item.input;
      const aiResponse = generateEmailResponseV2(
        inputData.customer_email,
        inputData.context
      );

      await datasetRunV2.createObservation({
        dataset_item_id: item.id,
        output: aiResponse
      });
    }

    // Score the new version
    // ... (same scoring code as before)

    console.log("✅ Completed v2 evaluation - compare results in the platform!");
    ```
  </Tab>
</Tabs>

Now you can **compare** the two runs side-by-side in the platform to see improvements!

***

## What You've Learned

Congratulations! You've completed a full evaluation workflow:

✅ **Created a dataset** with realistic test cases
✅ **Ran evaluations** on your AI outputs
✅ **Applied scores** using both LLM-as-a-Judge and custom rules
✅ **Viewed results** in the ABV platform
✅ **Iterated** by running a second evaluation with improvements

***

## Related Topics

### Expand Your Dataset

* Add 10-20 more test cases covering edge cases
* Include examples from production errors
* Balance positive and negative examples

**Read:** [Datasets Best Practices](/developer/evaluations/datasets)

### Advanced Scoring

* Use [LLM-as-a-Judge](/developer/evaluations/llm-as-a-judge) with custom criteria
* Add [Human Annotation](/developer/evaluations/human-annotation) for nuanced scoring
* Create [Custom Scores](/developer/evaluations/custom-scores) for domain-specific metrics

### Automation

* Set up [Remote Dataset Runs](/developer/evaluations/remote-dataset-runs) to run nightly
* Use [Prompt Experiments](/developer/evaluations/prompt-experiments) for A/B testing
* Track [Scores Data Model](/developer/evaluations/scores-data-model) over time

### Integration

* Link evaluations to [production traces](/developer/prompt-management/link-prompts-to-traces)
* Export data for [model fine-tuning](/developer/platform/api-data-platform/export-for-fine-tuning)
* Build [custom dashboards](/developer/platform/metrics/custom-dashboards) tracking quality metrics

***

## Common Questions

<Accordion title="How many test cases should I have?">
  Start with 10-20 high-quality test cases covering key scenarios. Add more as you find gaps. Quality > quantity—focus on representative, real-world examples.
</Accordion>

<Accordion title="Should I use LLM-as-a-Judge or custom scoring?">
  Use both! Custom scoring is fast and deterministic for clear criteria (e.g., "contains word X"). LLM-as-a-Judge is better for nuanced qualities like tone or helpfulness.
</Accordion>

<Accordion title="How often should I run evaluations?">
  * **During development:** Every time you change prompts or models
  * **In production:** Daily or weekly automated runs
  * **For experiments:** Before and after each change to measure impact
</Accordion>

<Accordion title="Can I evaluate production traces?">
  Yes! You can score production traces directly without creating datasets. This is called "online evaluation." See [Evaluations Overview](/developer/evaluations/overview) for details.
</Accordion>

***

## Troubleshooting

**Dataset not showing up?**

* Wait a few seconds for the data to sync
* Check your API key permissions
* Verify you're looking in the correct project

**Scores not calculating?**

* Ensure observations are created before scoring
* Check that score values are between 0 and 1
* Verify your scoring function returns the correct format

**Can't compare runs?**

* Both runs must be on the same dataset
* Scores must have the same names across runs
* Allow time for score aggregation to complete

For more help, see [Evaluations Troubleshooting FAQ](/developer/evaluations/troubleshooting-faq)

***

## Complete Example Code

Want the full working example? Here's everything in one place:

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    from abvdev import ABV

    abv = ABV()

    # 1. Create dataset
    dataset = abv.datasets.create(
        name="email-responses-tutorial",
        description="Quickstart tutorial dataset"
    )

    # 2. Add test cases
    test_cases = [
        {
            "input": {"customer_email": "Order damaged, need refund", "context": "Product: Coffee Maker"},
            "expected_output": {"polite": True, "addresses_refund": True}
        },
        {
            "input": {"customer_email": "Password reset help", "context": "Premium customer"},
            "expected_output": {"polite": True, "provides_steps": True}
        }
    ]

    for case in test_cases:
        dataset.create_item(input=case["input"], expected_output=case["expected_output"])

    # 3. Run evaluation
    dataset_run = dataset.run(name="Tutorial run")
    for item in dataset.items:
        response = f"Mock response to: {item.input['customer_email']}"
        dataset_run.create_observation(dataset_item_id=item.id, output=response)

    # 4. Add scores
    for observation in dataset_run.observations:
        observation.score(name="quality", value=0.8, comment="Good response")

    print(f"✅ Complete! View results for dataset run: {dataset_run.id}")
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    import { ABV } from "@abvdev/client";

    const abv = new ABV();

    async function runTutorial() {
      // 1. Create dataset
      const dataset = await abv.datasets.create({
        name: "email-responses-tutorial",
        description: "Quickstart tutorial dataset"
      });

      // 2. Add test cases
      const testCases = [
        {
          input: { customer_email: "Order damaged, need refund", context: "Product: Coffee Maker" },
          expected_output: { polite: true, addresses_refund: true }
        },
        {
          input: { customer_email: "Password reset help", context: "Premium customer" },
          expected_output: { polite: true, provides_steps: true }
        }
      ];

      for (const testCase of testCases) {
        await dataset.createItem({
          input: testCase.input,
          expected_output: testCase.expected_output
        });
      }

      // 3. Run evaluation
      const datasetRun = await dataset.run({ name: "Tutorial run" });
      const items = await dataset.items();

      for (const item of items) {
        const response = `Mock response to: ${item.input.customer_email}`;
        await datasetRun.createObservation({
          dataset_item_id: item.id,
          output: response
        });
      }

      // 4. Add scores
      const observations = await datasetRun.observations();
      for (const observation of observations) {
        await observation.score({
          name: "quality",
          value: 0.8,
          comment: "Good response"
        });
      }

      console.log(`✅ Complete! View results for dataset run: ${datasetRun.id}`);
    }

    runTutorial();
    ```
  </Tab>
</Tabs>

***

You're now ready to build comprehensive evaluation systems for your LLM applications! 🎉