> ## Documentation Index
> Fetch the complete documentation index at: https://docs.abv.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Evals Troubleshooting and FAQ

<AccordionGroup>
  <Accordion title="How do I create and manage Score Configs?" icon="sliders">
    Score Configs ensure scores follow a specific schema and standardize scoring across your team.

    **Create a Score Config:**

    1. Navigate to your project in the ABV UI
    2. Go to **Evaluations** → **Score Configs**
    3. Click **Create Score Config**
    4. Configure:
       * **Name**: e.g., `user_feedback`, `hallucination_eval`
       * **Data Type**: `NUMERIC`, `CATEGORICAL`, or `BOOLEAN`
       * **Constraints**: Min/Max for numeric, custom categories for categorical

    **Via API:**

    ```python theme={null}
    from abvdev import ABV

    abv = ABV(api_key="sk-abv-...")

    abv.create_score_config(
        name="correctness",
        data_type="NUMERIC",
        min_value=0,
        max_value=1,
        description="Measures factual accuracy"
    )
    ```

    **Manage Configs:**

    * Configs are **immutable** but can be archived
    * Archived configs can be restored anytime
    * Link scores to configs using `config_id` to ensure schema compliance

    See [Scores Data Model](/developer/evaluations/scores-data-model) for complete details.
  </Accordion>

  <Accordion title="I don't see traces in the dashboard. How do I troubleshoot?" icon="magnifying-glass">
    **Common causes and solutions:**

    1. **Events not flushed (short-lived apps):**
       * Python: Call `abv.flush()` before exit
       * JS/TS: Call `await abvSpanProcessor.forceFlush()` before exit

    2. **Incorrect API credentials:**
       * Verify your API key is correct
       * Check region (US: `https://app.abv.dev`, EU: `https://eu.app.abv.dev`)
       * Python: Use `abv.auth_check()` to verify credentials

    3. **Instrumentation not loaded:**
       * JS/TS: Ensure `import "./instrumentation"` is the FIRST import
       * Python: Initialize with `get_client()` or `ABV()`

    4. **Network/firewall issues:**
       * Verify your application can reach the ABV API
       * Check for proxy/firewall blocking requests

    5. **Sampling too aggressive:**
       * Check if sampling is filtering out traces
       * Temporarily set sample rate to 1.0 (100%) to test

    6. **Wrong project:**
       * Verify you're viewing the correct project in the ABV UI
       * Confirm API key belongs to the project you're viewing

    7. **For JS/TS with @vercel/otel:**
       * Use manual OpenTelemetry setup via `NodeTracerProvider`
       * The @vercel/otel package doesn't support OpenTelemetry JS SDK v2

    8. **Enable debug logging:**
       * Python: Set log level in code
       * JS/TS: Set `ABV_LOG_LEVEL="DEBUG"` in environment variables

    See [Troubleshooting FAQ](/developer/troubleshooting-faq) for general troubleshooting.
  </Accordion>

  <Accordion title="How do I capture user feedback for evaluation?" icon="comment">
    Capture user feedback as scores to evaluate LLM application quality.

    **Method 1: Frontend Collection (Browser SDK)**

    ```javascript theme={null}
    import { ABVClient } from '@abvdev/client';

    const abv = new ABVClient({ apiKey: 'sk-abv-...' });

    // Capture thumbs up/down
    abv.createScore({
      name: 'user_feedback',
      value: 1,  // 1 for positive, 0 for negative
      traceId: traceId,
      dataType: 'BOOLEAN',
      comment: 'User found this helpful'
    });
    ```

    **Method 2: Backend Collection (Python SDK)**

    ```python theme={null}
    from abvdev import ABV

    abv = ABV(api_key="sk-abv-...")

    # Categorical feedback
    abv.create_score(
        name="user_rating",
        string_value="excellent",  # or "good", "poor"
        trace_id="trace_id_here",
        data_type="CATEGORICAL",
        comment="User provided detailed feedback"
    )
    ```

    **Method 3: Human Annotation UI**

    Use [Annotation Queues](/developer/evaluations/human-annotation) for structured team reviews:

    1. Create Score Configs for feedback dimensions
    2. Create an Annotation Queue
    3. Assign team members to review traces
    4. Annotate traces directly in the ABV UI

    **Best Practices:**

    * Link scores to Score Configs for consistent schema
    * Use `trace_id` to associate feedback with specific interactions
    * Scores can be ingested before the trace is created (linked automatically)

    See [Custom Scores](/developer/evaluations/custom-scores) and [Human Annotation](/developer/evaluations/human-annotation) for details.
  </Accordion>

  <Accordion title="How do Score Configs ensure data consistency?" icon="shield-check">
    Score Configs enforce schema validation across your evaluation workflows.

    **Benefits:**

    * **Standardized scoring**: All team members use the same criteria
    * **Data validation**: Automatic validation of score values
    * **Type safety**: Ensures numeric/categorical/boolean consistency
    * **Schema evolution**: Archive old configs, create new versions

    **Example: Categorical Score Config**

    ```python theme={null}
    abv.create_score_config(
        name="sentiment",
        data_type="CATEGORICAL",
        categories=[
            {"label": "positive", "value": 1},
            {"label": "neutral", "value": 0},
            {"label": "negative", "value": -1}
        ]
    )
    ```

    When you create a score with this `config_id`, ABV validates that `string_value` matches one of the defined categories.

    **Example: Numeric Score Config with Constraints**

    ```python theme={null}
    abv.create_score_config(
        name="accuracy",
        data_type="NUMERIC",
        min_value=0.0,
        max_value=1.0
    )
    ```

    Scores outside the 0-1 range will be rejected.

    See [Scores Data Model](/developer/evaluations/scores-data-model) for configuration options.
  </Accordion>

  <Accordion title="Can I use scores without Score Configs?" icon="circle-question">
    Yes, Score Configs are **optional** but recommended.

    **Without Score Configs:**

    * Manually specify `data_type` for each score
    * No automatic validation of value ranges
    * Less consistency across team members

    **Example:**

    ```python theme={null}
    abv.create_score(
        name="custom_metric",
        value=42,
        trace_id="trace_id",
        data_type="NUMERIC"  # Must specify manually
    )
    ```

    **With Score Configs:**

    * Reference `config_id` to automatically set `data_type`
    * Automatic value validation
    * Standardized across all scores with that name

    ```python theme={null}
    abv.create_score(
        name="custom_metric",
        value=42,
        trace_id="trace_id",
        config_id="config_id_here"  # data_type set automatically
    )
    ```

    **Recommendation:** Use Score Configs for production evaluation workflows.
  </Accordion>

  <Accordion title="How do I link scores to traces, observations, or sessions?" icon="link">
    Scores can be attached to different levels of your application data.

    **Trace-level (most common):**

    ```python theme={null}
    abv.create_score(
        name="overall_quality",
        value=0.9,
        trace_id="trace_id_here"
    )
    ```

    **Observation-level (specific LLM call):**

    ```python theme={null}
    abv.create_score(
        name="hallucination",
        value=0,
        observation_id="observation_id_here"
    )
    ```

    **Session-level (multi-turn conversation):**

    ```python theme={null}
    abv.create_score(
        name="conversation_quality",
        value=0.85,
        session_id="session_id_here"
    )
    ```

    **Dataset Run-level (experiment performance):**

    ```python theme={null}
    abv.create_score(
        name="experiment_accuracy",
        value=0.92,
        dataset_run_id="run_id_here"
    )
    ```

    **Note:** Each score references **exactly one** of these objects.

    See [Scores Data Model](/developer/evaluations/scores-data-model) for use cases.
  </Accordion>

  <Accordion title="What's the difference between API, EVAL, and ANNOTATION scores?" icon="tags">
    The `source` field automatically categorizes how scores were created:

    | Source         | Description                            | Example Use Case                                      |
    | -------------- | -------------------------------------- | ----------------------------------------------------- |
    | **API**        | Scores created via SDK or API          | User feedback, runtime metrics, custom evaluations    |
    | **EVAL**       | Scores from LLM-as-a-Judge evaluations | Automated quality checks, hallucination detection     |
    | **ANNOTATION** | Scores from Human Annotation UI        | Manual reviews, annotation queues, team collaboration |

    **Automatic Assignment:**

    * SDK/API calls → `source="API"`
    * LLM-as-a-Judge runs → `source="EVAL"`
    * UI annotations → `source="ANNOTATION"`

    **Filter by source:**

    * View scores by source in the ABV UI
    * Query via API: `abv.get_scores(source="EVAL")`
    * Useful for comparing human vs automated evaluations

    This helps track evaluation provenance and compare different evaluation methods.
  </Accordion>
</AccordionGroup>

# Related Resources

<CardGroup cols={2}>
  <Card title="Scores Data Model" icon="database" href="/developer/evaluations/scores-data-model">
    Complete reference for scores and configs
  </Card>

  <Card title="Custom Scores" icon="code" href="/developer/evaluations/custom-scores">
    Implement custom evaluation workflows
  </Card>

  <Card title="Human Annotation" icon="users" href="/developer/evaluations/human-annotation">
    Team-based manual evaluation
  </Card>

  <Card title="LLM-as-a-Judge" icon="gavel" href="/developer/evaluations/llm-as-a-judge">
    Automated LLM evaluations
  </Card>
</CardGroup>