> ## Documentation Index
> Fetch the complete documentation index at: https://docs.abv.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Overview

> Implement comprehensive observability and tracing for LLM applications with complete visibility into inputs, outputs, tool usage, latencies, and costs

# How Observability & Tracing Works

**Observability** is the ability to understand what's happening inside your LLM application by examining its external outputs. **Traces** are the fundamental units of observability—individual records capturing complete LLM interactions from start to finish.

<Steps>
  <Step title="Install the SDK" icon="download">
    Add ABV to your application using the Python or JavaScript/TypeScript SDK. The SDK automatically instruments common LLM libraries (OpenAI, Anthropic, LangChain, LlamaIndex, etc.) with zero manual tracing code required for most use cases.

    ```bash theme={null}
    pip install abvdev  # Python
    npm install @abvdev/tracing @abvdev/otel  # JavaScript/TypeScript
    ```
  </Step>

  <Step title="Initialize with Your API Key" icon="key">
    Configure ABV with your project's API key (found in the ABV Dashboard). The SDK connects to ABV's backend and starts capturing telemetry automatically.

    <CodeGroup>
      ```python Python theme={null}
      from abvdev import ABV

      abv = ABV(api_key="your-api-key-here")
      ```

      ```javascript JavaScript/TypeScript theme={null}
      import { ABVClient } from '@abvdev/client';

      const abv = new ABVClient({ apiKey: 'your-api-key-here' });
      ```
    </CodeGroup>
  </Step>

  <Step title="Run Your LLM Application" icon="play">
    Make LLM calls as you normally would. ABV automatically captures:

    * **Inputs**: User queries, prompts, system instructions, few-shot examples
    * **Outputs**: LLM responses, tool calls, structured outputs
    * **Metadata**: Tokens (input/output), latency, costs, model parameters
    * **Context**: Sessions, users, environments, tags, custom metadata

    <CodeGroup>
      ```python Python theme={null}
      from openai import OpenAI

      client = OpenAI()

      # ABV automatically traces this call
      response = client.chat.completions.create(
          model="gpt-4",
          messages=[{"role": "user", "content": "Explain quantum computing"}]
      )
      ```

      ```javascript JavaScript/TypeScript theme={null}
      import OpenAI from 'openai';

      const client = new OpenAI();

      // ABV automatically traces this call
      const response = await client.chat.completions.create({
        model: 'gpt-4',
        messages: [{ role: 'user', content: 'Explain quantum computing' }]
      });
      ```
    </CodeGroup>
  </Step>

  <Step title="View Traces in the Dashboard" icon="chart-line">
    Navigate to the ABV Dashboard to explore traces. Each trace shows the complete interaction timeline, including:

    * Full prompt and response content
    * Token usage and costs broken down by model
    * Latency metrics (first token, total duration)
    * Tool/function calls with inputs and outputs
    * Error messages and stack traces for failures

    Filter by user, session, environment, tags, or metadata to find specific interactions. Share trace URLs with teammates for collaborative debugging.
  </Step>

  <Step title="Build on Your Data" icon="layer-group">
    Once traces are flowing, leverage ABV's advanced features:

    * **Evaluations**: Run systematic benchmarks on production data
    * **Prompt Management**: Version prompts and A/B test changes
    * **Cost Tracking**: Analyze spend by feature, user, or model
    * **Guardrails**: Catch problematic outputs before they reach users

    Your traces become the foundation for continuous improvement.
  </Step>
</Steps>

# Why You Should Set Up Tracing

<AccordionGroup>
  <Accordion title="Accelerate Development & Debugging" icon="magnifying-glass">
    Traditional debugging for LLM applications is painful. Print statements don't work for complex agent workflows. Logs are scattered across services. Reproducing production issues locally is nearly impossible due to non-deterministic LLM behavior.

    **With ABV tracing:**

    * See the complete interaction timeline for any request, including retries, tool calls, and nested LLM calls
    * Replay production traces with identical inputs to reproduce bugs deterministically
    * Compare successful vs. failed traces side-by-side to identify root causes
    * Share trace URLs with teammates for instant context sharing

    **Example workflow:**

    1. Customer reports an issue: "The chatbot gave me wrong refund information"
    2. Search traces by user ID to find their session
    3. Click the problematic trace to see the full conversation history
    4. Notice the RAG retrieval step pulled outdated policy documents
    5. Fix the retrieval logic and verify with new traces

    What used to take hours of back-and-forth with customers now takes minutes.
  </Accordion>

  <Accordion title="Track Costs in Real Time" icon="dollar-sign">
    LLM costs can spiral quickly, especially with production traffic. Without granular tracking, you don't know:

    * Which features or workflows are most expensive
    * Whether users are exploiting your system with excessive requests
    * If a recent code change increased costs
    * How much each customer/tenant costs to serve

    **ABV automatically tracks:**

    * Input/output tokens for every model call
    * Calculated costs using current provider pricing
    * Cost breakdowns by user, session, environment, model, or custom metadata
    * Cost trends over time with daily/weekly/monthly aggregations

    **Real-world savings:**

    * Discovered a feature using GPT-4 when GPT-3.5 would suffice → 10x cost reduction
    * Identified users triggering infinite retry loops → prevented runaway costs
    * Found prompt inefficiencies (e.g., redundant context) → reduced tokens 30%

    Set up [usage alerts](/developer/platform/administration/usage-alerts) to get notified when costs exceed thresholds.
  </Accordion>

  <Accordion title="Optimize Performance" icon="gauge-high">
    Latency matters for LLM applications. Users notice when responses are slow, and slow responses hurt conversion rates, engagement, and satisfaction.

    **ABV helps you optimize:**

    * Identify slow model calls, RAG retrievals, or tool executions
    * Measure time-to-first-token (streaming latency) vs. total duration
    * Compare latency across models (e.g., GPT-4 vs. Claude vs. Llama)
    * Track latency percentiles (p50, p95, p99) to catch tail latency issues
    * Correlate latency with cost to find the best price/performance tradeoff

    **Optimization patterns:**

    * Use faster models for simple tasks (gpt-3.5-turbo vs. gpt-4)
    * Implement caching for repeated queries (ABV integrates with prompt caching)
    * Parallelize independent LLM/tool calls
    * Stream responses to improve perceived latency
    * Set aggressive timeouts to prevent hanging requests

    Filter traces by latency ranges (e.g., >5 seconds) to find slow outliers.
  </Accordion>

  <Accordion title="Enable Systematic Evaluations" icon="flask">
    You can't improve what you don't measure. Manual testing doesn't scale, and gut feelings aren't data. ABV's [evaluation tools](/developer/evaluations/overview) require a foundation of production traces.

    **Evaluations unlock:**

    * **Prompt versioning**: Compare v1 vs. v2 on real user queries
    * **Model comparisons**: Test GPT-4 vs. Claude on your specific use case
    * **Regression detection**: Catch quality drops after code/prompt changes
    * **A/B testing**: Measure user satisfaction, conversion, or task success rates
    * **LLM-as-a-Judge**: Automatically score outputs for relevance, coherence, safety

    **Example evaluation workflow:**

    1. Export 100 production traces from the past week
    2. Create a dataset in ABV Evaluations
    3. Run both the current prompt and a new candidate prompt
    4. Compare outputs using LLM-as-a-Judge scoring
    5. Promote the winning prompt to production with confidence

    Without traces, you're guessing. With traces, you're measuring.
  </Accordion>

  <Accordion title="Streamline Customer Support" icon="headset">
    When customers report issues, support teams waste hours trying to reproduce problems or gather context. "It didn't work" isn't actionable without details.

    **ABV transforms support:**

    * Search traces by user ID, session ID, or custom metadata (e.g., order number)
    * See exactly what the user said, what the LLM responded, and what went wrong
    * Share trace URLs with engineering for instant context handoff
    * Filter by error status to proactively find and fix issues before users complain
    * Add comments on traces for internal notes and collaboration

    **Support playbook:**

    1. Customer: "I asked for a refund but got an error"
    2. Support agent searches traces by user email
    3. Finds the failed trace, sees the LLM tried to call a deprecated API
    4. Shares trace URL with engineering: "This API call is failing"
    5. Engineering fixes the integration, support confirms with new traces

    Time to resolution drops from days to hours. Customer satisfaction improves. Engineering gets actionable bug reports instead of vague complaints.
  </Accordion>
</AccordionGroup>

# Core Tracing Features

<AccordionGroup>
  <Accordion title="Sessions: Group Related Traces by User Journey" icon="diagram-project">
    A **session** groups multiple traces together to represent a complete user journey or job. For example:

    * A customer support conversation (10+ back-and-forth messages)
    * A document processing pipeline (upload → extract → summarize → classify)
    * A user's interactions during a single login session

    **Why sessions matter:**

    * See the full context leading up to an error (not just the final failed trace)
    * Measure end-to-end latency and cost for multi-step workflows
    * Analyze user behavior patterns across traces
    * Debug complex agent workflows where context builds over time

    **Setting a session ID:**

    <CodeGroup>
      ```python Python theme={null}
      # Option 1: Set session ID for all traces in a context
      with abv.observe(session_id="user-abc-session-123"):
          # All traces within this context belong to the session
          response1 = client.chat.completions.create(...)
          response2 = client.chat.completions.create(...)

      # Option 2: Set session ID for a single trace
      abv.trace(session_id="user-abc-session-123")
      ```

      ```javascript JavaScript/TypeScript theme={null}
      // Option 1: Set session ID for all traces in a context
      await ABV.observe({ sessionId: 'user-abc-session-123' }, async () => {
        // All traces within this callback belong to the session
        const response1 = await client.chat.completions.create(...);
        const response2 = await client.chat.completions.create(...);
      });

      // Option 2: Set session ID for a single trace
      ABV.trace({ sessionId: 'user-abc-session-123' });
      ```
    </CodeGroup>

    In the dashboard, click a session ID to see all traces in that session grouped together with aggregated metrics.

    See [Sessions](/developer/basic-features/sessions) for best practices.
  </Accordion>

  <Accordion title="Users: Link Traces to User Accounts" icon="user">
    A **user ID** identifies who triggered the trace. This enables:

    * Searching all traces for a specific user (for support or debugging)
    * Analyzing cost per user (identify expensive users or tiers)
    * Filtering traces by user cohorts (free vs. paid, region, etc.)
    * Personalizing evaluations (compare model performance by user segment)

    **Setting a user ID:**

    <CodeGroup>
      ```python Python theme={null}
      # Set user ID for all traces
      abv.init(api_key="...", user_id="user-abc-123")

      # Or set per-trace
      with abv.observe(user_id="user-abc-123"):
          response = client.chat.completions.create(...)
      ```

      ```javascript JavaScript/TypeScript theme={null}
      // Set user ID for all traces
      ABV.init({ apiKey: '...', userId: 'user-abc-123' });

      // Or set per-trace
      await ABV.observe({ userId: 'user-abc-123' }, async () => {
        const response = await client.chat.completions.create(...);
      });
      ```
    </CodeGroup>

    **User metadata:**
    You can also attach user metadata (email, name, subscription tier) to enrich traces:

    ```python theme={null}
    abv.init(
        api_key="...",
        user_id="user-abc-123",
        user_metadata={
            "email": "user@example.com",
            "tier": "enterprise",
            "region": "us-west"
        }
    )
    ```

    In the dashboard, filter by user ID or user metadata to analyze specific segments.

    See [User Tracking](/developer/basic-features/user-tracking) for GDPR-compliant user handling.
  </Accordion>

  <Accordion title="Environments: Separate Dev, Staging, and Production" icon="layer-group">
    **Environments** separate traces by deployment stage (development, staging, production). This prevents:

    * Test data polluting production metrics
    * Accidentally analyzing dev traces when investigating production issues
    * Mixing performance benchmarks across environments (dev is slower than prod)

    **Setting an environment:**

    <CodeGroup>
      ```python Python theme={null}
      import os

      abv.init(
          api_key="...",
          environment=os.getenv("ENV", "development")  # e.g., "production"
      )
      ```

      ```javascript JavaScript/TypeScript theme={null}
      ABV.init({
        apiKey: '...',
        environment: process.env.ENV || 'development'  // e.g., 'production'
      });
      ```
    </CodeGroup>

    **Common environment names:**

    * `development`: Local development machines
    * `staging`: Pre-production testing environment
    * `production`: Live user-facing application
    * `ci`: Continuous integration test runs

    In the dashboard, filter by environment to compare performance or isolate issues. You can also set different [sampling rates](/developer/basic-features/sampling) per environment (e.g., trace 100% in dev, 10% in production).

    See [Environments](/developer/basic-features/environments) for deployment strategies.
  </Accordion>

  <Accordion title="Metadata: Attach Structured Context to Traces" icon="database">
    **Metadata** is arbitrary key-value data attached to traces. Use it to add context that makes filtering and analysis precise:

    * Model parameters: `{"model": "gpt-4", "temperature": 0.7}`
    * Business context: `{"tenant_id": "acme-corp", "feature": "summarization"}`
    * Deployment info: `{"version": "1.2.3", "region": "us-west-2"}`
    * User context: `{"subscription_tier": "enterprise", "ab_test_group": "variant_b"}`

    **Adding metadata:**

    <CodeGroup>
      ```python Python theme={null}
      with abv.observe(metadata={
          "tenant_id": "acme-corp",
          "feature": "document-qa",
          "model": "gpt-4",
          "version": "1.2.3"
      }):
          response = client.chat.completions.create(...)
      ```

      ```javascript JavaScript/TypeScript theme={null}
      await ABV.observe({
        metadata: {
          tenantId: 'acme-corp',
          feature: 'document-qa',
          model: 'gpt-4',
          version: '1.2.3'
        }
      }, async () => {
        const response = await client.chat.completions.create(...);
      });
      ```
    </CodeGroup>

    **Querying by metadata:**
    In the dashboard, filter traces with advanced queries:

    * `metadata.tenant_id = "acme-corp"` → All traces for a specific tenant
    * `metadata.feature = "summarization"` → All traces for a specific feature
    * `metadata.version = "1.2.3" AND environment = "production"` → Traces for a specific deployment

    Metadata is indexed for fast querying. You can export filtered traces for evaluations or analysis.

    See [Metadata](/developer/basic-features/metadata) for best practices and advanced patterns.
  </Accordion>

  <Accordion title="Tags: Add Flexible Labels for Categorization" icon="tags">
    **Tags** are simple string labels that categorize traces for quick filtering. Unlike metadata (structured key-value pairs), tags are flat labels:

    * `["error", "timeout"]` → Mark traces with failures
    * `["high-priority", "customer-facing"]` → Prioritize critical workflows
    * `["experiment-v2", "canary"]` → Flag experimental features

    **Adding tags:**

    <CodeGroup>
      ```python Python theme={null}
      with abv.observe(tags=["experiment-v2", "high-priority"]):
          response = client.chat.completions.create(...)
      ```

      ```javascript JavaScript/TypeScript theme={null}
      await ABV.observe({
        tags: ['experiment-v2', 'high-priority']
      }, async () => {
        const response = await client.chat.completions.create(...);
      });
      ```
    </CodeGroup>

    **When to use tags vs. metadata:**

    * **Tags**: Simple categorization, boolean flags ("is this an error?"), filtering by presence
    * **Metadata**: Structured data with specific values ("which tenant?", "what model version?"), filtering by exact match or range

    In the dashboard, click a tag to filter all traces with that tag instantly. Tags appear as chips for visual scanning.

    See [Tags](/developer/basic-features/tags) for tagging strategies.
  </Accordion>

  <Accordion title="Trace IDs: Correlate Events Across Distributed Services" icon="link">
    In distributed systems, a single user request might trigger:

    1. API gateway (validates request)
    2. Backend service (calls LLM)
    3. RAG service (retrieves documents)
    4. Database (logs results)

    Without correlation, you can't connect events across services. **Trace IDs** solve this by propagating a unique identifier through the entire request lifecycle.

    **How it works:**

    1. Generate a trace ID when the request enters your system
    2. Pass the trace ID to every downstream service (via HTTP headers, message queues, etc.)
    3. Each service logs events with the same trace ID
    4. ABV groups all events by trace ID for end-to-end visibility

    **Setting a custom trace ID:**

    <CodeGroup>
      ```python Python theme={null}
      import uuid

      # Generate a trace ID (or extract from incoming request headers)
      trace_id = str(uuid.uuid4())

      # Pass to ABV
      with abv.observe(trace_id=trace_id):
          response = client.chat.completions.create(...)
      ```

      ```javascript JavaScript/TypeScript theme={null}
      import { v4 as uuidv4 } from 'uuid';

      // Generate a trace ID (or extract from incoming request headers)
      const traceId = uuidv4();

      // Pass to ABV
      await ABV.observe({ traceId }, async () => {
        const response = await client.chat.completions.create(...);
      });
      ```
    </CodeGroup>

    **Distributed tracing with OpenTelemetry:**
    ABV supports OpenTelemetry-compatible trace IDs. If you're already using OpenTelemetry, ABV automatically extracts the trace context and correlates spans.

    In the dashboard, search by trace ID to see the full request timeline across all services.

    See [Trace IDs & Distributed Tracing](/developer/basic-features/trace-ids-distributed-tracing) for advanced patterns.
  </Accordion>

  <Accordion title="Multi-Modality: Store Text, Images, Audio, and Files" icon="image">
    Modern LLM applications process more than just text. Vision models analyze images. Voice assistants transcribe audio. Document pipelines parse PDFs.

    **ABV captures all modalities:**

    * **Text**: Standard messages, prompts, completions
    * **Images**: Base64-encoded or URL-referenced images for GPT-4 Vision, Claude 4, Gemini Pro Vision
    * **Audio**: Transcriptions (Whisper), voice inputs, TTS outputs
    * **Files**: PDFs, CSVs, JSON, spreadsheets, code files

    **Example: Tracing a vision model call**

    <CodeGroup>
      ```python Python theme={null}
      response = client.chat.completions.create(
          model="gpt-4-vision-preview",
          messages=[{
              "role": "user",
              "content": [
                  {"type": "text", "text": "What's in this image?"},
                  {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}}
              ]
          }]
      )
      # ABV automatically captures the image URL and displays it in the dashboard
      ```

      ```javascript JavaScript/TypeScript theme={null}
      const response = await client.chat.completions.create({
        model: 'gpt-4-vision-preview',
        messages: [{
          role: 'user',
          content: [
            { type: 'text', text: "What's in this image?" },
            { type: 'image_url', image_url: { url: 'https://example.com/image.jpg' } }
          ]
        }]
      });
      // ABV automatically captures the image URL and displays it in the dashboard
      ```
    </CodeGroup>

    **In the dashboard:**

    * Images render inline for visual inspection
    * Audio files play directly in the trace viewer
    * Files are downloadable for local analysis
    * JSON/structured outputs are syntax-highlighted

    This eliminates the need to reconstruct multi-modal inputs manually or store attachments separately.

    See [Multi-Modality and Attachments](/developer/basic-features/multi-modality-and-attachments) for supported formats and size limits.
  </Accordion>
</AccordionGroup>

# Advanced Tracing Features

<AccordionGroup>
  <Accordion title="Model Usage & Cost Tracking" icon="dollar-sign">
    Every LLM call costs money. Without tracking, you don't know which features, users, or workflows are expensive. ABV automatically calculates costs for every trace based on:

    * Input tokens × provider's input price
    * Output tokens × provider's output price
    * Provider-specific pricing (cached tokens, batch discounts, etc.)

    **What ABV tracks:**

    * Input/output token counts
    * Total cost per trace (in USD)
    * Cost breakdowns by model, user, session, environment, metadata
    * Cost trends over time (daily, weekly, monthly aggregations)

    **Cost analysis queries:**

    * "Which users cost the most to serve?"
    * "How much did feature X cost last month?"
    * "Is GPT-4 or Claude cheaper for summarization?"
    * "Did costs increase after deploying version 2.0?"

    Set up [usage alerts](/developer/platform/administration/usage-alerts) to get notified when costs exceed thresholds (e.g., >\$100/day).

    See [Model Usage & Cost Tracking](/developer/basic-features/model-usage-cost-tracking) for detailed cost optimization strategies.
  </Accordion>

  <Accordion title="Masking Sensitive Data (PII, Secrets, Prompts)" icon="eye-slash">
    LLM traces often contain sensitive data:

    * **PII**: Emails, phone numbers, social security numbers, addresses
    * **Secrets**: API keys, passwords, tokens
    * **Proprietary prompts**: System instructions you don't want logged

    ABV supports multiple masking strategies:

    **1. Regex-based masking** (replace patterns with `[REDACTED]`):

    ```python theme={null}
    abv.init(
        api_key="...",
        mask_patterns=[
            r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",  # Emails
            r"\b\d{3}-\d{2}-\d{4}\b",  # SSNs
            r"sk-[a-zA-Z0-9]{32,}"  # OpenAI API keys
        ]
    )
    ```

    **2. Field-level masking** (exclude entire fields from logging):

    ```python theme={null}
    abv.init(
        api_key="...",
        exclude_fields=["messages.0.content", "metadata.api_key"]
    )
    ```

    **3. Custom masking functions** (dynamic logic):

    ```python theme={null}
    def custom_masker(data):
        if "email" in data.get("metadata", {}):
            data["metadata"]["email"] = "[REDACTED]"
        return data

    abv.init(api_key="...", masker=custom_masker)
    ```

    **Compliance:**

    * **GDPR**: Mask PII or exclude fields containing personal data
    * **HIPAA**: Redact PHI (protected health information)
    * **SOC 2**: Demonstrate data handling controls with audit logs

    See [Masking Sensitive Data](/developer/basic-features/masking-sensitive-data) for implementation guides and compliance patterns.
  </Accordion>

  <Accordion title="Log Levels: Filter Noise by Severity" icon="filter">
    Not all traces are equally important. **Log levels** let you prioritize signal over noise:

    * `DEBUG`: Verbose internal details (tool calls, intermediate steps)
    * `INFO`: Standard traces (successful LLM calls)
    * `WARNING`: Degraded performance (slow responses, fallbacks)
    * `ERROR`: Failures (API errors, timeouts, invalid outputs)

    **Setting log levels:**

    <CodeGroup>
      ```python Python theme={null}
      # Set minimum log level (only log WARNING and ERROR)
      abv.init(api_key="...", log_level="WARNING")

      # Or set per-trace
      with abv.observe(log_level="ERROR"):
          response = client.chat.completions.create(...)
      ```

      ```javascript JavaScript/TypeScript theme={null}
      // Set minimum log level (only log WARNING and ERROR)
      ABV.init({ apiKey: '...', logLevel: 'WARNING' });

      // Or set per-trace
      await ABV.observe({ logLevel: 'ERROR' }, async () => {
        const response = await client.chat.completions.create(...);
      });
      ```
    </CodeGroup>

    **Use cases:**

    * **Production**: Set to `WARNING` or `ERROR` to reduce noise and focus on issues
    * **Development**: Set to `DEBUG` for maximum visibility
    * **Cost optimization**: Only log `ERROR` traces to minimize ingestion costs

    In the dashboard, filter by log level to focus on failures or warnings.

    See [Log Levels](/developer/basic-features/log-levels) for severity guidelines.
  </Accordion>

  <Accordion title="Releases & Versioning: Track Code and Model Versions" icon="code-branch">
    You deployed a new prompt, updated the model, or changed the RAG retrieval logic. Did quality improve or degrade? **Releases** annotate traces with version information so you can:

    * Compare performance before/after a deployment
    * Identify regressions introduced by code changes
    * Correlate quality drops with specific versions
    * A/B test different versions in production

    **Setting a release version:**

    <CodeGroup>
      ```python Python theme={null}
      abv.init(api_key="...", release="v1.2.3")
      ```

      ```javascript JavaScript/TypeScript theme={null}
      ABV.init({ apiKey: '...', release: 'v1.2.3' });
      ```
    </CodeGroup>

    **Versioning strategies:**

    * **Semantic versioning**: `v1.2.3` (major.minor.patch)
    * **Git commit SHAs**: `abc1234` (unique per deployment)
    * **Timestamp-based**: `2025-01-15-v2` (human-readable)

    In the dashboard, filter traces by release version and compare metrics (latency, cost, error rate) across versions.

    See [Releases & Versioning](/developer/basic-features/releases-versioning) for deployment workflows.
  </Accordion>

  <Accordion title="Comments on Objects: Collaborate on Traces" icon="comment">
    Debugging complex issues often requires collaboration. Instead of copying trace URLs into Slack and losing context, **comment directly on traces** in the ABV dashboard.

    **Use cases:**

    * **Code reviews**: "This prompt could be more concise—too many tokens"
    * **Bug reports**: "@engineer This RAG retrieval pulled the wrong document"
    * **Customer support**: "User reported this response as unhelpful—needs investigation"
    * **Evaluations**: "Mark this output as incorrect for the next benchmark run"

    **Features:**

    * Markdown support for formatting
    * @mentions to notify teammates
    * Thread replies for discussions
    * Comments visible on trace, session, and evaluation pages

    In the dashboard, click the comment icon on any trace to add notes. Teammates get notified and can reply inline.

    See [Comments on Objects](/developer/basic-features/comments-on-objects) for collaboration workflows.
  </Accordion>

  <Accordion title="Trace URLs: Share Deep Links for Reproducible Reports" icon="link">
    Every trace has a unique URL. Share it to give teammates instant access to:

    * The full trace timeline (inputs, outputs, latency, costs)
    * Session context (all traces in the same session)
    * User metadata (who triggered this trace)
    * Comments and annotations

    **Example URL:**

    ```
    https://app.abv.dev/traces/tr-abc123456789
    ```

    **Use cases:**

    * **Bug reports**: Paste trace URL into GitHub issues or Jira tickets
    * **Customer support**: Share trace URL with engineering for instant context
    * **Code reviews**: Reference specific traces in pull request comments
    * **Evaluations**: Link to problematic outputs for manual review

    Trace URLs are stable (they don't expire) and respect your team's access controls (only team members with permission can view).

    See [Trace URLs](/developer/basic-features/trace-urls) for sharing best practices.
  </Accordion>

  <Accordion title="Sampling: Control Volume and Cost" icon="gauge">
    Tracing 100% of production traffic can be expensive, especially at scale (millions of traces/day). **Sampling** lets you capture a representative subset while reducing ingestion costs.

    **Sampling strategies:**

    **1. Rate-based sampling** (trace X% of traffic):

    ```python theme={null}
    abv.init(api_key="...", sample_rate=0.1)  # Trace 10%
    ```

    **2. Rule-based sampling** (trace errors, slow requests, specific users):

    ```python theme={null}
    def should_sample(trace):
        # Always trace errors
        if trace.get("error"):
            return True
        # Always trace slow requests
        if trace.get("latency_ms", 0) > 5000:
            return True
        # Sample 10% of everything else
        return random.random() < 0.1

    abv.init(api_key="...", sampler=should_sample)
    ```

    **3. Environment-specific sampling**:

    ```python theme={null}
    sample_rate = 1.0 if os.getenv("ENV") == "development" else 0.05
    abv.init(api_key="...", sample_rate=sample_rate)
    # Trace 100% in dev, 5% in production
    ```

    **Best practices:**

    * Start with 100% sampling until you understand your traffic patterns
    * Always trace errors and slow requests (use rule-based sampling)
    * Sample more aggressively in production (5-10%) than dev (100%)
    * Exclude health checks and monitoring probes from tracing

    Sampling saves costs without sacrificing observability. You still get representative data for debugging, cost analysis, and evaluations.

    See [Sampling](/developer/basic-features/sampling) for advanced sampling strategies.
  </Accordion>
</AccordionGroup>

# Related Topics

<CardGroup cols={2}>
  <Card title="Sessions" icon="diagram-project" href="/developer/basic-features/sessions">
    Group related traces by user journey or job to see end-to-end behavior and measure multi-step workflows
  </Card>

  <Card title="User Tracking" icon="user" href="/developer/basic-features/user-tracking">
    Link traces to user accounts for faster debugging, cost analysis by user, and GDPR-compliant data handling
  </Card>

  <Card title="Metadata & Tags" icon="tags" href="/developer/basic-features/metadata">
    Attach structured context and labels to traces for precise filtering and analysis
  </Card>

  <Card title="Cost Tracking" icon="dollar-sign" href="/developer/basic-features/model-usage-cost-tracking">
    Monitor tokens, runtime, and spend in real time to optimize model selection and reduce costs
  </Card>

  <Card title="Evaluations" icon="flask" href="/developer/evaluations/overview">
    Run systematic benchmarks on production traces to measure quality, compare prompts, and catch regressions
  </Card>

  <Card title="Prompt Management" icon="pencil" href="/developer/prompt-management/overview">
    Version, deploy, and A/B test prompts with automatic tracing integration for metrics by variant
  </Card>
</CardGroup>