> ## Documentation Index
> Fetch the complete documentation index at: https://docs.abv.dev/llms.txt
> Use this file to discover all available pages before exploring further.
# Trace IDs & Distributed Tracing
> Use custom trace IDs for distributed tracing across microservices, correlation with external systems, and precise trace lookups
# How Trace IDs Work in ABV
ABV uses the **W3C Trace Context standard** for trace IDs:
* **Trace IDs**: 32-character lowercase hexadecimal strings (e.g., `abcdef1234567890abcdef1234567890`)
* **Span IDs** (observation IDs): 16-character lowercase hexadecimal strings (e.g., `fedcba0987654321`)
By default, ABV generates random trace IDs and span IDs for every trace. This works fine for basic observability, but limits advanced use cases:
* **Distributed tracing**: Can't correlate events across multiple services
* **External system integration**: Can't map ABV traces to your support tickets, orders, or sessions
* **Programmatic access**: Can't fetch specific traces via API using your own IDs
**Default ID format:**
* Trace ID: Random 32-character hex string (W3C Trace Context compliant)
* Span ID: Random 16-character hex string
For advanced workflows, bring your own trace IDs.
ABV provides utilities to generate W3C-compliant trace IDs from any seed string. This creates deterministic, reproducible IDs from your external identifiers.
**Why deterministic IDs?**
* Same input always produces the same trace ID
* Correlate ABV traces with support tickets, order IDs, session tokens
* Re-generate the same trace ID later for scoring or retrieval
**Python:**
```python theme={null}
from abvdev import ABV
abv = ABV(api_key="sk-abv-...")
# Generate trace ID from external ID (always the same output for same input)
external_id = "support-ticket-12345"
trace_id = abv.create_trace_id(seed=external_id)
# trace_id: "a1b2c3d4e5f6789012345678abcdef12" (32 hex chars)
```
**JavaScript/TypeScript:**
```typescript theme={null}
import { createTraceId } from '@abvdev/tracing';
const externalId = 'support-ticket-12345';
const traceId = await createTraceId(externalId);
// traceId: "a1b2c3d4e5f6789012345678abcdef12" (32 hex chars)
```
Now you can regenerate this trace ID anytime using the same external ID.
Once you have a trace ID, pass it to ABV when creating traces or spans. This ensures all events use your custom ID instead of a random one.
**Python (decorator pattern):**
```python theme={null}
from abvdev import observe, ABV
abv = ABV(api_key="sk-abv-...")
@observe()
def process_user_request(user_id, request_data):
return f"Processed request for {user_id}"
# Generate trace ID from external ID
external_trace_id = "order-abc-123"
trace_id = abv.create_trace_id(seed=external_trace_id)
# Pass as special keyword argument
process_user_request(
user_id="user-456",
request_data={"query": "refund status"},
abv_trace_id=trace_id # Custom trace ID
)
```
**Python (manual span creation):**
```python theme={null}
with abv.start_as_current_span(
name="process-order",
trace_context={"trace_id": trace_id}
) as span:
# Your code here—this span uses the custom trace ID
result = process_order(order_id="abc-123")
```
**JavaScript/TypeScript:**
```typescript theme={null}
import { startObservation, createTraceId } from '@abvdev/tracing';
const externalId = 'order-abc-123';
const abvTraceId = await createTraceId(externalId);
const rootSpan = startObservation(
'process-order',
{ input: { orderId: 'abc-123' } },
{
parentSpanContext: {
traceId: abvTraceId,
spanId: '0123456789abcdef', // Arbitrary 16-char hex (parent doesn't exist)
traceFlags: 1 // Mark as sampled
}
}
);
// Your code here
rootSpan.end();
```
**Note:** When setting a custom trace ID in JS/TS, you must provide a `parentSpanContext` with an arbitrary `spanId`. This detaches the span from the active context.
For distributed tracing, propagate the trace ID from the entry point through all downstream services via HTTP headers, message queues, or RPC metadata.
**HTTP headers (W3C Trace Context standard):**
```
traceparent: 00---01
```
**Python example (service A → service B):**
```python theme={null}
# Service A: Extract trace ID and forward
import requests
from abvdev import ABV
abv = ABV(api_key="sk-abv-...")
current_trace_id = abv.get_current_trace_id()
# Forward to service B with traceparent header
response = requests.post(
"https://service-b.example.com/process",
headers={"traceparent": f"00-{current_trace_id}-{span_id}-01"},
json={"data": "payload"}
)
```
**Service B: Extract and use trace ID:**
```python theme={null}
# Service B: Extract trace ID from header
def handle_request(request):
traceparent = request.headers.get("traceparent")
if traceparent:
parts = traceparent.split("-")
trace_id = parts[1] # Extract trace ID
# Use the same trace ID in service B
with abv.start_as_current_span(
name="process-in-service-b",
trace_context={"trace_id": trace_id}
):
# Your code here
pass
```
Now both services log events under the same trace ID. In the ABV Dashboard, you see a unified timeline across services.
Retrieve the current trace ID at runtime for logging, debugging, or passing to external systems.
**Python:**
```python theme={null}
from abvdev import ABV
abv = ABV(api_key="sk-abv-...")
with abv.start_as_current_span(name="my-operation") as span:
# Option 1: Access via span
trace_id = span.trace_id
# Option 2: Get from ABV client
current_trace_id = abv.get_current_trace_id()
current_span_id = abv.get_current_observation_id()
print(f"Trace ID: {trace_id}")
print(f"Span ID: {current_span_id}")
```
**JavaScript/TypeScript:**
```typescript theme={null}
import { startActiveObservation, getActiveTraceId } from '@abvdev/tracing';
await startActiveObservation('my-operation', async (span) => {
const traceId = getActiveTraceId();
console.log(`Current trace ID: ${traceId}`);
// Use trace ID in logs or external systems
await logToExternalSystem({ traceId, event: 'operation-start' });
});
```
Store these IDs in your logs or databases to link back to ABV traces later.
# Why Use Custom Trace IDs?
Modern applications span multiple services: API gateways, authentication services, LLM backends, RAG retrievers, databases, caching layers. When a user request fails, you need to see the full journey—not just isolated logs from each service.
**The problem:**
Without shared trace IDs, each service logs events independently. You have:
* API gateway logs: "Request received at 10:45:12"
* Auth service logs: "User validated at 10:45:13"
* LLM backend logs: "LLM call failed at 10:45:15"
* No connection between these events
**The solution with custom trace IDs:**
1. API gateway generates a trace ID when the request arrives
2. Passes trace ID to auth service via HTTP header (`traceparent`)
3. Auth service extracts trace ID and uses it for its logs
4. Auth service forwards trace ID to LLM backend
5. LLM backend logs events with the same trace ID
6. ABV groups all events into one trace
**In the dashboard:**
Search for the trace ID and see the complete timeline:
```
10:45:12 [api-gateway] Request received
10:45:13 [auth-service] User validated
10:45:14 [llm-backend] LLM call started
10:45:15 [llm-backend] LLM call failed (timeout)
```
**How trace IDs propagate across services:**
```mermaid theme={null}
sequenceDiagram
participant Client
participant APIGateway as API Gateway
participant AuthService as Auth Service
participant LLMBackend as LLM Backend
participant ABV as ABV Dashboard
Client->>APIGateway: Request arrives
APIGateway->>APIGateway: Generate trace_id:
abc123...
APIGateway->>ABV: Log: [trace_id: abc123] Request received
APIGateway->>AuthService: Forward request
Header: traceparent: 00-abc123-...
AuthService->>AuthService: Extract trace_id from header
AuthService->>ABV: Log: [trace_id: abc123] User validated
AuthService->>LLMBackend: Forward request
Header: traceparent: 00-abc123-...
LLMBackend->>LLMBackend: Extract trace_id from header
LLMBackend->>ABV: Log: [trace_id: abc123] LLM call started
LLMBackend->>ABV: Log: [trace_id: abc123] LLM call failed
Note over ABV: All events grouped under
trace_id: abc123
LLMBackend-->>AuthService: Error response
AuthService-->>APIGateway: Error response
APIGateway-->>Client: Error response
```
**Implementation:**
```python theme={null}
# API Gateway (entry point)
trace_id = abv.create_trace_id(seed=f"request-{request_id}")
# Pass to downstream services via HTTP header
headers = {"traceparent": f"00-{trace_id}-{span_id}-01"}
response = requests.post("https://auth-service/validate", headers=headers, ...)
# Auth service extracts and uses trace ID
incoming_trace_id = extract_trace_id_from_header(request.headers)
with abv.start_as_current_span(name="validate-user", trace_context={"trace_id": incoming_trace_id}):
# Validation logic
pass
```
**Benefits:**
* Root cause analysis: See exactly where requests fail in multi-service workflows
* Performance optimization: Identify slow services in the critical path
* Debugging: Trace request flow end-to-end with one query
Your support dashboard, admin panel, or internal tools display user sessions, orders, or tickets. When investigating issues, you want to jump directly to the corresponding ABV trace without searching manually.
**The problem:**
ABV assigns random trace IDs (e.g., `f3a2b1c9-4567-8901-2345-6789abcdef01`). Your support ticket has ID `ticket-12345`. No connection between them.
**The solution with deterministic trace IDs:**
Generate ABV trace IDs from your external IDs:
```python theme={null}
# When logging the trace
ticket_id = "ticket-12345"
trace_id = abv.create_trace_id(seed=ticket_id)
# trace_id: "a1b2c3d4e5f6789012345678abcdef12" (deterministic)
# Store ticket_id in trace metadata for reverse lookup
with abv.start_as_current_span(
name="handle-support-ticket",
trace_context={"trace_id": trace_id}
) as span:
span.set_attribute("ticket_id", ticket_id)
```
**In your support dashboard:**
```python theme={null}
# User clicks "View ABV Trace" button for ticket-12345
ticket_id = "ticket-12345"
trace_id = abv.create_trace_id(seed=ticket_id) # Regenerate same ID
# Construct deeplink URL
abv_url = f"https://app.abv.dev/traces/{trace_id}"
# Redirect user to ABV trace
```
**Benefits:**
* Instant navigation from your tools to ABV traces
* No manual searching or copy-pasting trace IDs
* Better support workflows: "View this user's LLM traces" → one click
You're running A/B tests, experiments, or benchmarks. Each experiment has an ID (e.g., `experiment-2025-01-v2`). You need to fetch all traces from that experiment and score them programmatically.
**The problem:**
ABV's random trace IDs don't map to your experiment IDs. You can add metadata (`metadata.experiment_id = "experiment-2025-01-v2"`), but fetching and scoring requires API queries.
**The solution with deterministic trace IDs:**
Generate trace IDs that include your experiment ID:
```python theme={null}
# When running experiment
experiment_id = "experiment-2025-01-v2"
user_request_id = "request-789"
# Combine experiment ID + request ID for unique trace
seed = f"{experiment_id}-{user_request_id}"
trace_id = abv.create_trace_id(seed=seed)
# Log trace with custom ID
with abv.start_as_current_span(
name="experiment-run",
trace_context={"trace_id": trace_id}
) as span:
span.set_attribute("experiment_id", experiment_id)
# Run LLM call
```
**Later, score all traces from experiment:**
```python theme={null}
# Fetch all request IDs from experiment
request_ids = ["request-789", "request-790", "request-791"]
for request_id in request_ids:
# Regenerate trace ID
seed = f"{experiment_id}-{request_id}"
trace_id = abv.create_trace_id(seed=seed)
# Fetch trace via API
trace = abv.get_trace(trace_id)
# Score the trace
abv.score(trace_id=trace_id, name="quality", value=0.85)
```
**Benefits:**
* Programmatic access to specific traces using your IDs
* Batch scoring for experiments without complex queries
* Reproducible evaluations (same seed = same trace ID every time)
You want to fetch ABV traces programmatically via API, but you only know your external IDs (session tokens, order IDs, user IDs). ABV's random trace IDs force you to query by metadata, which is slower and more complex.
**The solution with deterministic trace IDs:**
Generate ABV trace IDs from your external IDs, then fetch directly by trace ID (fast, precise).
**Example: Fetch trace for a specific order**
```python theme={null}
import requests
# Your order ID
order_id = "order-abc-123"
# Regenerate ABV trace ID from order ID
trace_id = abv.create_trace_id(seed=order_id)
# Fetch trace via ABV API (direct lookup by trace ID)
response = requests.get(
f"https://app.abv.dev/api/public/traces/{trace_id}",
headers={"Authorization": f"Bearer {api_key}"}
)
trace = response.json()
print(f"Order {order_id} cost: ${trace['calculatedTotalCost']}")
```
**Benefits:**
* Direct trace retrieval by ID (no complex metadata queries)
* Faster API responses (indexed by trace ID)
* Simpler code—no need to parse query results
ABV's trace IDs follow the **W3C Trace Context** standard, making them compatible with OpenTelemetry and other observability tools. If you're already using OpenTelemetry, ABV integrates seamlessly.
**W3C Trace Context format:**
```
traceparent: 00---
```
* `00`: Version (fixed)
* ``: 32-character hex string
* ``: 16-character hex string
* ``: 01 (sampled) or 00 (not sampled)
**OpenTelemetry integration (Python):**
```python theme={null}
from opentelemetry import trace
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("my-operation") as span:
# Get W3C-compliant trace ID
trace_id = format(span.get_span_context().trace_id, "032x")
# Use in ABV (already compatible)
print(f"Trace ID: {trace_id}")
```
**Benefits:**
* Use the same trace IDs across ABV and other observability tools (Datadog, Honeycomb, Jaeger)
* Vendor-neutral: Switch tools without changing instrumentation
* Standards-based: Follow W3C best practices for distributed tracing
# Implementation Guide
Use the `@observe()` decorator to automatically trace functions. Pass custom trace IDs via the special `abv_trace_id` keyword argument.
**Setup:**
```bash theme={null}
pip install abvdev
```
**Basic usage:**
```python theme={null}
from abvdev import observe, ABV
import uuid
# Initialize ABV client
abv = ABV(
api_key="sk-abv-...",
host="https://app.abv.dev" # or "https://eu.app.abv.dev" for EU
)
@observe()
def process_user_request(user_id, request_data):
# Function logic
return f"Processed request for {user_id}"
# Generate custom trace ID from external ID
external_id = "session-abc-123"
trace_id = abv.create_trace_id(seed=external_id)
# Call function with custom trace ID
process_user_request(
user_id="user-456",
request_data={"query": "account balance"},
abv_trace_id=trace_id # Special keyword argument
)
```
**Key points:**
* `abv_trace_id` is a special keyword argument recognized by `@observe()`
* Trace ID must be a 32-character lowercase hexadecimal string
* Use `abv.create_trace_id(seed="...")` to generate W3C-compliant IDs from any string
For more control, create spans manually with custom trace contexts.
**Create span with custom trace ID:**
```python theme={null}
from abvdev import ABV
abv = ABV(api_key="sk-abv-...", host="https://app.abv.dev")
# Option 1: Use deterministic trace ID from external ID
external_id = "order-xyz-789"
trace_id = abv.create_trace_id(seed=external_id)
# Option 2: Use a predefined trace ID (must be 32 hex chars)
# trace_id = "abcdef1234567890abcdef1234567890"
# Create span with custom trace ID
with abv.start_as_current_span(
name="process-order",
trace_context={
"trace_id": trace_id,
"parent_span_id": "fedcba0987654321" # Optional, 16 hex chars
}
) as span:
print(f"This span has trace_id: {span.trace_id}")
# Your code here
result = process_order(order_id="xyz-789")
```
**Access current trace ID:**
```python theme={null}
with abv.start_as_current_span(name="outer-operation") as span:
# Option 1: Access via span
trace_id = span.trace_id
# Option 2: Get from ABV client
current_trace_id = abv.get_current_trace_id()
current_span_id = abv.get_current_observation_id()
print(f"Trace ID: {current_trace_id}")
print(f"Span ID: {current_span_id}")
```
Use the `@abvdev/tracing` package to create spans with custom trace IDs.
**Setup:**
```bash theme={null}
npm install @abvdev/tracing @abvdev/otel @opentelemetry/sdk-node dotenv
```
**Configuration (instrumentation.ts):**
```typescript theme={null}
import dotenv from 'dotenv';
dotenv.config();
import { NodeSDK } from '@opentelemetry/sdk-node';
import { ABVSpanProcessor } from '@abvdev/otel';
const sdk = new NodeSDK({
spanProcessors: [
new ABVSpanProcessor({
apiKey: process.env.ABV_API_KEY,
baseUrl: process.env.ABV_BASE_URL,
exportMode: 'immediate',
flushAt: 1,
flushInterval: 1
})
]
});
sdk.start();
```
**Create span with custom trace ID:**
```typescript theme={null}
import './instrumentation'; // Must be first import
import { createTraceId, startObservation } from '@abvdev/tracing';
async function main() {
// Generate deterministic trace ID from external ID
const externalId = 'support-ticket-252525';
const abvTraceId = await createTraceId(externalId);
// Start span with custom trace ID
const rootSpan = startObservation(
'process-ticket',
{
input: { ticketId: externalId },
output: { status: 'resolved' }
},
{
parentSpanContext: {
traceId: abvTraceId,
spanId: '0123456789abcdef', // Arbitrary 16-char hex
traceFlags: 1 // Mark as sampled
}
}
);
// Your code here
await processTicket(externalId);
rootSpan.end();
// Later, regenerate the same trace ID for scoring
const scoringTraceId = await createTraceId(externalId);
console.log(scoringTraceId === abvTraceId); // true
}
main();
```
**Important:**
* `parentSpanContext.spanId` must be a valid 16-character hex string
* The parent span doesn't actually exist—it's only used for trace ID inheritance
* Setting `parentSpanContext` detaches the span from the active context
Retrieve the active trace ID at runtime.
**Example:**
```typescript theme={null}
import './instrumentation'; // Must be first import
import {
startActiveObservation,
getActiveTraceId,
updateActiveTrace
} from '@abvdev/tracing';
async function main() {
await startActiveObservation('run', async (span) => {
// Update span metadata
span.update({
input: { query: 'What is the capital of France?' }
});
// Update trace metadata
updateActiveTrace({
metadata: { userId: 'user-123' }
});
// Get current trace ID
const traceId = getActiveTraceId();
console.log(`Current trace ID: ${traceId}`);
// Use in logs or external systems
await logEvent({ traceId, event: 'query-started' });
});
}
main();
```
**Use cases:**
* Log trace ID to external systems (Datadog, Splunk)
* Include trace ID in API responses for debugging
* Pass trace ID to downstream services via HTTP headers
For microservices architectures, propagate trace IDs using HTTP headers following the W3C Trace Context standard.
**Service A (Python): Send request with trace ID**
```python theme={null}
import requests
from abvdev import ABV
abv = ABV(api_key="sk-abv-...")
# Get current trace ID
with abv.start_as_current_span(name="call-service-b") as span:
trace_id = span.trace_id
span_id = format(span.get_span_context().span_id, "016x")
# Construct W3C traceparent header
traceparent = f"00-{trace_id}-{span_id}-01"
# Send request to service B
response = requests.post(
"https://service-b.example.com/process",
headers={"traceparent": traceparent},
json={"data": "payload"}
)
```
**Service B (Python): Extract and use trace ID**
```python theme={null}
from flask import Flask, request
from abvdev import ABV
app = Flask(__name__)
abv = ABV(api_key="sk-abv-...")
@app.route("/process", methods=["POST"])
def handle_request():
# Extract traceparent header
traceparent = request.headers.get("traceparent")
if traceparent:
# Parse W3C traceparent format: 00---
parts = traceparent.split("-")
trace_id = parts[1] # Extract trace ID
parent_span_id = parts[2] # Extract parent span ID
# Use the same trace ID in service B
with abv.start_as_current_span(
name="process-in-service-b",
trace_context={
"trace_id": trace_id,
"parent_span_id": parent_span_id
}
) as span:
# Process request
result = process_data(request.json)
return {"status": "success", "result": result}
# Fallback if no traceparent header
return {"error": "Missing traceparent header"}, 400
```
**Service A (TypeScript): Send request with trace ID**
```typescript theme={null}
import fetch from 'node-fetch';
import { startActiveObservation, getActiveTraceId } from '@abvdev/tracing';
await startActiveObservation('call-service-b', async (span) => {
const traceId = getActiveTraceId();
const spanId = span.spanContext().spanId;
// Construct W3C traceparent header
const traceparent = `00-${traceId}-${spanId}-01`;
// Send request to service B
const response = await fetch('https://service-b.example.com/process', {
method: 'POST',
headers: {
'traceparent': traceparent,
'Content-Type': 'application/json'
},
body: JSON.stringify({ data: 'payload' })
});
});
```
**Service B (TypeScript): Extract and use trace ID**
```typescript theme={null}
import express from 'express';
import { startObservation } from '@abvdev/tracing';
const app = express();
app.post('/process', async (req, res) => {
const traceparent = req.headers['traceparent'];
if (traceparent) {
// Parse W3C traceparent format
const [version, traceId, parentSpanId, flags] = traceparent.split('-');
// Create span with inherited trace ID
const span = startObservation(
'process-in-service-b',
{ input: req.body },
{
parentSpanContext: {
traceId,
spanId: parentSpanId,
traceFlags: parseInt(flags, 16)
}
}
);
// Process request
const result = await processData(req.body);
span.end();
res.json({ status: 'success', result });
} else {
res.status(400).json({ error: 'Missing traceparent header' });
}
});
app.listen(3000);
```
**Result:**
In the ABV Dashboard, search for the trace ID and see events from both services grouped together in one timeline.
If you're using OpenTelemetry directly (without ABV SDKs), trace IDs are managed by the OpenTelemetry SDK. ABV automatically ingests OpenTelemetry spans with their trace IDs.
**Access trace ID in OpenTelemetry (Python):**
```python theme={null}
from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("my-operation") as span:
# Get trace ID (returns an integer)
trace_id_int = span.get_span_context().trace_id
# Format as 32-character hex string
trace_id = format(trace_id_int, "032x")
print(f"Trace ID: {trace_id}")
# Set custom attributes
span.set_attribute("custom.trace_id", trace_id)
# Your code here
result = perform_operation()
# Set span status
span.set_status(Status(StatusCode.OK))
```
**Access trace ID in OpenTelemetry (JavaScript/TypeScript):**
```typescript theme={null}
import { trace } from '@opentelemetry/api';
const tracer = trace.getTracer('my-service');
const span = tracer.startSpan('my-operation');
// Get trace ID
const traceId = span.spanContext().traceId;
console.log(`Trace ID: ${traceId}`);
// Your code here
performOperation();
span.end();
```
**ABV integration:**
* ABV's span processors automatically extract trace IDs from OpenTelemetry spans
* No code changes needed—just configure ABVSpanProcessor
* Trace IDs in ABV Dashboard match OpenTelemetry trace IDs exactly
# Next Steps
Share deep links to traces for reproducible bug reports, customer support, and code reviews
Group related traces by user journey or job to see end-to-end behavior across multiple LLM calls
Attach structured context to traces for precise filtering, analysis, and correlation with business data
Learn more about OpenTelemetry-based SDKs and advanced instrumentation patterns