Skip to main content
ABV’s prompt management is highly available through multi-layer caching and robust infrastructure. For most applications, the default SDK caching provides sufficient resilience. However, mission-critical systems requiring absolute guarantees can implement pre-fetching or fallback prompts to eliminate any dependency on ABV’s availability.
Implementing guaranteed availability is usually unnecessary and adds complexity to your application. ABV Prompt Management is highly available due to multiple caching layers, and we closely monitor performance (status page). Consider these strategies only for mission-critical applications with zero-downtime requirements.

​
Understanding the Failure Scenario

Before implementing guaranteed availability, understand when and why prompt fetching can fail:
Typical request flow:
  1. Application starts, first get_prompt() call fetches from ABV API
  2. Prompt is cached locally in SDK (60-second TTL default)
  3. All subsequent get_prompt() calls return from cache (<1ms, zero network)
  4. After TTL expiry, background revalidation updates cache
  5. Cycle repeats
Network dependency: Only the first fetch and background revalidations require network access.ABV outage impact: If ABV becomes unavailable, cached prompts continue working until the cache expires. Even after expiry, stale-while-revalidate means the cached prompt is still served.Conclusion: Most applications experience zero impact from brief ABV outages due to caching.
Prompt fetching fails only when all of these conditions are true simultaneously:
  1. No cached prompt: Fresh application startup, or fetching a prompt name for the first time
  2. Network request to ABV fails: After retries (typically 3 attempts)
  3. No fallback configured: Application didn’t provide a fallback prompt
When this occurs:
  • Application deployment to new instances during ABV outage
  • Kubernetes pod restart during ABV outage
  • First use of a new prompt name when ABV is unreachable
Frequency: Extremely rare. Requires both ABV unavailability and cold start timing.Without guaranteed availability: get_prompt() raises an exception, application must handle error.
ABV’s high availability:
  • Multi-region deployment with automatic failover
  • 99.9% uptime SLA
  • Public status page with real-time monitoring
  • Multiple caching layers (SDK cache, API Redis cache, database fallback)
SDK resilience:
  • Caching eliminates network dependency for most requests
  • Stale-while-revalidate ensures zero downtime during cache updates
  • Automatic retries with exponential backoff
Practical reality: Most applications have other single points of failure (database, payment gateway, auth service) with similar availability profiles. Adding guaranteed availability for prompts while tolerating failures elsewhere provides minimal benefit.When to implement: Mission-critical systems (healthcare, finance, safety) with explicit zero-downtime requirements and comprehensive failure handling across all dependencies.

​
Option 1: Pre-Fetch Prompts on Startup

Fetch prompts during application initialization and exit if fetching fails:
Install dependencies:
pip install abvdev flask
Implementation:
from flask import Flask, jsonify
from abvdev import ABV
import sys

# Initialize Flask app and ABV client
app = Flask(__name__)
abv = ABV(
    api_key="sk-abv-...",
    host="https://app.abv.dev",  # or "https://eu.app.abv.dev" for EU
)

def fetch_prompts_on_startup():
    """
    Pre-fetch all prompts used by the application.
    Exit if any prompt fetch fails, preventing degraded startup.
    """
    try:
        # Fetch and cache all prompts used by your application
        abv.get_prompt("movie-critic")
        abv.get_prompt("summarizer")
        abv.get_prompt("translator")
        print("All prompts successfully fetched and cached")
    except Exception as e:
        print(f"CRITICAL: Failed to fetch prompts on startup: {e}")
        print("Application will not start in degraded state. Exiting.")
        sys.exit(1)  # Exit with error code

# Call during application startup (before accepting traffic)
fetch_prompts_on_startup()

@app.route('/get-movie-prompt/<movie>', methods=['GET'])
def get_movie_prompt(movie):
    # Prompt is guaranteed to be in cache (fetched at startup)
    prompt = abv.get_prompt("movie-critic")
    compiled_prompt = prompt.compile(criticlevel="expert", movie=movie)
    return jsonify({"prompt": compiled_prompt})

if __name__ == '__main__':
    app.run(debug=True)
Behavior:
  • ABV available at startup: Prompts cached, application starts normally
  • ABV unavailable at startup: Application exits with error code, orchestration system can retry or alert
Health checks: Container orchestration systems (Kubernetes, Docker Swarm) detect failed startup and prevent routing traffic to unhealthy instances.
Install dependencies:
npm install @abvdev/client express dotenv
Environment variables (.env):
ABV_API_KEY=sk-abv-...
ABV_BASEURL=https://app.abv.dev  # US region
# ABV_BASEURL=https://eu.app.abv.dev  # EU region
Implementation:
import express from "express";
import { ABVClient } from "@abvdev/client";
import dotenv from "dotenv";
dotenv.config();

// Initialize Express app and ABV client
const app = express();
const abv = new ABVClient();

async function fetchPromptsOnStartup() {
  /**
   * Pre-fetch all prompts used by the application.
   * Exit if any prompt fetch fails.
   */
  try {
    // Fetch and cache all prompts
    await abv.prompt.get("movie-critic");
    await abv.prompt.get("summarizer");
    await abv.prompt.get("translator");
    console.log("All prompts successfully fetched and cached");
  } catch (error) {
    console.error("CRITICAL: Failed to fetch prompts on startup:", error);
    console.error("Application will not start in degraded state. Exiting.");
    process.exit(1);  // Exit with error code
  }
}

// Call during application startup
await fetchPromptsOnStartup();

app.get("/get-movie-prompt/:movie", async (req, res) => {
  const movie = req.params.movie;
  // Prompt guaranteed to be in cache (fetched at startup)
  const prompt = await abv.prompt.get("movie-critic");
  const compiledPrompt = prompt.compile({ criticlevel: "expert", movie });
  res.json({ prompt: compiledPrompt });
});

app.listen(3000, () => {
  console.log("Server running on port 3000");
});
Run:
npx tsx server.ts
Test:
curl http://localhost:3000/get-movie-prompt/The-Lord-of-the-Rings
Configure Kubernetes to detect pre-fetch failures:Deployment manifest (deployment.yaml):
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: app
        image: my-app:latest
        env:
        - name: ABV_API_KEY
          valueFrom:
            secretKeyRef:
              name: abv-secret
              key: api-key
        # Startup probe: Wait for app to successfully start
        startupProbe:
          httpGet:
            path: /health
            port: 3000
          initialDelaySeconds: 5
          periodSeconds: 5
          failureThreshold: 3
        # Liveness probe: Restart if app becomes unhealthy
        livenessProbe:
          httpGet:
            path: /health
            port: 3000
          periodSeconds: 10
        # Readiness probe: Remove from service if unhealthy
        readinessProbe:
          httpGet:
            path: /health
            port: 3000
          periodSeconds: 5
Health endpoint implementation:
@app.route('/health', methods=['GET'])
def health():
    return jsonify({"status": "healthy"}), 200
Behavior: If pre-fetch fails (app exits with code 1), Kubernetes:
  1. Detects container exit
  2. Doesn’t route traffic to failed pod
  3. Attempts restart with backoff
  4. Alerts if restart limit exceeded

​
Option 2: Fallback Prompts

Provide hardcoded fallback prompts when ABV is unreachable:
Text prompt with fallback:
from abvdev import ABV

abv = ABV(
    api_key="sk-abv-...",
    host="https://app.abv.dev",
)

# Fetch prompt with fallback
prompt = abv.get_prompt(
    "movie-critic",
    fallback="Do you like {{movie}}?"  # Used if ABV unreachable
)

# Check if fallback was used
if prompt.is_fallback:
    print("WARNING: Using fallback prompt (ABV unavailable)")
    # Optional: Log alert, increment metric, etc.

# Compile and use (works same way for ABV or fallback)
compiled = prompt.compile(movie="Dune 2")
Chat prompt with fallback:
# Fetch chat prompt with fallback
chat_prompt = abv.get_prompt(
    "movie-critic-chat",
    type="chat",
    fallback=[
        {"role": "system", "content": "You are an expert on {{movie}}"},
        {"role": "user", "content": "Provide a review"}
    ]
)

# Check if fallback
if chat_prompt.is_fallback:
    print("WARNING: Using fallback chat prompt")

# Compile and use
compiled_messages = chat_prompt.compile(movie="Dune 2")
Key properties:
  • prompt.is_fallback (bool): True if fallback prompt is being used
  • prompt.compile(**vars): Works identically for ABV and fallback prompts
Text prompt with fallback:
import { ABVClient } from "@abvdev/client";

const abv = new ABVClient();

async function main() {
  // Fetch prompt with fallback
  const prompt = await abv.prompt.get("movie-critic", {
    fallback: "Do you like {{movie}}?"  // Used if ABV unreachable
  });

  // Check if fallback was used
  if (prompt.isFallback) {
    console.warn("WARNING: Using fallback prompt (ABV unavailable)");
    // Optional: Log alert, increment metric, etc.
  }

  // Compile and use
  const compiled = prompt.compile({ movie: "Dune 2" });
  console.log(compiled);
}

main();
Chat prompt with fallback:
async function main() {
  // Fetch chat prompt with fallback
  const chatPrompt = await abv.prompt.get("movie-critic-chat", {
    type: "chat",
    fallback: [
      { role: "system", content: "You are an expert on {{movie}}" },
      { role: "user", content: "Provide a review" }
    ]
  });

  // Check if fallback
  if (chatPrompt.isFallback) {
    console.warn("WARNING: Using fallback chat prompt");
  }

  // Compile and use
  const compiledMessages = chatPrompt.compile({ movie: "Dune 2" });
  console.log(compiledMessages);
}

main();
Key properties:
  • prompt.isFallback (boolean): true if fallback prompt is being used
  • prompt.compile(vars): Works identically for ABV and fallback prompts
Keep fallbacks in sync with production:
# Bad: Hardcoded fallback that diverges over time
prompt = abv.get_prompt(
    "movie-critic",
    fallback="Review {{movie}}."  # Doesn't match current production prompt
)

# Better: Periodically update fallbacks to match production prompts
# (e.g., copy production prompt to fallback during deployment)
FALLBACK_PROMPTS = {
    "movie-critic": "As a {{criticLevel}} critic, review {{movie}}.",  # Matches v5
    # Updated: 2025-01-15
}

prompt = abv.get_prompt(
    "movie-critic",
    fallback=FALLBACK_PROMPTS["movie-critic"]
)
Monitor fallback usage:
prompt = abv.get_prompt("movie-critic", fallback="...")

if prompt.is_fallback:
    # Increment metric
    metrics.increment("prompts.fallback.used", tags=["prompt:movie-critic"])

    # Log warning
    logger.warning(
        "Using fallback prompt for movie-critic. "
        "ABV may be unavailable. Check status page."
    )

    # Optional: Alert on-call if fallbacks are used frequently
    if fallback_usage_rate > 0.01:  # >1% fallback usage
        alert("High fallback prompt usage detected")
Understand fallback limitations:
  • No metrics tracking: Fallback prompts aren’t linked to traces, so you lose version-specific metrics
  • No config: Fallback prompts don’t include config field (model parameters, tools, etc.)
  • Maintenance burden: Must keep fallbacks updated manually
  • Version mismatch risk: Fallback may not match current production prompt
When to use fallbacks: Only for truly mission-critical applications where any downtime is unacceptable. For most applications, pre-fetching or accepting occasional startup failures is simpler.

​
Comparing Approaches

AspectPre-FetchingFallbacks
Startup behaviorFails if ABV unreachableAlways starts successfully
Operational modeFail closed (won’t run degraded)Fail open (runs with fallbacks)
Metrics trackingFull metrics (uses ABV prompts)No metrics during fallback
MaintenanceNo additional maintenanceMust keep fallbacks updated
Version mismatch riskNone (always uses latest from ABV)High (fallbacks can diverge)
ComplexityLow (one-time fetch)Medium (manage fallbacks)
Best forApplications that shouldn’t run degradedApplications requiring 100% uptime
Recommendation: Prefer pre-fetching for most mission-critical applications. It’s simpler and provides clearer failure modes. Use fallbacks only if your application absolutely must keep running during ABV outages.
Questions to ask:
  1. What’s the impact of a startup failure?
    • If deployment tools retry automatically, brief startup failures are harmless
    • If manual intervention is required, pre-fetching adds risk
  2. What’s your uptime requirement?
    • 99% (two nines): SDK caching is sufficient
    • 99.9% (three nines): Consider pre-fetching
    • 99.99% (four nines): Consider fallbacks or pre-fetching with multiple retries
  3. Do you have other single points of failure?
    • Database, auth service, payment gateway all have similar availability
    • If you tolerate those failures, why special-case prompts?
  4. Can you tolerate fallback degradation?
    • Fallback prompts may have lower quality than ABV-managed prompts
    • If quality is critical, pre-fetching (fail closed) is better
Most applications: Standard SDK caching provides sufficient availability without additional complexity.Mission-critical applications: Pre-fetch prompts at startup for clear failure modes.Absolute zero-downtime requirements: Use fallbacks, but understand the tradeoffs and maintenance burden.

​
Next Steps

Client-Side Caching

Understand how SDK caching provides resilience

Get Started with Prompts

Create and fetch prompts with ABV SDKs

Status Page

Monitor ABV’s real-time availability

Version Control

Manage prompt versions and deployments