Caching of Prompts in Client SDKs

abv prompts are cached client side in the sdks, so there's no latency impact after the first use and no availability risk you can also pre fetch prompts on startup to populate the cache or provide a fallback prompt cache hit when the sdk cache contains a fresh prompt, it's returned immediately without any network requests background revalidation when the cache ttl has expired, stale prompts are served immediately while it revalidates in the background cache miss when no cached prompt exists (e g , first application startup), the prompt is fetched from the api the api caches prompts in a redis cache to ensure low latency multiple fallback layers ensure resilience if redis is unavailable, the database serves as backup optional pre fetch pre fetching prompts during application startup ensures that the cache is populated before runtime requests this step is optional and often unnecessary typically, the minimal latency experienced during the first use after a service starts is acceptable see examples below on how to set this up optional fallback when both the local cache is empty and the abv api is unavailable, a fallback prompt can be used to ensure 100% availability this is rarely necessary because the prompts api is highly available, and we closely monitor its performance ( status page status page ) in the event of a brief service disruption, the sdk level prompt cache typically ensures that applications remain unaffected optional customize caching duration (ttl) the caching duration is configurable if you wish to reduce network overhead of the abv client the default cache ttl is 60 seconds after the ttl expires, the sdks will refetch the prompt in the background and update the cache refetching is done asynchronously and does not block the application python sdk \# get current `production` prompt version and cache for 5 minutes prompt = abv get prompt("movie critic", cache ttl seconds=300) js/ts sdk import { abvclient } from "@abvdev/client"; const abv = new abvclient(); // get current `production` version and cache prompt for 5 minutes const prompt = await abv prompt get("movie critic", { cachettlseconds 300, }); optional disable caching you can disable caching by setting the cachettlseconds to 0 this will ensure that the prompt is fetched from the abv api on every call this is recommended for non production use cases where you want to ensure that the prompt is always up to date with the latest version in abv python sdk prompt = abv get prompt("movie critic", cache ttl seconds=0) \# common in non production environments, no cache + latest version prompt = abv get prompt("movie critic", cache ttl seconds=0, label="latest") js/ts sdk const prompt = await abv prompt get("movie critic", { cachettlseconds 0, }); // common in non production environments, no cache + latest version const prompt = await abv prompt get("movie critic", { cachettlseconds 0, label "latest", }); optional guaranteed availability of prompts while usually not necessary, you can ensure 100% availability of prompts by pre fetching them on application startup and providing a fallback prompt please follow this guaranteed availability docid\ yeedfgywxhnbfmaan5v52 for more information performance measurement of inital fetch we measured the execution time of the following snippet with fully disabled caching you can run this notebook yourself to ve rify the results prompt = abv get prompt("perf test", cache ttl seconds=0) prompt compile(input="test") results from 1000 sequential executions using abv (includes network latency)