Typescript Guide

Installation and Setup

The ABV client library comes as a single npm package that includes everything you need. Install it in your project with npm, yarn, or pnpm:

npm install @abvdev/client

If you’re working with TypeScript, the type definitions are included in the package automatically. You don’t need to install separate @types packages. For projects that want explicit control over tracing behavior, you can also install the tracing package, though it’s already included as a dependency of the client:

npm install @abvdev/tracing

Client Initialization Patterns

The way you initialize the ABV client affects how you use it throughout your application. Let’s explore different patterns and when to use each. The simplest initialization creates a client with your API key directly:

import { ABVClient } from '@abvdev/client';

const abv = new ABVClient({
  apiKey: 'sk_...'
});

This works for quick prototypes or scripts, but hardcoding credentials isn’t recommended for production applications. Instead, use environment variables:

const abv = new ABVClient({
  apiKey: process.env.ABV_API_KEY
});

The client also checks for the ABV_API_KEY environment variable automatically, so you can simplify this further:

const abv = new ABVClient();

This pattern is cleaner and makes it impossible to accidentally commit credentials to version control. For applications that need to support multiple regions, specify the region during initialization:

const abv = new ABVClient({
  region: 'eu'  // 'us' (default) or 'eu'
});

Region selection affects which ABV infrastructure your requests route through. Choose the region closest to your users or the region that matches your data residency requirements.

Client Lifecycle and Reuse

Creating an ABV client is lightweight, but you should generally create one client instance and reuse it throughout your application. The client maintains connection pooling and handles request optimization internally. For a Node.js server application, create the client once at startup and export it for use across your application:

// abv-client.ts
import { ABVClient } from '@abvdev/client';

export const abv = new ABVClient();

Then import this client wherever you need it:

// route-handler.ts
import { abv } from './abv-client';

export async function handleRequest(userMessage: string) {
  const response = await abv.gateway.chat.completions.create({
    provider: 'openai',
    model: 'gpt-4o-mini',
    messages: [{ role: 'user', content: userMessage }]
  });

  return response.choices[0].message.content;
}

This pattern ensures efficient connection management and simplifies testing since you can mock the exported client in your test files.

Working with Types

TypeScript’s type system helps catch errors before runtime. The ABV client library provides comprehensive type definitions that make working with the gateway safer and more productive. Import types from the client package to annotate your code:

import type {
  ChatCompletionParams,
  ChatCompletionResponse,
  ChatMessage
} from '@abvdev/client';

These types help you build functions that work with gateway responses:

function extractResponse(response: ChatCompletionResponse): string {
  return response.choices[0].message.content;
}

function buildMessages(
  systemPrompt: string,
  userMessage: string
): ChatMessage[] {
  return [
    { role: 'system', content: systemPrompt },
    { role: 'user', content: userMessage }
  ];
}

When building the parameters object for a request, TypeScript ensures you provide valid values:

const params: ChatCompletionParams = {
  provider: 'openai',
  model: 'gpt-4o-mini',
  messages: [{ role: 'user', content: 'Hello' }],
  temperature: 0.7,
  max_tokens: 500
};

// TypeScript catches errors here
const response = await abv.gateway.chat.completions.create(params);

If you make a typo in the provider name or forget a required field, TypeScript flags it immediately. This catches bugs during development rather than in production.

Handling Streaming Responses

Streaming responses arrive as an asynchronous iterable, which you consume using a for-await-of loop. This pattern feels natural in modern JavaScript and TypeScript:

const stream = await abv.gateway.chat.completions.create({
  provider: 'openai',
  model: 'gpt-4o-mini',
  messages: [{ role: 'user', content: 'Tell me a story' }],
  stream: true
});

for await (const chunk of stream) {
  const content = chunk.choices[0]?.delta?.content;
  if (content) {
    process.stdout.write(content);
  }
}

Notice the optional chaining (?.) when accessing the content. Early chunks might not have content, or the structure might vary slightly. Optional chaining prevents runtime errors from null or undefined values. For applications that need to both display tokens as they arrive and store the complete response, accumulate chunks while streaming:

const stream = await abv.gateway.chat.completions.create({
  provider: 'openai',
  model: 'gpt-4o-mini',
  messages: [{ role: 'user', content: 'Explain TypeScript generics' }],
  stream: true
});

let fullResponse = '';

for await (const chunk of stream) {
  const content = chunk.choices[0]?.delta?.content;
  if (content) {
    fullResponse += content;
    process.stdout.write(content);
  }
}

console.log('\n\nComplete response length:', fullResponse.length);

This pattern works well for chatbots that display streaming responses to users but also need to save conversation history.

Error Handling Strategies

Gateway requests can fail for various reasons, and handling these failures gracefully improves your application’s reliability. TypeScript’s type system helps you handle errors correctly. The most basic error handling uses try-catch:

try {
  const response = await abv.gateway.chat.completions.create({
    provider: 'openai',
    model: 'gpt-4o-mini',
    messages: [{ role: 'user', content: 'Hello' }]
  });

  console.log(response.choices[0].message.content);
} catch (error) {
  console.error('Gateway request failed:', error);
}

For production applications, you want more sophisticated error handling that distinguishes between error types. Errors might be network issues, rate limits, authentication problems, or invalid requests. Each type suggests different handling:

async function makeRequest(
  messages: ChatMessage[]
): Promise<string> {
  try {
    const response = await abv.gateway.chat.completions.create({
      provider: 'openai',
      model: 'gpt-4o-mini',
      messages
    });

    return response.choices[0].message.content;
  } catch (error) {
    if (error instanceof Error) {
      // Check error message or properties to determine type
      if (error.message.includes('rate limit')) {
        console.error('Rate limited, need to slow down');
        throw new Error('Service temporarily unavailable');
      } else if (error.message.includes('authentication')) {
        console.error('Authentication failed, check API key');
        throw new Error('Configuration error');
      }
    }

    // Generic error handling for unexpected failures
    console.error('Unexpected error:', error);
    throw new Error('Request failed');
  }
}

For applications that need retry logic, implement exponential backoff. This pattern waits increasingly longer between retries, reducing load on the service:

async function makeRequestWithRetry(
  messages: ChatMessage[],
  maxRetries: number = 3
): Promise<string> {
  let lastError: Error | null = null;

  for (let attempt = 0; attempt < maxRetries; attempt++) {
    try {
      const response = await abv.gateway.chat.completions.create({
        provider: 'openai',
        model: 'gpt-4o-mini',
        messages
      });

      return response.choices[0].message.content;
    } catch (error) {
      lastError = error instanceof Error ? error : new Error(String(error));

      // Don't retry on authentication errors
      if (lastError.message.includes('authentication')) {
        throw lastError;
      }

      // On last attempt, throw the error
      if (attempt === maxRetries - 1) {
        throw lastError;
      }

      // Wait before retrying, with exponential backoff
      const delay = Math.pow(2, attempt) * 1000;
      await new Promise(resolve => setTimeout(resolve, delay));
    }
  }

  throw lastError || new Error('Max retries exceeded');
}

This retry logic handles transient failures like network blips while avoiding endless loops on permanent failures.

Building Conversation Context

Most AI applications involve multi-turn conversations where the model needs context from previous exchanges. Managing this context is a core part of implementing chat applications. The straightforward approach maintains an array of messages that grows with the conversation:

class Conversation {
  private messages: ChatMessage[] = [];

  constructor(systemPrompt: string) {
    this.messages.push({
      role: 'system',
      content: systemPrompt
    });
  }

  async sendMessage(userMessage: string): Promise<string> {
    // Add user message to history
    this.messages.push({
      role: 'user',
      content: userMessage
    });

    // Get response from model
    const response = await abv.gateway.chat.completions.create({
      provider: 'openai',
      model: 'gpt-4o-mini',
      messages: this.messages
    });

    const assistantMessage = response.choices[0].message;

    // Add assistant response to history
    this.messages.push(assistantMessage);

    return assistantMessage.content;
  }

  getHistory(): ChatMessage[] {
    return [...this.messages];  // Return copy to prevent external modification
  }
}

This pattern encapsulates conversation state and ensures the message history stays synchronized. Using it looks like this:

const conversation = new Conversation(
  'You are a helpful assistant that explains programming concepts clearly.'
);

console.log(await conversation.sendMessage('What is a closure in JavaScript?'));
console.log(await conversation.sendMessage('Can you give me an example?'));
console.log(await conversation.sendMessage('How is this different from a regular function?'));

Each message sees the full conversation history, allowing the model to reference earlier exchanges and maintain context. For long-running conversations, you might need to manage the context window size. Here’s a more sophisticated approach that limits history:

class Conversation {
  private messages: ChatMessage[];
  private readonly maxMessages: number;

  constructor(systemPrompt: string, maxMessages: number = 20) {
    this.messages = [{
      role: 'system',
      content: systemPrompt
    }];
    this.maxMessages = maxMessages;
  }

  async sendMessage(userMessage: string): Promise<string> {
    this.messages.push({
      role: 'user',
      content: userMessage
    });

    // Keep only recent messages (but always keep system message)
    if (this.messages.length > this.maxMessages) {
      this.messages = [
        this.messages[0],  // System message
        ...this.messages.slice(-(this.maxMessages - 1))  // Recent messages
      ];
    }

    const response = await abv.gateway.chat.completions.create({
      provider: 'openai',
      model: 'gpt-4o-mini',
      messages: this.messages
    });

    const assistantMessage = response.choices[0].message;
    this.messages.push(assistantMessage);

    return assistantMessage.content;
  }
}

This approach prevents the conversation from exceeding token limits by keeping only recent messages. The tradeoff is that the model loses access to earlier context.

Implementing Function Calling

Function calling lets models invoke functions you define, enabling structured outputs and tool use. While the gateway supports function calling, the implementation details vary by provider. This example shows the general pattern:

interface WeatherParams {
  location: string;
  unit: 'celsius' | 'fahrenheit';
}

async function getWeather(params: WeatherParams): Promise<string> {
  // In a real application, this would call a weather API
  return `The weather in ${params.location} is 72°${params.unit === 'celsius' ? 'C' : 'F'} and sunny`;
}

const tools = [{
  type: 'function' as const,
  function: {
    name: 'get_weather',
    description: 'Get the current weather for a location',
    parameters: {
      type: 'object',
      properties: {
        location: {
          type: 'string',
          description: 'The city and state, e.g. San Francisco, CA'
        },
        unit: {
          type: 'string',
          enum: ['celsius', 'fahrenheit']
        }
      },
      required: ['location']
    }
  }
}];

const response = await abv.gateway.chat.completions.create({
  provider: 'openai',
  model: 'gpt-4o-mini',
  messages: [
    { role: 'user', content: 'What\'s the weather in New York?' }
  ],
  tools
});

// Check if model wants to call a function
const message = response.choices[0].message;
if (message.tool_calls) {
  for (const toolCall of message.tool_calls) {
    if (toolCall.function.name === 'get_weather') {
      const params = JSON.parse(toolCall.function.arguments) as WeatherParams;
      const result = await getWeather(params);
      console.log(result);
    }
  }
}

This pattern defines functions the model can call, parses the model’s function call requests, executes the functions, and returns results to the model.

Framework Integration

The gateway integrates naturally with popular TypeScript frameworks. Here are patterns for common frameworks. For Express.js applications, add an endpoint that handles AI requests:

import express from 'express';
import { abv } from './abv-client';

const app = express();
app.use(express.json());

app.post('/api/chat', async (req, res) => {
  try {
    const { message } = req.body;

    const response = await abv.gateway.chat.completions.create({
      provider: 'openai',
      model: 'gpt-4o-mini',
      messages: [{ role: 'user', content: message }]
    });

    res.json({
      response: response.choices[0].message.content
    });
  } catch (error) {
    console.error('Chat error:', error);
    res.status(500).json({ error: 'Failed to generate response' });
  }
});

For Next.js API routes, the pattern is similar but uses Next.js request handlers:

// app/api/chat/route.ts
import { NextResponse } from 'next/server';
import { abv } from '@/lib/abv-client';

export async function POST(request: Request) {
  try {
    const { message } = await request.json();

    const response = await abv.gateway.chat.completions.create({
      provider: 'openai',
      model: 'gpt-4o-mini',
      messages: [{ role: 'user', content: message }]
    });

    return NextResponse.json({
      response: response.choices[0].message.content
    });
  } catch (error) {
    console.error('Chat error:', error);
    return NextResponse.json(
      { error: 'Failed to generate response' },
      { status: 500 }
    );
  }
}

For Next.js applications that need streaming responses to the browser, use server-sent events:

export async function POST(request: Request) {
  const { message } = await request.json();

  const stream = await abv.gateway.chat.completions.create({
    provider: 'openai',
    model: 'gpt-4o-mini',
    messages: [{ role: 'user', content: message }],
    stream: true
  });

  const encoder = new TextEncoder();
  const readable = new ReadableStream({
    async start(controller) {
      for await (const chunk of stream) {
        const content = chunk.choices[0]?.delta?.content;
        if (content) {
          controller.enqueue(encoder.encode(content));
        }
      }
      controller.close();
    }
  });

  return new Response(readable, {
    headers: { 'Content-Type': 'text/plain; charset=utf-8' }
  });
}

Testing Strategies

Testing code that calls AI models requires different strategies than testing deterministic functions. You can’t assert exact outputs since model responses vary, but you can test the structure of your code and ensure error handling works correctly. Mock the ABV client for unit tests to avoid making actual API calls:

// __tests__/chat-handler.test.ts
import { jest } from '@jest/globals';

// Mock the ABV client
jest.mock('./abv-client', () => ({
  abv: {
    gateway: {
      chat: {
        completions: {
          create: jest.fn()
        }
      }
    }
  }
}));

import { abv } from './abv-client';
import { handleChatRequest } from './chat-handler';

test('handleChatRequest processes messages correctly', async () => {
  const mockResponse = {
    choices: [{
      message: {
        role: 'assistant',
        content: 'This is a test response'
      }
    }]
  };

  (abv.gateway.chat.completions.create as jest.Mock).mockResolvedValue(mockResponse);

  const result = await handleChatRequest('Hello');
  expect(result).toBe('This is a test response');
});

For integration tests where you want to verify actual API behavior, make real requests but structure tests to be flexible about specific outputs:

test('gateway returns valid responses', async () => {
  const response = await abv.gateway.chat.completions.create({
    provider: 'openai',
    model: 'gpt-4o-mini',
    messages: [{ role: 'user', content: 'Say hello' }]
  });

  // Assert structure, not specific content
  expect(response.choices).toHaveLength(1);
  expect(response.choices[0].message.content).toBeTruthy();
  expect(response.choices[0].message.role).toBe('assistant');
  expect(response.usage).toBeDefined();
  expect(response.usage.total_tokens).toBeGreaterThan(0);
});

This approach verifies that the API integration works without depending on specific model outputs.

Next Steps

You now understand how to implement the gateway in TypeScript applications, handle errors, manage conversations, and integrate with frameworks. Here’s where to go next:

Python Guide

Learn how to implement the gateway in Python applications

Available Models

See all supported providers and models with pricing

LLM Gateway Overview

Understand the core concepts and architecture of the gateway

Quickstart

Get up and running with your first gateway request in 5 minutes

Getting Started

Basic Features

LLM Gateway

Guardrails

Evaluations

Prompt Management

Cookbook

SDKs

Platform

Support

Installation and Setup

Client Initialization Patterns

Client Lifecycle and Reuse

Working with Types

Handling Streaming Responses

Error Handling Strategies

Building Conversation Context

Implementing Function Calling

Framework Integration

Testing Strategies

Next Steps

Python Guide

Available Models

LLM Gateway Overview

Quickstart

​Installation and Setup

​Client Initialization Patterns

​Client Lifecycle and Reuse

​Working with Types

​Handling Streaming Responses

​Error Handling Strategies

​Building Conversation Context

​Implementing Function Calling

​Framework Integration

​Testing Strategies

​Next Steps

Python Guide

Available Models

LLM Gateway Overview

Quickstart

Installation and Setup

Client Initialization Patterns

Client Lifecycle and Reuse

Working with Types

Handling Streaming Responses

Error Handling Strategies

Building Conversation Context

Implementing Function Calling

Framework Integration

Testing Strategies

Next Steps