In-browser LLM integration with Vercel AI SDK

If you've been experimenting with creating applications that utilize local language models running directly in the browser using either Transformers.js, WebLLM or the new Prompt API in Chrome/Edge, you're likely familiar with the challenges. These include the need for custom hooks and UI components, the complexity of building robust integration layers to automatically fall back to server-side models when client-side (device) compatablity is an issue, and the significant differences in API specifications across different in-browser LLM frameworks.

Browser AI model providers for Vercel AI SDK

This is why I've recently open-sourced a [TypeScript community provider] that serves as the base [Vercel AI SDK] provider for client side in-browser AI models. It currently provides a model provider for Chrome & Edge's native browser AI models via the [JavaScript Prompt API], as well as a model provider for using open-source in-browser models with [WebLLM].

I am also working hard to include other popular browser-based AI frameworks such as [transformers.js] as a model provider.

It reached [700+ downloads] within the first week of launching and is now also featured on the official [Vercel AI SDK documentation website] as a custom model provider!

Vercel AI SDK

The Vercel AI SDK is a TypeScript toolkit designed to help you build AI-powered applications using popular frameworks like Next.js, React, Svelte, Vue and runtimes like Node.js. It is currently being downloaded 2 million times per week by developers around the globe!

Integrating large language models (LLMs) into applications is complicated and heavily dependent on the specific model provider (OpenAI, Anthropic, Google, etc.) you use. The AI SDK standardizes integrating artificial intelligence (AI) models across providers. This enables developers to focus on building great AI applications and not waste time on techincal details.

The AI SDK offers two main libraries:

• [AI SDK Core]: provides a unified API for interacting with LLMs, enabling functionality such as generating/streaming text, structured objects, tool calling as well as building complex AI agents.
• [AI SDK UI]: offers a set of framework-agnostic (React, Svelte etc.) hooks like useChat, useCompletion for rapidly building interactive chat and generative user interfaces, managing state and more.

You can switch between AI providers by changing a single line of code:

import { generateText } from "ai"import { openai } from "@ai-sdk/openai"const { text } = await generateText({  model: openai("o3-mini"), // change model provider here  prompt: "What is love?"})

With the increasing popularity of running local language models in the browser due to a [range of factors], there has been a missing piece in the ecosystem. While the AI SDK simplifies server-side model integration, there hasn't been a straightforward way to use these on-device models within the same framework. This is the gap that I am trying to fill.

The issue with in-browser LLMs

Community feedback reveals a common pain point: developers find it challenging to integrate in-browser LLMs into their applications.

The main issue is that the ways of using in-browser LLMs are fundamentally different, leading to API fragmentation:

• Transformers.js introduces its own [pipeline] API, support a range of NLP, Computer Vision, Audio and Multimodal tasks by leveraging ONNX Runtime for model execution.
• WebLLM provides an OpenAI-style API for seamless integration and leverages their own [MLCEngine], WebGPU and WebAssembly for model execution.
• The Prompt API (utilizing Gemini Nano and Phi4-mini) offers a native browser integration via the JavaScript LanguageModel namespace where everything is handled by the underlying API. This is probably the easiest one to use, but it is still experimental and not available without having to manually enable it behind feature flags.

Besides these API differences, it's also tricky to easily fall back to server-side models when the client device isn't able to run the local models due to hardware limitation (e.g., insufficient VRAM or unsupported environments) or browser compatablity issues (e.g. WebGPU still not being implemented [in some browsers])

This becomes easy with the AI SDK, enabling you to change model by just changing one line of code! The @browser-ai library extends the AI SDK by providing model providers for these in-browser capabilities. This architecture allows applications to prioritize local inference when browser capabilities and hardware support are detected, and developers can also easily create fallbacks to cloud-based models when local conditions are not met.

Installation

To streamline this process, I've built 2 (so far) npm model provider packages.

The @browser-ai/core package is the AI SDK provider for Chrome and Edge browser's built-in AI models. You can install it with:

npm install @browser-ai/core

The @browser-ai/web-llm package is the AI SDK provider for popular open-source models using the WebLLM inference engine. You can install it with:

npm install @browser-ai/web-llm

Usage

Just as in the AI SDK introduction code above, you can see they import { openai } from "@ai-sdk/openai" and use that in the streamText from the AI SDK.

To use the @browser-ai packages, all you have to do is import them and use them in the same way.

import { streamText } from "ai";import { browserAI } from "@browser-ai/core";const result = streamText({  model: browserAI(),  prompt: "Why is the sky blue?",});

[@browser-ai/core]

The provider will automatically work in all browsers that support the Prompt API since the browser handles model orchestration. For instance, if your client uses Edge, it will use [Phi4-mini], and for Chrome it will use [Gemini Nano].

[@browser-ai/web-llm]

The provider allows using a ton of popular open-source models such as Llama3 and Qwen3. To see a complete list, please refer to the official [WebLLM documentation]

Open source software (OSS)

It's truly inspiring to see the impact of open source, and it was a privilege to be included in the recent Vercel AI SDK tweet, celebrating the AI SDK contributors. Its success, with over 2 million weekly downloads, is a direct result of the collective effort of making AI development easier.

Example applications

If you want to test it out, you can chek out the [repository] for API references and example applications. Make sure to give it a star ⭐ if you feel like it!