RAG

@absolutejs/ragv0.22.2betaAI

Document retrieval, web research, durable workflows and owned web indexes for Bun and Elysia apps.

A standalone RAG runtime for Bun and Elysia apps covering the full pipeline: document ingestion and chunking, embedding, hybrid retrieval with reranking, source sync, and retrieval-quality evaluation. Vector storage is pluggable behind a single RAGVectorStore contract, with published adapters for Postgres (pgvector), SQLite, and Pinecone alongside a built-in in-memory store. It pairs with @absolutejs/ai for models and @absolutejs/search for web evidence. Web research and owned-index workflows share authorized plugins with React, Vue, Svelte, Angular, HTML and HTMX bindings.

#Installation

BASH

bun add @absolutejs/rag

#Capabilities

Overview

A standalone RAG runtime for Bun and Elysia applications covering document ingestion, chunking, embeddings, hybrid retrieval, reranking, source synchronization, evaluation, client primitives, and framework bindings.

Retrieval and storage

The built-in memory store supports development and tests. Published adapters provide PostgreSQL with pgvector, SQLite with optional vec0 acceleration, and Pinecone behind the same RAGVectorStore contract. Lexical and vector results can be fused, transformed, and reranked with provider or heuristic rerankers.

Retrieval channel requirements

Lexical and hybrid retrieval require a store implementing queryLexical. A backend without that capability raises an actionable error before vector embedding/search instead of silently returning vector-only or empty results. Explicit vector retrieval remains supported on vector-only backends. Configure a lexical-capable adapter before requesting hybrid retrieval; a mode flag does not add a missing backend capability.

Show 1 more

The built-in heuristic strategy preserves the requested retrieval mode when a query is scoped by source or document ID. Scope narrows the searchable corpus; it does not remove the need for exact keyword matches within that corpus.

Ingestion and source sync

The ingestion pipeline handles files, directories, uploads, URLs, PDFs, office documents, archives, images, and media transcripts. Scheduled connectors can keep collections synchronized from email, GitHub, sitemaps, feeds, directories, and S3-compatible storage.

Feed, sitemap and site-discovery sync sources use the bounded, DNS-pinned public web transport through final document ingestion. Private addresses and unsafe redirect destinations are rejected. Site discovery identifies itself as AbsoluteJSReader/1.0, respects agent-specific robots rules, Allow, wildcards and query paths, and checks each page redirect against its destination's policy. A robots 4xx response permits crawling except 429; network errors, 429 and 5xx abort the sync before reconciliation, preserving previously indexed documents. Robots policies are cached only within one sync run. A trusted host can inject fetchResource for isolated readers or deterministic tests; that adapter must preserve DNS pinning, response bounds and redirect validation. Direct URL and GitHub sync behavior is unchanged. Durable scheduling and corpus-wide crawl budgets remain separate from these source-level protections.

PDF uploads use the built-in PDF.js reader to decode compressed streams and font encodings, with page numbers retained in native text blocks for citations. The reader processes pages sequentially and does not impose an application intake page count or character limit. Upload/storage budgets and model context budgets belong to their respective layers; RAG chunks the extracted text for retrieval rather than sending the whole PDF to a model.

Show 1 more

Scanned or image-only PDFs need an explicitly configured OCR provider through createRAGPDFOCRExtractor({ provider }). Empty text layers fail clearly in the default extractor. Invalid or password-protected documents propagate reader errors; they are not indexed from raw PDF operators. The PDF worker is included in the server bundle, including compiled Bun applications.

Quality and evaluation

@absolutejs/rag/quality evaluates retrieval relevance and answer grounding, compares strategies and rerankers, and records runs against a baseline so retrieval changes can be tested before release.

Client and framework entry points

@absolutejs/rag/client and /client/ui provide browser-side primitives.

@absolutejs/rag/react, /vue, /svelte, and /angular provide framework bindings.

@absolutejs/rag/adapter-kit exposes the contracts used by vector-store adapters.

Show 10 more

@absolutejs/rag/ui exposes presentation-neutral UI contracts.

Pair the retrieval runtime with @absolutejs/ai when retrieved context should feed a model or streaming assistant.

Verbatim original text evidence

Use chunkRAGOriginalText({ sourceId, version, text }, options) when citations must resolve against an immutable text original. This opt-in path preserves whitespace and Unicode instead of normalizing or extracting document formats. Each chunk includes metadata.sourceLocator with the source ID, immutable version and UTF-16 start/end offsets. readRAGOriginalText validates identity, version and range before returning the exact original slice. Store and authorize the original separately; a locator is not an access grant.

createRAGOriginalTextTools({ collection, filter, loadSource, budget }) provides search_text_source and read_text_source AI tools. It requests real hybrid retrieval with diversity and verifies evidence against originals. filter is a server-owned scope; loadSource(id, version) must reauthorize every read and return null for inaccessible/deleted versions. The tools require budget: { maxTokens, countTokens } using the model tokenizer and a budget reserved by the AI context policy. Whole passages are selected within that budget; omitted passages are flagged rather than silently truncated. Add the final tools/instructions before budgeting the model request. Search traces are available through onTrace; they contain retrieval metadata and should not be copied wholesale into public logs. Servers may set searchTopK to an integer from 1 to 48 (default 12) for a smaller initial evidence lookup. Candidate ranking still considers up to 48 matches; authorization and token-budget checks are unchanged. A small initial result set does not establish that other facts are absent: retain broader search/read tools for missing or ambiguous evidence.

Keyword matching uses Unicode word segmentation and canonical normalization. English suffix rules only apply to ASCII words. This improves multilingual exact matches; it does not replace evaluation of the selected embedding model.

AI context policy compatibility

With the AI 0.1 context-policy release, RAG chat validates the final assembled retrieval context before every model request. Its contextPolicy config is forwarded to both WebSocket and SSE generation. Use a working token target or a saved-source recovery callback when appropriate. Providers without capacity support require an explicit contextPolicy: false raw opt-out. Older supported AI peers retain their previous behavior; upgrading RAG alone does not add the AI 0.1 capacity policy. No model-capacity numbers are defined in RAG.

Reversible quote references

createRAGQuoteReferences() creates a request-scoped registry for compressing already-verified original-text tool results. encodeToolResult(json) replaces passage text with sentence entries { citation, text }; resolve(citation) restores the exact registered sentence. Repeated sentences reuse a reference, and unknown references throw. Search and single-range read envelopes are supported; unrecognized tool results pass through unchanged.

Web research workflows

Bounded search and reading, schema extraction, field evidence review, company discovery, resumable batches and durable monitoring. Follow the Web Research Workflows guide for server configuration and all six presentation frameworks.

Owned web indexes

Scoped public-site crawling, durable coordination, versioned evidence, PostgreSQL retrieval, takedowns and model-generation lifecycle. The index exposes a SearchProvider for the same research runtime.

Pluggable vector stores

One RAGVectorStore contract with an in-memory store built in and Postgres, SQLite, and Pinecone adapters published separately, so swapping backends does not touch retrieval code.

Provider-agnostic embeddings

Embedding providers for OpenAI, Gemini, Mistral, Ollama, xAI, DeepSeek, and more, plus any OpenAI-compatible endpoint, behind a single provider interface.

Hybrid retrieval and reranking

Lexical and vector results fuse into hybrid search, with query transforms and rerankers (Cohere, Jina, Voyage, or heuristic) applied per collection.

File ingestion pipeline

Extractors for PDFs (including OCR), EPUB, office and legacy documents, archives, images, and media transcripts turn raw files, directories, uploads, and URLs into chunked documents.

Source sync and scheduling

Sync sources for email (Gmail, Microsoft Graph, IMAP), GitHub repos, sitemaps, feeds, directories, and S3-compatible storage keep collections current on a scheduler.

Retrieval evaluation suites

Evaluation suites score retrieval and answer grounding, compare strategies and rerankers, and persist run history so retrieval changes ship against a baseline.

Outcomes

What you can build

Build complete retrieval pipelines

Ingest files, URLs, office documents, archives, images, and media transcripts; synchronize durable sources; and search with lexical, vector, hybrid, transformed, and reranked retrieval.

Choose storage without changing the app

Swap memory, PostgreSQL/pgvector, SQLite/vec0, and Pinecone stores behind one contract and expose retrieval through framework-neutral or framework-specific clients.

Hardening checklist

Production guidance

Treat retrieval quality as a release gatePersist source checkpoints, make ingestion idempotent, pin embedding dimensions, test retrieval relevance and grounding, compare rerankers, and block releases that regress the evaluation baseline.

Follow in order

Troubleshooting path

Retrieval quality regressed

Separate ingestion, chunking, embedding, storage, lexical retrieval, vector retrieval, fusion, reranking, and answer grounding. Evaluate each stage against a retained baseline.

#Quick start

Partial snippet

Working example for Quick start.

import {
	createInMemoryRAGStore,
	createRAGCollection,
	ingestRAGDocuments,
	openaiEmbeddings,
	searchDocuments
} from '@absolutejs/rag';

const collection = createRAGCollection({
	embedding: openaiEmbeddings({
		apiKey: process.env.OPENAI_API_KEY ?? '',
		defaultModel: 'text-embedding-3-small'
	}),
	store: createInMemoryRAGStore()
});

await ingestRAGDocuments(collection, {
	documents: [{ id: 'intro', text: 'AbsoluteJS ships typed Bun primitives.' }]
});

const results = await searchDocuments(collection, {
	query: 'What does AbsoluteJS ship?',
	topK: 3
});

#Public websites and JavaScript rendering

Partial snippet

loadRAGDocumentFromURL loads a document; use prepareRAGDocument(doc).normalizedText for readable text. URL loading now honors response MIME types on extensionless URLs. For public websites, @absolutejs/rag/web provides readRAGWebpage with bounded responses, timeouts, prepared text, final URL, title, truncation and per-attempt retrieval diagnostics. It tries static HTML first and requests a browser for thin or empty application shells. A missing renderer returns rendering_required, not a claim that the website contains no information.

import { readRAGWebpage } from '@absolutejs/rag/web';
import { createPlaywrightWebRenderer } from '@absolutejs/rag/web/playwright';

const browser = createPlaywrightWebRenderer();
try {
  const page = await readRAGWebpage({
    url: 'https://example.com',
    render: browser.render,
  });
  // Check page.status and page.error before treating page.text as complete evidence.
} finally {
  await browser.close();
}

#Quick Start

Partial snippet

Create a collection over the built-in in-memory store, ingest documents, and run a similarity search.

import {
	createInMemoryRAGStore,
	createRAGCollection,
	ingestRAGDocuments,
	openaiEmbeddings,
	searchDocuments
} from '@absolutejs/rag';

const collection = createRAGCollection({
	embedding: openaiEmbeddings({
		apiKey: process.env.OPENAI_API_KEY ?? '',
		defaultModel: 'text-embedding-3-small'
	}),
	store: createInMemoryRAGStore()
});

await ingestRAGDocuments(collection, {
	documents: [
		{
			id: 'getting-started',
			text: 'AbsoluteJS ships full-stack primitives for Bun and Elysia.',
			title: 'Getting Started'
		}
	]
});

const results = await searchDocuments(collection, {
	query: 'What does AbsoluteJS ship?',
	topK: 3
});

#Postgres (pgvector) Backend

Partial snippet

Swap the in-memory store for pgvector by installing @absolutejs/rag-postgres; the collection API stays identical.

import { createPostgresRAG } from '@absolutejs/rag-postgres';

const { collection, store } = createPostgresRAG({
	storeOptions: {
		connectionString: process.env.DATABASE_URL,
		dimensions: 1536,
		distanceMetric: 'cosine',
		indexType: 'hnsw'
	}
});

#Vector store adapters

Published vector-store adapters that implement the RAGVectorStore contract from @absolutejs/rag. Each ships as its own package, so you install only the backend (and its heavy dependencies) you actually use.

@absolutejs/rag-pineconev0.0.13Pinecone vector-store adapter for @absolutejs/rag

@absolutejs/rag-postgresv0.1.0PostgreSQL (pgvector) vector-store adapter for @absolutejs/rag

@absolutejs/rag-sqlitev0.0.13SQLite vector-store adapter for @absolutejs/rag with optional native vec0 acceleration

Beta API

The package is pre-1.0 and its API surface is still settling; pin an exact version and review release notes when upgrading.

Subpath exports

Framework bindings live behind subpath exports (@absolutejs/rag/react, /vue, /svelte, /angular), and the adapter-kit subpath exposes the contract adapters build against.

Web research workflows Owned web indexes Search provider npm

Current package surface

What ships today

@absolutejs/ragv0.22.2 · betaAInpm Source

18entry points

Import surface · click to copy

PreviousAI Runtime · Streaming & Protocol NextIngestion & Chunking