2. Embedding and reranking stack: local open-source models

  • Status: Accepted

  • Date: 2026-06-13

Context

Claude has no embeddings endpoint, so the retrieval stack is an independent vendor/tech choice with two distinct stages:

  • Embedding — turns each chunk and each query into a vector stored in pgvector; powers the dense half of hybrid search.

  • Reranking — a slower cross-encoder that re-scores a candidate set (e.g. top-50) and reorders it to the final top-k. It reads query and document together, catching relevance that bi-encoder embeddings miss. This is the “not naive top-k” requirement and the single highest-leverage quality lever in the pipeline.

Three stacks were considered:

  1. Voyage AI (voyage-3 + rerank-2.5) — Anthropic’s recommended pairing. Best quality with least tuning, trivial infra, but hosted: requires an API key, network access, and chunks transit to a vendor.

  2. Local open-source (BGE embeddings + a BGE cross-encoder reranker) — runs in-process. No per-call cost, fully offline and private, but heavier containers and slower CPU reranking, and the eval/model-selection work is owned in-house.

  3. OpenAI + Cohere — solid hosted option, but spans three vendors and three keys to do what Voyage does in one.

At this project’s scale (a personal course corpus, ~10K–50K chunks) cost is negligible for every option and all three clear the quality bar for course Q&A. The deciding axis is therefore offline/privacy and learning value vs. operational simplicity, not cost or quality.

Decision

Adopt the local open-source stack: bge-m3 for embeddings and bge-reranker-v2-m3 (cross-encoder) for reranking, both running in-process within rag_core.

This is driven by the explicit goals of no per-call cost, keeping everything on-machine, and learning the RAG stack end-to-end (model loading, the eval loop, CPU/MPS performance).

Consequences

  • No per-call cost, fully on-machine, private, and works offline.

  • Container weight: only the component that loads the bge models carries PyTorch + weights (a multi-GB image). Model loading is isolated to the embed/ and rerank/ modules — no torch import at module top-level elsewhere — and weights are cached in a named Docker volume rather than baked into images, so the download happens once.

  • Reranking is the slow step on CPU: a cross-encoder scoring ~50 candidates per query is the cost of “not naive top-k.” It uses MPS on macOS and CPU in Linux containers/CI. The rerank candidate count is configurable to trade latency for quality, and model loads stay out of the request path.

  • We own the eval loop — a golden-set harness (labeled query → expected-doc) introduced in the embeddings phase — to validate retrieval quality and demonstrate that reranking improves it.

  • The embedding model’s output dimension is baked into the pgvector vector(N) column and its index. Changing embedders later means a re-embed plus a schema migration; this is the main lock-in and the reason the choice is fixed early.

  • Query and document vectors must come from the same model, so the embedder is pinned in config, not chosen per call.