retrieval, without a model
Text in, ranked passages out.
A retrieval pipeline for MoonBit: documents, splitters, a vector store in memory or in SQL, and retrievers that rank. Embeddings arrive through a contract, so nothing here calls a model — and the numbers in the panels below are computed by this library, compiled to JavaScript from the sources it ships.
The text is real input to the splitter; the numbers under each chunk are its code points. Drag the budget down and watch the seams land between words — never inside one, and never inside the fence.
A chunk is the unit
Everything downstream sees chunks: the embedder embeds them, the store files them, the retriever ranks them. What a chunk is decides what the library can ever find, so the splitters are the part worth choosing carefully.
| splitter | cuts at | keeps |
|---|---|---|
| Recursive | paragraph, line, space, character | nothing but the text; overlap carries the seam |
| Token | the budget your counter reports | the promise that a chunk fits the model's window |
| Markdown | headings, then block boundaries | the heading path, the fence, the indentation inside it |
| HTML | heading-delimited sections | the readable text, with h1 … h6 in the metadata |
A section over the budget is cut again; a code block is not. Half a program embeds to nothing useful.
The score is a cosine
Two vectors of the same width, divided by their lengths. That is the whole ranking function, and it is why a long passage and a short one about the same thing compare fairly: length is divided out.
query = (0.9, 0.1, 0.0, 0.1)
passage = (1.0, 0.0, 0.0, 0.0)
dot = 0.9
|query| = 0.911043…
score = 0.9 / 0.911043… = 0.9878783399072131The number on the right is not an illustration: it is what the engine returns for the first passage of the corpus below, and the same arithmetic the package ranks with on every target. Ties keep the order the documents arrived in, which is the only stable answer when several passages score exactly the same.
The same subject, three times
A plain search on a corpus that says one thing several ways fills the answer with that
thing: the harbour at dawn, the harbour at dusk, the harbour wall. MMR reranks the
candidates against each other, trading relevance for coverage: near
lambda = 1 it behaves like a plain search, near 0 like "the
most different thing still related to the question".
| passage | score |
|---|
The score under MMR is the relevance of the passage, not the MMR value that picked it — the number you would quote, not the number that decided the order.
One contract, three places to keep vectors
VectorStore is four methods. Where the distance is computed is the
implementation's business, and it is the only thing that changes between these three:
| store | the distance runs | what a search is |
|---|---|---|
| memory | in this process | score every document, sort, cut |
| sqlite | in this process | read the rows, apply the filter, score, sort |
| pgvector | in the database |
one statement: 1 - (embedding <=> $1), the filter in
WHERE, LIMIT on the server
|
The same standard cases run against all three — add, search with scores, limit, metadata filtering, delete by id, update by id, an empty store, a vector of the wrong width, and the same documents added in a different order.
Five contracts, pick one of each
A pipeline is assembled, not configured: choose a loader, a splitter, an embedder and a store, and the retriever on top of them does not know which you chose.
| contract | the method | implement it when |
|---|---|---|
| Loader | async load() |
documents come from somewhere new |
| Splitter | split(document) |
your corpus needs a chunking rule of its own |
| Embedder | async embed(texts) |
you have a model, hosted or local |
| VectorStore | add, delete, search |
vectors live somewhere this library does not ship |
| Retriever | async retrieve(query, limit) |
a question needs rewriting or reranking before it is answered |