guide

One pipeline, end to end

From an empty project to a ranked answer. Every output below is what the code printed when it ran, not a sketch of what it would print.

import {
  "moonbitstack/moonrag",           // documents, contracts, memory store, retrieval
  "moonbitstack/moonrag/split",     // recursive, token, Markdown and HTML splitters
  "moonbitstack/moonrag/load",      // text, CSV, JSON, Markdown, HTML loaders
  "moonbitstack/moonrag/index",     // incremental indexing
}

A loader, a splitter, an embedder and a store are the four pieces a pipeline is assembled from. moonbitlang/async is a dependency of anything that awaits a store or an embedder, so add it too if your project does not have it.

Load a CSV, split it, index it, and ask a question. The indexer is what makes the second run cheap; it is covered in step five.

async fn ingest(embedder : &@moonrag.Embedder, csv : String) -> Array[@moonrag.Scored] raise {
  let store = @moonrag.InMemoryVectorStore::new(embedder)
  let indexer = @index.Indexer::new(store)

  let documents = @moonrag.Loader::load(@load.CsvLoader::new("people.csv", csv))
  let splitter = @split.RecursiveSplitter::new(chunk_size=500, chunk_overlap=50)
  let pieces : Array[@moonrag.Document] = []
  for document in documents {
    for chunk in splitter.split(document) {
      pieces.push(chunk)
    }
  }
  let _ = indexer.index(pieces)

  let retriever : &@moonrag.Retriever = @moonrag.SimilarityRetriever::new(embedder, store)
  retriever.retrieve("who lives in London", limit=3)
}

The embedder is yours to write. HashingEmbedder is a deterministic stand-in that needs no network — good for tests and for getting the shape of a pipeline before a model is involved. With the examples' corpus and that embedder, the equivalent program prints:

documents to index: 3
indexed: added=3 updated=0 skipped=0

query: how do I add the package
  0.4644046100019301  [guide.html] Install
Add the package with moon add moonbitstack/moonrag.
  0.05634361698190111  [guide.html > Search] Search
Build a store, add documents, then search with a query vector.

query: cosine and magnitude
  0.366146046676159  [notes.md > Field notes] # Field notes

Cosine similarity ignores magnitude, so a long passage and a short one about the same subject compare fairly.
  -0.030556616567607043  [guide.html > Search] Search
Build a store, add documents, then search with a query vector.

second pass: added=0 skipped=3

Two things to read in that output: the score is a cosine, so it can be negative, and a retrieved chunk carries the heading path it came from — [guide.html > Search] is the metadata, not the text.

Chunking decides what the library can find. Four rules, one per splitter: paragraphs with an overlap, a token budget you supply, headings with a path, or headings in HTML. For Markdown, the syntax tree is read, so a # inside a fence is a comment and a code block is never cut.

let markdown = "# Install\n\nmoon add moonbitstack/moonrag.\n\n## Search\n\nBuild a store, then search.\n"
let splitter = @split.MarkdownSplitter::new(chunk_size=1000)
let chunks = splitter.split(@moonrag.Document::new(markdown))
assert_eq(chunks.length(), 2)
assert_eq(chunks[1].str_meta("h1"), Some("Install"))
assert_eq(chunks[1].str_meta("h2"), Some("Search"))

The heading text lands in the metadata under the key you chose, and every chunk carries the whole path down to it. A level you do not list does not split — it stays in the text of the section above:

let splitter = @split.MarkdownSplitter::new(headings=[
  @split.MarkdownHeading::new(1, "h1"),
  @split.MarkdownHeading::new(2, "h2"),
])

Defaults: all six levels split, headings are stripped from the chunk, overlap is zero.

InMemoryVectorStore is the reference implementation: documents in insertion order, every candidate scored on every search, filters applied before ranking. When the corpus has to outlive the process, the same contract is a table — and the database is chosen once, when the connection is opened.

pub async fn[T : @moondb.AsyncDriver] search_with(
  driver : T,
  embedder : &@moonrag.Embedder,
) -> Array[@moonrag.Scored] raise {
  let store = @sql.SqlVectorStore::new(driver, embedder, dimension=256)
  store.create()
  let _ = store.add([@moonrag.Document::new("the harbour at dawn")])
  let query = embedder.embed(["the harbour"])[0]
  store.search(query[:], limit=3)
}

dimension is the width of the embedding model, fixed at construction because the column is declared with it. On PostgreSQL with pgvector the search is one statement; on SQLite the rows come back and the cosine is computed in this process. A SQLite driver that blocks goes in through BlockingDriver:

let driver = @sql.BlockingDriver::new(@moonsqlite.SqliteDriver::open("app.db"))

This last line needs moon add moonbitstack/moonsqlite; the library itself depends on no driver.

Embedding is the expensive part, so a second run over an unchanged corpus must embed nothing. The indexer keeps a content hash per stored id:

let indexer = @index.Indexer::new(store)
let stats = indexer.index(chunks)                        // added / updated / skipped / deleted
let cleaned = indexer.index(chunks, cleanup=@index.Cleanup::Full)

unchanged

same id, same content hash: skipped, nothing embedded.

changed

same id, different text or metadata: the old copy is deleted, the new one added.

gone

Incremental deletes documents whose source appears in this run and whose id does not; Full deletes everything this run did not produce; None deletes nothing.

The source key defaults to source, the metadata key every loader writes. Moving a file changes the source but not the hash, so a moved file is not re-embedded.