From a first answer to a gateway

Install

Add the module once. It brings the contract, the event-stream reader and the wire formats; the subpackages you import decide what else is compiled, so a program that only asks for a completion does not carry the gateway.

moon add moonbitstack/moonmuse

A package is imported by its path under the module. The aliases below are the ones this guide uses:

import {
  "moonbitstack/moonmuse",
  "moonbitstack/moonmuse/openai",
  "moonbitstack/moonmuse/fake",
}

One request

A Request is a model name, the messages, and the knobs that change the answer. A field left out is left out of the wire body rather than sent as a default — the vendors do not agree on defaults, so guessing one would make the same request mean different things.

The scripted provider answers from a list, which is how the rest of this guide runs without a key or a network.

let provider : &@moonmuse.Provider = @fake.Fake::new([
  @moonmuse.Reply::new(
    @moonmuse.Message::assistant(
      "Air scatters blue light more than red, so the sky looks blue.",
    ),
    usage=@moonmuse.Usage::new(18, 14),
  ),
])
let request = @moonmuse.Request::new(
  "gpt-4o-mini",
  [@moonmuse.Message::user("Why is the sky blue?")],
  temperature=0.2,
  max_tokens=256,
)
let reply = provider.chat(request)
println("said: " + reply.text())
println("tokens: " + reply.usage.total().unwrap_or(0).to_string())

A Reply carries the message, a Finish reason, a Usage and the body that came back as raw. The three fields of Usage are optional because an endpoint that reports nothing is not the same as one that reports zero.

A provider per vendor

Each wire format is a package with a client. Only the constructor changes; the request and everything below it stay the same.

base is what points a client at a compatible endpoint instead of the vendor's own: a proxy, a gateway built with this library, a local server that speaks the same JSON. headers adds what an installation needs — a tenant tag, a beta switch — and extra on the request adds a field the shared vocabulary does not model.

let provider : &@moonmuse.Provider = @openai.Client::new(
  api_key="sk-…",
  base="http://127.0.0.1:12000/v1",
  headers=[("x-tenant", "acme")],
)
let request = @moonmuse.Request::new(
  "gpt-4o-mini",
  [@moonmuse.Message::user("Why is the sky blue?")],
  extra={ "reasoning_effort": Json::string("low") },
)

Streaming

stream calls back for each delta as it comes off the socket, then returns the reply those deltas assemble — the same value chat returns. A caller that ignores the callback still gets the whole answer, and the Done delta carries it too.

let provider : &@moonmuse.Provider = @fake.Fake::text(
  "Air scatters blue light more than red, so the sky looks blue.",
)
let request = @moonmuse.Request::new(
  "gpt-4o-mini",
  [@moonmuse.Message::user("Why is the sky blue?")],
)
let pieces = []
let reply = provider.stream(request, delta => {
  match delta {
    @moonmuse.Text(chunk) => pieces.push(chunk)
    @moonmuse.CallStart(name~, ..) => println("call " + name)
    @moonmuse.CallArgs(chunk~, ..) => println(chunk)
    @moonmuse.Done(_) => ()
  }
})
println("pieces: " + pieces.length().to_string())
println("whole answer: " + reply.text())

The four deltas are what every format reduces to: text, a call beginning, the arguments of that call arriving in pieces, and the end. A character split across two reads is held back until its remaining bytes arrive, so a callback never sees half of one.

Tools

A tool is a schema plus a function. Toolbox::call is the only place a call becomes a result, so a framework with its own loop uses it directly and gets the same behaviour — including the rule that a failing tool produces an error result rather than an exception.

let provider = @fake.Fake::new([
  @moonmuse.Reply::new(
    @moonmuse.Message::new(@moonmuse.Assistant, [
      @moonmuse.Part::call(
        "c1",
        "add",
        @moonmuse.parse_json_lossy("{\"a\":2,\"b\":40}"),
      ),
    ]),
    finish=@moonmuse.Calls,
  ),
  @moonmuse.Reply::new(@moonmuse.Message::assistant("The sum is 42.")),
])
let tools = @toolbox.Toolbox::new()
tools.add(@moonmuse.Tool::new("add", "Add two integers"), args => {
  let a = @moonmuse.json_int(args, "a").unwrap_or(0)
  let b = @moonmuse.json_int(args, "b").unwrap_or(0)
  (a + b).to_string()
})
let reply = @toolbox.run(
  provider,
  @moonmuse.Request::new("fake", [@moonmuse.Message::user("What is 2 + 40?")]),
  tools,
)
println("answer: " + reply.text())
println("provider calls: " + provider.hits().to_string())

run owns the loop: it sends each result back as a tool message and stops at limit rounds, which defaults to ten. A tool that raises, or one that was never registered, becomes a result with error set, because the model is the party that has to see it.

Retry, fall back, cache

Every layer is a provider around a provider, so they compose in any order. Retry honours retry-after and only retries what a second attempt could pass; fallback switches on the same classification; the cache key covers every field that decides an answer.

let primary = @fake.Fake::failing(503)
let secondary = @fake.Fake::text("Answered by the second provider.")
let gateway : &@moonmuse.Provider = @layers.cached(
  @layers.fallback([primary, secondary]),
  @layers.Memory::new(limit=16),
)
let request = @moonmuse.Request::new(
  "gpt-4o-mini",
  [@moonmuse.Message::user("Why is the sky blue?")],
)
let first = gateway.chat(request)
let second = gateway.chat(request)
println("answer: " + first.text())
println("primary attempts: " + primary.hits().to_string())
println("secondary attempts: " + secondary.hits().to_string())
println("second answer: " + second.text())

The second call never reached either provider: the answer was in the cache. @layers.routed({"claude-": a, "local/": b}) picks by model-name prefix instead, which is what a gateway over several vendors is.

Structured output

The schema is stated in the prompt and the reply is checked against it; a reply that does not fit goes back with the reason, up to a budget, and the last failure is raised rather than returned as a silent None.

let provider = @fake.Fake::text("{\"celsius\": 21}")
let schema = @moonmuse.parse_json_lossy(
  "{\"type\":\"object\",\"properties\":{\"celsius\":{\"type\":\"number\"}},\"required\":[\"celsius\"]}",
)
let celsius = @structured.ask(
  provider,
  @moonmuse.Request::new("fake", [
    @moonmuse.Message::user("How warm is it in Lisbon?"),
  ]),
  name="weather",
  schema~,
  decode=json => @moonmuse.json_int(json, "celsius"),
)
println("celsius: " + celsius.to_string())

The extractor finds the first balanced object in the answer, so a model that wraps its JSON in a sentence or a code fence still passes. fixes is the number of repair rounds; the default is two.

A gateway

serve returns an ASGI application. List the formats to offer and the endpoints come from the dialects, so a format written once is served, called and parsed by the same code.

let provider = @fake.Fake::text("Hello from the gateway.")
let app = @serve.serve(provider, dialects=@serve.default_dialects())
let client = @testing.Client::new(app)
let response = client.post(
  "/v1/chat/completions",
  body=@utf8.encode(
    "{\"model\":\"demo\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]}",
  ),
)
println("status " + response.status.to_string())
println(response.text())

Pass keys=@serve.StaticKeys::new([...]) to require a credential, and counter= to answer count_tokens. The application runs under any ASGI server, or mounted below a framework.

The same server answers as Anthropic, Gemini and Ollama when those dialects are listed — the endpoint table is on the formats page.

Observation

One wrapper records a client span per call and another records duration and tokens, both under the OpenTelemetry GenAI names. Message bodies stay out of the span unless content=true is passed.

let provider : &@moonmuse.Provider = @fake.Fake::text("Air scatters blue light.")
let recorder = @trace.Recorder::new()
let tracer = @trace.Tracer::new(
  rand=width => Bytes::make(width, b'\x01'),
  sink=recorder,
  sampler=@trace.parent_based(root=@trace.always),
)
let observed = @otel.traced(provider, tracer)
let request = @moonmuse.Request::new(
  "gpt-4o-mini",
  [@moonmuse.Message::user("Why is the sky blue?")],
)
let reply = observed.chat(request)
println("answer: " + reply.text())
println("spans: " + recorder.len().to_string())
for span in recorder.spans() {
  println("span: " + span.name)
}

@otel.metered(provider, registry) adds the token and duration instruments to a Prometheus registry; the names are the ones the convention fixes, so a dashboard matches them for this client as it would for any other.

A local model

An inference engine is a contract, and the text format the model was trained on is data. Nothing here reads weights: an engine turns a prompt into a continuation, and this package turns a request into the prompt and the answer back into text and calls.

struct Echo {
}

impl @local.Engine for Echo with fn generate(_self, prompt, _limit) {
  prompt
}

pub extend Echo with @local.Engine::{generate}

async fn main {
  let engine : &@moonmuse.Provider = @local.Local::new(
    Echo::{},
    template=@local.llama3(),
    calls=@local.Tagged,
    model="qwen2.5-7b",
  )
  let reply = engine.chat(@moonmuse.Request::new(
    "qwen2.5-7b",
    [@moonmuse.Message::user("Why is the sky blue?")],
  ))
  println(reply.text())
}

This engine echoes its prompt, which is why the answer is the template itself. A real one returns a continuation; the rest is unchanged.

An answer holds its calls as JSON inside the text, so @local.parse finds them, removes them from what the user sees, and numbers them:

let parsed = @local.parse(
  "It is 42. <tool_call>{\"name\":\"add\",\"arguments\":{\"a\":2,\"b\":40}}</tool_call>",
)
println("text: " + parsed.text)
for part in parsed.calls {
  match part {
    @moonmuse.Part::Call(id~, name~, args~) =>
      println("call " + id + ": " + name + " " + args.stringify())
    _ => ()
  }
}

@local.Tagged is the Qwen and Hermes format; Fenced and Bare cover the other two ways a model writes a call.

Targets

The contract, the dialects, the layers, the toolbox and the gateway build for native, wasm, wasm-gc and js. The clients and the streaming transport need a socket, which the HTTP library provides on native and wasm only; a program on the other two targets still has every wire format, which is what makes the four-pane page on this site possible.

Where to go next: the formats page for the field-by-field view, the API reference for every public name, and the repository for the examples under src/examples.