Reference
Every package, in order
Generated from each package's interface file and the comments above its declarations: 468 public declarations across 17 packages. A type is followed by its methods, because that is where a reader looks for them.
The contract
The request, the reply, the two traits, and the event stream reader.
moonmuse
moonbitstack/moonmuseThe shared vocabulary: messages, requests, replies, the two contracts, and the server-sent-events reader every streaming client needs.
How many tokens a request costs, before anyone answers it. A tokenizer is a large table that most callers do not need, so the contract lives here and the implementation that loads a vocabulary lives in moonmuse/count: a gateway is handed a counter, and a deployment that has no vocabulary says so instead of inventing a number.
One vendor's wire format, read and written in the same place. The client half turns a Request into a Call and a body into a Reply; the server half turns a body into a Request and a Reply into a body. Because both halves live here, a gateway that speaks a format out cannot drift from the client that speaks it in.
The one contract every model endpoint implements: a remote API, a local server, a scripted fake, or a layer wrapped around another provider. Swapping providers is a change to the constructor line, not to the calling code.
Assembles deltas into the reply they describe. Text accumulates in arrival order, argument fragments accumulate per call, and the Done reply wins when present.
Whether the stream already closed itself with a Done reply.
An empty assembly.
Feed one delta in.
The assembled reply: the Done one when the stream sent it, otherwise one built from what accumulated.
One HTTP request a dialect produced from a Request — verb, path, headers, body. A client only has to send it; nothing in it is provider-specific beyond what the dialect already decided.
A call whose body is bytes rather than JSON: speech, a file, a form.
A copy of the body with one field added or replaced.
A copy with a header set, replacing any previous value of the same name.
A call with the provider's JSON content type and no extra header.
A copy with a query parameter appended.
A copy that asks the endpoint to stream: the same body with stream set true. Every format this library speaks spells that field the same way, and a streamed request is otherwise the request it would have been.
One frame of a streamed response: the server-sent-events event name when the endpoint uses named events, and the payload. JSON endpoints that stream newline delimited objects leave event empty.
The frame's payload as JSON, or Decode when it is not JSON.
A named frame.
A frame with no event name.
A message: a role and its ordered parts.
An assistant message carrying one text part.
The tool calls this message requests, in order. Empty on a plain text reply.
Whether this message asks for a tool.
A message built from a role and its parts.
A copy with part appended. The tool loop grows the conversation this way.
The tool results this message carries, in order.
A system message carrying one text part.
The text parts joined by separator. What a caller shows a human, and what a provider without structured content receives.
A tool message carrying one result, addressed to the call id.
A user message carrying one text part.
What a provider says about one model. Prices are US dollars per million tokens, because that is the unit every vendor publishes and the one a cost estimate wants.
A model with everything unknown except its id.
Whether the model declares capability.
A model's answer: the assistant message, why it stopped, what it cost, and the response body as it arrived, for fields the shared vocabulary does not model.
The reply's tool calls.
A reply from an assistant message.
The reply's text.
One request, in the vocabulary every provider shares. extra is merged into the request body last of all, so a field this library does not model — a vendor's beta switch, a reasoning budget — still reaches the wire.
A request with the provider's defaults.
A copy with one more message — the tool loop appends the assistant's call and the tool's result this way.
A copy with the messages replaced.
One endpoint a dialect serves. path may contain a single *, which matches any run of characters — Gemini puts the model inside the path.
Whether this route answers verb at path.
A route for a verb and a path pattern.
A server-sent-events reader that takes the stream in whatever pieces the network delivers. A frame's fields may be split across two reads, and one read may carry several frames; the reader only dispatches at the blank line that ends a frame, so the boundaries of the transport never leak into the dialect.
Take one piece of the stream and return the frames it completed.
Dispatch whatever a truncated stream left behind. A server that closes without the final blank line has still sent a complete frame.
Whether a frame is in progress.
A reader with nothing buffered.
A tool the model may call: a name, a description, and a JSON Schema object for the arguments.
A tool with an empty object schema, for a tool that takes no arguments.
State carried across the frames of one streamed answer. Vendors number content blocks differently — OpenAI by tool-call index, Anthropic by block ordinal, Gemini by part ordinal — so the dialect keeps the counters here instead of in the client, and the same Turn serves the client reading a stream and the server writing one.
Add to a numeric scratch value and return the new total.
Append to a scratch value.
Whether a scratch value was recorded.
A fresh stream state.
Take the next content-block ordinal.
A numeric scratch value, or zero.
Set a numeric scratch value.
Record a scratch value.
A scratch value, or the empty string.
Count one frame and return its ordinal.
What a call cost. None means the provider reported nothing; a zero would claim the call was free.
Usage with both counts known and no cache report.
Input plus output, when both were reported.
Usage with nothing reported.
What a server-side request asked for, decoded into the shared vocabulary.
A capability a model may declare. The catalogue in moonmuse/models uses these to describe what an endpoint can actually do, so a caller can pick a model for a job instead of guessing from its name.
One event of a streamed answer. Done closes it with the assembled reply, so a caller that ignores the deltas still gets the same value as a non-streamed call.
Why generation stopped, in the four cases worth branching on plus a catch-all for a vendor's own reason string.
The Finish a vendor's reason string names. Shared by the four dialects, which differ in spelling but not in meaning.
One typed piece of a message. A message is an ordered list of these, so text and a tool call can share one assistant turn — which is what every provider returns when a model calls a tool and explains itself in the same breath.
A tool call part.
A tool result part.
A text part.
The text this part carries: a text part's own text, a result's content, and nothing for a call or an image.
Who authored a message. A tool exchange is a role of its own here rather than a part of a user message, because OpenAI, Anthropic, Gemini and Ollama all put it in a distinct place on the wire.
The lowercase name a role goes by on the wire, using the OpenAI spelling.
The kinds of endpoint a dialect can both call and serve.
How a request should use tools.
Everything a provider call can fail with. The four cases are the only distinctions a caller can act on: retry, parse differently, or stop asking.
A one-line description, for logs and for retry classification.
Whether asking again could plausibly succeed. A rate limit, a server fault and a dropped stream qualify; a malformed body or a missing capability does not.
The wait the endpoint asked for, when it named one.
The embeddings of a batch, checked against the batch length. A short answer is a broken endpoint, not a partial success, because the caller indexes vectors by input position.
Encode frames as a server-sent-events body, with the event name when there is one.
The array at key, or an empty one.
The value of a JSON boolean.
The value of a JSON number.
The text of a JSON string.
The boolean at key, when it is a boolean.
The member at key, when this is an object that has it.
The integer at key, when it is a number.
The number at key, when it is a number.
The numbers in an array value.
The object at key, or None.
The string at key, when it is a string.
The string at key, or fallback.
The strings in an array value, skipping anything that is not a string.
base with overrides merged over it, key by key. extra reaches the wire this way: a caller's field wins over the one this library wrote.
A JSON array of the numbers in values.
A JSON object from key/value pairs, in order.
An integer from a decimal string, or zero. Used where a stream carries an index in a text field; a vendor that sends something else should not crash the reader.
A JSON document from a string, or the empty object when the string is not JSON. A streamed tool call whose arguments never complete is left as {} so the loop can report the tool's own error instead of failing the whole turn.
Read a whole newline-delimited JSON body — Ollama's streaming shape and the fallback for endpoints that stream bare objects.
Read a whole server-sent-events body at once.
A JSON array of the strings in items.
Wire formats
One package per specification, each with a client and the reader and writer for its JSON.
openai
moonbitstack/moonmuse/openaiThe OpenAI wire format and its client. The dialect is pure and builds everywhere; the client files are limited to the backends moonfetch has transports for, which is why this package turns off the unused-import warning on the other two.
One batch job.
A connection to an OpenAI-compatible endpoint.
Implements @moonmuse.Provider
A copy pointed at another endpoint, keeping the credentials — how a caller reaches a second deployment of the same service.
One batch job, by id.
The batch jobs this account has run.
Delete a file.
One file, by id.
The files this account has uploaded.
One fine-tuning job, by id.
The fine-tuning jobs this account has run.
Generate images from a prompt.
Check text against the moderation model. A result with nothing flagged still carries the scores of the categories that were checked.
A client for base. api_key is left out for a local server that wants none; headers adds what a gateway needs, from an organisation id to a beta switch.
Answer through the Responses API. Its request shape differs from chat — input instead of messages, instructions for the system prompt — so the body is built here and the reply is read from output.
Synthesise speech. The answer is audio bytes, not JSON, so it comes back as bytes.
Start a batch job over an uploaded file.
A streamed answer, forwarded as the endpoint produces it. The same Turn the gateway writes with reads the stream here, so the deltas a client sees are the ones the frames carried, not a replay of the finished answer.
Transcribe an audio file. The file travels as a form, which is the only shape this endpoint accepts; language narrows the decoder when the audio is known.
Upload a file for a batch or a fine-tuning job.
One uploaded file.
One fine-tuning job.
One generated image: a URL, or base64 when the endpoint was asked for it.
What a moderation call found.
The OpenAI wire format.
Implements @moonmuse.Dialect
The fields the description gives CreateChatCompletionResponse.
The fields the description gives ChatCompletionResponseMessage.
The fields the description gives CreateChatCompletionRequest.
The fields the description gives CreateChatCompletionResponse.
The fields the description gives ChatCompletionTool.
The fields the description gives ChatCompletionRequestAssistantMessage.
The fields the description gives FunctionObject.
The fields the description gives ChatCompletionRequestSystemMessage.
The fields the description gives ChatCompletionMessageToolCall.
The fields the description gives ChatCompletionRequestToolMessage.
The fields the description gives ChatCompletionRequestUserMessage.
The one instance; it carries no state, so a dialect can be a value.
The fields the description gives Embedding.
The fields the description gives CreateEmbeddingRequest.
The fields the description gives CreateEmbeddingResponse.
The fields the description gives CreateImageRequest.
The fields the description gives ImagesResponse.
The fields the description gives CreateModerationRequest.
The fields the description gives CreateModerationResponse.
The keys request writes. parse treats every other key as a caller's own and keeps it in extra, so a request that arrives at a gateway leaves with the same fields it came with.
The fields the description gives CreateResponse.
The fields the description gives CreateSpeechRequest.
The fields the description gives CreateTranscriptionRequest.
One tool_calls entry as a call part.
The chat-completions body that answers reply.
The chat-completions call for request, with extra merged in last.
The request a chat-completions body asks for, with the fields this library does not model kept in extra.
The reply a chat-completions body describes. A non-success status becomes an LlmError rather than a reply with an empty message.
The body of /v1/completions.
The content of a message as OpenAI writes it: a plain string when it is all text, an array of typed parts when it carries images.
The embeddings body, the shape OpenAI's /v1/embeddings returns.
The error body OpenAI sends, for a status the client turned into an LlmError.
The finish reason as the wire spells it.
Whether a body's stream field asks for a streamed answer.
One message's wire form. A tool message expands to one message per result, because OpenAI pairs each result with exactly one call.
The messages a wire message describes. An assistant message that called tools and a tool message that answers them both map back to one shared message.
The model list body, listing what models reported.
Tool arguments arrive as a JSON string inside the JSON body, and a model can send a string that is not JSON. That is the tool's problem to report, not the transport's to crash on, so the unparsable case becomes an empty object.
The reply the accumulated chunks describe.
The body of /v1/responses.
The deltas one chunk carries. The last chunk closes the stream with Done, so a caller that only reads deltas still gets the final reply.
The frames that carry one delta out, in the shape a chat-completions stream uses.
How the request allows tools.
A tool as OpenAI writes it.
The usage object, from what the endpoint reported. A field it did not report stays absent rather than becoming a zero.
The usage an OpenAI body reports.
anthropic
moonbitstack/moonmuse/anthropicAnthropic's Messages API: the system prompt is a top-level field, max_tokens is required, tool results ride in user messages as content blocks, and usage names its two counts separately. The dialect keeps all of that in one place, read and written.
A connection to the Messages API.
Implements @moonmuse.Provider
Count the tokens a request would cost, without asking the model to answer it.
The Messages API has no embeddings endpoint.
A client for base, authenticating with api_key. version is the API version the endpoint expects; the default is the one this dialect implements.
A streamed answer, forwarded as the endpoint produces it.
The Anthropic wire format.
Implements @moonmuse.Dialect
The one instance; it carries no state.
The keys request writes; anything else in a served body is kept as extra.
The version header Anthropic requires. A gateway serving this format sends it too.
The error body Anthropic sends.
The Messages body that answers reply.
The Messages API call for request.
The request a Messages body asks for.
The reply a Messages body describes.
The messages a body message describes. A user message of tool_result blocks becomes tool messages; anything else keeps its role and blocks.
The model list in the shape Anthropic's console returns.
The reply the folded events describe.
The stop reason as Anthropic spells it.
The deltas one event carries.
The events that carry one delta out.
A tool as Anthropic writes it.
The usage an Anthropic body reports. Its two counts are named, and a cached read is reported apart from the input count.
gemini
moonbitstack/moonmuse/geminiGemini's generateContent shape: the model sits in the path, messages are contents with user and model roles, and a tool call has no id of its own — one is synthesized here so the shared vocabulary can pair calls with results.
A connection to the Gemini API.
Implements @moonmuse.Provider
Vectors for a batch, through batchEmbedContents. One call carries every input, which is what Gemini's batch endpoint is for.
A client for base, authenticating with api_key.
A streamed answer, forwarded as the endpoint produces it. Gemini streams from its own path and asks for events with alt=sse, so the call differs by more than a flag.
The Gemini wire format.
Implements @moonmuse.Dialect
The one instance; it carries no state.
The embeddings body for one input, in Gemini's shape.
The error body Gemini sends.
The finish reason as Gemini spells it.
The body that answers reply. The candidate count and model version are the fields Gemini always writes, so they are written here too.
The generateContent call for request.
The request a generateContent body asks for.
The reply a generateContent body describes.
The model named in a Gemini path, with the /models/ prefix and the method suffix removed.
The model list as Gemini's API returns it.
The reply the accumulated chunks describe.
The deltas one chunk carries.
The frames that carry one delta out, one chunk per event.
The usage a Gemini body reports.
ollama
moonbitstack/moonmuse/ollamaOllama's native API: /api/chat, /api/generate, /api/embed, /api/tags, /api/show and /api/version. Its options live under options, its stream is newline-delimited JSON rather than server-sent events, and its tool calls carry no id.
A connection to an Ollama server.
Implements @moonmuse.Provider
A client for base, which defaults to the port Ollama listens on.
A streamed answer, forwarded as the endpoint produces it. Ollama delimits its stream with newlines rather than server-sent events, which is the only difference here.
The version a server reports.
The Ollama wire format.
Implements @moonmuse.Dialect
The one instance; it carries no state.
The body that answers reply.
The chat call for request.
The request a chat body asks for.
The reply a chat body describes.
The embeddings body for /api/embed.
The error body Ollama sends.
The finish reason as Ollama spells it.
The reply the accumulated lines describe.
The body /api/show returns for a model.
The deltas one line carries.
The lines that carry one delta out.
The model list as /api/tags returns it.
A tool as Ollama writes it — the same function shape OpenAI uses.
bedrock
moonbitstack/moonmuse/bedrockAWS Bedrock as a provider. Its API signs every request with SigV4, which is an HMAC chain over the canonical form of the request, and streams answers in the binary vnd.amazon.eventstream framing rather than in server-sent events. Both are why this is a package of its own rather than a dialect: a Dialect maps JSON, and neither of these is JSON.
The credential a request is signed with. session_token is present for temporary credentials, which is what an instance role hands out.
A credential from a key pair.
One decoded frame.
A frame as it arrives, split into the pieces the consumer sees.
A frame's payload as the event it names.
The decoder's buffer. push returns the frames that are complete.
Whether anything is left over, which a well-formed stream never leaves.
A decoder with nothing buffered.
Take one piece of the stream and return the frames it completed.
What a signed request carries: the headers to send, authorization among them.
The x-amz-date value and the scope date for a Unix time in seconds. The two are the same day in UTC written two ways, and both go into the signature.
The canonical request: the form the signature is taken over. Headers are ordered by code unit, which is what the specification says and what String::lexical_compare does — String::compare is shortlex, so it orders by length first and would put host before content-type. headers must already be the ones that will be signed, sorted and lower-cased; the signer adds host, x-amz-date and the content hash, so a caller does not forget them.
The body that asks for an answer.
The reply a Converse body describes.
The CRC-32 the framing checks, in the reflected IEEE form the protocol names.
The hash of an empty body, which is what a GET sends.
A frame for a test or a stub server: the layout the protocol writes, CRC included.
One HMAC-SHA256, which the key chain and the signature both use.
The SHA-256 of a body, in the hex lowercase the signature writes.
Sign a request. headers are the ones the caller is sending beyond the three this adds; every one of them is signed, so what is signed is what is sent.
The four-step key: date, region, service, then the string "aws4_request".
The string that is signed: the algorithm, the stamp, the scope and the hash of the canonical request.
Around one call
Tools, schemas, templates, the layers that wrap a provider, the catalogue, the counter and the scripted provider.
toolbox
moonbitstack/moonmuse/toolboxTool execution and the tool loop: the one place a Call becomes a Result. An agent framework that has its own loop calls Toolbox::call; one that does not calls run.
One callable tool.
A set of tools addressable by name.
Add a tool. A later tool with the same name replaces the earlier one, so a caller can override a default without rebuilding the set.
Run one call and return its result. A tool that raises, or that is not registered, still produces a result: the model is supposed to see the failure and decide what to do, and a loop that raised instead would lose the conversation.
Whether a tool of this name is registered.
An empty set.
The tools as the model sees them, in the order they were added.
Ask, run whatever tools the model asks for, and ask again — at most limit rounds of tool calls. The reply that comes back is the last one the model produced, whether it was a final answer or a call the budget ran out on; the caller can tell by Reply::calls.
structured
moonbitstack/moonmuse/structuredStructured output: ask for a JSON object that matches a schema, validate it, and ask again with the failure in front of the model when it does not.
The first balanced JSON object in text. Models wrap answers in prose or a code fence often enough that requiring a bare object would fail more calls than it catches; string literals and escapes are respected, so a brace inside a string does not end the object.
The instruction that pins a reply to a schema.
The follow-up when a reply did not fit: the failure, then the same request.
prompt
moonbitstack/moonmuse/promptPrompt templates, rendered by moonpug: the same Jinja syntax a web template uses.
A prompt template.
Render and wrap in a message of role.
A template from its source text.
Render the template against values. A missing variable renders as the engine's default for it, which is what a template author expects.
A template that failed to render, with the engine's own message.
Render several templates as the messages of one request, in order. The first value of each pair is the role.
A value map from key/value pairs.
layers
moonbitstack/moonmuse/layersLayers: retry, fallback, rate limiting, caching and routing. Each wraps a provider and is itself a provider, so they compose in any order.
A place to keep answers. A process-wide map is the default; a shared store can implement the same two methods.
A provider that answers from cache when it can.
Implements @moonmuse.Provider
A chain of providers tried in order.
Implements @moonmuse.Provider
A provider behind a bucket.
Implements @moonmuse.Provider
A token bucket. rate is tokens per second and burst is the capacity.
A bucket that refills at rate tokens per second and holds burst of them.
An in-memory cache with a fixed number of entries, oldest evicted first.
Implements Cache
A cache of at most limit answers.
How many answers are held.
How often and how patiently to retry.
Three attempts, 200 ms doubling to 5 s.
The nominal wait before attempt number attempt (0 for the first retry).
A provider that retries chat and embeddings.
Implements @moonmuse.Provider
A copy of this provider with another policy.
A table of prefixes to providers.
Implements @moonmuse.Provider
The prefixes this table knows, longest first.
One attempt only.
Put a cache in front of provider.
Try each provider in turn. An empty chain answers with Unsupported rather than an index out of bounds.
Put limiter in front of provider.
A canonical key for a request: every field that can change the answer, in a fixed order.
Wrap inner. wait is the sleep, injectable so a test observes the schedule instead of living through it.
A provider that picks a route by the requested model.
Whether the next provider is worth trying after error.
models
moonbitstack/moonmuse/modelsThe model catalogue: context windows, output limits, list prices and capabilities, taken from each vendor's published documentation.
Every model this catalogue knows, grouped by vendor in the order above.
What one call would cost at list prices, from the usage the endpoint reported. None when the model is unknown or the usage did not report both counts, because a partial estimate reads as a cheaper call than it was.
The model with this id, when the catalogue knows it.
The models that declare capability.
count
moonbitstack/moonmuse/countCounting tokens needs a vocabulary, which is a large table most callers never load. It lives here rather than in the root package so that a gateway without a vocabulary does not download one, and the contract it fills is @moonmuse.Counter.
A counter over one vocabulary.
Implements @moonmuse.Counter
A counter over tokenizer. overhead is what the endpoint adds per message; the default is the four tokens the chat formats reserve for the role and the separator.
The tokens of text, which is the one number a tokenizer can give exactly.
fake
moonbitstack/moonmuse/fakeA scripted provider. Tests of everything above the wire use it: the tool loop, the layers, the structured-output repair, and the server that exports a provider as an API.
A scripted provider.
Implements @moonmuse.Provider
A provider that always asks for one tool call.
A deterministic vector per input: the same text always yields the same numbers, and two different texts almost always differ. Enough to test a retriever without a model.
A provider that fails every call with code, for the layers that retry or fall back.
A provider that fails fails times with code, then answers with reply — the shape a retry or a fallback is supposed to survive.
How many calls have been answered.
A provider that answers with replies in order.
The requests this provider saw, in order.
Replay the reply as deltas: text in five-character slices, then each call, then Done. A test that checks incremental handling sees more than one text delta.
A provider that always answers with one text.
Serving
A provider turned into an API, and the one place a call becomes bytes.
serve
moonbitstack/moonmuse/serveA gateway: any provider, exported in the four formats this library speaks. The wire shapes come from the dialects the client already uses, so a format is written once and both directions are the same code.
Who may call. Implement it for a key table, or for whatever an installation already has.
Limits on what a gateway accepts.
8 MiB of request and 4096 tokens of answer.
A table of bearer keys.
Implements Keys
Keys from a list, each mapped to itself.
Keys from a map of key to tenant name.
The credential a request presents: a bearer token, or Anthropic's own header.
The dialect list a gateway offers when the caller does not say: the two formats every client library speaks.
The path a scope asked for.
transport
moonbitstack/moonmuse/transportThe one place a dialect's Call becomes bytes on a socket. Everything above it is pure, which is what lets the same wire format be tested without a server.
A multipart/form-data body and the content type that names its boundary.
The bytes to send.
The content type to send.
An HTTP answer.
The last value of a header, case-insensitively.
The body as JSON, or Decode when it is not.
The retry-after header in seconds, when the endpoint sent one.
An answer whose body is read a piece at a time.
Give up the rest of the body and close the connection. Safe to call twice.
Raise the status of a failed answer, after reading the body that explains it.
The last value of a header, case-insensitively.
Whether the body has been read to its end.
The next piece of the body as text, or None at the end. A piece ends where the network ended it, except that a character split across two reads is held back until its remaining bytes arrive — a caller never sees half a character.
The next piece of the body as it arrived, without decoding. A binary framing — AWS's event stream, a media container — is read this way; next is the text one.
The retry-after header in seconds, when the endpoint sent one.
The status of the head.
Read the rest of the body as text. Used for the body of a failure, which is small.
How the frames of a streamed answer are delimited. Server-sent events carry a name and a blank line between frames; Ollama writes one JSON object per line.
The status of a response that failed, as an error. A success status passes through.
A form of text fields and at most one file. The boundary is derived from the field names and the file name, so the same form always produces the same bytes — which is what makes a recorded request comparable to a fresh one.
Ask for an answer and return once its head is in. A failed call is a Stream error, which is retryable; a 4xx or 5xx head is not raised here, because the body that explains it has to be read off the same stream.
Send one call to base + call.path. A transport failure becomes Stream, which is retryable: a refused connection is worth one more try, a malformed body is not.
Send a body this library did not encode as JSON: a file, an audio payload.
Send a form body, for the endpoints that take a file.
Drive a dialect's reader over an open answer, calling on_delta for each event as it arrives, and return the reply those events assemble. A stream that ends without a closing event still yields a reply: the events it did carry are the answer.
A local engine
An inference engine and a chat template behind the same contract.
local
moonbitstack/moonmuse/localA local model as a provider: a chat template, the text an engine produced, and the tool calls it wrote inside that text. The engine is a contract, so an inference engine can be wired in without this package knowing how weights are read.
The inference engine this package drives. An engine knows weights, tokens and a context window; it does not know Provider, HTTP, or which chat format a model was trained on — those are this package's job.
A provider backed by an inference engine.
Implements @moonmuse.Provider
A provider for engine. The template defaults to ChatML, the format most open models are tuned on; pass template=@local.llama3() for a Llama 3 model.
The prompt this request becomes. Rendering is where a template and its values meet, so a bad template reports itself here rather than after a model has been loaded.
A streamed answer. With no tools declared the model's text is forwarded as it is produced, so a chat streams token by token. With tools the answer is buffered first: a call is written inside the text, and half of one is not something a caller can act on.
An answer split into what is text and what is a call.
How a model writes a tool call.
A local generation failure.
The ChatML template: Qwen, Hermes, Yi and most models fine-tuned on their format.
The values a chat template is rendered with: the messages, the tools, and whether the assistant turn is left open for the model to continue.
The Llama 3 template, which names roles in a header block and ends turns with eot_id.
The text and calls of an answer written in calls style. Call ids are call_0, call_1, by position: a local model does not invent ids, and a caller needs something to match the results it sends back.
Observation
Traces and metrics under the OpenTelemetry GenAI names.
otel
moonbitstack/moonmuse/otelObservation for the model layer: a provider that reports what a call was and what it cost, as OpenTelemetry traces and Prometheus metrics. The names come from the semantic conventions in moontel, so a backend sees the same keys it sees from every other client.
A provider wrapped in metrics.
Implements @moonmuse.Provider
A provider wrapped in tracing.
Implements @moonmuse.Provider
A streamed call is one span, opened before the first event and closed after the last, with the time to the first event recorded as an event so a trace shows where the wait was.
Report provider's calls into registry. The instruments are created once, here, so a registry that already has them refuses the second wrapper rather than mixing two series.
Observe provider through tracer. Spans are named after the operation and the model, which is how the convention names them.