Documentation

The whole interface, on one sheet.

Everything below is the public surface. There is no hidden state, no clock, no file and no socket: a value goes in, bytes come out.

Install

moon add moonbitstack/moonfory

In moon.pkg:

import {
  "moonbitstack/moonfory",
}

The package is reached as @lib. Its dependencies are moonbitstack/moonvar for the numbers and moonbitstack/moondate for the time types; nothing else is pulled in.

The four verbs

A value goes in and comes out through dumps and loads. dump and load do the same over a Buffer or a Reader you already hold, for when a message is one part of a larger stream.

let bytes : Bytes = @moonfory.dumps(value)
let value : T     = @moonfory.loads(bytes)

@moonfory.dump(value, buffer)
let value : T = @moonfory.load(reader)

Implementing Codec

A type joins in by implementing Codec. write_data writes the body only: the type ID belongs to whoever knows the schema, so a list element and a struct field each name their own type without the body repeating it.

pub impl Codec for UserId with fn ty(_self) {
  TypeId::Int64
}

pub impl Codec for UserId with fn write_data(self, w) {
  w.write_i64(self.0)
}

pub impl Codec for UserId with fn read_data(r, _ty) {
  UserId(r.read_i64())
}

Bool, Int, Int64, UInt, UInt64, Byte, Float, Double, String, Bytes, Array[T], Map[K, V] and Value already implement it.

Type table

Strings are written as UTF-8. A reader also accepts LATIN1 and UTF-16, because pyfory writes LATIN1 for ASCII; only the writer's choice is fixed here.

Numbers, in one place

Int is written as INT32, Int64 as INT64, UInt as UINT32 and UInt64 as UINT64. The reader is looser on purpose: it accepts any width of the same signedness, so a peer's VARINT64 arrives as an Int without a byte of conversion. A 64-bit value asked for as a 32-bit type is refused rather than truncated.

Describing a struct

A struct's fields are a @moonmodel description — the same one an OpenAPI document or a CREATE TABLE would be generated from — plus the three things the wire needs that a description has no word for: the type ID the field goes out as, the tag that names it when the schema configures one, and whether the value may be shared.

let point = @moonfory.StructMeta::new("Point", 100, [
  @moonfory.FieldMeta::new("x", @moonfory.TypeId::VarInt32),
  @moonfory.FieldMeta::new("y", @moonfory.TypeId::VarInt32),
])

let person = @moonfory.StructMeta::new("Person", 101, [
  @moonfory.FieldMeta::new("name", @moonfory.TypeId::String),
  // a list says what it holds: the writer may leave it out of the header
  @moonfory.FieldMeta::new("tags", @moonfory.TypeId::List,
    element=Some(@moonfory.TypeId::String)),
  // a declared field sends no type ID, so the schema says which struct it is
  @moonfory.FieldMeta::new("home", @moonfory.TypeId::Struct,
    nullable=true, child=Some(point)),
])

From a #model.derive model, StructMeta::of fills the type IDs from the description's kinds and takes the child schemas by name; meta.model() hands the Model back, so one description can be published as a schema and used as a codec without being written twice. A kind with no wire type — Any — is refused rather than guessed at.

The fingerprint

Every struct carries a four-byte fingerprint of its schema and a peer checks it before it trusts a field: identifier,typeID,ref,nullable; per field in identifier order, hashed with MurmurHash3 x64 128 seeded with 47. A field of a user type writes 0 for its type ID and 0 for its ref flag, and a list writes its element as [type,0,0]. Under those rules the fingerprints here are the ones pyfory writes, byte for byte.

Configuration

Precedence. A call has three places to say what it wants. The library preset (config) is the floor. A Config passed as config= replaces it wholesale. The flat mirrors — xlang=, track_ref=, check_string=, depth=, fields=, graph= — then override single fields of that result. When a Config and a flat mirror disagree, the flat mirror wins, unless wins=Base is passed, in which case the record wins. A disagreement is silent by default; clash=Panic turns it into a Clashed error, and clash=Handle(f) lets you arbitrate.

@moonfory.dumps(v, track_ref=true)
@moonfory.dumps(v, config=my_config, depth=8)
@moonfory.dumps(v, config=my_config, xlang=false, clash=Panic)

Shared and circular values

MoonBit has no object identity to track, so identity is a key you choose and the value carries: Shared(key, value) is the value filed under that key, and Pointer(key) is the value already filed under it. The first is written out in full; the second is two bytes.

let node = @moonfory.StructMeta::new("Node", 300, [
  @moonfory.FieldMeta::new("val", @moonfory.TypeId::VarInt32),
  @moonfory.FieldMeta::new("next", @moonfory.TypeId::Struct, track_ref=true),
])

// A node whose `next` is itself: reference zero is the root, which this is.
let value = @moonfory.Value::Struct(node, [
  ("val", @moonfory.Value::Int(1L)),
  ("next", @moonfory.Value::Pointer(0)),
])

@moonfory.dumps(value, track_ref=true)  // 01 00 1b ac02 eecb234f 02 fe 00

Those are the bytes Go's fory writes for struct Node { Val int32; Next *Node }, byte for byte, and Go reads them back into a struct whose Next is itself. A value read from a peer is the same shape — the node it starts at is a Shared, and the field leading back to it is a Pointer — so it can be held without holding itself and sent again unchanged.

let once  = @moonfory.shared(7, "the same string")
let twice = [once, once]   // two references, one body
let bytes = @moonfory.dumps(twice, track_ref=true)

Turn track_ref on for both ends, or neither: a reader that is not tracking reads the first body and then a flag it does not expect. Python's pyfory cannot write a cycle at all — its serializer recurses — so Go is the peer the cyclic fixtures are checked against.

Interop, as a test

Every class of message here has been read from bytes another implementation wrote, and the bytes that go the other way have been read by that implementation: pyfory for Python and fory for Go. The fixtures are generated by scripts/interop.py and scripts/interop.go, run by hand, and checked in as MoonBit files the tests read — so CI runs the tests and needs neither language installed.

python3 scripts/interop.py   # src/fixtures_pyfory_test.mbt
cd scripts && go run .         # src/fixtures_go_test.mbt
moon fmt

A fixture is the bytes and nothing else, and the MoonBit side says what the value is; the expectation for each one is written out by hand, because a test that could be made to pass by changing both sides at once is not one.

The row format

The row format trades a little space for random access: eight bytes per field, whether the field needs them or not, so field i is always at bitmap + i × 8. A reader takes one field and leaves the rest alone.

let schema = @moonfory.RowSchema::new([
  { name: "id",   ty: @moonfory.RowType::Int64,  nullable: false },
  { name: "name", ty: @moonfory.RowType::String, nullable: false },
  { name: "age",  ty: @moonfory.RowType::Int32,  nullable: true  },
])

let row = @moonfory.Row::new(schema)
row.set(0, @moonfory.RowValue::Int64(7L))
row.set(1, @moonfory.RowValue::String("ada"))
row.set(2, @moonfory.RowValue::Null)

let cur = @moonfory.RowCursor::new(row.finish()[:], schema)
cur.get_field(1)   // RowValue::String("ada")
cur.is_null(2)     // true

The layout is the specification's standard variant: a null bitmap rounded to eight bytes, one eight-byte slot per field, then the variable data. An array keeps fixed-width elements at their natural width and variable ones as offset-and-size pairs; a map is a key array and a value array; a nested struct is a whole row inside the variable region.

Time

Through moonbitstack/moondate, so a MoonBit program never spells out epoch seconds:

let date   = @moonfory.date_of_days(19_000L)              // DATE
let moment = @moonfory.moment_of_parts(1_700_000_000L, 0) // TIMESTAMP
let span   = @moonfory.span_of_parts(3L, 500_000_000)     // DURATION
@moonfory.dumps(moment)                                     // and back

Nanoseconds are always the sub-second part, in [0, 1e9), so a value of 1.5 seconds is (1, 500_000_000) and a negative span borrows a second rather than carrying a negative nanosecond field.

The reference

Every public item has a page of its own: the API reference. The signature on each one is the one moon info prints, and the prose is the comment beside the declaration, so a page cannot describe something the package no longer does. CI regenerates the directory and refuses a diff; python3 scripts/api.py is the whole of it.

Errors

What is deliberately not here

Each refusal is a Refused variant with the offset that produced it, so a malformed message names where it stopped rather than "invalid input".

Implemented from the cross-language specification, the row format specification and the type mapping — not from another implementation's source.