Guide

From an empty module to a message that travels as bytes, and back. Every program on this page runs; the output under it is what it printed.

1 Install

Add the module and import the packages a program needs. A program that only encodes and decodes needs three of them.

1 terminal

moon add moonbitstack/moonproto

1 moon.pkg

import {
  "moonbitstack/moonproto/desc",
  "moonbitstack/moonproto/dynamic",
  "moonbitstack/moonproto/json",
  "moonbitstack/moonproto/known",
  "moonbitstack/moonproto/known/google/protobuf",
  "moonbitstack/moonproto/parser",
  "moonbitstack/moonproto/proto",
  "moonbitstack/moonproto/reflect",
  "moonbitstack/moonproto/text",
}

2 First message

A schema goes in as text. parser returns the descriptor shape, desc turns it into runtime descriptors, and a dynamic message holds the values. Nothing is generated ahead of time.

1 main.mbt

const SCHEMA : String = "syntax = \"proto3\";\n" +
  "package hello;\n" +
  "message Request { string name = 1; int32 count = 2; repeated int32 nums = 3; }\n"

fn main {
  let parsed = @parser.parse("hello.proto", SCHEMA) catch {
    _ => abort("bad schema")
  }
  let built = @desc.build_file(parsed) catch { _ => abort("bad descriptor") }
  let descriptor = built.messages[0]

  let message = @dynamic.Message::new(descriptor)
  let name = descriptor.field_by_name("name").unwrap()
  let count = descriptor.field_by_name("count").unwrap()
  let nums = descriptor.field_by_name("nums").unwrap()
  message.set(name, @reflect.String("moon"))
  message.set(count, @reflect.Int32(150))
  let list = message.mutable(nums).list()
  list.append(@reflect.Int32(1))
  list.append(@reflect.Int32(300))

  let bytes = @proto.encode(message) catch { _ => abort("encode failed") }
  println("binary bytes: \{bytes.length()}")

  let back = @dynamic.Message::new(descriptor)
  @proto.decode(bytes, back) catch {
    _ => abort("decode failed")
  }
  println("name: \{back.get(name).string()}")
  println("count: \{back.get(count).int()}")
  println("nums: \{back.get(nums).list().length()}")
  let again = @proto.encode(back) catch { _ => abort("re-encode failed") }
  println("same bytes again: \{again == bytes}")
}

binary bytes: 14 name: moon count: 150 nums: 2 same bytes again: true

A repeated field is read and written through the same field descriptor: mutable hands back the list, get reads it. Scalars use reflect.Value, so the compiler never sees a protobuf type.

3 Generated code

When the schema is known at build time, protoc-gen-mbt writes MoonBit source instead. The struct's fields are the protobuf fields; there is no hidden message behind it.

1 terminal

protoc -I . --plugin=protoc-gen-mbt=path/to/protoc-gen-mbt \
  --mbt_out=src hello.proto

1 generated

pub struct Envelope {
  mut i32 : Int32              // implicit presence
  mut name : String?           // explicit presence
  nums : Array[Int32]          // repeated
  counts : Map[String, Int32]  // map
  mut pick : Envelope_Pick     // oneof, as an enum
  mut unknown_fields : Bytes
}

Read and write through the generated accessors, or through reflect.Message if the code is generic. The struct also registers a MessageType, which is how Any finds its payload.

4 JSON and text

One message, three encoders: each walks the same descriptor, so the mapping is written once. This schema carries a 64-bit integer, a flag and a repeated string, which is where the JSON rules show.

1 main.mbt

const REPORT : String = "syntax = \"proto3\";\n" +
  "package report;\n" +
  "message Line { string title = 1; int64 total = 2; bool active = 3; repeated string tags = 4; }\n"

fn main {
  @known.register_all()
  let parsed = @parser.parse("report.proto", REPORT) catch {
    _ => abort("bad schema")
  }
  let built = @desc.build_file(parsed) catch { _ => abort("bad descriptor") }
  let descriptor = built.messages[0]
  let message = @dynamic.Message::new(descriptor)

  let title = descriptor.field_by_name("title").unwrap()
  let total = descriptor.field_by_name("total").unwrap()
  let active = descriptor.field_by_name("active").unwrap()
  let tags = descriptor.field_by_name("tags").unwrap()
  message.set(title, @reflect.String("quarterly"))
  message.set(total, @reflect.Int64(10_000_000_000L))
  message.set(active, @reflect.Bool(true))
  message.mutable(tags).list().append(@reflect.String("proto"))

  let binary = @proto.encode(message) catch { _ => abort("binary failed") }
  println("binary: \{binary.length()} bytes")
  let as_json = @json.encode(message) catch { _ => abort("JSON failed") }
  println("json: \{as_json}")
  let as_text = @text.encode(message) catch { _ => abort("text failed") }
  println("text: \{as_text}")

  // Each format parses back into a fresh message of the same descriptor.
  let from_json = @dynamic.Message::new(descriptor)
  @json.decode(as_json, from_json) catch {
    _ => abort("JSON decode failed")
  }
  println("total from JSON: \{from_json.get(total).int()}")
  let from_text = @dynamic.Message::new(descriptor)
  @text.decode(as_text, from_text) catch {
    _ => abort("text decode failed")
  }
  println("title from text: \{from_text.get(title).string()}")
  let from_binary = @dynamic.Message::new(descriptor)
  @proto.decode(binary, from_binary) catch {
    _ => abort("binary decode failed")
  }
  let agree = @proto.equal(from_json, from_binary) &&
    @proto.equal(from_text, from_binary)
  println("all three agree: \{agree}")
}

binary: 26 bytes json: {"title":"quarterly","total":"10000000000","active":true,"tags":["proto"]} text: title: "quarterly" total: 10000000000 active: true tags: "proto" total from JSON: 10000000000 title from text: quarterly all three agree: true

64-bit integers are strings in both directions. Enums travel by name, bytes in base64, and a field without presence is left out when it holds its zero value.

The text format is the one that reads as well as writes, and its grammar is wider than JSON's: fields separated by commas, semicolons or newlines, one element per repeated name, groups, octal and hexadecimal literals, and comments. Hand it title: "quarterly", total: 10000000000, tags: "proto" tags: "wire" and the encoder writes the two tags back one per line.

5 Editions

Edition 2023 and 2024 schemas are read the same way. Features resolve from the innermost declaration outward, so a file-level default can be overridden per field.

1 report.proto

edition = "2023";
package report;

option features.field_presence = IMPLICIT;

message Line {
  int32 total = 1;
  repeated int32 samples = 2 [features.repeated_field_encoding = EXPANDED];
}

With field_presence = IMPLICIT a scalar is generated as Int32; without it, as Int32?. repeated_field_encoding decides whether a repeated scalar is packed on the wire.

6 Extensions

An extension lives in the message's extension range and is named in brackets in text and JSON. The binary encoding is the same as any other field.

1 events.proto

syntax = "proto2";
package events;

message Event {
  optional string name = 1;
  extensions 100 to 199;
}

extend Event {
  optional string annotation = 100;
}

In text form the field is [events.annotation]: "..."; in JSON the key is the same bracketed name. A decoder resolves an unknown field number against the extension registry before it decides the bytes are unknown.

7 Well-known types

The google.protobuf types ship with the module, generated from the schemas the protobuf release carries. Register them once and Any, Timestamp, Duration, FieldMask and the wrapper types use their own JSON forms.

1 main.mbt

@known.register_all()
let stamp = @protobuf.Timestamp::new()
stamp.set_seconds(1_700_000_000L)
println(@json.encode(stamp) catch { _ => abort("JSON failed") })

"2023-11-14T22:13:20Z"

Without that call the descriptors are still available, but Any cannot resolve a payload and a timestamp prints as its two integer fields rather than as an RFC 3339 string.