Models and data — moonctl

The same spec, a SQL DDL file or a live database can become a model layer: structs, a declarative model with declared columns, CRUD calls, migrations, document stores and a cache in front of it.

A model from a spec

mctl gen model user.api writes one struct per type block, and with each struct its model:

terminal

$ mctl gen model user.api
mctl: generated user_model.mbt from user.api

user_model.mbt

///|
/// `LoginReq` model (generated from the `.api` `type` block).
pub(all) struct LoginReq {
  username : String
  password : String
}

///|
/// Declarative model for `LoginReq`: declared columns, a `@moondb.Row` decoder, and
/// the column projection an INSERT binds.
pub let login_req_model : @moonorm.Model[LoginReq] = @moonorm.Model::new(
  "login_req",
  [
    @moonorm.column("username", TextType),
    @moonorm.column("password", TextType),
  ],
  row => {
    username: row.text(0),
    password: row.text(1),
  },
  r => [
    ("username", @moondb.Text(r.username)),
    ("password", @moondb.Text(r.password)),
  ],
)

The model is the plain record plus four things the ORM needs and a struct cannot carry on its own: the column list, the row decoder, the INSERT projection, and the table descriptor. A migration pair follows — login_req_up creates the table, login_req_down drops it.

A model from DDL

mctl gen crud schema.sql reads CREATE TABLE statements and writes the same model, plus the calls for the table:

schema.sql

CREATE TABLE `orders` (
  `id` BIGINT NOT NULL,
  `user_id` BIGINT NOT NULL,
  `total` DOUBLE NOT NULL,
  `note` VARCHAR(200) NULL,
  `created_at` TIMESTAMP NOT NULL,
  PRIMARY KEY (`id`)
);

schema_model.mbt

pub(all) struct Orders {
  id : Int64
  user_id : Int64
  total : Double
  note : String
  created_at : String
}

pub let orders_model : @moonorm.Model[Orders] = @moonorm.Model::new(
  "orders",
  [
    @moonorm.column("id", IntType, primary_key=true),
    @moonorm.column("user_id", IntType, nullable=false),
    @moonorm.column("total", RealType, nullable=false),
    @moonorm.column("note", TextType),
    @moonorm.column("created_at", TextType, nullable=false),
  ],
  row => {
    id: row.int64(0),
    user_id: row.int64(1),
    total: row.double(2),
    note: row.text(3),
    created_at: row.text(4),
  },
  r => [
    ("id", @moondb.Int64(r.id)),
    ("user_id", @moondb.Int64(r.user_id)),
    ("total", @moondb.Double(r.total)),
    ("note", @moondb.Text(r.note)),
    ("created_at", @moondb.Text(r.created_at)),
  ],
)

And one function per call the table can answer:

the CRUD layer

pub fn orders_insert(sess : @moonorm.Session, record : Orders) -> @moondb.ExecResult raise @moondb.DbError
pub fn orders_all(sess : @moonorm.Session) -> Array[Orders] raise @moondb.DbError
pub fn orders_find_by_id(sess : @moonorm.Session, id : Int64) -> Orders? raise @moondb.DbError
pub fn orders_update(sess : @moonorm.Session, record : Orders) -> @moondb.ExecResult raise @moondb.DbError
pub fn orders_delete_by_id(sess : @moonorm.Session, id : Int64) -> @moondb.ExecResult raise @moondb.DbError

The lookup is named after the primary key the DDL declared — find_by_id here, find_by_code for a table keyed on code.

A live database

A schema is often in the server, not in a file. model mysql datasource and model pg datasource connect through moondb, read the column list and types, and write the model from what is actually there:

terminal

$ mctl model mysql datasource root:pass@tcp(127.0.0.1:3306)/shop
mctl: generated mysql_model.mbt

$ mctl model pg datasource postgres://app:pass@localhost:5432/shop
mctl: generated pg_model.mbt

Both need a reachable server: the command opens a connection, asks the catalogue for the tables and columns, and closes it. Nothing else in moonctl needs a database.

Documents

mctl model mongo user.api treats each type block as a document. The generated file carries the document and a typed repository, over one port that names the five calls a repository makes:

user_mongo.mbt

pub(all) struct Collection {
  name : String
  run : (MongoOp, Json, Json) -> Json raise MongoError
}

pub(all) struct LoginReqDoc {
  username : String
  password : String
} derive(FromJson, ToJson)

pub fn LoginReqRepo::find(
  self : LoginReqRepo,
  username : String
) -> LoginReqDoc raise MongoError {
  let reply = (self.coll.run)(
    FindOne,
    filter_of("username", @json.to_json(username)),
    empty_doc(),
  )
  match reply {
    Null => raise MongoMissing(collection=self.coll.name, key=username.to_string())
    _ =>
      @json.from_json(reply) catch {
        e => raise MongoBadDocument(collection=self.coll.name, why=e.to_string())
      }
  }
}

A driver binds run to insert-one / find-one / find / replace-one / delete-one and the repositories work. insert and all exist for every document; find, replace and delete appear when the type has a field to look a record up by. A field that names another type is stored as that type's document, so owner: User becomes owner : UserDoc. The file needs moonbitlang/core/json and no driver.

A cache in front

mctl model cache user.api writes the cache-aside layer for the same tables: read through on find, invalidate on write and on evict.

user_cache.mbt

pub(all) struct Cache[T] {
  get : (String) -> T? raise CacheError
  set : (String, T) -> Unit raise CacheError
  del : (String) -> Unit raise CacheError
}

pub fn login_req_find_cached(
  cache : Cache[LoginReq],
  username : String,
  load : (String) -> LoginReq? raise CacheError,
) -> LoginReq? raise CacheError {
  let key = login_req_cache_key(username)
  let hit : LoginReq? = (cache.get)(key) catch { _ => None }
  match hit {
    Some(record) => Some(record)
    None =>
      match load(username) {
        Some(record) => {
          (cache.set)(key, record) catch { _ => () }
          Some(record)
        }
        None => None
      }
  }
}

The port is generic, so the cache holds the record rather than the text it serialises to, and an in-memory implementation is generated alongside — memory_cache[T](), a Map behind the three calls. A cache that fails is a miss and a failed invalidation is swallowed: a broken cache costs latency, never a request.

Migrations

mctl migrate user.api writes the SQL a spec implies, one table per type block, with a primary key where a field is named id:

user-api_migrate.sql

CREATE TABLE IF NOT EXISTS `profiles` (
  `id` BIGINT NOT NULL,
  `name` VARCHAR(191) NOT NULL,
  PRIMARY KEY (`id`)
);

CREATE TABLE IF NOT EXISTS `login_reqs` (
  `username` VARCHAR(191) NOT NULL,
  `password` VARCHAR(191) NOT NULL
);

A spec says what a value looks like, not what the database should enforce, so the file carries columns and nothing else — indexes, uniqueness and foreign keys are yours to add. The mapping:

.api type	Column
`string`	`VARCHAR(191)`
`bytes`	`BLOB`
`bool`	`TINYINT(1)`
`int`, `int32`	`INT`
`int64`	`BIGINT`
`uint`, `uint32`	`INT UNSIGNED`
`uint64`	`BIGINT UNSIGNED`
`float`	`FLOAT`
`float64`	`DOUBLE`
`[]T`, `map[K]V`	`JSON`
`T` optional	the same column, `NULL`

Where next

A .proto can become a service, and an HTTP gateway in front of it. That is the gateway page.