Guide

From zero to a
deployed log

Twelve steps in three parts: a cluster in one process, the same cluster under failure, and the loop a deployment runs. Each step stands on its own and each one runs.

The code is compiled and the output is captured: every block below comes from src/readme/GUIDE.mbt.md, which moon test runs, and the assertions there pin every value printed. The simulator is seeded, so the same code prints the same transcript on every machine.

Part 1 · One process

A cluster that elects, replicates and applies

01
Add the package

moonraft is on mooncakes.io under moonbitstack; the source is moonbitstack/moonraft. The root package is a façade: every type is reachable as @raft.X wherever the sub-package that owns it lives.
```
# in your MoonBit project
moon add moonbitstack/moonraft
```

A first cluster

A Cluster is the batteries-included front: it wires the consensus core to a deterministic, single-seed network, so a whole cluster runs in one thread. Five nodes, and one of them wins the first election.

///|
test "guide: elect a leader" {
  let cluster = @raft.Cluster::new(["n1", "n2", "n3", "n4", "n5"], seed=7)
  let leader = cluster.run_until_leader(300).unwrap()
  println("elected \{leader} in term \{cluster.node(leader).term()}")
  assert_eq(leader, "n3")
  assert_eq(cluster.node(leader).term(), 1UL)
}

Output - verbatim

elected n3 in term 1

Propose, commit, read

Propose a command and wait until every node has committed it. committed_agrees compares the committed prefix across the cluster — the one guarantee a replicated log exists to provide.

///|
test "guide: replicate and read the commit index" {
  let ids = ["a", "b", "c"]
  let cluster = @raft.Cluster::new(ids, seed=11)
  let _ = cluster.run_until_leader(200)
  let _ = cluster.propose(b"set x = 1")
  let _ = cluster.run_until_committed(2, 300)
  println("all nodes committed through index 2: \{cluster.committed_agrees()}")
  for id in ids {
    println("  \{id}: commit_index = \{cluster.node(id).commit_index()}")
  }
  assert_true(cluster.committed_agrees())
  assert_eq(cluster.node("a").commit_index(), 2UL)
  assert_eq(cluster.node("b").commit_index(), 2UL)
  assert_eq(cluster.node("c").commit_index(), 2UL)
}

Output - verbatim

all nodes committed through index 2: true
  a: commit_index = 2
  b: commit_index = 2
  c: commit_index = 2

Your own state machine

Raft only orders bytes; meaning comes from your state machine, whose apply runs once per committed entry, in log order, on every replica. Commands here are UTF-16 text — s<key>=<value> sets a key, d<key> deletes one.

///|
/// The application state. `apply` is the only method consensus calls.
struct KV {
  data : Map[String, String]
}

///|
fn KV::new() -> KV {
  { data: Map([]), }
}

///|
fn KV::get(self : KV, key : String) -> String? {
  self.data.get(key)
}

///|
/// Commands are UTF-16 text: `s<key>=<value>` sets a key, `d<key>` deletes one.
impl @raft.StateMachine for KV with fn apply(self, command) {
  let s = command.to_unchecked_string()
  if s.has_prefix("s") {
    let body = s[1:].to_owned()
    let mut key = ""
    let mut value = ""
    let mut seen_eq = false
    for c in body {
      if !seen_eq && c == '=' {
        seen_eq = true
      } else if seen_eq {
        value = value + "\{c}"
      } else {
        key = key + "\{c}"
      }
    }
    self.data[key] = value
  } else if s.has_prefix("d") {
    self.data.remove(s[1:].to_owned())
  }
}

///|
/// Encode a string as the UTF-16LE bytes `to_unchecked_string` reads back.
fn KV::wire(s : String) -> Bytes {
  let arr : Array[Byte] = []
  for c in s {
    let code = c.to_int()
    arr.push((code & 0xff).to_byte())
    arr.push(((code >> 8) & 0xff).to_byte())
  }
  Bytes::from_array(arr)
}

Then each replica folds its own committed log into a fresh store — three nodes, three identical maps.

///|
test "guide: fold each replica's committed log into a store" {
  let ids = ["a", "b", "c"]
  let cluster = @raft.Cluster::new(ids, seed=9)
  let _ = cluster.run_until_leader(200)
  let _ = cluster.propose(KV::wire("sservice=moonraft"))
  let _ = cluster.propose(KV::wire("slang=moonbit"))
  let _ = cluster.run_until_committed(3, 300)
  for id in ids {
    let kv = KV::new()
    cluster.node(id).node().apply_committed(kv)
    println(
      "  \{id}: service = \{kv.get("service").unwrap()}, lang = \{kv.get("lang").unwrap()}",
    )
    assert_eq(kv.get("service"), Some("moonraft"))
    assert_eq(kv.get("lang"), Some("moonbit"))
  }
}

Output - verbatim

  a: service = moonraft, lang = moonbit
  b: service = moonraft, lang = moonbit
  c: service = moonraft, lang = moonbit

The whole program, with deletes and a self-check, is 04-kvstore.

Part 2 · Under failure

Crashes, partitions, reconfiguration, compaction

Every failure mode below is injected by the same deterministic network, so a scenario that reproduces once reproduces always — which is what makes a consensus bug fixable at all.

Crash the leader

Kill the leader and let a survivor take over. Both invariants that decide correctness under failure still hold: at most one leader per term, and every committed prefix agreed across the cluster.

///|
test "guide: crash the leader and re-elect" {
  let ids = ["a", "b", "c", "d", "e"]
  let cluster = @raft.Cluster::new(ids, seed=5)
  let first = cluster.run_until_leader(200).unwrap()
  let _ = cluster.propose(b"cmd-1")
  let _ = cluster.run_until_committed(2, 200)
  cluster.crash(first)
  let second = cluster.run_until_leader(400).unwrap()
  println("crashed \{first}; new leader \{second}")
  println("one leader per term : \{cluster.one_leader_per_term()}")
  println("committed agree     : \{cluster.committed_agrees()}")
  assert_eq(first, "a")
  assert_true(second != first)
  assert_true(cluster.one_leader_per_term())
  assert_true(cluster.committed_agrees())
}

Output - verbatim

crashed a; new leader b
one leader per term : true
committed agree     : true

Split the network

A partition cuts the cluster in two: two nodes on one side, three on the other. The minority cannot elect — a candidate needs a majority — while the majority side keeps its leader and keeps committing. Heal it and the two sides agree again.

///|
test "guide: a minority cannot elect, a majority carries on" {
  let ids = ["a", "b", "c", "d", "e"]
  let cluster = @raft.Cluster::new(ids, seed=13)
  let _ = cluster.run_until_leader(200)
  // Two nodes on one side of the split, three on the other.
  cluster.partition(["a", "b"], ["c", "d", "e"])
  cluster.run(200)
  let side = cluster.leaders()
  println("leaders after the split: \{side.length()}")
  println(
    "the majority side keeps committing: \{cluster.propose(b"after-split")}",
  )
  cluster.heal()
  cluster.run(200)
  println("after the heal, leaders: \{cluster.leaders().length()}")
  println("committed agree: \{cluster.committed_agrees()}")
  assert_eq(side.length(), 1)
  assert_eq(cluster.leaders().length(), 1)
  assert_true(cluster.committed_agrees())
}

Output - verbatim

leaders after the split: 1
the majority side keeps committing: true
after the heal, leaders: 1
committed agree: true

Change the membership

A configuration change is proposed like any other command and takes effect when it commits (§6). Adding a voter widens the quorum; a learner receives the log but never votes and never counts toward a quorum (§4.2.1).

///|
test "guide: add a voter, then a learner" {
  let cluster = @raft.Cluster::new(["a", "b", "c"], seed=7)
  let _ = cluster.run_until_leader(200)
  let _ = cluster.propose(b"init")
  let _ = cluster.run_until_committed(2, 200)
  println("voters before: \{sorted(cluster.node("a").voter_nodes())}")
  let _ = cluster.propose_conf(@raft.ConfChange::add("d"))
  let _ = cluster.run_until_committed(3, 200)
  println("voters after adding d: \{sorted(cluster.node("a").voter_nodes())}")
  let _ = cluster.propose_conf(@raft.ConfChange::add_learner("e"))
  let _ = cluster.run_until_committed(4, 200)
  println("learners: \{sorted(cluster.node("a").conf_state().learners)}")
  assert_eq(sorted(cluster.node("a").voter_nodes()), "a,b,c,d")
  assert_eq(sorted(cluster.node("a").conf_state().learners), "e")
}

///|
fn sorted(ids : Array[String]) -> String {
  let copy = ids.copy()
  copy.sort()
  copy.join(",")
}

Output - verbatim

voters before: a,b,c
voters after adding d: a,b,c,d
learners: e

Runnable versions: 05-membership and 06-learner.

Snapshot, and catch up

The log cannot grow without bound, so a leader folds its prefix into a snapshot (§7). A follower that was down while the leader compacted comes back to a log whose early entries are gone: the leader ships it the snapshot instead, and it catches up on its own.

///|
test "guide: a node that fell behind catches up from a snapshot" {
  let ids = ["a", "b", "c"]
  let cluster = @raft.Cluster::new(ids, seed=9)
  let leader = cluster.run_until_leader(200).unwrap()
  let lagging = ids.filter(fn(x) { x != leader })[0]
  for i in 0..<8 {
    let _ = cluster.propose(b"cmd-\{i}")
    let _ = cluster.run_until_committed(
      cluster.node(leader).node().last_log_index(),
      100,
    )
  }
  // Fold everything at or below index 5 into a snapshot.
  println("compacted at 5: \{cluster.compact_leader(5, b"payload")}")
  // While the follower is down, the leader keeps appending to a compacted log.
  cluster.crash(lagging)
  for i in 0..<4 {
    let _ = cluster.propose(b"more-\{i}")
    let _ = cluster.run_until_committed(
      cluster.node(leader).node().last_log_index(),
      100,
    )
  }
  let ahead = cluster.node(leader).commit_index()
  println("down at \{cluster.node(lagging).commit_index()}, leader at \{ahead}")
  cluster.restart(lagging)
  cluster.run(400)
  println("after the restart: \{cluster.node(lagging).commit_index()}")
  println("committed agree: \{cluster.committed_agrees()}")
  assert_eq(cluster.node(lagging).commit_index(), ahead)
  assert_true(cluster.committed_agrees())
}

Output - verbatim

compacted at 5: true
down at 9, leader at 13
after the restart: 13
committed agree: true

Part 3 · On a real transport

The loop a deployment runs

RawNode is the whole protocol as a state machine the caller drives: you own the clock, the disk and the wire.

The Ready / Advance loop

The simulator is the convenient front; a deployment runs RawNode instead. tick advances the clock, ready hands you a batch (entries to persist, messages to send, committed entries to apply), store persists it, advance acknowledges. No threads, no async, nothing hidden.

///|
test "guide: drive one node through tick, Ready and Advance" {
  let node = @raft.RaftNode::new("a", [], pre_vote=false)
  let raw = node.raw()
  raw.campaign()
  let mut applied = 0
  while raw.has_ready() {
    let ready = raw.ready()
    raw.store(ready)
    applied = applied + ready.committed_entries.length()
    raw.advance(ready)
  }
  println("entries applied after campaigning: \{applied}")
  raw.propose(b"set x = 2")
  while raw.has_ready() {
    let ready = raw.ready()
    raw.store(ready)
    applied = applied + ready.committed_entries.length()
    raw.advance(ready)
  }
  println("entries applied after the proposal: \{applied}")
  println("commit_index: \{raw.node().commit_index()}")
  assert_eq(applied, 2)
  assert_eq(raw.node().commit_index(), 2UL)
}

Output - verbatim

entries applied after campaigning: 1
entries applied after the proposal: 2
commit_index: 2

Restart from what you persisted

A Ready tells you exactly what must reach stable storage. Rebuild a node from a MemoryStorage that holds it and the term, the commit index and the log come back unchanged — the recovery path a deployment drives from its write-ahead log.

///|
test "guide: rebuild a node from what it persisted" {
  let node = @raft.RaftNode::new("a", [], pre_vote=false)
  let raw = node.raw()
  raw.campaign()
  let run = fn() {
    while raw.has_ready() {
      let ready = raw.ready()
      raw.store(ready)
      raw.advance(ready)
    }
  }
  run()
  raw.propose(b"set x = 3")
  run()
  let store = @raft.MemoryStorage::new()
  raw.node().node().save_into(store)
  println(
    "before restart: term=\{raw.node().term()}, commit=\{raw.node().commit_index()}",
  )
  let recovered = @raft.RaftNode::new("a", [], pre_vote=false)
  recovered.node().load_from(store)
  println(
    "after restart:  term=\{recovered.term()}, commit=\{recovered.commit_index()}",
  )
  assert_eq(recovered.term(), raw.node().term())
  assert_eq(recovered.commit_index(), raw.node().commit_index())
}

Output - verbatim

before restart: term=1, commit=2
after restart:  term=1, commit=2

A linearizable read

A leader that has been cut off must not answer a read from stale state. read_index asks for a fresh quorum round-trip; the ReadState that surfaces on the leader’s Ready is the proof that the read is safe at that commit index (§6.4).

///|
test "guide: a read the quorum has confirmed" {
  let ids = ["a", "b", "c"]
  let cluster = @raft.Cluster::new(ids, seed=17)
  let leader = cluster.run_until_leader(200).unwrap()
  let _ = cluster.propose(b"v1")
  let _ = cluster.run_until_committed(2, 200)
  let raw = cluster.node(leader).raw()
  raw.read_index(b"client-ctx-1")
  let mut reads = 0
  for _ in 0..<200 {
    cluster.run(1)
    if raw.has_ready() {
      let ready = raw.ready()
      raw.store(ready)
      reads = reads + ready.read_states.length()
      raw.advance(ready)
    }
    if reads > 0 {
      break
    }
  }
  println("read states confirmed by a quorum: \{reads}")
  println("safe to read at commit index: \{raw.node().commit_index()}")
  assert_true(reads > 0)
  assert_eq(raw.node().commit_index(), 2UL)
}

Output - verbatim

read states confirmed by a quorum: 1
safe to read at commit index: 2

Logging

The core logs through a one-method Logger trait and defaults to silence, so the consensus engine names no logging backend. Implement the trait and the diagnostics of an election arrive in order.

///|
/// The core logs through one method and defaults to silence, so a deployment
/// decides where diagnostics go.
struct Recorder {
  lines : Array[String]
}

///|
impl @raft.Logger for Recorder with fn log(self, _level, message) {
  self.lines.push(message)
}

///|
test "guide: capture what the core reports" {
  let recorder : Recorder = { lines: [], }
  let node = @raft.RaftNode::new("a", [], pre_vote=false, logger=recorder)
  let raw = node.raw()
  raw.campaign()
  while raw.has_ready() {
    let ready = raw.ready()
    raw.store(ready)
    raw.advance(ready)
  }
  println("records captured: \{recorder.lines.length()}")
  for line in recorder.lines {
    println("  \{line}")
  }
  assert_eq(recorder.lines.length(), 8)
  assert_eq(
    recorder.lines[recorder.lines.length() - 1],
    "a became leader at term 1",
  )
}

Output - verbatim

records captured: 8
  a switched to configuration voters=(a)
  a became follower at term 0
  newRaft a [peers: [a], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
  a is starting a new election at term 0
  a became candidate at term 1
  a received MsgVoteResp from a at term 1
  a has received 1 MsgVoteResp votes and 0 vote rejections
  a became leader at term 1

Already running moonlog? moonbitstack/moonraft/moonlog_bridge adapts its logger to the same seam.

///|
test "guide: the same seam takes a moonlog logger" {
  let bridge = @moonlog_bridge.MoonlogLogger::of(@moonlog.Hub::new())
  let logger = bridge.as_logger()
  logger.log(@raft.LogLevel::Info, "moonraft is ready")
  println("a hub with no sinks accepts the record and writes nothing")
  assert_true(true)
}

Output - verbatim

a hub with no sinks accepts the record and writes nothing

Where to next

Read it, run it, check it

Browse every public type and method in the API reference, watch the invariants hold in the live demo, or read how the port is kept honest against its reference implementation in the porting notes and the ledger.

The src/examples/ directory is the same tour as runnable programs — each one shows a single idea through either the deterministic Cluster or the message-driven RawNode:

00-helloworld — a minimal system: elect a leader
01-replicate — propose commands, confirm every node commits
02-fault-tolerance — crash, partition, drop packets; safety holds
03-operations — progress, leadership transfer, compaction
04-kvstore — a complete replicated key-value store on a StateMachine
05-membership — add & remove voters at runtime
06-learner — a non-voting learner, then promote it to a voter
07-checkquorum — a leader steps down when it loses its majority
08-snapshot — compact the log; a restarted node catches up
09-rawnode — the RawNode tick → Ready → Advance loop, by hand
10-storage — persist Raft state to MemoryStorage and recover it
11-readindex — a linearizable (ReadOnlySafe) read, confirmed by quorum

▶ See it running Full API

Local

Run the browser demo locally

The interactive demo builds from the same consensus core. Compile to WebAssembly, drop the artifact next to the page, and serve docs/ over http — an ES-module / fetch of .wasm needs http(s), not file://.

moon build --target wasm --release src/examples/browser
cp _build/wasm/release/build/moonbitstack/moonraft/examples/browser/browser.wasm docs/moonraft.wasm
python3 -m http.server 8099 --directory docs   # open http://localhost:8099/demo.html

From zero to adeployed log

A cluster that elects, replicates and applies

Add the package

A first cluster

Propose, commit, read

Your own state machine

Crashes, partitions, reconfiguration, compaction

Crash the leader

Split the network

Change the membership

Snapshot, and catch up

The loop a deployment runs

The Ready / Advance loop

Restart from what you persisted

A linearizable read

Logging

Read it, run it, check it

Run the browser demo locally

From zero to a
deployed log