MoonBit · the guide
moonzip is one package per format. This walks the path a caller actually takes: compress a buffer, put buffers in an archive, compress as they arrive, and choose the format that fits. Every snippet below runs as written; its output is what the library printed.
deflate is the raw codec (RFC 1951). gzip wraps it with
a ten-byte header, a CRC-32 and a length (RFC 1952). zlib wraps it
with a two-byte header and an Adler-32 (RFC 1950). All three round-trip to the
byte.
fn main raise {
let out : Array[Byte] = []
for _ in 0..<200 {
for b in "the quick brown fox jumps over the lazy dog. " {
out.push((b.to_int() % 256).to_byte())
}
}
let body = Bytes::from_array(out)
println("original -> \{body.length()} bytes")
let raw = @deflate.deflate(body[:])
println("deflate -> \{raw.length()} bytes, back to \{@deflate.inflate(raw[:]).length()}")
let gz = @gzip.gzip(body[:])
println("gzip -> \{gz.length()} bytes, back to \{@gzip.gunzip(gz[:]).length()}")
let zl = @zlib.zlib(body[:])
println("zlib -> \{zl.length()} bytes, back to \{@zlib.unzlib(zl[:]).length()}")
}original -> 9000 bytes deflate -> 87 bytes, back to 9000 gzip -> 105 bytes, back to 9000 zlib -> 93 bytes, back to 9000
A container that carries a checksum checks it. Flip one byte of the
trailing CRC and gunzip refuses, rather than hand back bytes
that look right and are not.
fn main raise {
let out : Array[Byte] = []
for _ in 0..<200 {
for b in "the quick brown fox jumps over the lazy dog. " {
out.push((b.to_int() % 256).to_byte())
}
}
let body = Bytes::from_array(out)
let gz = @gzip.gzip(body[:])
let bent = gz.to_array()
bent[gz.length() - 8] = (bent[gz.length() - 8].to_int() ^ 0xFF).to_byte()
try @gzip.gunzip(Bytes::from_array(bent)[:]) |> ignore catch {
Checksum(..) => println("tampered -> refused: the checksum does not match")
_ => println("tampered -> refused")
} noraise {
_ => println("tampered -> accepted, which would be the bug")
}
}tampered -> refused: the checksum does not match
The same input gives the same bytes every time: gzip writes a
zero modification time by default, so a build stays reproducible.
The writer can parse a block two ways. Greedy, the default,
takes the longest match at the spot the walk lands on and jumps to where
it ends — and it is byte for byte what the writer has always produced, so
a caller who omits the strategy gets the bytes they have always gotten.
LongMatch re-walks the block a few times, pricing a copy at
what the block's own codes charge for it, and keeps the parse that comes
out smallest: it pays for a longer match at a farther distance instead of
the shorter one close by.
fn main raise {
let out : Array[Byte] = []
for _ in 0..<200 {
for b in "the quick brown fox jumps over the lazy dog. " {
out.push((b.to_int() % 256).to_byte())
}
}
let body = Bytes::from_array(out)
let raw = @deflate.deflate(body[:])
let long = @deflate.deflate(body[:], strategy=@deflate.Strategy::LongMatch)
println("greedy -> \{raw.length()} bytes, back to \{@deflate.inflate(raw[:]).length()}")
println("long -> \{long.length()} bytes, back to \{@deflate.inflate(long[:]).length()}")
assert_eq(@deflate.deflate(body[:]), @deflate.deflate(body[:], strategy=@deflate.Strategy::Greedy))
}greedy -> 87 bytes, back to 9000 long -> 84 bytes, back to 9000
The greedy walk is always among the candidates LongMatch
measures, so on a body of noise or of one repeated sentence, where there
is nothing to choose between, it writes exactly what greedy does and no
more — on this body the longer match saves three bytes, on the one in the
table below it saves 2.8%, and it never writes more. The parse is paid at
write time, which is why it is opt-in and why the default keeps its bytes.
zip is random-access: a central directory at the end names every
member, so listing reads that and stops. tar is one header per file
and no compressed members; targz is a tarball through gzip, which is
all a .tar.gz ever was.
7z puts its directory after the folders it describes, so it cannot
be read from the front like zip. It can be read without holding it:
un7z_from takes anything implementing RandomAccess —
two methods, read(offset, len) and size() — reads the
directory, then takes one folder at a time. un7z is that same
reader over a block of bytes already in memory.
fn main raise {
let entries = [
@zip.Entry::new("readme.txt", b"a member of an archive\n", mtime=1700000000L),
@zip.Entry::new("data/table.csv", b"name,size\nreadme,24\n"),
]
let archive = @zip.zip(entries[:])
println("zip -> \{archive.length()} bytes, \{@zip.names(archive[:]).length()} members")
println("zip names -> \{@zip.names(archive[:]).join(", ")}")
let back = @zip.unzip(archive[:])
println("unzip -> \{back.length()} members, first is \{back[0].name}")
let files = [
@tar.Entry::new("readme.txt", b"a member of an archive\n", mtime=1700000000L),
@tar.Entry::new("data/table.csv", b"name,size\nreadme,24\n"),
]
let tarball = @tar.tar(files[:])
let gzipped = @tar.targz(files[:])
println("tar -> \{tarball.length()} bytes")
println("tar.gz -> \{gzipped.length()} bytes")
}zip -> 265 bytes, 2 members zip names -> readme.txt, data/table.csv unzip -> 2 members, first is readme.txt tar -> 3072 bytes tar.gz -> 170 bytes
Listing reads only the directory; unzip expands on demand, so a
caller can pull one member without touching the rest.
A member can be encrypted. ZipCrypto is the cipher the format called encryption
for fifteen years, and it is broken; WinZip AES is the real one — counter mode
under a PBKDF2-HMAC-SHA1 key, authenticated with an HMAC-SHA1 tag, at 128, 192
or 256 bits, in the AE-1 spelling that keeps the CRC or AE-2 that drops it. A
split archive is the same bytes in fixed-size pieces, archive.z01
through archive.zip, that rejoin into an ordinary archive.
fn main raise {
let entries = [@zip.Entry::new("secret.txt", b"attack at dawn")]
let sealed = @zip.zip(entries[:], password="hunter2", aes=256)
let opened = @zip.unzip(sealed[:], password="hunter2")
println("aes-256 -> \{opened[0].name}, \{opened[0].data.length()} bytes")
let parts = @zip.zip_split(entries[:], 64)
println("volumes -> \{parts.length()} pieces: \{@zip.volume_names("archive", parts.length()).join(", ")}")
let joined = @zip.unzip_split(parts)
println("rejoined -> \{joined[0].name}, \{joined[0].data.length()} bytes")
}aes-256 -> secret.txt, 14 bytes volumes -> 3 pieces: archive.z01, archive.z02, archive.zip rejoined -> secret.txt, 14 bytes
A wrong password is caught by the two-byte verification value, and a tampered payload by the tag, both before any byte is returned.
A response body arrives in chunks; a file is read four kilobytes at a time. The
Writer/Reader pair takes it a piece at a time and never
holds the whole thing — only the 32 KiB window a back-reference can reach, plus
four bytes of running checksum. push returns what is ready;
finish flushes the rest and checks the trailer.
Every codec carries the pair. What each one holds between calls is its own
unit — a block for deflate, lz4, zstd and
a brotli meta-block, a chunk for snappy and xz, a
block for bzip2 — and lzma, which has no blocks at
all, holds the dictionary a copy can reach into and carries its range coder
from token to token.
The archive writers carry it too: zip.Writer and
sevenz.Writer hand back each member's bytes as the member is
added, so an archive is never held. A sevenz.Writer produces its
thirty-two-byte signature header last, though, because the field that says
where the directory sits is the length of the packed data — write the
members out as they come, then put signature() in front. The
matching reader is the one place the streaming is one-sided: a 7z directory
follows the folders it describes, so a reader cannot cut members until the
last byte has arrived.
fn main raise {
let out : Array[Byte] = []
for i in 0..<4000 {
for b in "the quick brown fox jumps over the lazy dog. " {
out.push((b.to_int() % 256).to_byte())
}
out.push((i % 251).to_byte())
}
let source = Bytes::from_array(out)
println("body -> \{source.length()} bytes")
let writer = @gzip.Writer::new(level=9)
let packed : Array[Byte] = []
let mut at = 0
while at < source.length() {
let stop = if source.length() - at < 4096 { source.length() } else { at + 4096 }
for b in writer.push(source[at:stop]) { packed.push(b) }
at = stop
}
for b in writer.finish() { packed.push(b) }
let gzipped = Bytes::from_array(packed)
println("gzip -> \{gzipped.length()} bytes (\{100 - gzipped.length() * 100 / source.length()}% smaller)")
let reader = @gzip.Reader::new()
let got : Array[Byte] = []
let mut at2 = 0
while at2 < gzipped.length() {
let stop = if gzipped.length() - at2 < 777 { gzipped.length() } else { at2 + 777 }
for b in reader.push(gzipped[at2:stop]) { got.push(b) }
at2 = stop
}
for b in reader.finish() { got.push(b) }
println("gunzip -> \{Bytes::from_array(got).length()} bytes, matches: \{Bytes::from_array(got) == source}")
}body -> 184000 bytes gzip -> 1983 bytes (99% smaller) gunzip -> 184000 bytes, matches: true
The home page runs the same algorithm in the browser; here it is in MoonBit, on a body fed back in pieces that fall inside headers and codes.
A stream can also be cut into messages without ending it. flush
writes a synchronising flush — the empty stored block, four bytes of
00 00 FF FF — and the reader stops exactly there, keeping its
window so the next message may match against the last one. reset
on either side drops the window, which is the other setting a WebSocket
connection negotiates for context takeover. The window is a parameter both
ends agree on — eight to fifteen bits, 256 bytes to 32 KiB — and a writer may
also be given an output budget: push refuses the message that
would cross it, raising TooBig, rather than expanding the whole
input first.
fn main raise {
let messages = [
"the quick brown fox jumps over the lazy dog. ",
"the quick brown fox jumps over the lazy dog. ",
]
let writer = @deflate.Writer::new(level=6)
let parts : Array[Bytes] = []
for text in messages {
let body = @utf8.encode(text)
let piece : Array[Byte] = []
for b in writer.push(body[:]) { piece.push(b) }
for b in writer.flush() { piece.push(b) }
parts.push(Bytes::from_array(piece))
}
println("message 1 -> \{parts[0].length()} bytes")
println("message 2 -> \{parts[1].length()} bytes, same body")
let reader = @deflate.Reader::new()
let mut total = 0
for part in parts {
for b in reader.flush(part[:]) { total = total + 1 }
}
println("read back -> \{total} bytes")
}message 1 -> 49 bytes message 2 -> 8 bytes, same body read back -> 90 bytes
The second message is eight bytes because the whole of it is one back-reference into the first. Two writers would have sent 49 twice.
The numbers are one 184,000-byte body of English text with a periodic tail,
printed by src/examples/stream and src/examples/formats.
They are the same body the home page's
strata show — here as a table, with what each format is for.
| Format | Bytes | Of original | Reach for it when |
|---|---|---|---|
| gzip | 1,983 | 1.1% | A single file on a wire — HTTP, a log, a download. |
| zlib | 2,667 | 1.4% | The deflate HTTP content coding; a stream wrapped by someone else's framing. |
| deflate (dynamic) | 2,661 | 1.4% | Raw DEFLATE when no container is wanted at all. |
| lz4 block | 1,997 | 1.1% | Decode speed above everything — a cache, a transport. |
| lz4 frame | 4,576 | 2.5% | lz4 with a header, a content checksum and skippable frames. |
| snappy block | 9,884 | 5.4% | The framing Google uses — protobuf on the wire, Parquet on disk. |
| snappy frame | 12,439 | 6.8% | Snappy with per-chunk masked CRC-32C. |
| brotli | 682 | 0.4% | Static assets on the web; it is the codec that serves this page's text. |
| zstd | 3,042 | 1.7% | A general replacement for gzip and zlib, with a fast decoder. |
| bzip2 | 853 | 0.5% | When the setup cost is worth it on highly repetitive text. |
| xz | 468 | 0.3% | Distribution tarballs, where size matters most and decode time does not. |
| lzma alone | 415 | 0.2% | The codec xz and 7z are built on, on its own. |
| — nothing | 184,000 | 100% | The body, uncompressed, for reference. |
brotli takes a quality from 0 to 11 — higher is slower
and never larger — and the row above is quality 9, the default, which already
models literals by context, splits the command and literal streams into two block
types each when the cut pays, and codes words the static dictionary knows as
references rather than letters. Quality 10 and 11 add the one thing 9 does not
do: they choose the parse instead of taking it, walking the block for the
cheapest path to every position and keeping whichever round encoded smallest.
Pick by what the caller does with the bytes. Servicing a request: lz4 or snappy. Storing a file people download: gzip or zstd. Shipping a tarball: gzip for speed, xz for size. Every function above is documented in the API reference.