moonurl.

Which component an octet is going into, and so which octets may stand for themselves (RFC 3986 §2.2, §3).

& is ordinary inside a path and a separator inside a query. The grammar answers that, not the caller.

Bare

Unreserved only — everything else is escaped. This is what one piece of a query goes out as, since & and = inside a value must not read as the separators they look like.

Host

reg-name (§3.2.2): unreserved and sub-delims.

Userinfo

userinfo (§3.2.1): reg-name and :.

Segment

One segment (§3.3): pchar, so / is escaped and stays one segment.

Path

A whole path (§3.3): pchar and /.

Query

query (§3.4): pchar, / and ?.

Fragment

fragment (§3.5): the same set as query.

Whole

A whole URI reference: everything the grammar allows anywhere, % included. What a Location header carries — the value is already a URI, and only the octets no URI may contain need escaping.

What a decode accepts. Past either bound the whole input is refused rather than cut short, so a handler never acts on part of a form.

pairsInt32

Most pairs.

sizeInt32

Most bytes in one pair as it arrives, still encoded.

Limits by name, each defaulting to the preset's.

The name–value pairs of an application/x-www-form-urlencoded string: a URL's query, or a form body (WHATWG URL Standard §5). Order and repeated names are kept, as URLSearchParams and Go's url.Values keep them.

Add a pair at the end, keeping any already under name.

Read an application/x-www-form-urlencoded string (WHATWG URL Standard §5.1).

& separates pairs and the first = separates a name from its value; + is a space here and nowhere else in a URL. Escapes are undone to octets and the octets read as UTF-8, a broken sequence becoming U+FFFD. An empty piece between two & is no pair at all.

test {
  let form = @moonurl.Query::decode(b"q=moon+bit&tag=a&tag=b")
  assert_eq(form.get("q"), Some("moon bit"))
  assert_eq(form.values("tag"), ["a", "b"])
  assert_eq(form.encode(), "q=moon+bit&tag=a&tag=b")
}

Write the pairs as application/x-www-form-urlencoded (WHATWG URL Standard §5.2): letters, digits and *-._ stand for themselves, a space is +, and every other octet of the UTF-8 is %XX.

Every pair, in order.

The first value under name.

Whether any pair is named name.

A query holding pairs, in order.

Remove every pair under name.

Leave exactly one pair under name, holding value: the first one's place if there was one, the end if not.

Put the pairs in order by name, keeping pairs that share a name in the order they arrived (WHATWG URL Standard §5.1, URLSearchParams.sort).

The order is that of the UTF-16 code units of the names, which is what a browser sorts by, so \u{1F308} comes before \u{FB03}: its high surrogate is the smaller unit even though the code point is the larger. Query::encode keeps whatever order it is given, so this is how two bodies with the same pairs come out byte for byte alike. It does not touch the pairs' values.

test {
  let body = @moonurl.Query::decode(b"b=2&a=1&b=3")
  body.sort()
  assert_eq(body.encode(), "a=1&b=2&b=3")
}

Every value under name, in order.

Why a string is not a URI reference. Only the authority can fail: §B splits any string at all into the five top-level components, and does not split that one.

Portat~ : Int32

A port that is not digits, at this offset.

Hostat~ : Int32

An IP-literal with no closing bracket, at the opening one's offset.

Nameat~ : Int32

A host that is not a host: an empty one where one is needed, a code point the grammar forbids in one, or a name the IDNA rules refuse.

Addressat~ : Int32

A number that reads as an IP address and is not one.

Schemeat~ : Int32

Input with no scheme, and no base to take one from.

Pairslimit~ : Int32

A form-encoded string with more pairs than the limit.

Pairlimit~ : Int32, got~ : Int32

One form-encoded pair longer than the limit, in octets.

Which grammar decides what a string means. The two disagree about almost everything a browser has to get right, and both are in the standard library of every language that talks to the web.

Rfc3986

RFC 3986: the generic grammar, where http://h:80/ keeps its port and http:g is a URI with scheme http.

Whatwg

The WHATWG URL Standard: special schemes with their default ports, backslashes as soliduses, IPv4 in any of its spellings, and a host name run through IDNA.

A URI reference taken apart (RFC 3986 §3).

No scheme makes it relative (§4.2). No host means no authority, which is not an empty host — file:///tmp has one, mailto:x@y has none.

Fields stay percent-encoded. Decoding on the way in would lose the difference between a / that separates two segments and a %2F inside one.

schemeString?

The scheme with no colon, https rather than https:.

userinfoString?

user:password as the authority spelled it, both still escaped.

hostString?

A name in ASCII, an IPv4 address in dotted decimal, or an IPv6 address without its brackets.

portInt32?

The port as a number. Nothing means the authority wrote none, which is not the same as the scheme having a default.

pathString

The path, always a string: a reference with an authority and nothing after it has the empty path, not no path.

queryString?

What follows the ?, still escaped. Nothing means no query at all, which a bare ? distinguishes from an empty one.

fragmentString?

What follows the #, still escaped.

The authority as one string, brackets restored around an IP-literal (§3.2).

Take a reference apart.

Under Rfc3986 any string splits (§3), and only the authority can be malformed. Under Whatwg the string is what a browser would make of it: a reference that does not parse is refused rather than guessed at.

base resolves a relative reference against one, in whichever grammar is asked for. The WHATWG algorithm takes the base as text and parses it again, so a base that is itself not a URL is refused here too.

test {
  // RFC 3986 keeps the reference as it was written.
  let url = @moonurl.Url::decode("HTTP://EXAMPLE.test:80/a/../b?q#f")
  assert_eq(url.encode(), "HTTP://EXAMPLE.test:80/a/../b?q#f")
  let whatwg = @moonurl.Url::decode(
    "HTTP://EXAMPLE.test:80/a/../b",
    syntax=@moonurl.Whatwg,
  )
  assert_eq(whatwg.encode(), "http://example.test/b")
}

Put the reference back together (RFC 3986 §5.3).

A reference with a scheme and no fragment is a URI (§3); anything else is a relative reference or a same-document one.

A reference this shape, for building one rather than reading one.

Syntax-based normalisation (RFC 3986 §6.2.2): scheme and host to lower case, escapes normalised, dot segments out.

Not §6.2.3: dropping :80 is true of http and false in general, and this package holds no register of schemes.

The origin of a URL (§4.7): what two URLs have to share to be the same site, written scheme://host:port.

A blob: URL carries the origin of the URL inside it. Everything else that is not a scheme with an authority of its own — file:, data:, mailto: — has an origin no other URL shares, which is written "null".

test {
  let url = @moonurl.Url::decode(
    "https://example.test/a",
    syntax=@moonurl.Whatwg,
  )
  assert_eq(url.origin(), "https://example.test")
  let file = @moonurl.Url::decode("file:///tmp/x", syntax=@moonurl.Whatwg)
  assert_eq(file.origin(), "null")
}

The query as pairs, read the way URLSearchParams reads it (WHATWG URL Standard §5.1).

test {
  let url = @moonurl.Url::decode(
    "https://h/?a=1&a=2&b=x+y",
    syntax=@moonurl.Whatwg,
  )
  assert_eq(url.params().values("a").length(), 2)
  assert_eq(url.params().get("b").unwrap(), "x y")
}

Resolve reference against this base (RFC 3986 §5.2.2, strict).

Strict: a reference that repeats the base's scheme stays absolute. The non-strict reading is for pre-1998 parsers and turns http:g into something it does not say.

test {
  let base = @moonurl.Url::decode("http://example.test/a/b/c")
  assert_eq(
    base.resolve(@moonurl.Url::decode("../d")).encode(),
    "http://example.test/a/d",
  )
  assert_eq(
    base.resolve(@moonurl.Url::decode("//other.test/x")).encode(),
    "http://other.test/x",
  )
}

This URL with its fragment replaced, the leading # of the value optional. An empty value takes the fragment away.

This URL with its host replaced, and its port too if the value carries one.

An empty value is a host with nothing in it, not a missing host: the URL keeps its shape and loses its authority's middle.

This URL with its host replaced and its port left alone.

This URL with its password replaced.

This URL with its path replaced. A URL whose path is opaque — mailto:, data: — has no path to replace, and keeps the one it has.

This URL with its port replaced. An empty value takes the port away, which is not the same as setting it to zero.

This URL with its query replaced, the leading ? of the value optional. An empty value takes the query away.

This URL with its scheme replaced, or this URL if the value is not a scheme.

A URL may change between special and non-special schemes only through another special one, and only a URL with no credentials, port or file host can become file; the setter leaves anything else as it was, as the standard's own setter does.

test {
  let url = @moonurl.Url::decode(
    "http://example.test/a",
    syntax=@moonurl.Whatwg,
  )
  assert_eq(url.set_scheme("https").encode(), "https://example.test/a")
  assert_eq(url.set_scheme("1bad").encode(), "http://example.test/a")
}

This URL with its username replaced.

The domain parser of §3.3: the host name a browser would make of text.

Strict is for a name that has to be one: CheckHyphens, the STD 3 rules and the DNS lengths are all on, and invalid Punycode is refused.

Lenient is the browser: an ASCII string is taken as it stands, lowercased, whatever IDNA would say about the label — a name that reads as Punycode but is not still addresses something. Everything else goes through UTS 46.

test {
  assert_eq(@moonurl.domain("MÜNCHEN.example"), "xn--mnchen-3ya.example")
  assert_eq(@moonurl.domain("xn--a"), "xn--a")
}

Percent-encode octets for kind (RFC 3986 §2.1).

Octets, not text: what travels inside a URI is whatever the caller encoded it as. escape_text is the one that assumes UTF-8.

spare is for percent coding that is not going into a URI at all — gRPC's grpc-message keeps all of printable ASCII but %, which is no component of RFC 3986's. A URI component names its kind and leaves spare alone.

Percent-encode text as UTF-8 (RFC 3986 §2.5).

test {
  assert_eq(@moonurl.escape_text("a b/c", kind=@moonurl.Path), "a%20b/c")
  assert_eq(@moonurl.escape_text("a b/c", kind=@moonurl.Segment), "a%20b%2Fc")
}

Whether every % is a complete escape and nothing outside kind's set is there unescaped — the check unescape does not make.

Percent-encoding normalised (RFC 3986 §6.2.2.2): hex digits in upper case, and escapes of unreserved octets written out plainly.

Take out the . and .. segments (RFC 3986 §5.2.4).

The section's loop rather than a split-and-rejoin: the two differ at exactly the inputs that matter — a trailing .., a path that climbs above its own root, /..//.

Undo percent-encoding (RFC 3986 §2.1).

A % without two hex digits after it stands for itself. This runs on request targets and form bodies, where refusing the whole message over one stray % helps nobody; escaped is for the caller who would rather be told.

Undo percent-encoding and read the result as UTF-8, replacing what is not.

The strictest component, for when nothing says. Escaping an octet that did not need it changes nothing; leaving one that did changes where the component ends.

A thousand pairs of up to 1 MiB each: Starlette's max_fields and max_part_size, and the parameterLimit of Node's qs.

The grammar Url::decode uses when nothing says.

Why a name is not the name the standard would accept.

Shapeat~ : Int32

A label that is not in NFC, begins with a combining mark, or begins with xn-- where that prefix is not allowed (V1, V4, V6).

Hyphenat~ : Int32

A hyphen where check_hyphens forbids one (V2, V3).

Dotat~ : Int32

A full stop inside a label (V5).

Disallowedat~ : Int32, got~ : Char

A code point the mapping table does not allow here (V7).

Std3at~ : Int32, got~ : Char

ASCII outside a to z, 0 to 9 and -, while std3 is on (U1).

Punycodeat~ : Int32

An xn-- label that does not decode, decodes to nothing, or decodes to ASCII alone (P4).

Joinerat~ : Int32

A joiner out of context (C1, C2).

Bidiat~ : Int32

A label the bidi rule refuses (B1 to B6).

Emptyat~ : Int32

An empty label (A4_2, X4_2).

Lengthlimit~ : Int32, got~ : Int32

A name or label past its length limit (A4_1, A4_2).

What UTS 46 leaves to the caller. Its two presets cover the two callers there are: a browser reading a host name out of a link, and a zone file.

transitionalBool

Treat ß, ς and the two joiners the way IDNA2003 did. Deprecated by the standard, and kept because some names still exist in that spelling.

check_hyphensBool

Refuse a label with a hyphen where one may not be: leading, trailing, or in the third and fourth positions of a label that is not an A-label.

check_bidiBool

Refuse a name whose right-to-left labels break the bidi rule of RFC 5893.

check_joinersBool

Refuse a zero width joiner or non-joiner outside the context that allows one (RFC 5892 Appendix A).

std3Bool

Refuse ASCII outside a to z, 0 to 9 and -, the STD 3 rules.

dns_lengthBool

Refuse a name past the DNS limits: 253 octets, 63 to a label.

ignore_invalid_punycodeBool

Pass an xn-- label that is not Punycode through untouched instead of refusing it. The URL standard asks for this: a browser keeps a label it cannot read rather than dropping the host.

A name as far as the standard could convert it, and what it objected to on the way. Conversion does not stop at the first error, so a caller that wants to show the user what is wrong has the rest of the name too.

nameString

The converted name.

errorsArray[Fault]

Everything wrong with it, in no particular order.

Whether the name is one the standard accepts.

Why a label is not Punycode.

Digitat~ : Int32

A character that is not a Punycode digit, at this offset.

Overflowat~ : Int32

A number that runs past the code space, at this offset.

Decode one label (RFC 3492 §6.2).

Both cases of digit are accepted: a name may have been upper-cased on its way here, and case carries nothing in Punycode.

Encode one label (RFC 3492 §6.3).

The label is not split and no xn-- is written: that prefix belongs to UTS 46, which decides when a label needs encoding at all.

test {
  assert_eq(@idna.encode("münchen"), "mnchen-3ya")
  assert_eq(@idna.decode("mnchen-3ya"), "münchen")
}

A name as ASCII, the form that goes on the wire (UTS 46 §4.2).

test {
  let converted = @idna.to_ascii("münchen.example")
  assert_eq(converted.name, "xn--mnchen-3ya.example")
  assert_eq(converted.is_ok(), true)
}

A name as Unicode, the form that is read (UTS 46 §4).

What a host name gets: no STD 3, no hyphen rule (an A-label always has a hyphen in the fourth position), the bidi and joiner rules on, and no DNS length check, which the resolver is better placed to make.

What a zone file gets: everything options turns off, turned on.

Why a template cannot be read, or a value cannot be filled in.

Bracketat~ : Int32

A { that is never closed, or a } that opens nothing, at this offset.

Syntaxat~ : Int32, got~ : Char

A character that may not stand in a template where this one does.

Operatorat~ : Int32, got~ : Char

A reserved operator: =, ,, !, @ or | (RFC 6570 §2.2).

Lengthat~ : Int32, limit~ : Int32, got~ : Int32

A :N prefix that is not one to four digits from 1 to 9999.

Compositename~ : String

A :N prefix on a variable bound to a list or a map, where the standard says the modifier does not apply (RFC 6570 §2.4.1).

A template (RFC 6570 §2), kept as the pieces it was read into.

The variables a URI spells out, in the order the template names them, or nothing if the URI is not one this template could have produced.

A template is a pattern, not a grammar: a value that holds the separator is read as two values. Escapes are undone, so a name comes back as it was.

test {
  let template = @template.Template::decode("/users/{id}{?page}")
  assert_eq(template.capture("/users/42?page=2").unwrap(), [
    ("id", "42"),
    ("page", "2"),
  ])
}

Read a template (§2).

test {
  let template = @template.Template::decode("http://h{/list*}{?q,page}")
  assert_eq(template.encode(), "http://h{/list*}{?q,page}")
}

The template as it was written, the way §2 spells it.

Fill the template in (§3.2).

test {
  let values : Map[String, @template.Value] = Map([])
  values.set("who", @template.Value::Text("moon"))
  values.set("list", @template.Value::List(["a", "b"]))
  let template = @template.Template::decode("{/who}{?list}")
  assert_eq(template.expand(values), "/moon?list=a,b")
}

What a variable holds (RFC 6570 §2.3): one string, a list of them, or a set of named ones.

TextString

A string.

ListArray[String]

A list, whose items are expanded in order.

PairsArray[(String, String)]

An associative array: names with values, expanded in order.

trait Variables {
  fn get(Self, StringView) -> Value?
}

Where the values of a template come from. Implement it for whatever holds them — a map, a configuration record, a request's query.

test {
  let values : Map[String, @template.Value] = Map([])
  values.set("q", @template.Value::Text("moon bit"))
  assert_eq(
    @template.Template::decode("?q={q}").expand(values),
    "?q=moon%20bit",
  )
}

A map of names to values is the obvious place to look.

Why a string is not a data URL.

Scheme

It does not begin with data:.

Comma

Nothing separates the header from the payload (§3 requires a comma).

Escapeat~ : Int32

A % with fewer than two hex digits after it. Only under strict.

Base64at~ : Int32

The payload is not base64, at this offset into it.

A data URL taken apart: what the octets are, and the octets (RFC 2397 §3).

data is already decoded. Which of the two spellings the URL used is a property of the text, not of what it carries.

kindString

The media type with its parameters, as the URL spells it.

dataBytes

The payload.

The charset the payload's text is in (RFC 2397 §2).

US-ASCII when nothing says otherwise, which is the default media type's own parameter rather than a guess of ours.

Read a data URL.

strict turns the two things browsers let pass into errors: base64 with whitespace or the wrong padding, and a % that is not an escape. Left off, a lone % stands for itself and whitespace is dropped.

test {
  let greeting = @dataurl.Url::decode("data:text/plain,Hi%20there")
  assert_eq(greeting.kind, "text/plain")
  assert_eq(greeting.data, b"Hi there")
  assert_eq(greeting.encode(base64=true), "data:text/plain;base64,SGkgdGhlcmU=")
}

Write the URL out.

base64 defaults on, as webpack's asset modules do (generator.dataUrl.encoding). Turn it off for SVG and anything else a human may want to read in the URL.

A media type equal to kind is left out; reading one back puts it there again.

A data URL carrying these octets.

What a data URL means when it names no media type (RFC 2397 §2).