I wrote the same email parser three times, and the hard part was making them fail identically

@mailkite/mail-parse is MIT-licensed, and its test suite is really two conformance proofs stacked together. One implementation generates the truth; the other two are checked against it, once for the emails that parse and once for the emails that break. Here’s both halves at a glance, before any of the code that makes them true:

The reason any of this matters is that inbound email is a swamp. Not the protocol, the content. The moment you accept mail from the open internet you stop getting the tidy RFC 5322 messages from the spec and start getting whatever thirty years of mail clients, marketing tools, and misconfigured servers actually emit: base64 that isn’t padded, a Content-Type charset that doesn’t exist, boundaries that never close, winmail.dat, 8-bit bytes in a header that swears it’s ASCII. A parser that runs on that input has to degrade well, not throw.

import { parse } from "@mailkite/mail-parse";

// bytes in: a Buffer/Uint8Array, a string, or any (async) iterable of chunks (a stream)
const msg = await parse(rawMimeBytes);

msg.from;        // { address, name? }
msg.subject;     // RFC 2047 decoded
msg.text;        // decoded text/plain
msg.attachments; // [{ filename, mimeType, content, size }]
msg.diagnostics; // typed, non-fatal degradations: it never throws on bad input

The same call, and the same Message shape, exists in the Python and Go ports.

Why three languages at all

The honest answer: the email doesn’t get to pick where it lands.

Our inbound path parses raw MIME at the SMTP edge, a Node process on a plain VPS: no CPU cap, the natural home for streaming a 20 MB message straight to object storage without buffering it. But the same parsed-message shape also has to be producible inside a Cloudflare Worker (buffered, a different runtime), and the SDKs developers actually call live in a spread of languages.

If those parsers drift, you get the worst class of bug: an email that produces one JSON shape in the Node path and a subtly different one somewhere else. No stack trace. No crash. Just a support ticket that says “the attachment is missing” for one customer and no one else, with no way to reproduce it, because the email that triggered it is gone.

So “three languages” wasn’t a flex. It was a constraint that forced a discipline: there is one parser, expressed three times, and I need a mechanical way to prove they’re the same.

Making success identical: one golden, generated from the reference

The first half is conformance testing, and the trick is to have a single source of truth rather than three hand-written expectation sets that rot independently.

The TypeScript package is the reference implementation. A script runs it over the 15 gold .eml fixtures (the pathological ones, collected from real breakage) and serializes the full parsed result to a parse_golden.json: subject, from, recipient count, the text body, the HTML body, every attachment’s metadata, and the sorted diagnostic codes.

Then Python and Go each assert field-for-field against that same file:

# tests/test_streaming.py — Python asserts against the TS-generated golden
def test_every_fixture_matches_ts(self):
    for path in sorted(glob.glob(os.path.join(FIXTURES, "*.eml"))):
        name = os.path.basename(path)
        g = self.golden[name]                       # the TS reference's output
        msg = parse(open(path, "rb").read())
        self.assertEqual(msg.subject, g["subject"], "subject")
        self.assertEqual(msg.from_.address if msg.from_ else None, g["from"], "from")
        self.assertEqual(len(msg.to), g["toCount"], "toCount")
        self.assertEqual(msg.text, g["text"], "text")
        self.assertEqual([a.filename for a in msg.attachments],
                         [a["filename"] for a in g["attachments"]], "attachments")
        self.assertEqual(sorted(d["code"] for d in msg.diagnostics),
                         g["diagnostics"], "diagnostics")   # the whole shape

// parse_test.go — the SAME golden JSON, asserted from Go
func TestTsParity(t *testing.T) {
    golden := loadGolden(t, "testdata/golden/parse_golden.json")
    for name, g := range golden {
        msg := Parse(readFixture(t, name))
        if strOrNil(msg.Subject) != strOrNil(g.Subject) {
            t.Errorf("%s subject: got %q want %q", name, strOrNil(msg.Subject), strOrNil(g.Subject))
        }
        // ...from, toCount, text, html, attachments, diagnostics — field-for-field over 15 fixtures
    }
}

The nice property: the golden is generated, not authored. When the reference parser’s behavior changes, the golden regenerates, and the Python and Go tests fail until they match. There’s no world where the three implementations silently diverge on a fixture and everyone’s test suite stays green. The fixtures include TypeScript’s exact 8-bit-header quirks, so “close enough” doesn’t pass.

That covers the emails that parse. It’s the emails that don’t that taught me the real lesson.

Making failure identical: a fingerprint, not a stack trace

When a MIME parser hits something it can’t cleanly handle, the naive move is to throw. That’s wrong for inbound email for two reasons: one broken part shouldn’t sink the whole message, and, more subtly, an exception is a terrible unit of aggregation. Ten thousand deployments hitting the same malformed-boundary bug should be one signal, not ten thousand log lines with slightly different byte offsets.

So the parser never throws on bad input. Every degradation emits a typed diagnostic, and the part worth showing you: a failure signature, a deterministic, PII-free hash of the structural features of what broke.

interface FailureFeatures {
  libVersion: string;
  scope: 'envelope' | 'structure' | 'part'; // header block? assembly? one leaf part?
  diagnosticCodes: string[];                 // e.g. ["BOUNDARY_NOT_CLOSED"], order-independent
  contentType?: string;                      // the offending leaf's declared type
  disposition?: string;
  transferEncoding?: string;
  byteSignature?: string;                    // hex of the leading bytes: STRUCTURE, never content
  headerNames?: string[];                    // header NAMES present, never their values
  mailerFamily?: string;                     // X-Mailer normalized, e.g. "outlook"
  structurePath?: string;                    // "multipart/mixed>multipart/alternative>application/ms-tnef"
}

interface FailureSignature {
  hash: string;                 // fnv1a64(canonicalize(features)): 16 hex chars, the dedup key
  features: FailureFeatures;
  rollup: SignatureRollup[];    // coarser hashes: "a whole scope is failing" vs a precise group
}

Notice what’s not in there: no subject, no addresses, no body bytes. The signature describes the shape of a failure (a base64 attachment that won’t decode, an HTML part with a bogus charset, a winmail.dat at a particular position in the tree) using only structural facts. That’s what makes it safe to emit from a library running on other people’s mail: the fingerprint can leave the box, the email never does.

And because it’s a pure hash of canonicalized structural features, the same broken email produces the same hash everywhere. That’s the second conformance proof, the one that actually matters. The expected hashes are captured from the TypeScript computeSignature() and asserted verbatim in Python and Go:

# tests/test_signature.py — hashes captured from the TS reference, asserted in Python
def test_part_tnef_signature_matches_ts(self):
    sig = compute_signature({
        "scope": "part",
        "diagnosticCodes": ["BOUNDARY_NOT_CLOSED"],
        "contentType": "application/ms-tnef",   # a winmail.dat leaf
        "transferEncoding": "base64",
    })
    self.assertEqual(sig["hash"], "f55154fda8f2cadd")           # byte-identical FNV-1a-64
    scope = next(r for r in sig["rollup"] if r["level"] == "scope")
    self.assertEqual(scope["hash"], "c3c0a940c0ea88e6")         # and the coarser roll-up

I picked FNV-1a (the 64-bit variant, a 16-hex-char digest) deliberately: a handful of lines, no dependencies, trivially portable, so “compute this hash” means the same thing in three languages without pulling in a crypto library or hoping two implementations of something fancier agree on edge cases. The canonicalization is the real work (lowercase the types, strip noisy params, bucket byte-signatures to known magic numbers, reduce the mailer to a family, sort the codes so order doesn’t matter); the hash is just the cheap, deterministic seal on top.

The payoff: a malformed email that breaks in the Node edge and the same email replayed through the Python SDK don’t just both fail, they report the same signature, land in the same dedup bucket, and (once a threshold is crossed) file one GitHub issue with a structural description precise enough to write a regression test from. Cross-language observability falls out of cross-language determinism for free.

Where the off-the-shelf parsers fit (honest alternatives)

If you only live in one language and don’t need the cross-runtime fingerprint, reach for the mature single-language option first:

• Node: mailparser or postal-mime (we lean on postal-mime for our own Workers-side build).
• Python: the capable email package in the standard library.
• Go: net/mail plus mime/multipart for the structure.

Any of them will turn a well-formed message into fields with little or nothing to vendor.

What none of them give you is the thing this post is about: the same parsed shape and the same failure fingerprint across all three runtimes. That guarantee only earns its keep if, like us, you parse the same mail in more than one place and need a bug in one to be provably the same bug in the others. If you don’t, a stdlib parser is the right call, and I’d tell you to use it.

The thing I’d tell past-me

Two lessons, both slightly counterintuitive:

1. Generate your conformance oracle; don’t hand-write it per language. One reference implementation plus a generated golden beats three lovingly-maintained expectation files that drift the day you’re not looking. The languages get to have different idioms internally (streams in Node, synchronous middleware in Python and Go) as long as they’re forced through the same external truth.

2. Design your failures to be aggregatable, and you get portability and privacy as side effects. The instinct is to make errors rich: full context, the offending bytes, a stack trace. For a library that runs on data you’re not allowed to see, the opposite is right: make errors structural and hashable. A PII-free fingerprint is the thing you can compare across languages, dedup across deployments, and safely emit from someone else’s process. Determinism is what makes it identical across three parsers; structure-only is what makes it safe to emit at all.

The parser is MIT-licensed and lives here: Node/TS, Python, Go. It grew out of building MailKite, which we build (inbound email turned into a webhook), and that’s where the appetite for “the same broken email must behave the same everywhere” came from. But the parser stands on its own: if you just need to turn MIME into clean, typed JSON without a service in the loop, take it and ignore the rest of us.

If you’ve fought the email-content swamp, I’d genuinely like to hear which message finally made you write your own parser. Mine was a winmail.dat inside a multipart/mixed that three off-the-shelf libraries each mangled a different way. In our signature scheme that failure has a name now: f55154fda8f2cadd. It’s the same on all three parsers, and I’ll never forget it.

Related: Receiving email is the part nobody warns you about on why the inbound content swamp is the hard direction, and Handling email attachments without losing the £ on the charset-and-encoding half of the same problem.