Gonemaster Packet Cache File Format

This document specifies the on-disk format produced by gonemaster --save and consumed by gonemaster --restore. It covers:

• the JSON schema,
• the checksum contract,
• how each cache kind is represented,
• strict vs. lenient parsing behavior.

The format is implemented by engine/cachefile .

Compression

Cache files can optionally be stored as a single gzip stream wrapping the same JSON document. The CLI selects compression in two ways:

• gonemaster --save FILE.gz - any save path with a case-insensitive .gz suffix is gzip-compressed automatically.
• gonemaster --save FILE --save-compress - explicit opt-in; useful when the file name does not end in .gz.

--restore does not need a matching flag: it sniffs the leading two bytes of the file (1f 8b, the gzip magic per RFC 1952) and decompresses transparently. Files saved without compression are read as plain JSON.

Limits

gonemaster --save FILE --save-max-entries N (or cachefile.WithMaxEntries(n)) refuses to write the file if the exported cache would contain more than N entries, counted across all kinds. The check runs before the file is written, so a rejected save never leaves a partial file behind. The default of 0 disables the guardrail.

Performance and scaling

Save and restore are both linear in the number of entries. The cachefile benchmarks (go test ./engine/cachefile -bench BenchmarkCachefile) give representative figures for a nameserver-only cache on an Apple M4:

On-disk size is roughly 150-400 bytes per entry for plain JSON (a typical cached response base64-encoded), so 100,000 entries is on the order of 15-25 MB uncompressed. gzip compresses this several-fold; the exact ratio depends on how many distinct responses the cache holds (many identical responses compress dramatically, a diverse cache much less), and it adds roughly a third to save time.

The practical ceiling is memory, not CPU or disk: Restore decodes the entire file and materializes every packet in memory before the run starts, so peak memory scales with the file size (restoring 100,000 entries allocates on the order of hundreds of MB). For very large caches, prefer --save-compress for the on-disk footprint and --save-max-entries to bound growth, or split the cache across multiple files.

File layout

A single JSON object:

{
  "format": "gonemaster.packet-cache",
  "version": 2,
  "checksum": "<lowercase sha-256 hex>",
  "entries": [ ... ]
}

(*) checksum is always written by --save and always verified when present. It is optional only so that hand-crafted fixtures can omit it; omission is a warning in lenient mode and an error in strict mode.

Checksum contract

The checksum covers the entire file with the checksum field itself set to the empty string, serialized using encoding/json.Marshal (no indentation, Go struct field order). It is lowercase hex SHA-256.

Verification rules:

• Present and correct → file accepted.
• Present and incorrect → Import / Restore return an error (“checksum mismatch”), regardless of mode.
• Absent → warning in lenient mode; error in strict mode.

The pretty-printed form produced by Save (indented JSON with a trailing newline) includes checksum, but the checksum value itself is computed against the canonical, non-indented, checksum-stripped form. Consumers that re-serialize an already-validated file must recompute the checksum if they want to preserve roundtripping.

Entries

Every entry carries a kind discriminator that selects the remaining fields:

• "nameserver" - one cached response for a specific nameserver IP.
• "recursor" - one cached response for the internal recursor.
• "asn" - one cached ASN lookup result for a queried IP.
• "axfr" - one cached zone-transfer result for a nameserver IP.

Unknown kind values are warnings in lenient mode (the entry is skipped) and errors in strict mode.

kind: "nameserver"

{
  "kind": "nameserver",
  "address": "192.0.2.53",
  "key": "example.com./A/IN",
  "answer_from": "192.0.2.53:53",
  "message": "<base64-encoded DNS wire packet>"
}

A cached empty (nil) response is encoded with "no_message": true and no message:

{
  "kind": "nameserver",
  "address": "192.0.2.53",
  "key": "example.com./A/IN",
  "no_message": true
}

kind: "recursor"

{
  "kind": "recursor",
  "name": "example.com",
  "qtype": "A",
  "qclass": "IN",
  "nameservers": [
    { "name": "ns1.example", "address": "192.0.2.1" },
    { "name": "ns2.example", "address": "2001:db8::1" }
  ],
  "message": "<base64-encoded DNS wire packet>"
}

An empty nameservers array (or omission) denotes a lookup that started from the root servers.

The recursor cache only stores non-empty responses; there is no no_message form for recursor entries.

kind: "asn"

{
  "kind": "asn",
  "ip": "192.0.2.10",
  "asns": [64496, 64497],
  "prefix": "192.0.2.0/24",
  "raw": "64496 64497 | 192.0.2.0/24 | US | arin | 2001-01-01",
  "code": "AS_FOUND"
}

ASN entries are stored on hit and on miss (EMPTY_ASN_SET, ERROR_ASN_DATABASE), so a restored cache avoids both successful and failed lookups.

kind: "axfr"

An AXFR result is a stream of resource records rather than a single message, so it has its own shape. An available transfer stores the ordered records in rrs; each element is a base64 (std) single-record DNS message (the fork has no standalone RR codec).

{
  "kind": "axfr",
  "address": "192.0.2.53",
  "name": "example.com",
  "qclass": "IN",
  "rrs": ["<base64 single-record DNS message>"]
}

A refused or errored transfer is stored with no_transfer and no rrs, so a restored run reproduces AXFR_FAILURE offline:

{
  "kind": "axfr",
  "address": "192.0.2.53",
  "name": "example.com",
  "qclass": "IN",
  "no_transfer": true
}

An entry must carry either rrs or no_transfer, never both and never neither. Only the records streamed by the consumer are captured, so an available entry typically holds just the leading SOA.

axfr is an additive kind: version stays 2, and older binaries skip unknown-kind entries in lenient mode.

Strict vs. lenient parsing

Import, Restore, and Load accept functional options:

• WithStrict() - turn every non-fatal warning into an error.
• WithWarnf(fn) - route non-fatal warnings to a callback.

On the CLI, --cache-strict selects WithStrict() for both --restore and --cache-stats.

In lenient mode (default) the following produce warnings but keep processing:

• Missing checksum.
• Unknown top-level field (e.g., a future stats section).
• Unknown field inside an entry.
• Entry with an empty or unknown kind (the entry is skipped).

In strict mode the same conditions are hard errors.

Any base64 decode failure, DNS unpack failure, invalid IP address, version mismatch, or checksum mismatch is always a hard error regardless of mode. So is an axfr entry that carries both rrs and no_transfer, or neither.

Minimal example

{
  "format": "gonemaster.packet-cache",
  "version": 2,
  "checksum": "0000000000000000000000000000000000000000000000000000000000000000",
  "entries": [
    {
      "kind": "nameserver",
      "address": "192.0.2.53",
      "key": "example.com./A/IN",
      "answer_from": "192.0.2.53:53",
      "message": "PQ4BAAABAAEAAAAAB2V4YW1wbGUDY29tAAABAAE="
    },
    {
      "kind": "recursor",
      "name": "example.net",
      "qtype": "NS",
      "qclass": "IN",
      "nameservers": [
        { "name": "ns1.example", "address": "192.0.2.1" }
      ],
      "message": "E+6BAAABAAAAAAAAB2V4YW1wbGUDbmV0AAACAAE="
    },
    {
      "kind": "asn",
      "ip": "192.0.2.53",
      "asns": [64496],
      "prefix": "192.0.2.0/24",
      "code": "AS_FOUND"
    }
  ]
}

Replace the placeholder checksum with the real SHA-256 when writing a file by hand, or let gonemaster --save produce it.