Same layout, new words

Comparing two PDFs means choosing what “the same” means: the same bytes, the same text, the same pixels or the same look. Each has public tools, and each tool passes something another one fails. I ran six of them on pairs of 10-page documents with one kind of change each, from a re-export with nothing changed to a stamp on one scanned page, and measured perceptual hashes and pdfcmp at sizes from 1 to 100 pages. Perceptual hashes, the tolerant option, record where a page’s text sits and little of what it says: a page with 24 of its 25 lines rewritten still scored 0.85 to 0.94, and in documents of 5 pages or more the mean of the page scores hid it. pdfcmp, my Go library, scores each page by its hashes and by a MinHash of its words, and lets the worst page decide. It also calls a page different when its text or a small tile of it changed. It gets every pair below right, but I wrote both it and the pairs, so that shows what they cover, not a ranking. On documents with text it was faster than diff-pdf, pdf-visual-diff and pdf-diff at every size; on scans, which it has to render, diff-pdf and pdf-diff were faster.

Which pages changed

The question that started pdfcmp is how a document’s content changes from one version to the next: a contract after review, a report regenerated with new numbers. Whether two files are equal is the easy part. The useful answer is which pages changed and by how much.

Long documents make that expensive. A pixel diff renders every page of both files, and on 100 pages that took diff-pdf 1.8 s and pdf-visual-diff 76 s. A byte hash is instant and useless after a re-export: every pair below whose pages all match still differs in bytes, because the creation dates do.

Summarising with an LLM is where knowing which pages changed pays. A document too long for one prompt gets split, usually by page or by section, and each piece is summarised on its own. When the next version arrives, the summaries of unchanged pages can be kept and only the changed pages sent again: one call instead of 100 when one page of a 100-page document changed. A provider’s prompt cache doesn’t do this for you. Anthropic’s needs “100 percent identical prompt segments” up to a cache breakpoint, so a changed page invalidates every cached prefix that contains it, and an entry lives 5 minutes by default (prompt caching). Caching across versions needs a key per page. When a page’s summary comes from its text alone, an exact hash of the extracted text is that key, and a similarity score decides whether a page that changed is worth summarising again. A summary that reads a chart or an image needs the look in its key too, and a scanned page has no text, so its image is the only thing to compare.

Four meanings of “the same”

diff-pdf has 4,335 stars on GitHub. pdf-visual-diff, the snapshot library for Jest, was downloaded 155,640 times from npm in the month to 3 November. JoshData’s pdf-diff has 527 stars and 1,798 downloads a month from PyPI. pdftotext comes with poppler. diff-pdf-visually, at 33,518 PyPI downloads a month, was going to be a seventh, but with ImageMagick 7.1.2 and 6.9.13 it calls every pair different, identical renders included, because it reads ImageMagick’s normalised PSNR as decibels (issue #20).

What each tool misses

The test documents are generated A4 PDFs: 25 lines of 9 or 10 words per page from a 64-word vocabulary, with line 13 an invoice total. Only the middle page changes. One more pair is the same document rewritten by Ghostscript’s pdfwrite, which changes the bytes and the content streams and keeps the text and the look. Four come from outside that set. One is pdfcmp’s own test documents with their shapes redrawn from another seed under the same text. The others start from ten pages of generated photo-like images saved as JPEG at quality 92: the same images at quality 60, as one scan saved at two settings might be, and the quality 92 file with page 5 swapped for another image, or with a 120 × 60 pixel stamp drawn on it. Every tool ran with its defaults: diff-pdf compares pixels at 300 DPI, pdf-visual-diff renders with pdf.js at 144 DPI and diffs with Jimp at zero tolerance, pdf-diff diffs the text layers word by word, and pdfcmp calls a document different when a page’s text changed, a tile of a page changed, or its score is below 0.95.

Every tool but cmp passes the pair that differs only in bytes. The pixel tools fail the Ghostscript rewrite, whose pages render a few grey levels apart, and the resaved images: a visual regression test wants that, and for two saves of one document it’s the wrong answer. Those are their defaults. diff-pdf with --channel-tolerance=30 passes the resaved images and still catches the stamp and the swap, for 9 of 10, but it fails the rewrite at tolerances of 0, 10 and 30. At 30 the rewrite still differs by up to 4,948 pixels on a page and the digit by 252, so no per-page pixel allowance passes the one and catches the other. The text tools catch every change to the words and pass everything without them: the redrawn shapes, the swapped image and the stamp. pdfcmp’s scores for the digit and the stamp, 0.95 and 0.99, are not below the threshold. Both still count as different, by the text rule and the tile rule described below. pdfcmp skips rendering a page only when its text, its object count and a hash of every object’s type, bounds, colours and image data all match. That check is most of why it’s fast on documents with text. The redrawn shapes fail it and get rendered.

Wall time from hyperfine on an Apple M2 Pro at a load average of 1.6 to 4.2. Each cell is the mean of as many runs as fit in 3 seconds after a warmup, at least 3 and at most 10:

pdfcmp runs PDFium compiled to WebAssembly, and compiling the module costs about 0.9 s. pdfcmp keeps the compiled module in a cache directory, so only the first run pays: with the cache emptied before each of 20 runs the 1-page pair took 890 ms, and with it filled 44 ms. A program that uses the library compiles it once per process, and there the 100-page pair with the same content took 214 to 216 ms, one run at each of three DPIs. pdfcmp is faster on the rewritten 100-page pair than on the unchanged one because the changed page is in its sample, and failing it ends the comparison. On the scans no page has text that could settle it without rendering, so pdfcmp renders all ten pages of both files, and diff-pdf and pdf-diff were faster; pdf-diff found no words to compare. pdftotext | cmp was the fastest on every pair.

A rewritten page still looks like itself

dHash shrinks a page to 9 × 8 grey pixels, compares each pixel with the one to its right and records which is darker. pHash shrinks it to 32 × 32, takes the discrete cosine transform and keeps the 8 × 8 lowest frequencies. Neal Krawetz’s posts on pHash and dHash describe both as ways to match a picture that has been resized or recoloured, and read a dHash distance of 1 to 10 bits as “potentially a variation” of it. A 72 DPI render of an A4 page is 595 × 842 pixels, so each of pHash’s 32 rows averages about 26 rows of pixels, and the lines on the test pages are 9 mm, or 25.5 pixels, apart. A row of the grid is about one line of text, and a rewritten line from the same vocabulary puts about as much ink in that row as the line it replaced. On the page below, rewriting 24 of its 25 lines moved dHash by 10 bits, inside Krawetz’s range for a variation, and pHash by 4.

Scored by the hashes alone, 1 minus the share of their 128 bits that differ, the changed page came out as below, over 1, 5, 10, 30 and 100 pages at 72, 150 and 300 DPI: 15 cells per change, one run each, because the documents and the hashes are deterministic.

A small mark on a scan is lost the same way. The 120 × 60 pixel stamp covers 0.7% of its 840 × 1188 page, and the page still scored 0.992. So pdfcmp also cuts every page it renders into a 64 × 90 grid of tiles, compares each tile’s mean grey level between the two files, and calls the page different when one tile moved by more than 6 levels. The stamp moved its tile by 120 levels or more and the swapped image by 87. The JPEG re-saves moved no tile by more than 2.2 and the Ghostscript rewrite none by more than 2.7. Over four seeded scans, each re-saved at three pairs of JPEG qualities and stamped four ways, the closest a re-save came to the line was 1.6 levels and the closest a stamp came was 81. Like the text rule below, a changed tile decides the verdict and leaves the score alone. Used as a score, it would rank the stamp below a page rewritten from top to bottom. A page whose text changed is left to the text rule, so --text-tolerant, which lets small text changes pass, isn’t undone by the ink those changes move.

Averaging hides the page

A document of many pages needs one score, and the obvious one is the mean of its pages. A mean over n pages divides one page’s drop by n:

The lowest page score doesn’t shrink with the document. pdfcmp scores a document by its worst page:

// worstSimilarity returns the lowest similarity among page results: the
// document is only as similar as its least similar page.
func worstSimilarity(results []PageResult) float64 {
  if len(results) == 0 {
    return 1.0
  }
  worst := results[0].Similarity
  for _, r := range results[1:] {
    worst = math.Min(worst, r.Similarity)
  }
  return worst
}

With the hashes alone, the mean catches the 24-line and the 12-line rewrites only in the one-page documents, 3 cells of 15 each. The worst page catches the 24-line rewrite in all 15 and the 12-line one in 13. Neither catches a rewritten line or a changed digit, because there the hashes barely move.

Reading the words

pdfcmp extracts each page’s text anyway. For MinHash, the text is split into words, and every run of three consecutive words, a shingle, is hashed to 64 bits. Each of 128 seeded hash functions then keeps the smallest value it sees over the page’s shingles:

for _, s := range shingles {
  for i, seed := range minHashSeeds {
    if h := mix64(s ^ seed); h < m[i] {
      m[i] = h
    }
  }
}

The Jaccard similarity of two pages is the number of shingles they share over the number in either. Two pages agree on a slot with probability equal to it (Broder, 1997), so the fraction of the 128 slots that agree estimates it. pdfcmp scores a page whose text differs by its hashes times 0.95, capped by that estimate:

How many of the 15 cells each way of scoring calls different:

A changed digit is where any similarity score runs out. It touches 3 of a page’s 235 or so shingles, so the true Jaccard similarity is 0.975, and with 128 slots the estimate’s standard error is √(0.975 × 0.025 / 128) ≈ 0.014: a changed total is within noise of an unchanged page, and it scores 0.935 to 0.950. So the score doesn’t decide whether the text changed. A page whose extracted text hashes differently is different whatever it scores, and the score says by how much: 0.95 for the digit, 0.0078 for the 24 rewritten lines. --text-tolerant hands the verdict back to the score, for a caller who wants a new date in a footer to pass. Pages with fewer than 60 distinct shingles on both sides, about six lines, get no estimate, because there one changed word would move it by more than 10%. Shingling two pages and comparing their signatures takes 105 to 110 µs (go test -bench, 10 runs, two 25-line pages, Apple M2 Pro).

The other common text signature is SimHash, which Google used for near-duplicate web pages. I built it next to MinHash. SimHash estimates a different measure, the cosine similarity of shingle counts: a shingle that occurs ten times votes ten times, so repeated template text outweighs whatever changed around it. On pdfcmp’s templated test pages, one sentence ten times over with a different value at the end of each line:

SimHash estimates its measure well, and the measure calls these pages 96% alike. One rewritten line in the test documents leaves the cosine at 0.950 to 0.954, on the threshold, where the Jaccard similarity is 0.905 to 0.912. Generated invoices and reports are templates, so MinHash went in.

pdfcmp runs all of this cheapest first: a hash of the bytes, the page counts, the first and last pages at 72 DPI, then √n + 2 pages including the middle one, and every page only when that sample passes. A first or last page below 0.5 ends it, and so does a sampled page below the threshold or one whose text or a tile changed. So by default the score is the worst page pdfcmp compared, and a document with several changed pages can come back naming only the first one found. --sampling all compares every page, which is what a per-page cache needs.

Pick the meaning before the tool. For a cache of summaries made from text, the key is each page’s exact text, and similarity only decides whether a changed page is worth paying for again. For two saves of one scan the meaning is the look, which perceptual hashes and a grid of tile means judge and a pixel diff at its defaults can’t. For a page of text a perceptual hash sees the layout, so the words have to be compared too, and a document is only as similar as its least similar page.