noblogs-analysis

Scripts for analyzing Browsertrix WACZ archives of Noblogs websites.

/!\ WARNING /!\
This has been LLM generated and I did not spent much time to check the code. I did run it against wacz files I have and did not notice any wrong doing.
But use this a your own risk!
I'm not responsible for anything that happens when using this code.
/!\ WARNING /!\

Find missing embedded resources

find_missing_embedded_resources.py detects resources that were referenced in archived HTML but were not captured in the WACZ. It is intended to find cases where Browsertrix missed a PDF or another file because it appeared only in an <object data="..."> or <embed src="..."> element rather than in a normal <a href="..."> link.

The script uses only the Python standard library and does not extract the WACZ files to disk.

Basic usage

Scan all WACZ files directly inside one directory:

./find_missing_embedded_resources.py /path/to/archive-directory

Scan a directory and all its subdirectories:

./find_missing_embedded_resources.py --recursive /path/to/archives

The script prints the current directory and WACZ filename as it works, including an overall file counter. This also applies when the shell expands multiple directory arguments, for example:

./find_missing_embedded_resources.py /path/to/archives/seg_a*

In an interactive terminal, progress is updated in place on a single line. If standard error is redirected to a file or pipe, each WACZ gets a separate line so that the resulting log remains readable. Progress is written to standard error, so it remains visible without contaminating redirected JSON output.

You can also pass individual files or multiple paths:

./find_missing_embedded_resources.py site-one.wacz site-two.wacz /path/to/more-waczs

By default, only affected WACZ files and same-origin resources are displayed. For example:

AFFECTED noblogs-acabb.wacz (1 missing of 1 embedded references)
  missing: https://acabb.noblogs.org/files/2025/08/acabb-poster-v1.pdf
    found in: https://acabb.noblogs.org/materials/ as <object data>

Scanned 1 WACZ(s): 1 affected, 0 error(s).

Options

Show unaffected archives as well:

./find_missing_embedded_resources.py --recursive --show-clean /path/to/archives

Include missing resources hosted on external websites:

./find_missing_embedded_resources.py --recursive --include-external /path/to/archives

Also check <iframe src="..."> references. Iframes are normally requested by the browser, so this check is optional:

./find_missing_embedded_resources.py --recursive --include-iframes /path/to/archives

Produce a machine-readable report:

./find_missing_embedded_resources.py --recursive --json /path/to/archives > missing-embeds.json

Run ./find_missing_embedded_resources.py --help for the complete command-line reference.

How detection works

For every WACZ, the script:

1. Reads captured URLs from its CDX index.
2. Streams the HTML responses stored in its compressed WARC files.
3. Extracts URLs from <object data> and <embed src> elements.
4. Reports embedded URLs that have no corresponding capture.

Each occurrence is reported separately. The same missing resource may therefore appear more than once when it was embedded by multiple archived pages.

The script can only identify references present in the archived HTML. It cannot detect resources added to the live website after the crawl, or elements created only in a page's final JavaScript DOM when that markup was not stored in the WARC response.

To make Browsertrix discover these resources in a future crawl, retain its default link selector and add selectors for embedded objects:

--selectLinks 'a[href]->href' \
--selectLinks 'object[data]->data' \
--selectLinks 'embed[src]->src'