How to Automate Snapshot Downloads Using PHP

Kaudo

Manually clicking through Wayback Machine snapshots works - if you're after just one page. But if you're auditing an entire domain, collecting evidence, or reconstructing digital history across dozens or hundreds of URLs, bookmarks and browser tabs won’t cut it.

This is where PHP steps in.

Yes, the same PHP that's run half the internet for decades can also help you build a lightweight, repeatable snapshot downloader. No frameworks, no extensions - just a few lines of server-side code to query archive.org, fetch the versions you need, and save them locally.

It’s not scraping just for fun. It’s controlled digital preservation - on your own terms.

Let’s walk through how it works, and what kind of work it can support.

Why Automate Snapshot Retrieval?

Automation pays off when you’re:

  • Tracking how a site has changed over time

  • Recovering content from a dead or expired domain

  • Creating an offline copy of someone else’s public web presence (for research or OSINT)

  • Feeding content into an investigation, timeline, or profile reconstruction

All of these require more than just casual browsing. They require structure - and archive.org’s CDX API gives you structured access to all available snapshots for a given domain.

Core Workflow with PHP

Here’s the basic approach using plain PHP:

  1. Query the CDX API for a list of archived captures.

  2. Parse the returned JSON to extract timestamps and original URLs.

  3. Use file_get_contents() or cURL to download each snapshot.

  4. Save the results locally as .html files.

Example PHP Script:

 
<?php
$domain = "example.com";
$cdxUrl = "https://web.archive.org/cdx/search/cdx?url={$domain}&output=json&filter=statuscode:200";

$response = file_get_contents($cdxUrl);
$data = json_decode($response, true);

// Skip header row
array_shift($data);

foreach ($data as $snap) {
    $timestamp = $snap[1];
    $originalUrl = $snap[2];
    $snapshotUrl = "https://web.archive.org/web/{$timestamp}id_/{$originalUrl}";

    $html = @file_get_contents($snapshotUrl);
    if ($html !== false) {
        $filename = "{$domain}_{$timestamp}.html";
        file_put_contents($filename, $html);
        echo "Saved: $filename\n";
    } else {
        echo "Failed to fetch: $snapshotUrl\n";
    }
}
?>

This will give you a clean archive of full HTML pages - one file per snapshot, with readable filenames. You can then expand the script to extract content, filter for certain types of URLs, or even log metadata.

Filtering and Optimization

Not all captures are useful. Some are redirects, error pages, or duplicates. The CDX API supports filters to reduce noise:

  • filter=statuscode:200 for valid pages

  • filter=mimetype:text/html for real page content

  • collapse=digest to avoid identical snapshots

You can also add limit= or from= parameters to scope the results down to a specific year or range.

This kind of filtering is especially helpful when you’re doing SEO audits, backlink tracking, or context verification - where one subtle change can shift meaning over time.

Going Beyond the HTML

With a little more effort, your PHP script can go further:

  • Store timestamps and URLs in a database for later reference

  • Extract <title>, headers, or meta tags for analysis

  • Compare snapshots to track updates or removals

  • Integrate with NLP tools or machine learning pipelines (yes, even in PHP)

This becomes especially useful in OSINT workflows. When you're building narratives from archived bios, public statements, or forum posts, automation turns scattered fragments into timelines with substance.

Respect Archive.org

Remember, archive.org is a public service. Don’t hammer it with hundreds of requests per second.

Add small delays using sleep(1) or similar functions. And always be mindful of the ethical implications of what you’re archiving and why. Just because it's online doesn't mean it's fair game for misuse.

PHP Still Holds Its Own

People joke about PHP, but the truth is - it’s perfectly capable for this kind of task. It handles HTTP requests, text processing, and file I/O just fine. And if you’re already working in a PHP environment, there’s no reason to switch languages just to automate archive queries.

So go ahead - build your own local time machine.

You should also read: