Build log · Software & apps

North Korea Web Archive — 37 Sites Recovered

Software & appsdone1 versions37 sites84,841 URLs428 MBDNS forensics
▶ Browse the archive▶ Pyongyang Times📄 Raw spec (.md)📁 Portfolio

reflection not written yet · every fact below is generated from this project’s own spec and files

What it is

The entire .kp country domain stopped resolving, so 37 North Korean websites were rebuilt from the Wayback Machine — 84,841 URLs indexed, 1,255 pages, 428 MB, browsable offline. Includes the DNS forensics that explained why a whole TLD vanished and the three crawler bugs the rebuild exposed.

Version history

Straight from the project’s own card data — what each version added.

V1

37/44 hosts mirrored from the Wayback Machine; 1,255 pages, 5,752 files, 428 MB. Portal + per-site browse, verified by re-parsing 13,141 local references.

Files on the site

What broke, and what I’d change

Not written yet. This is the part only I can write — what actually went wrong, and what I’d do differently. It lives in builds/reflections/nk-archive.md (see _TEMPLATE.md); as soon as that file has words in it, this section fills in on the next build.

Everything else on this page — the spec, the version history, the file counts — is generated from the project’s own build notes and files, so nothing here is guesswork.

Build spec

The project’s own build notes, rendered. Open the raw .md ↗

Show the spec

North Korea Web Archive — build spec

Goal

Build a browsable offline copy of every publicly known website in the DPRK, starting from a single site that would not open (airkoryo.com.kp). Publish the list plus the captured HTML to the portfolio.

What the failure actually was

airkoryo.com.kp returned a name-resolution error — which reads like "the server is down". It was not: curl --resolve www.airkoryo.com.kp:80:175.45.176.69 returned 200 OK, 13,110 bytes, <TITLE>Air Koryo</TITLE> in ~25 s at ~0.5 KB/s. The website was alive the whole time; only the lookup was broken.

The root zone still delegates kp. → ns1.kptc.kp / ns2.kptc.kp (175.45.176.15 / .16), which sit inside the same /22 as the websites. Probing them directly:

  • UDP/53 → timeout (filtered)
  • TCP/53 apex → NOERROR
  • TCP/53 child (kcna.kp, naenara.com.kp, rodong.rep.kp) → SERVFAIL
  • Cloudflare DoH → EDE(22): No Reachable Authority at delegation kp

Both authorities live inside the link that closes, so when it closes every .kp host vanishes at once. That is why it presented as "the country's internet went down" rather than "one site broke".

Pipeline

  1. Enumerate — no zone transfer is possible, so hostnames come from three sources: public

    2. DPRK-site trackers, passive-DNS history (api.mnemonic.no), and institutions appearing in archived links. Result: 44 candidate hosts.

  2. Inventory — query the Wayback CDX API per host and write inventory.json incrementally so

    3. the job is resumable. Critical: use matchType=domain, not a prefix match, or every host archived under www. reports zero captures (cost 24 false "never archived" results on the first pass).

  3. Mirror — breadth-first from each site's real front page, same-host only, depth 2, 45-page

    4. cap, up to 220 assets fetched 3-way parallel. Seeds must include /, /index.html, www.-prefixed and :80 variants plus every netloc the CDX knows, because archives store www.host:80/-style URLs.

  4. Rewrite — strip the web.archive.org/web/<ts>id_/ wrapper, then relativise every

    5. root-absolute reference so the mirror browses from file:// with no server. Missing assets fall back to the archive URL.

  5. Verify — re-parse every saved page and resolve each local reference against the

    6. filesystem. Never eyeball it.

Three bugs worth remembering

| Symptom | Cause | Fix |
|---|---|---|
| 45 pages, 1 file on disk | article URLs are index.php?newsID=…; the query was truncated to 36 chars, so all IDs collapsed onto one filename | hash-suffix the full query → 1 file became 79 |
| sites with no CSS/images | host check compared raw netloc; archived URLs are www.host:80, never equal to host, so every asset was rejected as off-site | compare .hostname, never the netloc |
| 10 sites "never archived" | archive.org signals a rate-limit with TCP connection refused, not HTTP 429; the logger read that as an empty archive | treat connect-failure as throttle: 20–90 s backoff, 4 tries |

The third produced a confidently wrong conclusion about sites among the largest in the set (Pyongyang Times, Kim Chaek University of Technology). After the fix: failed URLs per site 198 → 6, coverage 27 → 37 hosts.

Result

  • 37 of 44 hosts mirrored; 7 have no capture anywhere (kiyctc, knic, cooksfriends,

    • korelcfund, ryongnamsan.edu, silibank, star-co).

  • 84,841 URLs indexed, 1,255 pages, 5,752 files, 428 MB.
  • Link resolution 9,698 / 13,141 (73.8%); the rest are links to pages beyond the 45-page cap.
  • Four sites render unstyled — their CSS genuinely 404s in the archive (verified, not assumed).
  • Live door: 14/14 probes to 175.45.176.0/22 on 80/443 refused.

Deliverables

  • demos/nk-archive/index.html — the hub list, generated from the manifests (no hand-typed

    • numbers).

  • demos/nk-archive/sites/<host>/ — the 37 mirrored trees, browsable offline.
  • demos/nk-archive-analysis.html — long-form write-up: DNS forensics, method, bug table,

    • limits, evidence.

  • This spec.

Verification performed

  • Hub and analysis page rendered at 390 / 768 / 1440 px: no horizontal overflow, tables scroll

    • inside their own container.

  • Every hub row link fetched → 200 against the local static server.
  • Live: hub, analysis page, a sample of site trees, and the card markers.

If this is ever rerun

Delete a site's _manifest.json to force a redo. Raise MAX_PAGES for deep mode. Keep THREADS=3 — higher concurrency re-triggers archive.org's throttle and nets out slower.