Skip to content

Distilling 50,900 JSON files down to what actually matters

When we migrated Chronicle of Philanthropy from Brightspot to WordPress, the content came over — but not cleanly. Months later, we were still tracking down missing articles that return 404 errors, content with broken formatting, and taxonomy gaps where articles never got associated with the right issues, packages, or series. The problems were real and the complaints were coming in regularly, but diagnosing them at scale was nearly impossible.

The root issue: we had one way to interrogate the source data, and it wasn’t good. A custom-built tool that a single developer maintained — every query or data pull meant going to him, and every change to what we needed meant more overhead on his end. Brightspot had given us a full export — 50,900 JSON files, one per piece of content — but in a form that was nearly unusable for direct investigation. Each file was essentially a full server-rendered page object: navigation menus, ad slot configurations, paywall config, SEO boilerplate, social widgets, authentication state. The actual content — headline, body, authors, tags, publication metadata — was buried somewhere inside all of that.

Parsing or querying these files in that state meant wading through noise on every read. Writing scripts against them was painful. Spotting patterns across thousands of files was out of the question. Before we could answer questions like “which articles are missing their issue association?” or “which pieces of content have malformed body blocks?”, we needed to get the files down to just the content.

That cleanup had been on the backlog for about six months. Then I spent an afternoon with Claude and it was done.

The problem with the raw export

Brightspot renders pages server-side and its JSON exports reflect that — each file includes everything the frontend needs to display it, not just the editorial content. For a content audit or migration investigation, that distinction matters a lot.

To give you a concrete example: this is a real ad block that appeared inside article body content across thousands of files, sitting right alongside actual paragraph text:

{
  "_template": "/dfp/GoogleDfpAdModule.hbs",
  "adIndex": "4dd7131d-fd56-4cf1-a95b-2b66294ed539",
  "adSizeMaps": "[[[992, 0], [970, 90], [970, 250]], [[768, 0], [728, 90]], [[320, 0], [300, 250]]]",
  "ampEnabled": false,
  "pageType": "article",
  "sizes": "[[970, 250], [970, 90], [728, 90], [320, 50]]",
  "timedRefresh": true
}

That’s a DFP ad slot — display dimensions, lazy-load config, responsive size maps. It has nothing to do with what was written or published. Multiply that across 50,900 files, add navigation trees, paywall configuration, user auth state, and social sharing metadata, and you end up with a 4 GB export where over 80% of the data is noise.

What I asked Claude to do

I described the problem, shared a sample file, and asked Claude to write a Python script that would strip the page-chrome keys and recursively prune ad blocks and other junk nested deeper in the structure. The key constraints were:

  • Keep all actual content keys — headline, subHeadline, articleBody, authors, tags, canonicalLink, and a handful of others
  • Delete known noise keys at the top level — navigation, paywall config, SEO metadata, ad slots, social widgets
  • Recursively prune nested junk — ad blocks and promo cards scattered throughout article body content
  • Sort cleaned files into subdirectories by template type — ArticlePage, NewsletterPostPage, PackagePage, and so on
  • Handle the weird edge cases — 11 files in the export were actually rendered HTML with .json filenames; those needed to be quarantined rather than parsed

The script Claude wrote handled all of it. From there, we ran a few more passes in subsequent conversations: stripping redundant image crop variants, then merging editorial metadata — tags, issue associations, package relationships — from the originals.

The results — and what comes next

The cleaned export came in at 469 MB, down from 4 GB. Every file is organized by template type. The content keys we actually care about are right at the top level, not buried under layers of page furniture.

More importantly, the files are now workable. We can query across all 50,900 of them in seconds.

  • We were able to assign nearly 2,000 posts to the magazine issue they appeared in
  • We were able to re-relate hundreds of articles to our grant-funded work
  • We were able to fix several broken author archives
  • We were able to find and repair dozens of missing articles

That work — which is the actual goal — is now straightforwardly possible.

Why this worked

The cleanup script itself wasn’t technically hard. Any competent Python developer could write a recursive JSON pruner in a few hours. The bottleneck was that no one had carved out those hours, because the task kept losing priority to things that were more visibly urgent — like the 404 complaints landing in our inbox.

Claude didn’t replace a developer here. It removed the activation energy. I described what I needed, we iterated on a couple of edge cases together, and the script was done before lunch. The six-month backlog item became an afternoon task, and now the real investigation work has somewhere to start.

That’s a pattern I expect to keep finding uses for.

Tags: