Loopshift

One pipeline, two repositories. A Python CLI scrapes local businesses from Google Places, audits each website against about thirty-five technical defects, scores the opportunity, and pushes the result into a Node dashboard used for agency outreach.

Grid search, because Places lies about coverage

The Places API caps results per query, so a single search for "Toledo" returns a fraction of what's there and gives you no indication that it did. The scraper instead walks a configurable NxN grid of coordinates with tunable spacing and radius — three presets plus arbitrary custom grids — and dedupes across cells. Coverage becomes a function of grid density rather than of what the API felt like returning.

Scoring is a flat dictionary, one source of truth, summed at the end. The highest-value signal is the absence of a website entirely, worth 20 points — a business with no site is a better prospect than one with a slow site. Which creates a small problem, because the database is keyed on domain and these leads have no domain:

domain = normalize_domain(r.get("website", ""))
if not domain:
    slug = re.sub(r"[^a-z0-9]+", "-", (r.get("name","") or "unknown").lower()).strip("-")
    domain = f"nosite-{slug[:50]}"

A synthetic key, prefixed so it's obvious in the table what it is.

The seam runs both directions

The documented direction is the CLI POSTing audit results to /api/prospects/ingest behind a bearer key.

The undocumented direction is more interesting: the dashboard shells out to the scraper. A click builds a python -m prospector --push argv, spawn()s it, and streams stdout to the browser over SSE with a thirty-minute SIGTERM timer. So a button in the web app launches a Python process on the host, which then HTTP-POSTs back into the same server that started it.

It is not the architecture I would draw on a whiteboard. It is the architecture that let one person ship a scraper and a dashboard without building a job queue for a system with one user.

The upsert is the careful part

Re-scanning a prospect has to update the measurements without destroying what you've learned about them. ON CONFLICT (domain) DO UPDATE uses COALESCE(EXCLUDED.x, prospects.x) for identity fields — a re-scan that fails to find a phone number must never null out the one you already had — but bare EXCLUDED.x for measured fields, where the fresh audit should always win.

And prospect_outreach is left untouched entirely, so sales state survives re-scanning. That one is the difference between a tool you can run twice and a tool you run once and then never trust again.

1,457 lines of scraper against 2,128 lines of tests, 183 test cases, 2,445 cached domain audits.