EB/

Spiderbot

How 58 utility sites get chosen, built, and pruned.

What is Spiderbot?

Spiderbot is my pipeline for running programmatic SEO utility sites: calculators and tools built from one shared template. It scores niches by real search demand, builds only what the data supports, and cuts what does not perform. 58 sites are live today.

The thesis

Finished work should stay finished, and the data should decide what gets built.

Why does Spiderbot work?

  1. Utility pages stay finished.A calculator is built and verified once, then earns with almost no repeat labor. Formats that need a fresh production run for every unit, like video, re-pay the full cost every time. Every candidate faces one filter: does the unit of output stay finished after a single pass?
  2. The site is the unit, and count is the lever.Each site comes from a shared template with flat fixed costs, so traffic grows with the number of winning sites, not with hours worked.
  3. Big utility queries are held by weak sites.Keyword difficulty scores count backlinks and miss topical saturation, so they mislead. The better test is how many low-authority sites already rank on page one: three or more in the top ten is winnable, zero or one is a wall.
  4. Tool data lies, so the gates run cheap-first.Keyword tools can report volume that is many times the real figure on low-demand clusters. Totals only decide the order of the verification queue. A second data source and a human read of page one decide the rest.
  5. Google's feedback is slow, so ship in cohorts.Search Console takes 30 to 90 days to say what worked. Building faster than that just makes copies of an unproven template. Each cohort waits on the last one's data, and dead sites are pruned at day 90.
  6. It compounds in vintages.Each year's builds pay out the following year. The easy topics get used up, so later cohorts go into deeper verticals with the same gates.

What would prove it wrong?

Monetization is a test, not an assumption. Templated utility sites may not clear ad-network review, or may earn less per visit than planned. The first five sites are the deliberate test, and the rest wait for signals before applying.

The pool of open topics is finite. Measured directly, the number of high-demand topics without an entrenched winner is far smaller than a raw keyword export suggests. Growing past it means smaller clusters and less revenue per site.

Google may see templates as low value. Pages that get indexed but never get impressions would show up in the day 30 and day 60 data, and that is the signal to stop and change the approach.

First real signals, about a month in. Impressions across the portfolio are growing week over week, clicks are still close to zero, and one site has already been declined for ads as low-value content. That points to a track-record problem more than a template problem, and the day 60 and day 90 checkpoints will say which.

The workflow

Click a stage

Seed

Generate candidate topics from an outside topic space instead of from what the portfolio already covers, so the pipeline can find domains it has no foothold in.

Tooling
Python scripts; topic groups crossed with query patterns; keyword data from DataForSEO
Rule
Weight seeds toward the topic groups that historically produce big head terms.
Output
A few thousand seed queries.

What survives

One measured run
~100
seed queries
~76
candidate clusters
~9
clear every gate

Most ideas die at the gates, by design. Cheap filters run first so the paid checks only see survivors.

What has running it taught me?

  1. Some queries can't be won, however good the page is.My height converter reached page one for searches like “5’3 in cm,” drew thousands of impressions a day, and got almost no clicks. Google answers those searches itself with a built-in converter above every result. Now every target phrase gets checked in a live search for an answer box before anything is built.
  2. Search volume is not traffic.Some calculator searches show tens of thousands of searches a month, yet every page on page one, big sites included, gets almost none of it. Before trusting a volume figure, I check what the pages already ranking actually receive.
  3. Slow indexing was a discovery problem, not a quality problem.Across the portfolio, about 230 pages were stuck before Google ever crawled them, against zero that it crawled and turned down. The fixes are internal links from pages that are already indexed, real outside links, and time, not rewriting pages Google hasn't read yet.
  4. Ad networks judge a track record, not a page.One of the oldest sites was declined for display ads as low-value content even though its pages are substantial. A month-old domain with zero clicks reads as unproven no matter how good each page is, so ad applications now wait for real traffic.
  5. Committed is not shipped.A fix for one site sat in git for a week while the live site ran the old version. Now every change is confirmed by checking the live page, not the commit log.
  6. The builder should never grade its own work.Independent audits keep catching real errors before launch: a Bible text that was a years-old edition, a country's outdated currency, a sign bug in a map tool. Each site is checked by a separate pass that re-pulls the source data.

What is it built with?

Next.js and Vercel for the sites, Python for the pipeline, DataForSEO, Ahrefs and Mangools for research, Google Search Console and GA4 for measurement.