31 sources surveyed, 9 crawlers, 77,446 real records with a completeness flag on every snapshot, and a written taxonomy of why each source refused. No headless browser, no proxies, no scraping library, no paid API. Below: how I read a source, the failure modes that cost the most, where I draw the line, and what I would do in the first 90 days here.
Writing a fetch loop is an afternoon. The cost is finding out which of your assumptions about each source were wrong, and the only way to find out is to run it and measure the output. Two of my own strongest early conclusions were wrong, and both were wrong in the same direction: a source that refuses one request is not a source that is unavailable.
A crawler that crashes is a good day. The expensive failure is a parse that returns zero rows into a pipeline reporting success, or a pager being silently ignored so the crawler re-reads page one forever while the log looks healthy. Everything I build is instrumented against those two.
A partial crawl produces confident wrong answers for everything past where it stopped. That has to be a stored field, defaulted to incomplete, and it has to reach the interface. This is the discipline that transfers directly to selling datasets to a lab.
Wynd operates a distributed crawler on residential bandwidth contributed by around 3M nodes, roughly 100 Gbit/s, and sells structured datasets to frontier labs on top of it. The moat is the residential exit: a data centre cannot fake a household IP, and that is the one input a competitor cannot simply buy more servers to replicate. The exposure sits in the same place.
| Player | Supply model | What they sell | Gap versus Wynd |
|---|---|---|---|
| Bright Data | Aggregated residential + datacentre proxies | Proxies, managed collectors, off-the-shelf datasets. Won Meta v Bright Data in Jan 2024 on the finding it scraped only public logged-out data. | The incumbent and the legal reference point. Supply is bought rather than incentivised, so the cost curve rises with volume where a contributor network's flattens. |
| Oxylabs, Zyte | Proxy pools, managed scraping | Infrastructure sold to engineers. Zyte maintains Scrapy. | Tooling companies selling to people who scrape. Wynd sells the output, which is a different buyer with a much larger budget. |
| Apify | Marketplace | Other people's extractors, rented. | Breadth without ownership of supply or of dataset quality. |
| Common Crawl | Public good | A free web corpus every lab already has. | The real competitor for any raw-pages pitch. It is why the paid market has to sell freshness, coverage and structure, which is exactly where an extraction specialist earns their seat. |
| Honeygain, IPRoyal | Bandwidth sharing, no token | Residential exits to proxy buyers. | Same supply mechanism, no dataset layer and no incentive flywheel. They also carry the category's reputational history. |
Common Crawl sets the floor for raw pages, so value accrues to whatever a general crawl cannot produce: sources that need a session, a form, a mandatory parameter, or an endpoint that only the page's own JavaScript knows about. Those do not yield to more bandwidth. They yield to somebody sitting down and reading the page. That is the seam this role occupies, and it is the reason a specialist is being hired alongside a crawling engineer.
Almost every source is one of five shapes and a single request tells you which. This is the classification my crawlers and the demo both run on.
| Shape | The tell | How to read it | Seen on |
|---|---|---|---|
| Plain paginated GET | ?page= in the pager and a real table in the source | Walk the pager, asserting page N differs from N−1. A bare GET with no search returned an entire 21,111-row register across 1,056 pages. | 4 sources |
| Form POST | A POST form whose fields you can read off the page | Post the observed names in the observed spelling, then re-request page one with an explicit page size rather than reusing the POST response. | 2 sources |
| Session-held portal | A per-session token in the form action, namespaced fields, a session id in the path | Criteria live in the session, so carry a cookie jar and re-read the token every run. Verify the container's real paging parameters rather than the framework's documented ones. | 1 source |
| Embedded JSON | A large page with no table at all | Brace-scan the blob with a scanner that respects string state. One 2.4 MB request returned 1,937 entries with no pagination, and 1,930 records are held after dedup. | 1 source |
| Targeted query endpoint | A front end calling a separate route, refusing to list without a query | The one shape not to copy. It answers one record at a time by design, so query it live and store nothing about the people in it. | 5 sources |
Scrape Doctor takes any URL, fetches it twice (bare and browser-shaped), and reports the shape, the platform tells, the endpoints the page's own JavaScript calls, the pager parameter and whether it is zero-indexed, and whether a refusal was mine or theirs. Built for this application. scrape-doctor.leverlabs.workers.dev
| What the JD asks for | Where I have already done it | Detailed in |
|---|---|---|
| Write, test and refine code that extracts data from various online sources, reliably | Nine extractors against nine different source shapes, each with its own observation log written into the file header: what was seen, on what date, and what remains unverified. Every one is running. | §02 |
| Handle pagination and dynamic content loaded with AJAX | Six pagers, one of which ignores its framework's standard parameters and needed its own. One source renders rows in JavaScript, so I read its script and posted to the JSON endpoint it calls. One publishes no table at all and hides the dataset in the page source. | §02, §04 |
| Clean and format extracted data to a quality standard | Name normalisation across Malaysian naming conventions (honorifics, patronymics, spelling variants), status derived from validity windows rather than assumed, and per-field fill counts as the sign-off gate instead of row counts. | §04 |
| Store and manage the scraped data, optimising for access speed and integrity | An identity key chosen by what the source vouches for, a merge that preserves first-seen dates and matches through a queue so identically-named people stay distinct, disappearances kept and marked rather than deleted, and completeness carried as its own field. | §05 |
| Regularly monitor, identify and resolve issues to maintain continuous flow | The part I think is actually the job. Archived raw bodies for every parse, a collapse floor so an empty result cannot overwrite good data, and checkpoint plus resume so an interrupted crawl does not re-request what it already read. | §06 |
None were visible from reading the pages. Each was found by running the extractor and measuring what came out.
page 1 of 1056 never matches a \s regex, so the crawler cannot see the page count and walks blind.Source-assigned id first, then the registration number, then the normalised name. Never key on an attribute that changes: keying on address turns a house move into a disappearance plus a new arrival.
Keep it, mark it absent, leave its last-seen date alone. But only when the fetch read the whole source. Inferred from a partial crawl it becomes a false claim about everyone past the stopping point.
Assuming completeness is the dangerous direction, so the field defaults to false when absent and the interface says what was not read. A buyer of a dataset needs this more than a consumer does.
Take the extractors nobody wants to own. Read the last quarter of breakage and classify it by which of the eight gates failed. Establish whether a silent zero-row parse is currently detectable, and by whom. Ship the smallest instrumentation that closes that gap.
Take the targets the general crawl cannot produce cleanly: session portals, mandatory-parameter searches, JS-rendered grids with an endpoint behind them. Turn the per-source detective work into a shared classification the whole team applies, so the second one is faster than the first.
Per-field fill rates and completeness per source, tracked over time rather than at sign-off. That is the artefact a lab buying a dataset actually wants, and it is also the earliest warning that a parser has quietly rotted.
Sending the headers a browser sends, carrying a session the site expects, and calling the endpoint the page's own JavaScript calls are all being a well-formed client. Solving a challenge, logging in, creating accounts, or rotating addresses to evade a rate limit are not. The same rule holds in law: every scraping case that went badly in court turned on authentication or circumvention rather than on collection. hiQ lost on contract grounds for using fake accounts. Bright Data won on the finding it never scraped logged in. Reddit v Perplexity is pleaded as circumvention of rate limits. Logged out and non-circumventing is the defensible zone, and it is a rule you can put in code.
Every number comes from a system I run. Record counts are read from the live snapshots, page counts and register sizes are what each source reported on the probe date, and the failure modes are drawn from the run logs and the adapter headers. Nothing here is illustrative. Role details from the public job posting, 2026.