Every large language model starts life as a pile of text. Not a neat, curated library, but a sprawling mix of web pages, code, forum threads, manuals and reference articles. Someone had to find all of it, download it and clean it up. The models get the headlines, while the data work behind them rarely does.
Yet much of a model’s quality is decided before training starts, by what went into the corpus and what was filtered out. If you work in data engineering, the process will feel familiar. It’s an extract, transform and load pipeline, just at a scale few teams ever touch. Here’s how it works, from the first crawled URL to a training-ready dataset.
Where the text comes from
When people say a model was “trained on the internet”, they usually mean a blend of several kinds of source:
- Open web crawls. Common Crawl, a non-profit, has spent years archiving the public web and releasing its crawls for free. Many well-known open datasets start there and then filter it heavily.
- Reference and knowledge sites. Encyclopedias, documentation, Q&A forums and preprint servers are full of well-structured explanations. Dataset builders often give them extra weight.
- Code. Public repositories teach models to program, and they bring their own licensing questions.
- Books. Public-domain books help models handle long, coherent passages.
- Licensed and private data. A growing share comes from deals with publishers and platforms, plus data written or labelled by people.
The word “public” is doing a lot of work here. A page anyone can open without logging in is public, but still subject to copyright, terms of use and privacy rules.
From a URL list to raw pages
Teams that crawl for themselves usually begin with a seed list. It might come from sitemaps, an existing archive or a curated set of domains. The crawler then works through a queue, often called the frontier, fetching pages and adding the new links it finds.
Across millions of domains, the hard questions are about scheduling. How often should a site be revisited? How many requests can a server reasonably handle? Which pages need a full headless browser because JavaScript builds the content?
Rendering is slow and expensive, so many pipelines use it only where plain HTML falls short. Good crawlers also store the URL, fetch time, status code and headers, so every document can be traced later.
Cleaning is where the real work happens
Raw HTML is mostly not training material. Menus, cookie banners, footers and ads often outweigh the article on a page. Much of what gets crawled never reaches the final dataset. What survives usually passes through:
- Text extraction. Boilerplate is stripped away so only the main content remains.
- Language identification. Each document is tagged by language, which matters for models meant to handle more than English.
- Quality filtering. Simple rules remove pages that are too short, mostly symbols or full of repeated lines. Some teams add classifiers trained to spot well-written, informative text.
- Deduplication. The same press release can appear on hundreds of sites. Exact and near-duplicate detection keeps the model from memorising repeated text.
- Privacy and safety passes. Email addresses, phone numbers and other personal details are masked or removed, and harmful content is filtered out.
One more check is easy to overlook. Not every response is the page you asked for. A crawler may get back a CAPTCHA, a rate-limit notice or a regional error page that looks like a normal response. If nobody catches these, block pages end up in the corpus looking like real content.
When a public archive isn’t enough
Open crawls have limits. Each one is a snapshot, so recent events and fast-changing pages lag behind. Coverage is uneven across languages and regions. Niche material, like one industry’s technical documentation, may be thin or missing.
That’s why teams building specialised models for law, finance, healthcare or retail often run targeted crawls of their own. Some also need pages as they appear in a particular country. Prices, stock levels and local news can all change with the visitor’s location.
The fetch layer and the IP address question
Crawling yourself brings a practical problem. Websites see the IP address behind every request, and many limit how much traffic one address can send. Many sites are also stricter with traffic from cloud and hosting ranges, which is where crawlers typically run.
Location is the second issue. A crawler in one data centre sees every site from a single country. That leaves gaps in a dataset meant to cover several markets.
Residential proxies route requests through IP addresses that internet providers assign to home connections, in the countries you choose. That spreads traffic across many addresses and lets a crawler see regional versions of pages. It isn’t a licence to ignore a site’s limits, though. Responsible teams still cap request rates per domain and back off when a server pushes back.
When comparing providers, check country and city targeting, IP depth in your markets and whether you pay only for successful requests. Then validate what comes back, since a proxy can’t tell real content from a polite refusal.
Consent is now part of the spec
The robots.txt file is decades old, but it has never mattered more than it does now. Several AI companies publish the names of their crawlers, so site owners can allow or block them separately from search engines. A growing number of publishers now do.
Respecting those signals is the baseline, not the finish line. Site terms still apply, and in many jurisdictions personal data stays personal data even when it’s publicly visible. A clear record of each document’s origin makes these obligations easier to meet. This isn’t legal advice, so check the rules for your use case.
Training is only the first collection job
Gathering data doesn’t stop once a model is trained. Training data goes stale, so many AI products now fetch fresh pages at question time through retrieval-augmented generation, or RAG. AI agents go further, browsing sites step by step to finish a task.
Each job puts different demands on the collection layer. Bulk training crawls care about coverage and deduplication, retrieval cares about speed and freshness, and agents need stable sessions. ProxyEmpire’s guide to how LLM teams gather web data compares these jobs side by side, from AI extraction to browsing agents. It also explains why the fetch layer is still the hardest part to get right.
What this means for data teams
If your organisation plans to build or fine-tune its own models, the data pipeline deserves as much planning as the model. A few habits pay off early:
- Start from open archives and crawl yourself only where they fall short.
- Log provenance from day one, including the URL, fetch time, status and every filter a document passed.
- Treat block pages as a data quality issue, not just an operations headache.
- Honour opt-outs and rate limits, even when nobody would notice if you didn’t.
Models will keep changing every few months. Teams that know where their data came from will find it far easier to explain, fix and improve them.
