You’ve probably never used Brave Search. Well, not directly. But the AI you ask questions might have, and it might have found your website there. On Nov. 2, the New York Times sued Brave, the company behind the Brave browser and Brave Search. The Times says Brave copied more than 30 billion webpages “without identifying itself or providing website owners with any mechanism to block its crawler,” then sold that content to AI companies. Brave hasn’t answered yet, so for now those are claims. But why can’t you keep its crawler out without keeping Google out too? Brave explains that itself, right there on its help page.
Brave Sells Its Search Index To Cohere, Mistral And Perplexity
Brave sells access to its search index to AI companies, through a product called the Brave Search API. When an AI assistant needs something recent from the web, it doesn’t always go and read the web itself. It can pay to search someone else’s index instead. Brave’s About page says the API is “Used by some of the biggest names in AI, including Cohere, Mistral, Perplexity, and You.com.” Its AI page says the index holds “over 30 billion pages, with millions of new pages indexed each day.”
The Times’s complaint opens by naming five AI companies it says Brave supplies: Anthropic, Perplexity, Mistral, xAI and Cohere. None of them is being sued. Claude is the fuzzy one, because the evidence is old. The complaint points to a 2023 Brave post about Anthropic training Claude 2. In March 2025 Anthropic added Brave Search to its list of subprocessors, the outside companies it uses, which TechCrunch reported. The same week, Simon Willison, a developer who writes a widely read blog about AI tools, asked Claude to search for pelican facts, and the 10 results it cited matched the same search on Brave exactly. Whether Claude still uses Brave today, I couldn’t confirm.
Brave’s Crawler Doesn’t Have A Name, So You Can’t Block It Without Blocking Google
Websites decide which crawlers may read them in a small file called robots.txt, and that only works because crawlers have names. OpenAI and Anthropic each publish the names of theirs, so you can let one company’s crawlers in and turn the other’s away.
Brave’s help page, the one at the top of this article, says its crawler “does not advertise a differentiated user agent.” In plain words, Brave’s crawler doesn’t wear a name tag, and Brave gives a reason: Some websites let Google in and nobody else, so a crawler with Brave’s name on it would get turned away. Brave says it follows Google’s rules instead: whatever Googlebot isn’t allowed to read, Brave won’t read either.
So you can’t let Google in and keep Brave out. You get both, or neither. To get out of Brave’s index, the same page points you to the noindex tag, which tells Google to drop the page too. The Times calls this Brave’s “refusal to release the name of its web crawler bot.”
My Website Is In Brave’s Index, And My Robots.txt Can’t Even Mention It
A Brave search for site:nohacks.co on Nov. 3 showed 16 of my pages on the first page of results, 13 of them articles.
Brave’s result even shows the homepage description I replaced earlier that same day, so what Brave serves is a stored copy, not a live look at my website. My robots.txt names 13 AI bots, from GPTBot to Bytespider, and blocks two of them, Meta’s and ByteDance’s. There’s no line I could add for Brave. It’s hard to block a name that doesn’t exist.
Checking yours takes about 30 seconds: go to search.brave.com and type site: and your domain. Everything that shows up is in the index Brave sells.
A Search Engine Sends People Back, And An Index Sold To AI Doesn’t Have To
Being in a search engine is a deal every website understands. Someone searches, sees your page, clicks, and lands on your website. Brave Search’s ordinary results work like that, though Brave has added AI answers of its own, called Ask Brave, which The Times is also suing over.
Selling the index is a different deal. The complaint quotes Brave describing one part of it, “LLM Context,” as handing over “extracted page chunks and source metadata in a shape and length that satisfies the needs of an LLM without need for further scraping.” So the AI company gets your page, already chopped up, and writes an answer from it. Whether that answer sends anyone back to you is the AI company’s call. It isn’t yours, and it isn’t something Brave is selling. That’s the part that bothers me.
A Court Will Decide Whether Brave Can Sell Its Index To AI Companies
The New York Times is claiming copyright infringement and getting around its paywall, among other things, and the Daily News, the Chicago Tribune and the Orlando Sentinel filed a separate case the same day. Brave hadn’t responded to either when I wrote this. It has argued its side before, though: when News Corp sent it a cease-and-desist, Brave sued first, saying it was only indexing the web, which “all search engine operators must do to exist,” and in July 2026 News Corp sued back, saying Brave hides its crawlers so publishers can’t detect or block them, Semafor reported. A ruling on The Times’s pages won’t automatically cover anyone else’s, but it will say a lot about every website in that index, and that’s probably yours too. So I’ll be watching this one.
- Should I Block AI Crawlers At Robots.txt Or Server Level?
- The Click Economy Is Over: How AI Search Is Forcing Publishers To Rethink Revenue
- Indexed Claude Chats Show Why Disallow Is Not Noindex
This post was originally published on No Hacks.
