llms.txt on the top 1,000: 200 is not a file

If you count llms.txt adoption by asking which sites return 200 for /llms.txt, you’ll more than double the real number. I checked the top 1,000 domains today. 246 answered 200. 114 of them actually served an llms.txt file. The other 132 sent a login page, a “not found” page with a 200 status, their home page, or a single-page app shell.

This post is the measurement, the three ways a naive check goes wrong, and a 20-line script that doesn’t.

What I measured, and the limits first

List: Tranco list 56WKN, generated 2026-10-05, top 1,000 entries. Tranco ranks registrable domains by combined traffic signals, so it includes infrastructure domains (CDN hostnames, API domains, ad servers) that have no website at all.
When and from where: 2026-10-06, 06:30 to 06:37 UTC for the files, with the dead-link re-check in the following hour, from one ordinary connection in Japan. Some sites answer differently by country (a few redirected me to /jp/ or /ja-jp/ pages), so a run from elsewhere will differ at the edges.
Requests: one GET each to https://<domain>/, /llms.txt and /llms-full.txt, with a desktop Chrome user agent and Accept: text/plain,text/markdown,*/*. If https://<domain> didn’t answer, I tried https://www.<domain>. Redirects were followed.
Not counted as “no file”: 70 domains answered /llms.txt with 401 or 403. Some of those may have a file that a bot-protection layer kept from me. I count them as unknown, not as absent.
One snapshot. No second pass on the presence check. Status codes for a static file are much steadier than for home pages, but a single run is a single run.

742 of the 1,000 domains answered HTTPS at all. That’s the denominator for everything below unless I say otherwise.

The headline numbers

What `/llms.txt` returned	Domains
An llms.txt-shaped text file	114
200, but an HTML page	122
200, but a redirect to some other page served as text (my `Accept` header asked for text, and a few servers converted the page to markdown)	8
200, but not a file at all (an IP echo service, a JSON error)	2
404 or 410	388
401 or 403 (unknown)	70
Anything else (400, 5xx, 202, timeouts)	38

So 114 of 742 reachable domains (15%) serve an llms.txt. In the top 100 it’s 8 of the 75 that answered: cloudflare.com, github.com, fastly.net, wordpress.org, digicert.com, adobe.com, opera.com and sentry.io. 16 of the 114 get there by redirecting to a sibling domain’s file, for example unity3d.com to unity.com and macromedia.com to adobe.com.

17 domains also serve llms-full.txt, and every one of them has an llms.txt too. Worth knowing: llms-full.txt is not in the llms.txt proposal at all. It’s a convention that documentation platforms popularised.

Three ways a naive check lies

1. A 200 that is an HTML page

122 domains answered with an HTML page and status 200. 37 of them serve that HTML at /llms.txt itself, which is what a single-page app does when it routes every unknown path to the same shell. instagram.com, linkedin.com, pinterest.com and chatgpt.com are in this group. The other 85 redirected first: 38 to a home page, the rest to login screens or to a “not found” page that returns 200 (netflix.com lands on /NotFound, tiktok.com on /404).

Any tool that reports “llms.txt found” on status alone counts all 122. Check the Content-Type, check that the first non-blank byte isn’t <, and check where you ended up after redirects.

2. A text file that isn’t an llms.txt

The proposal requires exactly one thing: the file starts with an H1, the name of the site or project. Everything else (a blockquote summary, H2 sections holding lists of [name](url) links) is the recommended shape.

108 of the 114 start with an H1.
97 have the full shape: H1 first, at least one H2 section, at least one list item that is a link.
The 6 that don’t start with an H1 open with a “Generated by …” line (playstation.com, moloco.com), a bare URL (cisco.com), an H2 (agora.io, trendyol.com), or straight into a link list (salesforce.com). A generator tool wrote the first two, and nobody read the first line afterwards.
11 have the H1 but no linked list sections. porkbun.com has 14 H2 sections and no links at all; it’s a prose summary of the site. That’s allowed by the letter of the proposal, but an agent reading it has nowhere to go next.
9 have more than one H1.

Size is the other surprise. The median file is 11,378 bytes, which is fine. 12 are over 100 KB and the largest, salesforce.com’s, is 1,021,703 bytes. At the rough rate of four bytes per token that’s about a quarter of a million tokens. Version 2 of the proposal, published in August 2026, says the file “stays small enough to fit in context” and that the detail belongs behind the links. A 1 MB index is a sitemap with extra steps.

The same version says links should point to LLM-friendly content, such as a markdown copy of each page at page.md or page.html.md. Of the 107 files that contain links, 74 link to no .md or .txt URL at all and 8 point mostly to them. I only looked at the extension, so a site that serves markdown by content negotiation on a normal URL is undercounted here. Still, most files today are a list of ordinary HTML pages.

3. Dead links, and HEAD requests that lie about them

For every file with links I took up to 20 of them, spread evenly through the file, 2,043 links from 107 files in total. I sent HEAD first and fell back to GET only on errors.

HEAD said 81 of them were 404 or 410. When I re-fetched those 81 with GET, only 42 were actually gone. dynatrace.com answers HEAD with 404 on pages that GET returns with 200, which accounts for 15 of the 39 false alarms on its own. If your link checker uses HEAD to save bandwidth, it will report rot that isn’t there.

The 42 confirmed dead links (2.1% of the sample) sit in 13 files, and they bunch up: 10 of 20 sampled links on moloco.com and 9 of 20 on weather.com are gone. 40 of the 42 point at the site’s own host. These files were written once and the site moved on underneath them.

I wanted a staleness number from Last-Modified too, and it doesn’t work. 74 of the 114 files send the header. 24 of those are dated the same day I fetched them, because the file is generated on each request, and one says 1 February 1980. Dead internal links are a better staleness signal than any date the server gives you.

Check your own

This prints one verdict per host. It needs bash and curl.

check_llms() {
  local host=$1 tmp
  tmp=$(mktemp)
  IFS=$'\t' read -r code ctype final < <(curl -s -L -o "$tmp" -A 'Mozilla/5.0' \
    -w '%{http_code}\t%{content_type}\t%{url_effective}\n' "https://$host/llms.txt")
  final=${final%$'\r'}
  local first
  first=$(sed -e '1s/^\xEF\xBB\xBF//' "$tmp" | grep -m1 -v '^[[:space:]]*$')
  printf '%-22s %s %-28s %s\n' "$host" "$code" "${ctype%%;*}" "$final"
  if [ "$code" != 200 ]; then echo "  -> no file ($code)"
  elif [[ "$ctype" == *html* || "$first" == \<* ]]; then echo "  -> HTML page answering 200, not an llms.txt"
  elif [[ "${final%%\?*}" != */llms.txt ]]; then echo "  -> redirected to $final, not an llms.txt"
  elif [[ "$first" != "# "* ]]; then echo "  -> text, but the first line is not an H1: ${first:0:60}"
  else echo "  -> llms.txt, $(wc -c < "$tmp") bytes, $(grep -c '^## ' "$tmp") H2 sections, $(grep -cE '^\s*[-*] \[' "$tmp") list links"
  fi
  rm -f "$tmp"
}
check_llms vercel.com; check_llms netflix.com; check_llms moloco.com; check_llms pages.dev

What it printed for me today:

vercel.com             200 text/plain                   https://vercel.com/llms.txt
  -> llms.txt, 4868 bytes, 9 H2 sections, 25 list links
netflix.com            200 text/html                    https://www.netflix.com/NotFound?prev=https%3A%2F%2Fwww.netflix.com%2Fllms.txt
  -> HTML page answering 200, not an llms.txt
moloco.com             200 text/plain                   https://www.moloco.com/llms.txt
  -> text, but the first line is not an H1: Generated by Moloco, this is an llms.txt file designed to he
pages.dev              200 text/html                    https://www.cloudflare.com/products/pages/
  -> HTML page answering 200, not an llms.txt

The script doesn’t check links. For that, take the URLs out of the file and GET them; don’t trust a HEAD 404.

What I’d change on a site today

If you don’t have an llms.txt, the first thing is to make sure /llms.txt returns a real 404. A 200 HTML page there is worse than nothing, because a careless checker will tell you you’re done. If you do have one: start it with # Your Site, keep it under a few tens of kilobytes, and re-run a link check whenever you restructure URLs, since that’s what killed the dead links above.

And an llms.txt only helps if crawlers can fetch your pages in the first place. In my earlier measurement, 19 of 79 Amsterdam bike-shop sites refused at least one AI crawler at the server, mostly against their own robots.txt. The free checker tests that half: robots.txt against what your server actually answers GPTBot, ClaudeBot and PerplexityBot. It doesn’t look at llms.txt.

The raw data (one JSON line per domain, the files I fetched, the probe and analysis scripts) is kept with the date above; ask in the comments if you want the per-domain table.

If you’d rather have a person go through one site end to end, the paid mini audit is on the same page as the checker. It checks AI-crawler access and machine-readability and includes paste-ready fixes (a corrected robots.txt, CDN/WAF steps for bot 403s, meta and canonical fixes, JSON-LD and llms.txt skeletons), all marked review before deploying. It queries no AI engine and measures no citations or rankings.

Correction 2026-10-06: an earlier version said the paid audit measures what AI engines answer; it checks crawler access and machine-readability only.