Journal
Measurements, dated.
What the work measured and what it taught, each piece with its date. Pieces first published elsewhere say so, and their canonical address points there.
- Your CDN can overrule robots.txt: finding the layer that refuses AI crawlersIn 79 reachable Amsterdam bike-shop sites, 19 refused at least one AI crawler while a browser got 200, mostly against their own robots.txt. How to tell which layer refused (CDN, web server or plugin), and the fix for each.First published on dev.to · 8 min read
- Your robots.txt allows GPTBot. Your server may still answer 403.In a convenience sample of 23 reachable Japanese tourism sites, 6 refused GPTBot or ClaudeBot at the server while their robots.txt welcomed them. How to test your own site with curl in under a minute, and what the check cannot tell you.First published on dev.to · 4 min read