This is a heavily interactive web application, and JavaScript is required. Simple HTML interfaces are possible, but that is not what this is.
Post
Martin Splitt
divingfor.fun
did:plc:zbpbrhdce5y6i4hpw3a3j6x7
Hey Jeremy, this isn't quite right. Nothing changed here.
If there is no robots.txt, crawling is assumed to be allowed.
If there is one, Googlebot follows the rules in it.
If your server gives a 5xx, Googlebot doesn't know if there's a robots.txt or not and then failsafes to "no crawling now"
2026-01-15T07:34:12.040Z