This is a heavily interactive web application, and JavaScript is required. Simple HTML interfaces are possible, but that is not what this is.
Post
Martin Splitt
divingfor.fun
did:plc:zbpbrhdce5y6i4hpw3a3j6x7
Basically if a site doesn't have a robots.txt, crawling is assumed to be allowed.
If it does have one the rules will be followed.
If it can't be read and can't be determined to exist then crawling is assumed to be disallowed.
See https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3.1
2026-01-13T05:36:05.659Z