This is a heavily interactive web application, and JavaScript is required. Simple HTML interfaces are possible, but that is not what this is.
Post
Seirdy
seirdy.one
did:plc:cjiy3gbhtim6tw6nsts4eiwu
I've replaced the comments in my robots.txt with a detailed page on scrapers I block: https://seirdy.one/meta/scrapers-i-block/
Includes info on the multiple blocking-approaches & criteria I use, commonly-blocked scrapers I allow, & more fact-checking than more comprehensive alternatives.
#scrapers #RobotsTxt
2024-12-10T05:44:27.656Z