If the hits on a page all look like garbage, or I come across a blank page because there's nothing there but requesters I've blocked from my sight, I can flip through a stack of pages pretty quickly... Or at least I could before today. Now I keep seeing "You have exceeded the maximum allowed page request rate for this website." over and over again. WTF? Has this been happening to any of you?
Did you get that on mturk.com or on some other site? Sounds like the server in question is throttling you (which is different from an ISP throttling you). Sometimes a webmaster will do that to thwart site rippers / scrapers.
Yeah, it's mturk. If it's the server doing it, then it's mturk doing it, because surely they own their own server. I don't understand the ripper/scraper part. What are they afraid of, exactly? All I'm trying to scrape off that website is pennies, and that only by finding something worth doing as quickly as possible, so I can earn them.
Sometimes that happens to me if I am running to many scripts on that page. Or maybe because you have so many blocked requesters? I don't know.
Same with me. It seems to have to do with running too many scripts and going between pages too fast. I've tried to slow down now.