Panos Ipeirotis has another interesting post up on his blog, along with a new tool. He's put together a list of MTurk's top 1000 requesters in terms of how much money they've spent on rewards so far in 2009. Check it out: http://behind-the-enemy-lines.blogspot.com/2009/08/top-requesters-on-mechanical-turk.html
The quick hits are not represented well. I can't even find AEXLR15TK0KXH - Content Review in the top 300. I know they should be.
Could have sworn I posted something above explaining that Panos only scrapes MTurk a few times an hour, which is why some requesters will be undervalued or not represented, but appears I was imagining things. Would be nice if Amazon themselves would share their data as it would be complete, but I think it's a pretty safe bet that such will never happen.
Hi guys, Some comments on the limitations of the tool: I crawl the website every 15 minutes, logging in using my own account. So, if there is a qualification test, then I cannot really capture the HIT. Same thing if my Turker account does not meet the requirements for the HIT. Also, HITs that are posted and completed in less than 15 minutes may not be captured as the crawler captures only tasks that appear at NN:00, NN:15, NN:30, and NN:45. My analysis indicates that there are not that many tasks that disappear within 15 minutes, compared to the total amount of tasks posted. (Initially, I was crawling every 1 hour and increased the frequency to every 15 minutes but I do not seem to capture many more HITs than before.) The major limitation of the tool is the fact that it cannot capture the value of tasks that get multiple Turkers to complete the same HIT. So, there is definitely some undervaluing factor there, depending on the requester. For some requesters the reward amount is close to accurate, for some others it may be a 2x to 5x underestimate. And for survey-like HITs it may be even a 10000x underestimate. Cheers, Panos
Interesting stuff. Actually, I've bookmarked this blog's page with posts labeled "mechanical turk" [http://behind-the-enemy-lines.blogspot.com/search/label/mechanical%20turk] since when I first came to know of this blog via ChatBox here, thanx to ergo. Though I'd like to admit that I haven't had the opportunity to read more of the posts due to certain limitations of my own. BTW, the page at http://hyperion.stern.nyu.edu/mturk/requesters.php did take time in loading & did load successfully on my Firefox browser [version 3.0.11], however the page loaded was absolutely blank. Tried to reload a few times in vain. Fortunately, it loaded well on my Opera browser (due to which I didn't have to try it on my Google Chrome, IE or Flock browser).
I tried loading the page on FF 3.5.2 (Windows Vista) and it seems to work fine. I am using some Google Libraries to display the table, and it is not truly compatible across all-browsers. Not sure what I can do other than wait and see Can you upgrade FF to the latest version and see if the problem persists?
It works on FF 3.0.7 (I'm waiting for certain addons to update so I can) Just don't be too fast hitting the page forward button, once about a second has no issues.
Can't do that for the time being for specific reasons. But this isn't a big deal as it seems I'm the only one having problem so far. So most probably the fault lies in my system. And I didn't close the tab the very moment the page was completely loaded, but was empty. I waited for at least 5 minutes [sometimes it takes about half minute for an image to appear even after the page (happens only in cases where the page consists of an image, the ones with .jpg extension & alike) is completely loaded with this current Firefox version of mine] to see if the content loads, but it didn't. Anyways, as stated earlier, the problem lies in my system.
In case someone hasn't noticed the Top-1000 Requesters list now has RSS feeds thanks to an outside source. I'm really surprised someone else here didn't say anything about it.
The outside source is me. I'm still tweaking how the feeds work, so although people are welcome to use them they're going to be a little more useful in the near future. The main limitations are: * Feeds are cached for 10 minutes. This can probably be reduced to 5 minutes, I'll have to see how things go. * If a requester adds more HITs to an existing HIT group, you'll never be notified of that. Will be doing something about this. * Some HITs I can't link to directly because I lack the qualification to view the HIT; so in those cases I link to the list of that requester's HITs on MTurk. Don't expect this to change. By the way, I'll be releasing a small script to add RSS feeds directly to MTurk for Firefox and Opera users, which will make it a lot less hassle to subscribe.