Hey all, I wanted to give people a heads up for tomorrow. We are planning to launch a very large batch of these newer "Rating" hits. We are looking for people who love trivia, history, science, and general knowledge to rate answers that come back from other turkers and a group of related answers from a search engine. We have a total of 19 THOUSAND questions with answers. Thats right, 19 THOUSAND. And we need your help. The goal is to have all of these rated by the weekend's end. Do you think you guys have what it takes!? We've bumped the price from 8 cents to 9 cents per task. These HITs will be a bit easier, because we now provide multiple answers from turkers and the source URL in which they found their answer. We hope this is sufficient compensation - it's the rest of our money in our budget for these questions. The result of this MegaHIT will determine how much money we get from grants/investments for our research. Just a reminder that we'll be monitoring worker quality throughout the HIT. We ask 3 different people to rate each question/answer combination, so we can detect whenever a specific worker isn't agreeing with the rest of the crowd. It's really easy to spot scammers on our HITs (we also designed attention checks in there) - so please be honest with your work! Please keep an eye out tomorrow morning starting at 9AM CENTRAL TIME for these HITS to pop up. Search "BlueJay Labs" to find them. I will be posting here before I open the gates, so watch this post for a headsup! I created this forum as a discussion. Please ask any questions and put any discussion here so we can get a community going to help each other out on some of the questions you might come accross. I'll be hanging out here all day tomorrow answering any questions you have. I'll be posting stats so you get an update on how much work is left throughout the week. FAQ Why the limited time? 10 minutes? It shouldn't take more than a couple minutes to complete each hit.. and you'll get fast over time. 10 minutes is more than enough and it prevents 'hoarding' of the batch near the end of the run. Worth it? We hope so! We spent a lot of time trying to sneak as much money as we can into this HIT. Some of the questions will be hard, and some of them will be really easy. The average time from the last hit was just under 1 minute with the median being around 45 seconds. If you like science, history, trivia, and everything in between we think you'll like our HIT. There is plenty of other work on mturk, so if you get tired feel free to take a break - we have so much work to do that you'll have plenty to come back to! So much clicking!? A little protip: You can click the top of a table column to set the rating of all of the radio buttons to the same value. To go fast, you can select the 'acceptable' and 'good' answers individually, then click the 'bad' column button once to set the remainding radio buttons as 'bad.' If you click the 'bad' column button again, it will override all the responses in that table to 'bad.' How much research do I have to do?! Some of the questions will be really obvious - like "How many fingers on my hand?" while some of them will be hard "What president's children had a pony?" The first section is mechanical turk answers. These answers were already researched by humans, so it might be best to just go through and make sure their answer is correct and then check off the variations of the same answer in the other tables. You shouldn't spend more than 3 minutes on a single HIT (and that should be pretty rare). Most HIT's should only take no more than 1-2 minutes. If it takes you longer, you may be overthinking it. Quals? We put an autogrant qualification on our HIT. This will let us softblock any scammers. We also require you have a 98%+ at 5,000 HITs and up. Sorry, we don't allow exceptions. Majority Rules!? NO! We dont reject the work anyone legitatemly does. We ask 3 people to perform each task to get a better idea how good each answer is - some people are more optimistic and some people or more pesimistic. This is a scientific approach and not used to block/reject anyone because their opinions may not match completely with the majority. Got what it takes!? Well here you go! You can find the hits by searching our name: https://www.mturk.com/mturk/searchbar?selectedSearchType=hitgroups&requesterId=A1ZCUBP2G0ZGZM Thanks -BJ
Time to nap! Should we expect quick approval on these or will it be sometime after the batch has been chewed through?
It should be fairly quick. We'll be doing an analysis batch two or three times a day to scrape out any scammers with a softblock, and the rest will be approved automatically.
What does this mean for the other type of hit? The one for exceptional workers.. will there still be work in that regard?
All - I'm delaying this to 9AM Central Time, so that way everyone can get rest tonight for the big day tomorrow. I want to make sure I'm all ready to handle the onslaught!
Oh thank god, I thought I was gonna have to try and figure out how to be awake and asleep at the same time. Also, I have a question. Let's say I'm not the same as someone on the majority thing. Is that an immediate strike? Or does it have to be repetitive? Also, is being wrong like marking yellow if others mark red, or is it marking green when others mark yellow or red? Just curious how it works
Short answers: You have to be pretty repetitive. Long answer: We've tested this system last week and out of 150 workers only 1 was flagged and that was because they just weren't reading the instructions correctly. The system monitors your responses over time and when a threshold is hit then the worker is flagged. When a worker is flagged, we get an alert that lets us know their worker ID and then we manually go through their responses to see if there is a pattern. We'll then provide feedback onto what they might be doing wrong and give them a strike. Just to put this in perspective though - I doubt we would give out these strikes more than a couple of times - if you put 1-5 brain cells into these hit's you'll automatically agree 95% of the time. It's when you only agree 60% of the time that we start saying "hey, wait a minute - something might be going wrong with this guy." I really wouldn't worry about it though Thanks for the question - Hope to see you out there!! BJ
So if the question is how much water is in something and the answers are both correct but one has an extra word, would this be bad or acceptable?