Last year I received some great feedback from the MTurk Forum in the preparation of my HIT batch, so I thought I'd ask here again before releasing a new batch of HITs. I'm planning a HIT to help me evaluate the success of a computer application that produces vocabulary quiz questions automatically. The HIT I'm planning is basically a multiple-choice word quiz to test how good the items are. Take a look at a sample copy of the HIT here. I'm planning the following Quals: HIT Approval Rate > 99% HITs Approved >= 10000 Location is US There are 120 items in the quiz and I estimate experienced Turkers will finish it in less than one hour (40-50mins), and the pay'll be $5.00. I have two points of concern: Does the task and the pay appear attractive enough to Turkers with these quals? There are ten check items and I'm requiring a success rate of at least 80% (8/10). Does this seem too high a bar (for the task) and would it dissuade good workers from even accepting the HIT? If you have any other comments or suggestions, I'm eager to hear them. I really appreciate the support of the MTurk Forum Community. Sincerely, Ralph PS. I will announce the HITs here again just before they go live so that people here will have a heads up.
Hi Ralph, I only have about 8,000 approved, so I won't qualify, but I will give my quick opinion. My guess is that with an 80% success rate some workers will choose not to do the HIT. Words can be subjective, and what may be an obvious answer to you might not be to others. In Example 2, I could see a small amount of people choosing "shifts" as a possible correct answer. Perhaps a 70% rate would be better. Your pay amount seems fair for the task. Too bad, it looks like fun. Sorry I will miss it. Good luck with your research.
Dear ShoppingNinja, Thank you very much for your feedback. I don't think I'm that particular about the HITs approved qual, so I might lower it to something so that you can take part (7,500?). I'll make it clear when I announce the HITs later here. You are right about Example 2: "shifts" is not an implausible option. However, all of the check items are more like Example 1 where the answer is quite definite. I've even evaluated them in advance with some native speakers and responses are highly consistent. Still, I can imagine good workers might be hesitant to take the risk. I'll consider whether to lower the minimum requirement to 70%. Anyway, please keep an eye out here for an announcement of the HITs after a couple of days or so. In the meantime, good luck with your turking! Sincerely, Ralph
Thank you to all you members who have taken the time to look at the draft HIT. I will finalize it and make it live in a day or two. If you're interested in doing this HIT, please watch here for an advance notice before it goes live. Sincerely, Ralph
Any chance of opening this up to Canadian workers as well? Amazon makes it easy to include multiple locations (you can choose up to 15). Thanks for your consideration!
Dear swiftkick. Sorry not to reply more promptly: I've been getting the final version ready to go. Thanks for your suggestion. Indeed, I will make it available to CA as well as US. For my research purposes, that will be a uniform group. I will soon make an advance announcement of the HITs. Keep watching this thread. Sincerely, Ralph
Dear MTurk Forum users, Thanks for your attention to this thread. I will shortly (5-10 mins) make the HIT discussed in this thread available. Here are some specifics to help you find it: Title: Multiple-choice word quiz Requester: Ralph L. Rose Qualifications: HIT Approval Rate > 99% HITs Approved >= 10000 Location is CA, US I had hoped to lower the HITs approved qual, but the requester UI limits the numeric options, so I need to go with 10000. Sorry. I will initially request 5 assignments and collect the results to check that things are working fine, and then later request an additional 45 assignments (i.e., 50 total) after a day or so. So, even if you miss one of the first 5, please keep your eyes open for the later ones. Happy Turking, Ralph
There is still one of the first five assignments remaining, it seems, but things seem to look fine with the HIT. I will therefore request the remaining 45 assignments. If you're looking to do this HIT, please watch for it in about 5-10 minutes or so. Thanks! Ralph
Sorry, I seem to have hit a bit of a snag. This is my first time using Mechanical Turk after the fee structure changed and there's a couple of things I didn't understand. Anyway, at the moment, I will request only four more assignments (total 9 so far). Later, I will figure out how to get the other 41. (sigh...)
Thank you very much to all workers who did my HIT so far. I will now be requesting a further 41 assignments in the next 5-10 minutes. Please look for them. However, a message to workers who have already completed the HIT: Unfortunately, I cannot accept multiple submissions, so please do not repeat the HIT. I have sent an e-mail via Mechanical Turk to confirm your completion, so if you're not sure, please double-check your mail. Thanks for your understanding and cooperation. Sincerely, Ralph