Hello Turkers: We are looking to create 2 or 3 HITs (to start) looking for opinions from individuals that have worked or are currently working in software development and or quality assurance. We are looking for your advice, experience and suggestions for formatting our questions, ensuring question clarity and using qualifications. We also want to pay fairly for good responses. The following are some clarification questions: 1) I have read that it is advised to perhaps do a small batch test first -- what is small? Opening it up to 10 people? And seeing how they respond, what is their feedback? Any thoughts on this? 2) The way I have structured our questions at the moment, the first question is essentially a qualifying question. If the Turker's response is "no" they would not be qualified to answer the following 6 questions. Clarification questions: (a) Would this thus best be set-up as a batch of 7 independent HITs with the first HIT being a qualifying HIT? They are all multiple choice questions, with one have an option to "fill in a blank for other." Or do you think it would be best to do one qualification hit followed by a second hit with a survey of 6 questions all in one HIT (so two HITs in total)? (b) If we have a second and a third batch of HITs, we can make them only available to the folks that successfully answered the first batch? 3) And lastly (for now) what would be a good price per hit for multiple choice questions? Thank you for your help in advance. Please feel free to share any other advice you have for new requesters. Alison
Is there a better place to post these questions? I was advised I should talk to this forum or a forum before posting, but evidently this is not the place? ***update*** I have set up a Google survey with 7 hits, but still wondering how best to set-up the qualifications so that we get good data and we promote good working environment for workers.
Hey all, Perhaps you are more interested in getting paid to help? I have set up a test hit, should take 20 seconds or less. Pays $0.05. Search "software" or "quality assurance." Hit is by Bas Hamer. Please take a look and then pop over for feedback. Bonus, if you have worked in software development or quality assurance previously, we will add you to our list of workers (this is of course dependent on your answer to the hit). Thank you! Thank you!
I'll answer as best I can: 1. The format that a lot of people like to see from requesters is a "qualification batch". Obviously the following numbers are up to your funding: Let's say you have 10,000 hits to do. Take 100 of them and release them, with 3 workers per batch, for 300 hits total. Pick 100 hits that YOU know the answer to. APPROVE THEM ALL, this is important, because you don't want to set the bar with penalizing people. You're investing in getting a good workforce. Then, manually go through all 300 hits and check their answers. You should have a decent number of workers who submitted good answers for every hit they did. Take a bunch of them, grant them a qual, and put up a 1000 hit production batch. See how quickly it gets done, how useful the answers are, and revisit your qual allocations, granting or revoking quals as necessary. This is a great method for workers AND requesters, because we don't worry about a requester unjustly rejecting us, and you know you have a batch of workers who are capable of doing good work. The only downside is the initial investment, but look at it this way: When you have a batch of 500 quality workers who you know do good work, you won't have to do another qual batch again, and you'll have an almost guaranteed workforce. 2. This would be MUCH better with a qualification test, not hit. You will need to use the mturk command line tools, but you will set up a qualification test with your one question. Have it verify with an answer key (can be set up when you create the test). The right answer grants the qual with 100, the wrong answer grants with 0. Then, add the qual to the hit with a required value of 100. Here is documentation about creating the qualification type, and here is documentation about the files used. The documentation is sparse at best, if you need help with specifics let me know, but the example files in the commandlinetools download will help a lot. 3. This is impossible to answer without knowing more. A good rule of thumb is to pay about $7.50/hr, so break that down it becomes $0.125/minute. If you want to get your stuff done super quick and your budget allows, double it if it's a short quiz. If your quiz is going to take a minute, paying $0.25 will make it fly off the shelves. As generic advice, make sure your quals are tight. I recommend usually >5000 approved, >95% approval rate. If you want to be extra sure, you can do >98% and >10k, or if you want it to be more open >1k >95%. I wouldn't go lower than 95%. Also, I'd recommend locking it to US/UK/Canada. FYI, as much as I love this forum, it isn't very good for rapid responses. I recommend stopping by MTC or the IRC to talk to workers who are more active. Also, post in the daily hits thread if you want to get a quick response, nobody really checks other forums. Best of luck! Let me know if you have any questions or need clarification on anything!
Thank you! Exactly the type of information I was looking for to complete and compliment what I had read in the Mechanical Turk guides.