Setting expectations for a survey.

Discussion in 'Do My HIT!' started by virgilp, Apr 9, 2014.

  1. virgilp

    virgilp Member

    Joined:
    Apr 9, 2014
    Messages:
    7
    Likes Received:
    0
    Hi,
    I'm trying to do some research using MTurk, and I want to see what is the expected cost.

    So, here is the job: you receive a brief for some design-related work (e.g. "create an ad"), and have to think up what phrase you would use to search for an image. Then you perform the search on my site, and rate about 60 result sets (5 images per "result set" - i.e. you have to look at 300 images).

    For this experiment - the turkers would have to be qualified, (to have used paid stock images before, e.g. from shutterstock, istock, dreamstime, fotolia etc.). Also I ask that they record the session (e.g. using quicktime on OSX, or ScreenRecorder from Microsoft on Windows) and upload it to a site (e.g. youtube or vimeo)

    Depending on how long you take to analyze those images, the task should take between 15 and 25 minutes. I'm thinking I should pay 10$/HIT. Does that sound fair to you?
     
  2. bobbie01

    bobbie01 User

    Joined:
    Jan 29, 2013
    Messages:
    11,348
    Likes Received:
    62
    I think that sounds like a more than fair price. What will you do to ensure that you get qualified workers? A HIT paying that amount is going to go like hotcakes and you need to make sure you are getting what you pay for. Good luck (and welcome to the forum).
     
  3. virgilp

    virgilp Member

    Joined:
    Apr 9, 2014
    Messages:
    7
    Likes Received:
    0
    Cool, thanks.
    To your question: that's why I ask for the video; I have quite enough experience to know if someone is faking it, and I'll reject the fake jobs. I'm also planning to ask for "master qualification" (or whatever it's called), so that I don't get "one-time accounts" applying for the job in the hope that they'll fool me.
     
  4. Tjololo

    Tjololo Pony
    Staff Member

    Joined:
    Jun 14, 2013
    Messages:
    13,720
    Likes Received:
    105
    While the Master's qualification is useful for that, something far more useful is to set simple thresholds. Setting an "approved hits is not less than 5000" and "HIT approval rating is not less than 99" will get you a lot more (think thousands instead of hundreds) of responses a lot faster than the Master's qual will.

    We as workers have no idea how to get the Master's qualification. I have >10,000 hits approved, 6 rejected, and I don't have it. I know others with >100,000 approved and ~40 rejected and they don't have it. I know still others with ~4000 approved and many rejected who DO have the Master's qual. These numbers aren't exact (except for mine), just examples, but it illustrates the point: There are many very very good workers who simply do not have the Master's qual for one reason or another.

    I think for your instance, an internal qualification would be best. Do a single qualifying hit that pays maybe $1 or $2, but make sure everyone know's it's a qualifying hit. Then, approve every submission, but only grant the qual to the good workers as you find them.

    There is merit to rejecting bad work, and that is a requirement in some instances. However, if it's at all possible to avoid it, I recommend looking at the alternatives. Rejections are a HUGE deal for workers. In order to offset a single rejection, a worker needs 2000 approvals. This makes the rejection statistically insignificant. Because of this, there's nothing that can ruin your reputation more than rejections.

    I'm not saying to accept bad work by all means. Any work you can't use, it's not right that you should have to pay for. What I'm saying is you should try to take all possible means to prevent those workers from taking your hits at all. If a worker actually tries, and puts forth real effort, but doesn't give you the work you're looking for? That's an instance where you should pay them for their time, but not allow them to do any more of your hits. This is where the internal qual comes into play; for workers who are not "up to snuff", you can just remove their qual and never have to worry about them doing work again.

    The hit sounds good though! I would normally caution against requiring a screen recording, but for a $10 hit that doesn't seem to be bad. Some are more cautious of their security, however, so you'll be losing some potentials because of that. It's a trade-off for valuable workers vs potentially useless information. This again would be a good use for a qual hit. A $0.50 that takes 5 minutes would be considered "good pay" for most workers. If I'm doing the math right, it would be 100 images to be 5 minutes. You could then get a large number of responses, parse through them for "good" workers, and assign them a qualification for your main work.

    I realize this is kindof a wall of text, but I wanted to give you some quick information. You can get a lot more information if you ask in the daily "Greatest hits" thread. Hopefully this helps!

    As per your mod request, I've forwarded it on to the appropriate people. You will most likely receive communication from them for the final switchover.
     
  5. virgilp

    virgilp Member

    Joined:
    Apr 9, 2014
    Messages:
    7
    Likes Received:
    0
    Thanks for the info. Ok, I won't be using "masters" qualification, but the filters you suggest.
    I've learned in the past (used mturk once before) that HIT rejections are big deal for turkers; but in this case, there is no way one can fail at the task (the task simply asks for your opinion, all you need to do is be honest) - the only way one can fail is by trying to do the HIT without meeting the requirements. In this case, a "reject" seems appropriate :)
     
  6. Tjololo

    Tjololo Pony
    Staff Member

    Joined:
    Jun 14, 2013
    Messages:
    13,720
    Likes Received:
    105
    Ah, yes definitely in that instance a rejection is warranted. But, those can be mitigated. Typically when you hit 5000 approved you're in it for the long haul, and you're not just randomly submitting hits.

    Thanks for the rapid response, I look forward to seeing your hits out there! As I mentioned, feel free to stop by the daily hits thread and see what others think.
     
  7. virgilp

    virgilp Member

    Joined:
    Apr 9, 2014
    Messages:
    7
    Likes Received:
    0
    I've published it (a first test batch of 5 hits). Don't know how to give the link - it's "Image Search Quality Survey "
     

Share This Page