Lots of new HITs on labeling human pose in the images

Discussion in 'Do My HIT!' started by TurkPoseEstimation, Mar 29, 2013.

  1. TurkPoseEstimation

    TurkPoseEstimation Verified Requester

    Joined:
    Mar 29, 2013
    Messages:
    65
    Likes Received:
    0
    Hi,

    I have submitted quite a few HITs on labeling body poses of people and we will add more HITs with this task in the future. Please take a look:

    The groupId is 2F7PLSP75KLO22CULB8QMQVJRD9ZTA

    Its a rather simple task that everyone can do. The default reward is 2 cents. Those people that are able to accurately complete the task will receive a qualification an an upgrade in payment.

    I am also very interested in feedback regarding the labeling tool used in the task.

    all the best,
    M.
     
  2. TurkPoseEstimation

    TurkPoseEstimation Verified Requester

    Joined:
    Mar 29, 2013
    Messages:
    65
    Likes Received:
    0
    You can also find the HITs by searching for
    "Label body parts of the person in the image" or "Max Planck"

    Best,
    M.
     
  3. gfsincere

    gfsincere User

    Joined:
    Mar 24, 2013
    Messages:
    168
    Likes Received:
    0
    Wait, so about 2 minutes of work for 2 cents? lol ok.
     
  4. justlookin

    justlookin User

    Joined:
    Jan 20, 2013
    Messages:
    1,122
    Likes Received:
    0
    Don't worry, if you do a great job you get a raise to .03 per HIT. That's about .90 cents an hour.
     
  5. Shego

    Shego User

    Joined:
    Feb 4, 2009
    Messages:
    475
    Likes Received:
    0
    Ninety whole cents? Pretty darn close to irresistible...let's see; If I work on these for 3 hours I can buy a 4-pack of toilet paper. Oh no, wait..I've got enough for that in my couch cushions. Sigh...smh
     
  6. TurkPoseEstimation

    TurkPoseEstimation Verified Requester

    Joined:
    Mar 29, 2013
    Messages:
    65
    Likes Received:
    0
    Fair enogh :)

    The key issue right now is the quality of labeling, the HITs have to be cheap because we have to average the results from multiple workers to get a decent quality. This is of course not fair to workers who try to do their best, but have to compensate for others that are not as careful.

    We can definitely pay more for HITs, but can only do that for people that are careful and reliable.

    I did about one hundred of our HITs to test the interface, and the average time per HIT is about a minute. I guess this requires a bit of routine, which you get after some time.

    In any case, if you decide to invest more time in our HITs I am open to suggestions.
     
  7. Haekon

    Haekon User

    Joined:
    Mar 14, 2013
    Messages:
    206
    Likes Received:
    0
    Maybe I'm just old fashioned, but I think I'd just prefer to pick cotton. The pay is better.
     
  8. admiraljohn

    Joined:
    Oct 2, 2012
    Messages:
    2,581
    Likes Received:
    0
    How about we offer him some constructive ideas rather than just offering rude and snippy comments about his pay rate? If people do nothing but complain like this when requesters post here, requesters will eventually stop posting here.

    Generally, TurkPoseEstimation, work is only considered worth doing when you average about 10 cents a minute. Offering a task that takes two minutes and offers ten percent of that may not net you many quality workers.

    You may want to consider increasing the rate for these HITs if you want to attract good workers.
     
  9. raitch

    raitch User

    Joined:
    Mar 12, 2013
    Messages:
    132
    Likes Received:
    0
    You could always set up a qualification test for the workers who are willing and able to do good work for you. You can weed out "bad" workers and make it so only those who meet your requirements can do the task. Then up the pay for those qualified workers and everyone's happy.
     
  10. shyaamist

    shyaamist User

    Joined:
    Mar 27, 2012
    Messages:
    415
    Likes Received:
    0
    At that pay rate, the only way you are going to get quality work is with a custom qual. No way is almost anyone, unless they are really poor/bored/bad etc gonna do this work they way it is!
     
  11. TurkPoseEstimation

    TurkPoseEstimation Verified Requester

    Joined:
    Mar 29, 2013
    Messages:
    65
    Likes Received:
    0
    Thanks for the feedback, I was originally very exited about the possibility to have qualification tests for the workers, but it turned out that the quialification tests do not support the "external question" type that I use to implement interface in this HIT. Take a look here:

    AWSMechTurk/2007-06-21/AWSMechanicalTurkRequester/ApiReference_ExternalQuestionArticle.html

    It says
    I might be mistaken at this point, and will need to dig more into this issue. For now I am thinking of granting qualification based on the current performance of the workers, and then paying more to workers with qualification.
     
  12. VixSin

    VixSin User

    Joined:
    Mar 6, 2013
    Messages:
    275
    Likes Received:
    0
    I get this thinking to a degree, but it is an insult that a requester knew they were paying at max $1.20 an hour and expecting quality. You have to pay for quality. Even if there was an incentive to get a qualification, I wouldn't touch these for less than .15, and that is if I am very bored (this hasn't happened in awhile). That is the reality. Most of us here give quality work, we deserve more for our time than this.

    Qualification test is the way to go. Some requesters do a number of 0.00 hits and give out qualifications based on those results. This may be a better option for you possibly.
     
  13. TurkPoseEstimation

    TurkPoseEstimation Verified Requester

    Joined:
    Mar 29, 2013
    Messages:
    65
    Likes Received:
    0
    Alright, so following the discussion here we have a new policy.

    The tasks for qualified users will be paid 10 cents per HIT. We also have a set of qualification tasks that are paid 3 cents per HIT. In order to obtain a qualification one needs to perform at least 25 qualification HITs correctly.

    All the best and hope you enjoy working on our HITs. We are a research institution working on algorithms for computers to understand contents of images e.g. to make cars automatically recognize pedestrians on the street or cameras to focus on faces of people. We do appreciate your contribution.
     
  14. TurkPoseEstimation

    TurkPoseEstimation Verified Requester

    Joined:
    Mar 29, 2013
    Messages:
    65
    Likes Received:
    0
  15. justlookin

    justlookin User

    Joined:
    Jan 20, 2013
    Messages:
    1,122
    Likes Received:
    0
    I think you deserve a lot of credit for taking another look at the pay rate on your HITs and setting up a qualification test.
     
  16. Flora

    Flora Admin
    Staff Member

    Joined:
    Jul 16, 2012
    Messages:
    70,597
    Likes Received:
    18,623
    Looks interesting. I am doing them now. I am finding that the blue font is hard to read against the black box and in the pictures too.

    I did 25 of them. I enjoyed them once I got the hang of the instructions.
     
    #16 Flora, Mar 30, 2013
    Last edited by a moderator: Mar 30, 2013
  17. leftfield

    leftfield User

    Joined:
    Oct 4, 2012
    Messages:
    968
    Likes Received:
    0
    Thank you for coming here and communicating with us on a personal level. I appreciate the chance to help you in your research.

    Would it be a possibility to get some sort of mid-review? I have read the directions thoroughly but I'm still not sure if I am doing the work to your satisfaction. Could you look at some today and approve or reject?

    I see the auto-approve time on the qualification is 15 days. I would hate to have put forth all this effort for $0.75 and a chance at qualification only to get 25 rejections because I didn't quite understand.
     
  18. shyaamist

    shyaamist User

    Joined:
    Mar 27, 2012
    Messages:
    415
    Likes Received:
    0
    Ok, gave 30 of the quals a try. I know I did the first few wrong because I missed one of the items in the tutorial/examples. Now we wait and see if they approve in a speedy manor. At that time, I will decide if its worth it.
     
  19. Blaize

    Blaize User

    Joined:
    Jan 16, 2013
    Messages:
    147
    Likes Received:
    0
    Well I hope these quals will be up for a while. I'm far too chicken to do 25 hits with the risk of rejection and only 1 TO. :( Did two to see if they approve
     
  20. bobbie01

    bobbie01 User

    Joined:
    Jan 29, 2013
    Messages:
    11,348
    Likes Received:
    62
    Thanks for being gracious and listening to the suggestions! I have done a few also but I want to make sure I am doing them correctly before I do any more.
     

Share This Page