Now I'm scared to do them. I've done I think 40 today. I stopped after hearing about rejections. Perhaps he will come here and explain himself.
Sounds like technical error. I would hope so. It's impossible to get all rejected unless he was just clicking anywhere just to get paid. I really want to give him benefit of the doubt. He approved all of mine the other day, 20-30 or so.
I know. I don't know if they were really bad and should of been rejected. I guess I will just do some and see what happens. Crossing fingers.
While I suppose it's possible I wasn't doing them the way he wanted in some way, I did them exactly the same as I did the qual hits. So if they weren't what he wanted, he should not have granted the qual. That's the whole point of the qual isn't it? So that he can get workers who are doing it the way he wants? I think I just got a lot of pictures that had no lower body shown, and I think he rejected those for not having enough points in them.
I went by examples he had in the tutorial. I looked at the pictures closely and saw what he did with the ones that were not visible in the picture. Leave those points out since you cannot see them, is what I did.
Same thing happened to me. I not only got my first rejection, but my second rejection as well. At least I only did 9 of them? Never again though.
I wish he had at least given feedback to say what he didn't like about it. Sorry about your rejections.
I had to reject these HITs because they had *only* 3 facial keypoints labeled, nothing else. It has been clearly stated in the task description and in the tutorial that all the keypoints inside the image have to be labeled. The tutorial also included multiple images with all the keypoints, how this did not ring the bell I don't know. My mistake was that I initially accepted a bunch of such HITs because they sliped through the system. That is why people continued to submit such work, and eventually did quite a bit of it.
Ah I see, that explains it. My 50 were approved so I'm happy to hear that I did mine correctly. I followed the directions to a T and went to it over and over to make sure I got it right. I enjoy doing your HITS. Thanks!
I understand that we rejected a few HITs recently a bit hastily. In the majority of cases the HITs were rejected because they did not include all the keypoints that where within the image boundaries. I will to two things to address that: 1) I will review all the rejected HITs from the users with qualification and give feedback here. 2) In the future I will review all the rejected HITs, and we will not reject HITs unless the annotation is clearly bogus, as was for example the case when results of some HITs contained nothing but 3 facial keypoints. The rejected HITs will also come with clear description for the reasons why HIT is rejected. I also added some more images for labeling, here is the link for qualified workers: https://www.mturk.com/mturk/preview?groupId=2EFB3Q39Z0B8QZHXQ55V0TVRKPFVK3 and here are the qualification tasks: https://www.mturk.com/mturk/preview?groupId=2PQMB3Q39Z0D20MF3UVE3STVRT8JU5
I am looking at the HITs rejected so far and the majority of them miss sigificant portion of keypoints or have the occluded/visible keypoints mislabeled. Common mistake seems to be forgetting to annotate the facial keypoints (this is in contrast to other rejected HITs that annotated facial keypoints only). Here are some examples: http://datasets.d2.mpi-inf.mpg.de/amt-human-pose/tutorial/examples/r_ex1.png http://datasets.d2.mpi-inf.mpg.de/amt-human-pose/tutorial/examples/r_ex3.png In other cases the occlusion labels were the problem. In this example the right/left knee and hip are marked as occluded, which was considered a mistake. http://datasets.d2.mpi-inf.mpg.de/amt-human-pose/tutorial/examples/r_ex2.png In this example the left arm is marked occluded, while in fact it is poorly visible, but not occluded: http://datasets.d2.mpi-inf.mpg.de/amt-human-pose/tutorial/examples/r_ex4.png There was a certain tollerance for such errors in the system, but I must admit that in a very small number of cases the correct HITs were rejected because of the occluded/visible issue. Just to make sure in the future all of the HITs will be checked manually before rejection. Only mistakes such as missing significant number of keypoints, or obviously wrong occluded/visible labeling of significant number of keypoints will be penalized with rejection.
Thank you for your explaination! There are always two sides to a story! I really like these hits and look forward to doing them but I did get a little scared. I did a few and they were all approved so I guess I am good to go. Thanks again and look forward to working on your hits!
Did anyone attempt the .15 hits? I looked but it wanted me to outline a person that was behind 2 other people. I started to attempt it and then returned it. lol Where did the .10 hits go? I was just feeling confident with them lol
Finally done with 25 qualification HITs...I am a bit nervous now...although I am pretty sure I did them fine...
Hi, thank you very much for your work, the majority of results look really good. I processed the last batch of HITs and was more careful with rejections this time. I will need to work on another project for the next 10 days or so, but will post more HITs after that. We sill have quite a bit of images that need to be labeled.