Just another story that validates my reason for not choosing hits that are graded based on the work of other turkers. A few days ago I worked on a couple dozen .25 hits. The object was to use the link to visit another site and then ascertain whether a sales tax was charged to orders shipped to the state of Illinois. Easy enough. After the first couple of hits, I wasn't comfortable with the possibility that the web site was giving me all the info I needed up front, without actually ordering something from them. So I called the businesses instead, using the phone numbers listed on the web sites. And this is how I proceeded with the rest of the hits I did, except for a couple of them. Lo and behold, I noticed yesterday that the requester was beginning to approve the hits. I was happy. but later, when I checked again, I saw that he had rejected TEN of my hits! His reason, "Other wokers who did this hit gave the opposite answer." I was stunned. Then I was pissed! He had not stated in the hit that approval of our work would be based on the answers of other turkers. If he had, I would not have accepted the work. I wrote to the requester: "Interesting. I wonder if the other workers took the time to actually call these businesses and speak with a real person to get their information. Because I did, and that's how I came to the conclusions I did." Sometimes even your best still isn't good enough.
Shego, I feel bad for you. I don't work on hits which are rated by other workers, if I know before hand. This is a crazy way to approve or reject hits.
I agree that it's not a wise way to decide which hits to approve. For all he knows, those other people could have just been answering the same thing on every hit they did without even looking because there were only two choices to make--yes, or no. Maybe requesters over estimate the work ethic of turkers. Most of us are decent, honest people, but we all know there are just as many who don't care.
That must be the reason for so many rejects on 'Are these items different'. Companies with employees, would not usually hand out the same task to three of the people they are paying decent money for. So why does it happen on mturk? Put in a qualification system, pay people a fair amount, and they should be honest if the directions are clear. Just imagine if they paid 75 cents a hit just to one person, for 4 or 5 minutes work. Thats still decent enough to try your best for those few minutes and provide good results. In your case, you made a smart choice to call, instead of going though the ordering process, which sometimes wants a CC number and everything else before it tells you the shipping costs or tax. It's sad that multiple people with wrong answers can overpower one person with right answers. And it's said that people that put out good paying hits probably get some garbled trash back too. That's were a good qualification system would help, and should be used.
I think honest workers are getting penalized without any fault. Few days back I was working on a hit which paid about $0.10. There was a survey to be answered which was three pages long and must have about 15-20 multiple choice questions on each page. Since, I bothered to read, understand and answer each question, I ran out of time when I had just three answers left to click. I was bugged, not for losing the money but by the fact that others who will click on answers without reading will be able to complete that hit on time. I sent a mail to that requestor and asked him to increase the time on his hit if he wanted honest answers for the survey. I got to know from his reply that he is a university student who has put up this hit for his research project. I don't know how accurate his findings will be on this hit.
Most people do those surveys because they generally pay okay and don't take super long. But I too, have run out of time on hits that required reading or thoughtful answers and it's maddening!
Although I do admire that folks are doing surveys on some of the topics that are posted, their data and results have to be terribly skewed since the results are not going to be statistically reliable or valid (because of the selection process). That should be factored into their results.
Sonica and Shego I am one of those mTurkers who love to do surveys and the easy is way to get around the time limited is to do the survey first before accepting it. About 90% of the surveys will ask you to open a new browser and paste on the URL that they give you (or click on the URL) this will allow you to do the hit without accepting it first, when you complete the survey and get the completion code ( or what ever is required to comfirm that you did do the hit) go back to mTurk accept the hit and just paste the code in to the comfirmation box and submit. This has worked for me every time. Brus51
Brus51, I do the same whenever possible, but this particular hit did not allow me to proceed before accepting it. I was not interested in that hit, I was doing it because someone here had a problem with submissions. Some days back there was a problem at mturk, where hits submitted were not showing up in our account.
i'm annoyed with the HITS that ask for your opinion, then get rejected. it's like, if you wanted a right or wrong answer on an opinion question, don't ask for someone's opinion!! the HIT got rejected, which hurts my rating. i'm ticked!