Arjun Jayaram

Discussion in 'General' started by battlemedic, Apr 1, 2010.

  1. battlemedic

    battlemedic New Member

    Joined:
    Apr 1, 2010
    Messages:
    2
    Likes Received:
    0
    Watch out for Arjun Jayaram. They hand out some hella vague hits and love the reject button.

    I was running a 99.7 with a 600 HITs completed, with my only reject being for something I mindlessly forgot to click.

    I sent a message asking why about nine out of twelve hits were rejected. No reply. No feedback. Its been four days. Joy :(
     
  2. JuliaJolie

    JuliaJolie Member

    Joined:
    Apr 1, 2010
    Messages:
    7
    Likes Received:
    0
    He's a sack of sh1t!!

    He's a cheating, lying, piece of shit. Someone on proboards found his Facebook profile and he looks like a smug c**t, too. Sorry that this is my first post and it's full of swears but I've never felt like I've wanted to smash someone's face in more in my life. I sent him a polite message at first, then after I read that messages were going ignored, I reported him to Turk and said I would pursue it to the highest point. I know for a fact I double and triple-checked those hits. He used us to spam. His Facebook wall even mentions cameras!:mad::mad::mad:

    (Apologies if you're a proboards member and have seen my other rant)
     
  3. arjunj

    arjunj Member

    Joined:
    Apr 1, 2010
    Messages:
    6
    Likes Received:
    0
    This is Arjun Jayaram. I opened the account in my name; but did not do the automatic rejection or approval. Engineers in my group are responsible for the approval or rejection.

    I understand the frustration. I had a talk wth my engineers and this is what I understand.

    * There are several Turks who click without even reading the question.
    * The team put in some pre scored hits into the mix as control questions.
    * These I was told are questions that a normal person who reads the messages can answer easily. It is done to prevent people from just randomly selecting items.
    * The team rejected the hits where the control questions wrong

    I think the logic seems to be right. But I would like to understand (please don't flame - just rational discussions) what we did wrong. We would like to improve our proceses and keep adding more tasks (several hundred of thousands) of hits to Mechanical Turk in the months ahead.

    I hear your pain and I am not going to tolerate anyone in my team rejecting hits without proper deligence. I will be checking this board. Please don't personally attack me. I did not do this myself. I look forward to creating a very healthy Mechanical Turk community with well paying hits. But want to avoid random hits being returned as well.


    Thanks and Regards
    Arjun
     
  4. JuliaJolie

    JuliaJolie Member

    Joined:
    Apr 1, 2010
    Messages:
    7
    Likes Received:
    0
    Why should anyone bother to do your HITs anymore after those mass rejections? Do you have any idea how bad my approval rate is now? There are a lot of tasks that I cannot do anymore thanks to your rejections.

    I had a 99% (well, over 99.something) approval rate and it has now gone down to 97%.

    I had just 7 rejections before I (literally) took your HITs and those rejections were in my first few days of Turking.

    I now have 49 rejections. Those were ALL from you.

    You don't seem to have bothered answering anyone's Dashboard messages, as can be seen on other Turking messaging boards. Please give me one good reason why I should bother to do your HITs anymore. There are a lot of scammers on Turk ("Fill out this form with your personal info and maybe we'll pay you $10!!") and your mass rejections speak volumes. Most Requesters try to smooth things over but all you're doing is offering excuses and absolutely zero reparations. People worked very very hard on your HITs, and I was one of them. I triple-checked my work. Not a single answer was incorrect. None of them should have been rejected. How can you possibly explain yourself?:mad:
     
  5. arjunj

    arjunj Member

    Joined:
    Apr 1, 2010
    Messages:
    6
    Likes Received:
    0
    Hi Julia,

    I understand your frustration. I am pained by the trouble this is causing to people.

    We are building software in Artificial Intelligence. The data that you are providing is
    'teaching' the software. So if the data is incorrect, the software is useless. We spent thousands of dollars earlier and all most of the data that we got was random clicks. So we implemented some process changes.

    Coming back to the current situation - What can we do forward that will make it easier or more fair? I am much better off knowing that we have a good team of trustworthy individuals to do the hits. I am thinking in the lines of giving this set of Turks the first crack at the hits. As I mentioned, we will have hundreds of thousands of hits in the months ahead. I look forward to your recommendations.

    BTW: I checked the emails - It goes to a general message box; does not come to me. There were hundreds of them. Most of them thanking for the money they made and several people angry at the rejections. Not practical reply to every one of them personally.

    Look forward to your suggestions. Please understand again that this is not me personally. It was done by engineers in my group trying to improve the process without realizing the frustrations they caused to many people.

    Thank You
    Arjun Jayaram

    I am going to review all the control questions tomorrow with my team. If they are subjective and can be interpreted differently depending on who is reviewing it, I will take some corrective action.

    Also, thank you for bringing this to my attention.

    Thanks
    Arjun
     
    #5 arjunj, Apr 2, 2010
    Last edited by a moderator: Apr 2, 2010
  6. RancherMum

    RancherMum Member

    Joined:
    Apr 2, 2010
    Messages:
    5
    Likes Received:
    0
    I also did a fair amount of these HITs, most of which were rejected, bringing my approval rating from 99.8% down to 93.4%! And with no explanation as to why they were rejected.

    I read every one (Some of them twice just for clarification), but most of them took less than a minute or two to read through and click, could this be why they were rejected? I know they weren't wrong, it's pretty easy to read a snippet and decide whether the person wants to buy a camera or not.

    Would really like to get this fixed as 1- I did the work accurately and honestly, so I should be paid for it and 2- It has completely destroyed my HIT rating. It is quite frustrating to do all that work, only to log in a day or two later and realize you completely wasted your time.
     
  7. mle

    mle New Member

    Joined:
    Apr 2, 2010
    Messages:
    1
    Likes Received:
    0
    I am grateful that I did not waste too much of my time on your hits. I only did 13. But I will not be doing any more because 7 were rejected. My stats cannot take that rejection rate. I would suggest offering examples and better instructions. Although I still wonder about you being a scammer.
     
  8. mysconnie

    mysconnie Active Member

    Joined:
    Jan 29, 2009
    Messages:
    28
    Likes Received:
    0
    The hardest part about the hit to the approval rating is that we all start at 100%, but it doesn't take much to bring it down, and when it goes down significantly, it's insanely hard to get it back up again. If you put your name on something, you had better be prepared to accept the consequences. You have done irreparable damage to the scores of a lot of turkers. Don't expect any sympathy from them. I only did three. All three were rejected-and don't tell me I screwed up, I know better.
     
  9. RancherMum

    RancherMum Member

    Joined:
    Apr 2, 2010
    Messages:
    5
    Likes Received:
    0
    Been thinking about some of the comments included in these HITs, and imho many of them were a bit subjective, to be honest. For example: "Dog ate my camera" - is this person interested in buying a camera? Hmm, could be since the dog ate it, but they don't actually say "Dog ate my camera, need to go buy a new one." Perhaps they don't have the money to replace said camera and won't be looking to buy a new one. Maybe they're waiting to sell said dog and won't be interested in buying a camera until then. Hard to tell from a 4 word snippet of information. I'm just saying.

    There were also a few that were not in English [I recall seeing Spanish, Portuguese, and German, just off the top of my head]. I translated those to the best of my ability [aka Google], but perhaps the translations were a bit off?

    Other than that, and the fact that some of the HITs took less than a minute for me to finish, I can think of no reason as to why these HITs were rejected by your 'team'.

    BTW, I did about 60 of these HITs for you and 31 were rejected. So painful to see your rating drop by nearly 7% when you KNOW without a doubt that you did the work properly.
     
  10. Dana

    Dana New Member

    Joined:
    Apr 2, 2010
    Messages:
    2
    Likes Received:
    0
    Hi. I'm new to this board (hiyas), I registered just to post in this thread.

    I first did these camera hits on March 25. On that day I did 18 of them. It wasn't until several days later that the results came back (1 accepted, 17 rejected).

    Had I realized, I wouldn't have done any more of these hits, of course, but since he was so slow to accept/reject, I attempted to do these again a day or so later.

    But when I attempted to do these again, I found out I was not qualified to do these hits anymore.

    Looking at the hit info, I found that the hit creator had set the qualification to "hit approval rating is 97"

    On March 25, my hit approval rating was 97.x. When I tried to do more the next day or so, my hit approval rating had risen to 98.x, making me no longer qualified.

    I sent a note to the requester, Arjun Jayaram, through Mturk's "Contact Requester" system. I suggested that perhaps the qualification they would want to use was "hit approval rating not less than 97," as "hit approval rating is 97" disqualifies not only under 97 but over 97 as well.

    I had never sent a note to a requester before. The form said that my email address would be included in my message.

    So, I did expect some kind of acknowledgement from the requester. I had taken my time to help him.

    I got no reply.

    A day or so later, the hits still had the wrong qualification and, thinking perhaps I had submitted the message incorrectly, I wrote again.

    Still no reply.

    A day or so later, I saw that the hit qualification had indeed changed, to:

    HIT approval rate (%) is less than 100 97 You meet this qualification requirement
    HIT approval rate (%) is greater than 94

    Sloppy, but better.

    Since this change put me back into the qualified range, I did some more of these hits, about 35 of them, with, as it turned out, the same dismal competency on the requester's side

    At the time I did the later hits, the results of the first hits still hadn't come back.

    The guy had his qualification wrong when I first did them. I took the time to bring this to his attention, twice, and got no response at all for my trouble. And at the end, for doing about 50 or so hits, I had an absolutely ridiculous rate of rejection from him, that ended up moving my approval rating back into the 97% range, ironically.

    Really bad all over on this requester's part.
     
  11. RancherMum

    RancherMum Member

    Joined:
    Apr 2, 2010
    Messages:
    5
    Likes Received:
    0
    You gotta love it. Just logged on to mturk to peek at HITs and noticed that Arjun Jarayam has put another nearly 25,000 HITs up about camera purchases. Funny thing is, it now says that I am not qualified to work on these HITs, gee I wonder why???? [not that I would at this point.]

    I know that I am new to this and all, but shouldn't the old mess be taken care of first before a new one is created? Or is this how it always is in mturk land? Is there a way to let Amazon know what's up so that a bunch of other turkers don't face the same crap that we're dealing with?

    And yeah, Dana, I also emailed this person telling him that his rating was set to exactly 97, instead of 97+, no response, no thank you whatsoever, just 60% of my HITs rejected with no reason as to why.
     
  12. snoogiebug

    snoogiebug New Member

    Joined:
    Feb 11, 2009
    Messages:
    2
    Likes Received:
    0
    ok so tell us arjun, are you going to pay the hits we have done or are we stuck with shitty approval ratings? I enjoy these hits and was looking forward to doing more untill i got almost all i did rejected... i do quality work and expect to be paid for not scammed out of it by people who obviously don't know the english language.....i am hoping you make this right ..there are plenty of turkers who would work these hits ,but yu and your so called team have ruined our ratings and we are not able to do the work we once did because of it...... let's see some kind of retribution for this!!!!!
     
  13. brus51

    brus51 Member

    Joined:
    Feb 6, 2009
    Messages:
    11
    Likes Received:
    0
    Stay away from Rebecca J Passonneau

    Stay away rom this person, she gets you to do the hits then tells you, that you have to complete all 15 hits to get paid. It does not matter if you only do 1 or 14.
     
  14. James Franzen

    James Franzen Member

    Joined:
    Apr 3, 2010
    Messages:
    7
    Likes Received:
    0
    arjunj,

    It is very cool of you to take the time to work with us on this. I am new at this, I did a bunch of hits for you. Over half of them were rejected. I agree with what is being said by a lot of people that some of the statements were very vague and ambiguous. The one that stands out the most to me is "My dog ate my camera" this statement could be taken diffrent ways.

    I would like to personally see an example of some hits that were rejected of mine. I am thinking that that was one of the "test" answers and that is why I had so many rejections.

    Also, some of the options were in a different language and the hit did not say anything about having to know a different language to be able to do the hit.

    I think that instead of having answers that are predetermine you should use the majority rules. but not best of 3. The odds of 2 other people cheating on the question is too high. Or maybe having a test you have to take before you can do the hits.

    I enjoyed the Hits that I did for you but really need an solid reason why majority of my hits were rejected before I even think about doing any others by you.
     
  15. arjunj

    arjunj Member

    Joined:
    Apr 1, 2010
    Messages:
    6
    Likes Received:
    0
    Hi,

    I am having a meeting with the engineers on Monday evening. We started an email chain on this. I will update you on the details. To set expectations, we will try to find a way to redo some (not all) of the rejections (in whatever way is appropriate on Turk).

    I have taken a lot of feedback from the message boards back to my team. Some of it is very useful and will help us improve the process. You can see more hits being posted perhaps not in my name (at the minimum, learnt to be more careful about that :) ) .

    I looked at some of the replies on email. One person admitted "I click on the hits by random. By just chance, I should get atleast 20% right. why did you reject a 100%?" Turks like that are the reason why we need processes. There is no value in random clicks. These jobs are there because machines cannot do them. A machine can do random clicks a lot faster than people.

    Thanks
    Arjun
     
  16. JuliaJolie

    JuliaJolie Member

    Joined:
    Apr 1, 2010
    Messages:
    7
    Likes Received:
    0
    Are you kidding me? I'm not the person who emailed you that, but I don't think you understood that person's email. What they obviously meant was that IF - *IF* - they clicked randomly, they SHOULD have at least 20% correct answers STATISTICALLY. No-one would actually admit to clicking randomly because, first of all, that's idiotic and gets you banned from Turk, and secondly, it's asking for rejection and no pay for the work you did.


    I and all of the other posters on both proboards and this board are NOT bots or machines. We did the work, for a ridiculously small amount of money to start with, and were then rejected. We are here voicing our opinions because you took time and money away from us and what you did is unconscionable. If you spent hours upon hours doing a tedious, low-paying (to the point of insulting) job, and then never got paid AND had your reputation (i.e. our approval rates) sullied, how exactly would you feel?

    My guess is you would probably sue, because that's loss of time and earnings. Do you really think you can get away with this? I've been giving you the benefit of the doubt so far but you're just full of flimsy excuses and empty promises. You're not interested in repairing your damage at all. Good luck trying to stay on Turk, because I'll be keeping an eye out for massive batch hits that even remotely resemble your camera crap.

    BTW, *really* smart job on basically admitting that you're going to play the system by putting up new HIT batches under a "different name" so you can con more of us into working for you for free. How stupid do you think we are?
     
  17. CaseyMetro

    CaseyMetro New Member

    Joined:
    Apr 5, 2010
    Messages:
    3
    Likes Received:
    0
    I use Twitter and am familiar with reading tweets. I was born and raised in USA and I went to college here, so i know how to read english. I know I double-checked my answers, and did not deserve to have most of my submissions rejected.

    I had 3000 HITs under my belt when I first started doing these camera HITs. Since I did several hundred of these, my quality rating went from a high 99% to 90.2%. As a result, I am no longer qualified to do many of the HITs I was comfortable with.

    I have to do another 3000 HITs without rejection to bring my quality back up to 95%!

    I too notified Arjun Jayaram that he had incorrectly set the qualification at exactly 97%. Perhaps the same type of logic error was used in the quality control/rejection criteria?

    You are so right about Rebecca J Passonneau. I was struggling to bring up my quality score from these camera HIT rejections so her scam did not help!

    The instructions stated to not accept this HIT if there were less than 15 available. I am certain there were 15 available. In fact, I did 2 out of the 3 that had 15 HITs available and puposely did not do the HIT that had only 14 available.

    I did them one after the other until it returned me to the 'HITs Available' screen, which I assumed meant that I had done all 15.

    When I checked my dashboard for the day, I had submitted only 14 of each. I wrote her an email right away, explaining that there were 15 available when I started, and I did all that the system allowed.

    No luck. All 28 were rejected, tanking my quality score again.

    BTW, this is her: http://www1.ccls.columbia.edu/~beck/

    Is she really as nice a person as her picture appears or does she seem to be laughing at all the people she scammed? We'll see after she replies to the 3 emails I have sent her by now.
     
    #17 CaseyMetro, Apr 5, 2010
    Last edited by a moderator: Apr 5, 2010
  18. arjunj

    arjunj Member

    Joined:
    Apr 1, 2010
    Messages:
    6
    Likes Received:
    0
    Hi,

    We have started the process to pull the data for analysis. I will have some more information in a couple of days. I will post updates on this board. Thank you for your patience.

    Thanks and Regards
    Arjun Jayaram
     
  19. arjunj

    arjunj Member

    Joined:
    Apr 1, 2010
    Messages:
    6
    Likes Received:
    0
    Hi,

    I have some more data. As I had mentioned earlier, we looked at the data more closely. I also followed up with Amazon Turk based on some feeback from the board.

    * I asked about creating blank hits
    * Inviting specific people to do the hits.

    Here is the feedback from Amazon.

    "Greetings from Amazon Turk,

    Thank you for contacting Mechanical Turk about choosing the workers for your hits. At this time we do not have a way to choose specifically who you want to work on your hits. In creating your HITS you can limit by approval rating and country. You may want to post the question to the discussion boards to see if other requesters have been in a similar situation and see what feedback you can get.

    The Resource Center includes sample use cases and customer interviews demonstrating how businesses are using Amazon Mechanical Turk for a wide variety of work, including website moderation, data governance, metadata creation, and more. The Resource Center also includes a User Guide, a Best Practices Guide to help Requesters optimize their approach, and a section to help Requesters choose the right tool for creating and managing HITs."

    ... Please post on the board if you were able to get past this. In the meanwhile, we will also be changing some of the control question process going forward.

    Thanks
    Arjun Jayaram
     
  20. zshipp

    zshipp New Member

    Joined:
    Apr 9, 2010
    Messages:
    2
    Likes Received:
    0
    I too am someone who got burned by this poorly designed HIT, but was fortunate to at least only submit 20 or so instead of some of my friends who submitted hundreds and are now suffering financially because of the damage to their acceptance rate . I also sent an email to the requester about that handful of HITS and got no reply. For what its worth I had a 99.6% acceptance rate with over 40K hits since Sept., so hopefully you believe I complete this work in good faith. I also am a graduate student and have ample command of the English language. Assuming the arjunj here is the actual requester and is trying to resolve this in good faith (his or her claim that people were emailing and thanking for the money makes me skeptical) , two things need to happen:

    1. Probably more important than the money which would have been paid for the HITS (though should be paid too), it is damage to the hit acceptance and inability to do other hits that is most vexing to the workers. For example, I have 2 friends, both graduate students, who completed these hits in good faith were adversely affected. One had an acceptance rate drop from >99% to less than 95% and couldn't complete ongoing work for a requester that would have paid him ~$60 twice a week for a few hours work. He will be unable to do so until he does another 4,000 hits or so perfectly to bring his average up, so he'll lose a couple hundred bucks in the near-term. The other friend was just starting with MTurk and now has an acceptance rate so low that he can't do any of the better paying hits and has given up completely.

    2. You don't need elaborate mechanisms to fix your perceived problem of people randomly clicking like preferred workers or comparing best of 3 or whatever. My guess is that it is just an issue of the clarity of your test questions. Don't use "Lost my camera" or "Dog ate my camera" as those are ambiguous at best. ("I lost my camera, so I'm going to go look for it." My dog ate my camera, so its lucky I have a back-up and don't have to buy a new one.") A conscientious worker would mark those as negatives to help the requester avoid false positive returns. Just use straight-forward, clear-cut test sentences. "I'm shopping for a new Canon to replace the one I just dropped in a lake." "This new camera I bought is so good I won't have to replace it for years." etc.

    Sigh. Something tells me I just spent 10 minutes whistling into the wind, but at least you all have an experienced Turker's mostly unbiased view of the situation.

    Another thought occurs to me - neither here nor in the other Turker Nation forum have you given a concrete example of one of your control questions. There are examples of what us saying what we consider bad, but not you. If you truly think there was nothing wrong with your HIT design other than thresholding, post a couple of the Tweets on which you based your rejections and get some feedback. I think in the other forum you said you looked them over yourself and didn't see a problem, but clearly there must be one.
     
    #20 zshipp, Apr 9, 2010
    Last edited by a moderator: Apr 9, 2010

Share This Page