MET-LAB, a controversial requester

Discussion in 'Requesters' started by gabriel, Aug 25, 2012.

  1. gabriel

    gabriel Active Member

    Joined:
    Apr 25, 2012
    Messages:
    37
    Likes Received:
    0
    Hi mates,
    I wish to share with you my email correspondence with this requester ( Met Lab <moodswings.metlab@gmail.com>; Id=A3C29DUUMQ9WLQ)
    He seems to be reliable despite of many bad reviews on turkopticon.
    My message to him:
    #On Thu, Aug 16, 2012 at 2:45 AM
    Hi,
    I do a batch of your Hits, majority approved. I was endeavouring to do my best with all of them, yet there are some rejected with the reason: " FAILED VERIFICATION". What that it's supposed to means?
    I kindly ask this in order to do it better on the future, if indeed I made any mistake (but sincerely I think it's a system error somehow, as I didn't draw aleatory circle, squares or something else)
    On the other hand I must set out that if you use some kind of "Majority Rule" as criteria of selection, the level of subjectiveness that is implied in these tasks, would turn them out in a roulette, and I don't wanna play it, you know, every rejection it affects my approval rate.#

    His reply:

    #Hi Gabriel-
    I am not a scammer. I am a real person. Though you will find there are many of us who run this activity and respond to emails, my name is Erik Schmidt and I am a PhD candidate in Electrical and Computer Engineering at Drexel University. We are collecting this data to hopefully develop intelligent systems that understand emotion in music.
    Given the nature of this data collection process it is very important that participants take the task seriously and perform it honestly. It is therefore paramount that we identify and remove data from those who are not doing so.
    Unfortunately, we have found this activity attracts a great number of people who do not take the task seriously. Examples of this are drawing a square or a triangle in the mood grid for every song they annotate, putting the cursor in the same spot for every song, or just quickly moving the cursor all over the grid in a nonsensical fashion.

    We also find many examples of people who do not understand the mood grid. For example, they label an extremely angry and intense Rage Against The Machine song as being very calm and happy. Or we also see people who rate the same song very inconsistently. This often happens the first time someone tries the HIT. We urge workers to not get frustrated if this happens, please read the instructions about the meaning of the mood grid and try again. You may need to practice the HIT more than once to get it down. Also note that you can navigate to our URL and practice as many times as you want before accepting the HIT.
    I also must acknowledge that people on our end failed to respond to emails in the past. That was not acceptable and is partly why many workers became frustrated. Those individuals are no longer involved, and we are working to do better on our end. Please let us know how things go with the HIT. If you have technical issues, let us know what browser and version you are using. We recommend the latest version of Safari, Chrome, or Firefox. Our HIT is still running and accept/reject verification will be completed when it ends early next week. Anther round of HITs will likely go up right after. If you feel you were treated unfairly, then please let us know. Thanks!

    -Erik
    #
    And the second reply from him:
    Hi Gabriel,
    Keep in mind that all of our HITs have to pass verification. This means that a human observes some of the songs you annotated. The ratings for these songs are compared to a panel of experts. We are *not* looking to see that you agree exactly with the experts. We *are* looking to see that you note that the emotion *changes* in those musical pieces. Remember, you are providing second by second ratings. If 20 experts provide data showing that the emotion changes in some way, but you leave your cursor in the same position for the entire clip then that indicates you are not engaged in the task and is grounds for rejection. We are looking to see that when you label the same piece of music twice that you produce the same rating.
    He attached this images with explanations:
     

    Attached Files:

  2. melissa71642

    Joined:
    Jan 25, 2009
    Messages:
    201
    Likes Received:
    0
    Nothing really controversial about this one, I don't think. He's just asking that you pay attention and carefully read the instructions before working on the hits or you may get a rejection. I consider a controversial requester to be one that doesn't respond to emails and just goes around passing out blocks and rejections willy-nilly, not one that's willing to explain their process to workers. This guy is exactly the kind of requester most of us on Mturk actually want posting work :)
     
  3. ohsostrange

    Joined:
    Aug 13, 2012
    Messages:
    3,760
    Likes Received:
    1,334
    A subjective task with a high level of rejections? Sure doesn't sound like the kind of requester I have any interest in working for.
     
  4. melissa71642

    Joined:
    Jan 25, 2009
    Messages:
    201
    Likes Received:
    0
    I wasn't aware of a high level of rejections. I was just going by what I read in the email that he sent back and the pictures that clearly explain why rejections were happening. I'll pass on this one as well and take it all back what I said above LOL :)
     

Share This Page