ProductRnR - Any current issues?

Discussion in 'ProductRnR' started by TAC, Mar 6, 2013.

  1. TAC

    TAC User

    Joined:
    Sep 15, 2012
    Messages:
    474
    Likes Received:
    0
    Just wondering if anyone here is doing these HITs. Looks like there's some recent TO negative feedback but it's not completely helpful. :confused:

    These are easy HITs but don't want to get burned if others are having problems.
     
  2. zEric7x

    zEric7x User

    Joined:
    Dec 5, 2012
    Messages:
    655
    Likes Received:
    0
    I am not sure how this requestor was in the past but currently they seem pretty good. I not had any rejections and I heard they would disqualify you now than reject you.
     
  3. gabriel

    gabriel Active Member

    Joined:
    Apr 25, 2012
    Messages:
    37
    Likes Received:
    0
    ProductRnR Quality score issue
    After that old time when this requester gave mass rejection and no communication, about one year ago I received an email with apologies from him and things have been starting to go better.
    I've done over 30k hits in time(Identify adult images) with no rejection because they have introduced this system with Quality scores.
    So far so good, but recently my Quality score 1.1 reached to 13. Even if all of my other quality scores have excellent values regarding the red limits, I can't work anymore on them due to this damn score 1.1.
    I've messaged the requester for many times by now asking what is that about, but to no avail.
    It seems he has returned to the old bad habit - zero communication.
    Does anybody know what this score is about?
    Are there any workers in the same situation (on the last reviews on turcopticon I see similar complaints)
    Hoping they'll have a look around here and give us a reply.
     
  4. ProductRnR

    ProductRnR New Member

    Joined:
    Jan 29, 2014
    Messages:
    3
    Likes Received:
    0
    Hi Gabriel,

    This is ProductRnR. Thanks for doing our HITs. The reason why you are not qualified to do our work is because the quality of your judgments does not meet our expected level.

    Labeling quality is critical to us. We look at quality from different perspectives and calculate quality scores based on them. In order to maintain high quality score, a worker is required to follow the guideline.

    You can still continue to work on tasks from other requesters in Mturk.

    Thanks.
     
  5. gabriel

    gabriel Active Member

    Joined:
    Apr 25, 2012
    Messages:
    37
    Likes Received:
    0
    I'm impressed of your quickly reply here, even if it seems as an automatic impersonal reply.
    I've have been worked on "Indentify Adult images" for over a year and done about 40.000. All of my quality values are very good except of that score1.1 which suddenly reach to 13.
    I wrote you many messages entreating you to double check this issue but it seems you mark my email address as spam or something because no reply.
    This aspect below might be a prove that something goes wrong in your part regarding that score 1.1:
    Currently I'm still able to work on the another type of hits with basic identical criteria " Adult multiple images"
    How this could be explained if I didn't manage well the categorization criteria?
    Please again you to double check that score, reset it and keep a good communication again.
    I do assure you I'm endeavouring to submit good work and pay attention to every hit.
    I need you be cooperative not a laconic impersonal message: "You can still continue to work on tasks from other requesters in Mturk." !
    Thanks a lot for your patience yet.
     
  6. csrsathish

    csrsathish New Member

    Joined:
    Jul 25, 2012
    Messages:
    3
    Likes Received:
    0
    Seems the requester is done with the job.

    Hi all

    This is one of the highest paying requester. But for the past one month or so, the work is very less. Seems he has done with his work. If this requester is not available, money making become 1/4. If we dont find a requester like him, then no use sitting in mturk.
     
  7. mrtrentsd

    mrtrentsd User

    Joined:
    Apr 9, 2013
    Messages:
    371
    Likes Received:
    1

    I am having issues with the Identify pages containing videos thatdon't play (REWARD INCREASED).


    I last did these on Feb 27 - I did 2. Beginning March 3, my Quality score 15.1 is greater than 80 score has changed three times. It was 87 on 3/3,86 and then 80 on 3/4 and now 79 on 3/5.

    I can not see a reason for these scores to change 4 times days after doing two HITS. I have written the requester about this issue, but have not had a response yet. Maybe there is a flaw in his scoring bot, but I don't know. I fired up all the videos and let them play, but still lost the qual.

    I would suggest if you do these particular HITS, do as many as you can because you will lose your qual. I got paid, but I can not do anymore.




     
    #7 mrtrentsd, Mar 5, 2014
    Last edited by a moderator: Mar 5, 2014
  8. kcasepr

    kcasepr User

    Joined:
    May 5, 2013
    Messages:
    115
    Likes Received:
    0
    A large part of Rnr scoring system has always seemed to be what everyone else marked that item as well. On something like videos playing correctly: 1) the status can change from computer to computer, 2) a large portion of the people won't have the patience for this hit and will do them incorrectly, 3) they didn't define what a playing video page is in enough detail, so, people will disagree.

    I seriously doubt that many people will hold a high accuracy score on this one for long.
     
  9. mrtrentsd

    mrtrentsd User

    Joined:
    Apr 9, 2013
    Messages:
    371
    Likes Received:
    1
    3 hours later my Qual is back up to 86!?!?! (change #4) Still have not done any HITS since Feb 27. I just wish it made sense. Hopefully RnR will reply in some way.


    *** Well, its one day (March 6th) after the above posts, and now my qual is 80. 5 changes in that one qual score since submitting those two hits on Feb 27.

    March 7th: Qual back to 85. Again, no HITS done since Feb 27. So I decided to do a HIT - just one. I will post an update when there is one.

    March 8th: Woke up to see that my qual dropped to 79, then 3 hours later it is 78. Did one HIT yesterday. Two videos did not load, but my guess is that the score is a majority rules. I bet there are some third world turkers using a VPN are running a script and just pounding through these, which in turn f**k up my score.

    March 11: No HITS since march 7, but now my qual is back to 84 after changing twice on March 8th.

    Well, enough of these updates. No rhyme or reason to the qual changing so much.

    Good luck everyone.
     
    #9 mrtrentsd, Mar 5, 2014
    Last edited by a moderator: Mar 11, 2014

Share This Page