Returned Hits

Discussion in 'General' started by kenaboo, May 6, 2012.

  1. kenaboo

    kenaboo New Member

    Joined:
    May 4, 2012
    Messages:
    3
    Likes Received:
    0
    Hi everyone!
    Will returned hits hurt my approval rating? Sometimes I click on a hit, and find it difficult to complete, so I return it. I figure returning it is better than taking a chance on doing it wrong.
     
  2. OneEye

    OneEye Member

    Joined:
    May 4, 2012
    Messages:
    10
    Likes Received:
    0
    I was wondering the same thing. I have only been at this about a week but I have not seen a HIT with qualifications based on return percentages or hit submission percentages, and I know its preferred to return it as opposed to doing it wrong or incorrect.

    Sometimes I have to return a hit because when I accept it says there is no more work but it keeps the job open. I actually abandoned a few by accident cause I did not realize these jobs were still considered open at first. That being said there still may be some way our return % plays into things and I would also like to know for sure.
     
  3. riazthemturkhelper

    Joined:
    Jan 14, 2012
    Messages:
    1,640
    Likes Received:
    0
    Hi Friends the the below mails from me to Mturk Team and the mail I got reply will be a better answer to both of your questions


    Here is the mail me send to Mturk Team

    And here is the reply got from Mturk Team
    But from my suggestion please don't return any Oscar HITs as that will affect your Quality Score(Nothing to do with Approval Rate) to become a Trusted Worker.

    Hope this post will help you as expected :).
     
  4. gourab

    gourab User

    Joined:
    Apr 24, 2012
    Messages:
    91
    Likes Received:
    0
    I just hate Crowdflower Hits,after doing 4-5 Hits,they just keep saying,"thanks for your hard work "and blah blah blah.....then I have to return their hits,I cannot see this thing before accepting the hit!once you accept hit and see this message. Awful.
     
  5. riazthemturkhelper

    Joined:
    Jan 14, 2012
    Messages:
    1,640
    Likes Received:
    0
    +1, yes you are right. So me blocked them forever.
     
  6. OneEye

    OneEye Member

    Joined:
    May 4, 2012
    Messages:
    10
    Likes Received:
    0
    As I was saying earlier I had this issue, but if you do have this happen, I learned not to return the hit, but keep it open, at least for a bit cause if more jobs open up for that hit it will let you work on one of those. So if this happens i just check the jon every 5 mins or so to see if one has opened up. This will help me sometimes.


    And thanks for the help riazam that helps tremendously. That's exactly what i was wondering. Your the best!!!!
     
  7. kenaboo

    kenaboo New Member

    Joined:
    May 4, 2012
    Messages:
    3
    Likes Received:
    0
    Thank you Riazam, that was a great help!
     
  8. mralaska

    mralaska User

    Joined:
    Nov 7, 2013
    Messages:
    1,230
    Likes Received:
    0
    It has come to my attention that this post is being held up as "proof" that return ratios do not matter. In an effort to set the record straight and for the sake of incoming links I need to wake up this sleeping thread to post some samples that demonstrate everything Amazon said about abandonment rate and submission rate was categorically wrong. The only valid argument is the magnitude of effect but I maintain even the potential of harm is something that I want to avoid. It is a bit ironic that searching for "Submission rate" on google turned up so many people complaining about getting locked out of hits for this very reason that it made it difficult to find the actual HITs to prove it to the detractors on TO. Following are a few examples I did uncover:

    ken Reimer 3/19/2014 (today):
    Qualifications Required: HIT abandonment rate (%) is less than 5
    http://mturkforum.com/showthread.ph...tional-Monday!&p=714854&viewfull=1#post714854

    Product RNR currently and has for quite some time required HIT abandonment rate (%) is less than 20
    https://www.mturk.com/mturk/searchbar?selectedSearchType=hitgroups&requesterId=A2BAP2QO7MMQI9

    TR 02-06-2014 required 95 submission rate:
    http://mturkforum.com/showthread.ph...shing-Wednesday!!/page131&p=673685#post673685

    OCMP 04-23-2013 required 95 submission rate:
    http://mturkforum.com/showthread.php?7398-Can-t-Find-Great-HITs-4-23&p=226595&viewfull=1#post226595

    Extension 02-11-2014 requires 90 submission rate:
    http://mturkforum.com/showthread.ph...zing-Tuesday!!&p=681995&viewfull=1#post681995

    Brenden Lake 01-30-2014 required 75 submission rate:
    http://mturkforum.com/showthread.ph...sty-Thursday!!&p=663249&viewfull=1#post663249

    PackageFox 03-14-2014 required 80 submission rate:
    http://www.***************/showthre...-m-Thirsty-too&p=105305&viewfull=1#post105305

    There are many more examples and they span the years going back at least to 2012 when the email in question was posted.
     
  9. Bigfaced_Honies

    Joined:
    Mar 8, 2014
    Messages:
    2,391
    Likes Received:
    0
    The main reason it's pretty stupid for requesters to use abandonment/submitted %s as a qual is that some requesters set their HITS up in such a way that you can't even tell if a HIT is available/can be done until you accept it, necessitating returns.

    They're of course free to use it as a criteria for whether or not you can do their HIT, but it's still pretty clueless of them.
     
  10. mralaska

    mralaska User

    Joined:
    Nov 7, 2013
    Messages:
    1,230
    Likes Received:
    0
    One can debate who is being clueless. One can even argue we are clueless for accepting that kind of abuse.

    There is never a valid reason to force a person to accept a task to see what it is and such behavior should not even be allowed. Usually they are simply ways to get free work out of people because even if you return it you are giving them data. The most common cases are demographic screeners that they do not even bother setting up a separate link to screen. The most legitimate requesters will post a paid screener. Some requesters hide their instructions just so they can track to see how many people are reviewing the HIT without taking it (by returning it). It is all free data to them.

    Other tasks that hide their intent are more devious. For instance, in the last couple days Michael Toomim (as CrowdClearinghouse) posted HITs that hide the task then surprise you with something most consider reprehensible such as writing a friendly letter to a convicted child molester. His current published project includes tracking how many people are willing to continue with such work after they find out they are doing something unpleasant so when you return the HIT you are actually giving him half his data for free! He gets a bargain on the rest via rejects and uses hard blocks also but that is another story. There are other requesters that will ask for stuff like increasingly personal questions and then track at which point you have had enough by returning the HIT. There are many ways to exploit the return ratios to extract free data. These are only a few examples.

    On the flip side, for many types of batches it makes perfect sense to utilize the submission ratio and I am surprised that it is not used more often. It is my concern is that more requesters will discover a manner to use it as a useful tool to get higher quality work done in a more efficient manner. For that reason I do everything within my power to try to recover my submission ratio to an acceptable level and that means turning down tasks if it is known they are looking to get free work via task returns.

    Obviously there are some returns that are not avoidable on our end, such as software issues and servers going down, but it would be a lot simpler and natural to maintain a high submission ratio without catering to the exploitation of cheap and illegitimate requesters.
     
  11. mralaska

    mralaska User

    Joined:
    Nov 7, 2013
    Messages:
    1,230
    Likes Received:
    0
    With the removal of the ratios from our new "improved" dashboard it has caused a new eruption of confusion on TO so I am updating this thread.

    Amazon has changed their cut and paste responses to now indicate Mturk "no longer" gives requesters access to screen for abandonment rate and submission rate. Unfortunately, this is yet another assertion that is not supported by the facts.

    One can easily see for themselves that these qualifications are accessed at least a few times a day by monitoring for them. Using DCI New HIT Monitor you can cut and paste the following search terms in and verify they are still in affect and occasionally with significant ratios required:

    "HIT return rate",
    "HIT abandonment rate",
    "HIT submission rate",

    What Amazon did do is recently change the documentation of the API. The new documentation has moved the type ID references but the API still supports the old documentation which has been used by requesters all along and is easily available to new requesters which at least some have successfully located it. When I was recently informed of how the requester interface is set up it took about 3 minutes to dig up the documentation needed to screen for abandonment rate, submission rate, or return rate.

    [​IMG]

    One can only speculate as to Amazon's motives. I personally feel they are doing a disservice to their requesters by making the information more difficult to access but it pales in comparison the disservice they do to us with misleading information that can affect the tasks we are able to accept.
     
  12. Tribune

    Tribune User

    Joined:
    Nov 4, 2013
    Messages:
    1,390
    Likes Received:
    0
    Kudos to mralaska for bringing these questions to light for a general audience in his TurkOpticon comments!

    With comments recently re-enabled on TO, we're bound to see a return to commentary by the well-meaning but ill-informed, to the effect that returned HITs don't "really" matter. (I addressed these points in a post on Google Group "TO-discuss").

    Thanks, mralaska, for digging out the information on the API, too!
     

Share This Page