TOTW - New batch, Updated design (yet again)

Discussion in 'Taste of the World' started by TasteOfTheWorld, Jun 25, 2013.

  1. TasteOfTheWorld

    Joined:
    Jun 7, 2013
    Messages:
    55
    Likes Received:
    0
    We just posted another small batch for workers who have done >1000 HITs for us. The design has changed again - based on some of the recent results, we realized there was too much ambiguity between "closed" and "I don't know" so they are combined now. We also updated the instructions to be clearer about special cases.

    We did receive feedback from some who were upset that we removed the Comments. The reason we did this was that we didn't have a good way of dealing with these in our database. Hopefully with the updated choices there'll be fewer questions, but feel free to email us if anything does come up.

    For those of you who don't qualify, stay tuned!! This should be our last hurdle before we open the flood gates again and start posting larger batches available to more workers.

    -Your friends at ToTW
     
  2. riyazalikhan

    Joined:
    Mar 8, 2012
    Messages:
    369
    Likes Received:
    0
    How can i know my no of TOTW hits done? i request for qualification... i have done earlier but not confidence that i cross 1000 mark!!!
     
  3. TasteOfTheWorld

    Joined:
    Jun 7, 2013
    Messages:
    55
    Likes Received:
    0
    If you view your qualifications you can see what range you're in - we've got separate ones set up for those who have done >1000, 500-1000, and 100-500.

    If you want to know the exact number, just private message us with your worker ID.
     
  4. Turkalicious

    Joined:
    Jun 4, 2012
    Messages:
    1,642
    Likes Received:
    0
    -waits [im]patiently for the glorious flood gates of ToTW to open-

    Thank you for keeping us updated!
     
  5. TasteOfTheWorld

    Joined:
    Jun 7, 2013
    Messages:
    55
    Likes Received:
    0
    A new batch is up! I wouldn't call this the full flood gates opening quite yet, but you should have HITs available to you regardless of any of our qualifications. Post questions here, or contact us directly. Enjoy!

    PS - we noticed that certain browsers were having trouble with expand/collapse, so we've made the Update Details section expanded by default.
     
  6. Khalinov

    Khalinov User

    Joined:
    Dec 12, 2011
    Messages:
    873
    Likes Received:
    0
    I see the one batch that requires “ToTW Reviewer is not 100”

    So if I am not 100, what am I? FWIW, I have been doing these for weeks.

    Nemmind, I found it in my quals.
     
    #6 Khalinov, Jun 25, 2013
    Last edited by a moderator: Jun 25, 2013
  7. Turkalicious

    Joined:
    Jun 4, 2012
    Messages:
    1,642
    Likes Received:
    0
    Never mind, I see :)
     
  8. riyazalikhan

    Joined:
    Mar 8, 2012
    Messages:
    369
    Likes Received:
    0
    I seen my score is 50 for TOTW Reviewer is 100, Can i get score to do hits?
     
  9. bumonskateboard6

    Joined:
    Jul 20, 2012
    Messages:
    287
    Likes Received:
    0
    I noticed I had a qual of 50 but had to run out and that batch is gone now, will you be posting more "lower" qualification batches in the future since these are great paying!
    Thanks
     
  10. TasteOfTheWorld

    Joined:
    Jun 7, 2013
    Messages:
    55
    Likes Received:
    0
    We will be posting more soon for those with a score of 50 for TOTW Reviewer. (Probably tomorrow!)
     
  11. riyazalikhan

    Joined:
    Mar 8, 2012
    Messages:
    369
    Likes Received:
    0
    I sent you PM with my turk id for no of hits done? Please update.
     
  12. Khalinov

    Khalinov User

    Joined:
    Dec 12, 2011
    Messages:
    873
    Likes Received:
    0
    So there are two different sets of quals, then?

    What does it take to get the reviewer qual? 1240 people have it, 354 have the other three. Yet the work for the reviewers is just laying there. Apparently, they don’t want the work as badly as the 100-500, > 500 and > 1000 people do.

    Seems more than a little convoluted.
     
    #12 Khalinov, Jun 25, 2013
    Last edited by a moderator: Jun 25, 2013
  13. TasteOfTheWorld

    Joined:
    Jun 7, 2013
    Messages:
    55
    Likes Received:
    0
    We've only recently starting experimenting with quals, so my apologies if things seem convoluted. We created the quals for number of HITs completed to identify those who are most familiar with our HITs, so we could test out our new design.

    We're still experimenting with the reviewer qual, but it's a combination of number of HITs done, as well as some manual reviews we recently conducted. There are VERY few people who have a score of 100 to start, but our aim is to add more.

    Thanks for bearing with us as we update our process!
     
  14. BuddysGirl

    BuddysGirl User

    Joined:
    Feb 14, 2013
    Messages:
    1,520
    Likes Received:
    1
    ugh...only have a 50 Qual here.
     
  15. TurkingBear

    Joined:
    May 8, 2013
    Messages:
    16,225
    Likes Received:
    5
    I for one love this new qual. Being one of the ones with the reviewer qual.
     
  16. reebs

    reebs User

    Joined:
    Feb 13, 2013
    Messages:
    4,138
    Likes Received:
    18

    Me too, Bear. Me too. :lol:
     
  17. MSUHolly

    MSUHolly User

    Joined:
    Jan 25, 2013
    Messages:
    2,114
    Likes Received:
    0
    Pretty sure 1240 people have a reviewer qual but what score they have on it is not determined. Some people might have a 50, some 100 or some score in between. Just because it says 1240 doesn't mean there are actually 1240 able to work on the batch that was up until this morning. I am one of the people with the 100 score and along with a few friends we sat up all night working on them.

    SO I don't think it's an issue of us not wanting the work bad enough. It's just as TOTW has stated, they are testing a new qual and not everyone has it so there were few people working on it. Let them work out their testing with people they trust and maybe you'll get moved up at some point. I highly doubt insulting totw or degrading the workers who were chosen to do it is the route to getting on their good side though. Good luck.
     
  18. Khalinov

    Khalinov User

    Joined:
    Dec 12, 2011
    Messages:
    873
    Likes Received:
    0
    I don’t see where anyone insulted ToTW or its workers, though I suppose people can always find what they are looking for.
     
  19. TasteOfTheWorld

    Joined:
    Jun 7, 2013
    Messages:
    55
    Likes Received:
    0
    Hi all - Just to give you a little more info about the Reviewer qual...

    We realized that when we submitted a batch intended for those without the Reviewer=100 qual, that MTurk wouldn't allow anyone to work on it unless they had some value for that qual. So, we granted everyone else a score of 50 to start. Don't think of it as a letter grade (i.e., 50 is not an "F"). It's just a starting point for us to allow you to keep working on our HITs while we continue to evaluate and promote more workers.

    FYI, here are some of the things we're evaluating:
    -Accuracy of answers: we have been sprinkling our batches with known answers. Though we haven't been automatically approving or rejecting HITs based on this, we are using it to feed into the Reviewer qual. Note that we are not looking for all answers to be "open and in business" - we need to have solid evidence (official website, official Facebook page, reviews on third party sites) that a restaurant is open. The latest instructions should be clear on this.
    -Updates: we appreciate when workers take the time to update details (e.g., street address, URL). It's important to us that we have the most accurate info for a restaurant. We've gotten a few complaints that the collapse/expand link doesn't work in all browsers, so we're making it expanded by default to allow everyone to make updates.
    -Quality of URLs provided and attention to instructions: we have made our instructions more explicit about what types of URLs are best. We also appreciate when workers provide more than one URL as evidence (e.g., an official Facebook page as well as a third party website).

    We'll have another batch up this afternoon, stay tuned...
     
  20. TurkingBear

    Joined:
    May 8, 2013
    Messages:
    16,225
    Likes Received:
    5
    I believe the reviewer qual was a success. They were all gone in less than a day with the people who were already on board.
     

Share This Page