Please Join Me On A Crusade To Fix Bad CrowdFlower Test Questions!

Discussion in 'Requesters' started by liladybugz36, Nov 4, 2012.

  1. liladybugz36

    Joined:
    Dec 24, 2011
    Messages:
    224
    Likes Received:
    1
    Fellow Turkers,
    I am on a crusade to have CrowdFlower correct bad "test" questions. I have started this week. There are a ridiculous number of bad "test questions" in each hit which lower our percentages and do not give us the proper credit we deserve, even blocking us from jobs at times. This takes away time from our work, and undermines us as employees leaving us frustrated and looking quite awful in percentages at times which lead some people to feelings of unnecessary inadequacy. PLEASE help try to fix a broken system and force CrowdFlower to fix this issue!

    Take the time to submit a small copy of every incorrect test question you come across (make sure they are "bad" test questions) with your job id and contributor id(found under the help button in the hit): Ask CrowdFlower to reverse your rating and fix the incorrect question that you found. Thank you for your help. Together we can make a difference!!!

    Sincerely,
    liladybugz26

    *I am not self righteous or anything, just fed up with injustice. Please be diligent in order for this to work. It only takes a minute and honestly, you could really make a difference here!
     

    Attached Files:

    #1 liladybugz36, Nov 4, 2012
    Last edited by a moderator: Nov 4, 2012
  2. chaos

    chaos User

    Joined:
    Jun 14, 2012
    Messages:
    912
    Likes Received:
    0
    This is going to make absolutely no difference, sorry. When you get a "test question" wrong, there's a box right there where you can let them know if there is a problem with it.
     
  3. naturegirl

    naturegirl User

    Joined:
    Aug 30, 2012
    Messages:
    701
    Likes Received:
    0
    I agree with chaos in the sense that CrowdFlower has a history of providing work that pushes (or exceeds) the limits of what is reasonable to ask for tiny pay. Based on that alone, this is not a company that is motivated to make your work experience significantly more fair or positive for you, no matter what you tell them.

    There are many ways we can all spend our Turking time. There are also many injustices in the world. What CrowdFlower does isn't right, but their pay would/should never make enough of a difference in anyone's life to warrant any extra effort to communicate with them.

    But really, I shouldn't speak for others. So what I will say is just that I feel that my time has been proven to be better spent and much better paid with other requesters, so I ignore CrowdFlower. Nevertheless, I wish you luck, liladybugz.
     
  4. liladybugz36

    Joined:
    Dec 24, 2011
    Messages:
    224
    Likes Received:
    1
    Yes but.....
     
    #4 liladybugz36, Nov 4, 2012
    Last edited by a moderator: Nov 4, 2012
  5. liladybugz36

    Joined:
    Dec 24, 2011
    Messages:
    224
    Likes Received:
    1
    Yes but if enough people complain then maybe they will do something about it if people are diligent. It is not necessary to shoot down the idea. It is worth a try.
     
  6. liladybugz36

    Joined:
    Dec 24, 2011
    Messages:
    224
    Likes Received:
    1
    Thank you. I have contacted them 5 times this week and have had ratings reversed and hopefully "test" questions fixed. I have not worked for them in many months because of this but they have had some decent hits lately so I jumped back into the pot and all the frustration came back.

    I am just surprised that with all of the bad test questions there are only a fraction of them reported. It only takes a minute to report them and if people that do work on CrowdFlower hits diligently report this issue, they will either be forced to ignore a slew of people and stop responding or be a responsible company and fix the error at the source.

    They have recently posted a comment in the FAQ section of the dashboard page that they were working to correct this issue, so why not encourage them to put a move on it?

    Nothing ever changes with inaction. If you don't ever work for CrowdFlower then I can see how it would not be a concern, but if you do then at least care enough about being manipulated to do something about it. Sure they won't listen to one person but it's hard to ignore an entire workforce.
     
    #6 liladybugz36, Nov 4, 2012
    Last edited by a moderator: Sep 18, 2013
  7. naturegirl

    naturegirl User

    Joined:
    Aug 30, 2012
    Messages:
    701
    Likes Received:
    0
    I do admire your initiative and energy for this, liladybugz! Had not heard that they posted that comment in their FAQ. Again, good luck.
     
  8. vordai

    vordai User

    Joined:
    Aug 21, 2012
    Messages:
    51
    Likes Received:
    0
    liladybugz36,

    I usually remember to mark the test questions as unfair, but I'll make more of an effort to report every one I find. I don't do too many CF (I like the twitter ones because they're amusing), but every little bit helps! Hope other people join your cause.
     
  9. mTurkMeister

    Joined:
    Aug 8, 2012
    Messages:
    246
    Likes Received:
    0
    I have to agree with Chaos on this; the approach being advocated will make no difference. If you want to get CrowdFlower's attention, don't work for them. If enough people did that, they would have to make changes. They might choose to do better quality checking, they up the rewards or something else. But I am convinced that completing tasks and complaining a lot is fine with them and they are OK with the situation.

    I think the recent ProductRNR situation supports my contention. If not familiar with it, I suggest you read the thread.
     
    #9 mTurkMeister, Nov 4, 2012
    Last edited by a moderator: Nov 4, 2012
  10. ash breaks stuff

    Joined:
    Sep 20, 2012
    Messages:
    874
    Likes Received:
    0
    Can you give me an example of a bad test question?
     
  11. liladybugz36

    Joined:
    Dec 24, 2011
    Messages:
    224
    Likes Received:
    1
    I am very familiar with ProductRNR. I have been on mTurk for a couple of years. CrowdFlower is not going to go away, trust me. They still do post some good hits, they do not reject and just because some people do not like them doesn't mean they don't have a strong following.

    I am not a fan of theirs at the moment by any means and rarely work for them, but I am promising that when I do and I come across bad test questions I will hold them accountable.

    They do have some high paying sentiment hits, and an occasional hit that pays well like the one today (taxi addresses). People are not going to ban them. Not all of their hits pay .01 cent, and it's the decent ones and the promise of no rejections that keep people coming back.

    I don't understand why people who do not like the idea have to bash it? Don't people believe it's right to stand up for fairness? I have done only a total of probably 250 hits for them in 6 months and the disgrace of the whole thing moves me to want to do something about it instead of sitting and letting a company take advantage of people. If you don't want to, fine... but why bash what I think is a good idea?

    I am satisfied with the money I make on mTurk and if I want to spend my time standing up for what I believe in instead of making the extra .05 cents it takes to file a complaint when I find an error, and if other people choose to do the same, why is it anybody's place to tell me or anyone else that it is a bad idea and worthless?

    Standing up for yourself is never worthless and even if the battle is lost and nothing changes at least I can say I didn't just stand by, and thanks to my convictions I have more dignity than I would if I would have just allowed it to happen and watched without doing a thing.
     
    #11 liladybugz36, Nov 4, 2012
    Last edited by a moderator: Sep 18, 2013
  12. liladybugz36

    Joined:
    Dec 24, 2011
    Messages:
    224
    Likes Received:
    1
    For instance today a forum user was blocked (I received the same question and my rating was lowered) for selecting benefit/fundraiser in a hit. It came back as wrong saying it should have been fundraiser/benefit. This is irrelevant to the task and means the same thing also - there was no fundraiser/benefit option. It is questions like these that are regular with CrowdFlower.
     
  13. HappyFish

    HappyFish User

    Joined:
    Sep 8, 2012
    Messages:
    1,778
    Likes Received:
    0
    I have been marked wrong for answering "neutral" to a sentiment. They marked it wrong, saying "It should have been marked "Neutral". I never heard back and was blocked from that HIT, as well as similar ones.
     
  14. liladybugz36

    Joined:
    Dec 24, 2011
    Messages:
    224
    Likes Received:
    1
    I received a detailed response from CrowdFlower since I've enacted my efforts. I thought some people on the board might be interested in what they had to say.

    Hi Rachael,

    Thank you so much for this post.

    I cannot tell you just how much we appreciate efforts of this sort. Although it may not always seem to be the case, we really are trying very hard to improve the experience for our Contributors. Unfortunately we are a small company of ~35 people (with only 4 on the Community Support Team), so it continues to be a challenge to adequately allocate resource everywhere we need it. This often results in long delays in our response time, which we know is frustrating.

    And that is exactly why efforts like this are incredibly helpful. It is of the utmost importance for us to have accurate 'test' questions...on so many levels. First of all, incorrect 'test' questions negatively affect your experience with CrowdFlower. If you keep finding yourself incorrectly marked wrong, a) it's very frustrating and b) we might unnecessarily reject honest/good answers. If we keep building and building on your frustration, then you'll likely give up on CrowdFlower, which is terrible for us. As our business is crowdsourcing, it is absolutely imperative that we retain our crowd by not letting good people like you disappear.

    Let me assure you that when we unnecessarily reject honest/good answers because of faulty 'test' questions, it's not in an effort to scam or cheat people by any means. Our business is data collection / verification, etc. And our clients require that we provide them high quality data in a timely manner. Which is to say when we reject answers, our system requires that we ask another person to complete the same task. This makes the process take longer (which means we might miss deadlines). But it also compromises the quality of our data, as the next person we ask might not be as diligent in their efforts as the person we incorrectly rejected.

    In short, 'test' questions are our way of ensuring that we deliver high quality data to our clients. If the 'test' questions are bad, then our system breaks. As a result, it is so important for us to find and correct the bad apples. But that is sometimes challenging for us to do. A team of less than 10 people actually develops each of the tasks as well as creates all of the 'test' questions for each of those tasks. And with the very large number of tasks we are working on every day, that means that bad 'test' questions often slip into the system. It also means that some 'test' questions start to go stale (ie: websites go down, links stop working, new answers choices aren't accounted for, etc).

    To be able to identify these issues quickly is so important to us. And that is why well-documented evidence (screenshots, descriptions, etc) that our 'test' questions are faulty are so valuable. We really do act on it. We really do appreciate it. And we plan to soon roll out efforts to more clearly demonstrate how much we do appreciate that feedback.

    I assure you that we are working on ways to improve our system and your contributions to that process are invaluable.

    Thanks again for your efforts.

    Best,

    Brian
    &
    The Community Support Team
     
  15. naturegirl

    naturegirl User

    Joined:
    Aug 30, 2012
    Messages:
    701
    Likes Received:
    0
    I would imagine you found that very validating, liladybugz. :) So setting my personal opinions about CF aside, congratulations to you!
     
  16. liladybugz36

    Joined:
    Dec 24, 2011
    Messages:
    224
    Likes Received:
    1
    Thank you, many people have put a lot of effort into this that work on CrowdFlower tasks. I littered the Crowdflower board with my agenda and people responded strongly. I am glad that I recieved such a detailed response but until it's fixed there's work to be done. Thank you for your congrats!

    Sincerely,
    Rachael Jones
     
  17. liladybugz36

    Joined:
    Dec 24, 2011
    Messages:
    224
    Likes Received:
    1
    For those who say a single person can't make a difference...I received this letter today from CrowdFlower. Here is to prove that for every action in the world creates action and for every inaction creates NOTHING! :)
    cheers~

    Hi,
    >
    > We're writing you because we've recognized you as a very valuable
    > Contributor -- either because you've worked on many many tasks, or you're
    > incredibly helpful in our support system, or you've provided some great
    > feedback that we'd like to engage with you more about.
    >
    > Our Contributors are an incredibly important part of our process at
    > CrowdFlower. Without our Contributors we have no Crowd. And without that
    > our name wouldn't make any sense and our business wouldn't exist.
    >
    > We are a small, young company and thus we know that our process is far far
    > from perfect. There are bugs. Countless bugs (broken tasks, incorrect
    > test questions, payment issues, etc). And although it is frustratingly
    > difficult for our Support Team to get to your issues in a timely manner,
    > everyday we spend trying to figure out ways to improve the experience for
    > our Contributors. This involves conversations with Task Owners,
    > conversations with our engineers, and conversations with the sites you
    > access our tasks from.
    >
    > And to this end, we would love to have conversations with our Contributors,
    > as you are the most important piece. So, if you would be willing, we would
    > love to set up a Skype (or phone) conversation with you in the next week or
    > so...to hear your honest feedback...in your words. Please let us know if
    > this is something you'd be interested in doing. It would be incredibly
    > helpful.
    >
    > Thanks so much for your consideration.
    >
    > Best,
    >
    > Brian
    > & The Community Team @ CrowdFlower
     
  18. bayonjoset

    bayonjoset User

    Joined:
    Nov 18, 2012
    Messages:
    162
    Likes Received:
    0
    You've done a great thing, Hats off dude
    :fencing: victory
     
  19. Maggiemw

    Maggiemw User

    Joined:
    Aug 3, 2012
    Messages:
    139
    Likes Received:
    0
    Way to go, lillady! You are one of the people who gives me hope that things can change for the better on Mturk if we turkers are willing to take the time to be stubborn, determined, proactive and constructive, just like you have been.

    Apathy and pretending that "if you ignore it, it will go away" or "it's somebody else's business, not mine, I have better things to do" will only lead to more of the same garbage, in my not-so-humble opinion.

    Hats off to you once again!
     
  20. naturegirl

    naturegirl User

    Joined:
    Aug 30, 2012
    Messages:
    701
    Likes Received:
    0
    Good luck with the call, OP. I'm looking forward to hearing how it goes!
     

Share This Page