Please Join Me On A Crusade To Fix Bad CrowdFlower Test Questions!

Discussion in 'Requesters' started by liladybugz36, Nov 4, 2012.

  1. brian_cf

    brian_cf Verified Requester

    Joined:
    Feb 5, 2013
    Messages:
    13
    Likes Received:
    0
    Hello Everybody,

    And thanks for the introduction liladybugz26! We've really appreciated the feedback that she has been providing us over the past few months. And a big part of that stems from her willingness to start forum threads like this one. Our software and process is far from perfect, so we cherish any organized attempts to provide us helpful feedback.

    CrowdFlower is always trying to make improvements to the experience of our Contributors. Unfortunately, as a small company with limited resources it can often take a frustratingly long time for those improvements to be realized. But we do have some exciting things coming down the line.

    1) We're nearly finished with a complete redesign of our Contributor Task interface. We think the changes there will make the tasking experience much more intuitive and enjoyable. We've even been using Contributors during the entire development process, isolating a small group of folks to test and provide feedback on every feature before it gets released.

    2) We're close to having new skills tests available to Contributors via their Contributor Dashboard. These tests will be used to identify our most skilled members and provide them exclusive access to higher paying tasks. These skill tests will also hopefully allow Contributors the ability to more easily identify tasks that are most suited to their interests.

    3) Much like Mturk, CrowdFlower also has many requestors that use our software to create and release tasks. We know that some of these requestors have been posting some problematic tasks as of late. As a result, we're soon going to add some transparency into which tasks have been verified by CrowdFlower and those which have not reached that distinction. This will help us and the community to quickly identify any problematic requestors using our platform.

    4) And particularly pertinent to this thread, we're about to begin a complete overhaul of our Test Question QA system. Our efforts should result in a MUCH improved QA system that doesn't needlessly reject good work and will hopefully lessen the frustration you often encounter in CrowdFlower tasks. I can talk more about that in the near future.

    5) This month we plan to have our first ever Contributor Question & Answer Webinar. We want to provide a venue to answer Contributor questions directly as well as provide more transparency as to our process. If you've ever been interested in any of the ins and outs of our platform, this will be a great way to have your questions answered. If you'd like to submit a question for us to answer, I'll share a link as a secondary post. As this is my first time posting, I can't provide any links just yet. :p

    In short, we know there are issues with CrowdFlower tasks, but we really are making strides to improve the Contributor experience. And we certainly do listen to our Contributors and value all the feedback we receive. CrowdFlower has been absent from Turk forums in the past due to our wanting to respect the users of those sites. We've wanted to give ourselves a voice, but felt like we might better serve your community by remaining absent. But liladybugz26 convinced me that it would be a good idea to provide a voice to CrowdFlower and allow Turkers the chance to engage with us in this venue. I hope that our conversations can be as helpful and constructive as possible.

    Thanks!

    Brian @ CrowdFlower
     
  2. brian_cf

    brian_cf Verified Requester

    Joined:
    Feb 5, 2013
    Messages:
    13
    Likes Received:
    0
    If you'd like to submit some questions for consideration to be used in our webinar, please use the link below:
    Click Here!
     
  3. liladybugz36

    Joined:
    Dec 24, 2011
    Messages:
    224
    Likes Received:
    1
    Thank you so much for updating everybody on where CrowdFlower is in your new processes, and for joining in on the forum. I greatly appreciate it and I am positive many other people will as well.
     
  4. TAC

    TAC User

    Joined:
    Sep 15, 2012
    Messages:
    474
    Likes Received:
    0
    Ok, I'm officially shocked.

    Complained about an unfair answer, they said negative I said neutral because it was a negative balanced by a positive. While working on the same HIT batch, it was reversed. Couldn't have been more than 5 minutes and my accuracy went back to 100%.

    I did type in all caps so maybe that got attention quick?
     
  5. Heather P

    Heather P Member

    Joined:
    May 14, 2013
    Messages:
    17
    Likes Received:
    0
    I am a new turker and within the almost 3 weeks that I have been on Mturk I have decided they are total slave labor. I noticed and reported the test questions you are talking about but Its ridiculous even trying to deal with them so I don't. Good luck.
     
  6. Shego

    Shego User

    Joined:
    Feb 4, 2009
    Messages:
    475
    Likes Received:
    0
    Hehe..a small company of 35 people that is worth millions...they suck!
     
  7. liladybugz36

    Joined:
    Dec 24, 2011
    Messages:
    224
    Likes Received:
    1
    There is great news on this topic that I so avidly campaigned for many months ago and I just want to update anybody who is following this thread.

    CrowdFlower initiated a program that I will be a part of to help make test questions fair and accurate. I do not know if I am at liberty to disclose more information, but I'm certain as the program progresses the CF staff will issue details.

    I just wanted people to know that CF has not stopped listening and I'm grateful that they are ensuring quality to crowdworkers such as myself.

    I appreciate the complaints on this thread; however, what have your complaints done to change anything? Maybe if you do something useful with them it will enact change, or why bother spending your energy posting negative things on a thread about a company you don't care for when you could be using your energy to make money or make a difference for a fellow worker somewhere, somehow by lobbying the requester about those negative things?

    Sincerely,
    Rachael M Jones
     
    #67 liladybugz36, Sep 8, 2013
    Last edited by a moderator: Sep 18, 2013
  8. liladybugz36

    Joined:
    Dec 24, 2011
    Messages:
    224
    Likes Received:
    1
    This is the second time within a span of four months that you have posted nothing but useless negatives on this thread. Everyone wants to see wrongs righted and thats what my hope was then and still is. Your posts have an agenda of their own that you should really take to a unique thread instead of attempting to dampen positivity.
     
  9. Charlie Turkelly

    Joined:
    Jul 15, 2012
    Messages:
    1,126
    Likes Received:
    0
    Keep fighting the good fight, Rachael.

    [​IMG]
     
  10. kimg

    kimg User

    Joined:
    Apr 7, 2013
    Messages:
    1,382
    Likes Received:
    0
    This is amazing! I stopped working for CrowdFlower because of this problem, and the fact that I got the "no more work available" message every time I accepted a hit. I would write a paragraph long reason as to why my answer was in fact not wrong, but I never got any response back from them. I think because I got these bogus questions and answers wrong, they have blocked me from working for them. What you've done is amazing and I commend you for not giving up!
     

Share This Page