CrowdFlower

Discussion in 'CrowdFlower' started by brian_cf, Feb 8, 2013.

  1. brian_cf

    brian_cf Verified Requester

    Joined:
    Feb 5, 2013
    Messages:
    13
    Likes Received:
    0
    Hello Everyone!

    I wanted to introduce myself (and CrowdFlower) to all the wonderful Turkers on the mTurk Forum. It's been a long time coming. CrowdFlower has been absent from Turk forums in the past due to our wanting to respect the users of those sites. We of course have wanted to give ourselves a voice, but felt like we might better serve your community by remaining absent. But recent communication I've had with liladybugz26 (and most recently with with RWStein) convinced me that it would be a good idea to provide a voice to CrowdFlower and allow Turkers the chance to engage with us in this venue. I hope that our conversations can be as helpful and constructive as possible. We'll still manage our support task support via our support forum at GetSatisfaction, but hope to be as engaged in this forum as possible.

    CrowdFlower is always trying to make improvements to the experience of our Contributors. Unfortunately, as a small company with limited resources it can often take a frustratingly long time for those improvements to be realized. But we do have some exciting things coming down the line.

    1) We're nearly finished with a complete redesign of our Contributor Task interface. We think the changes there will make the tasking experience much more intuitive and enjoyable. We've even been using Contributors during the entire development process, isolating a small group of folks to test and provide feedback on every feature before it gets released.

    2) We're close to having new skills tests available to Contributors via their Contributor Dashboard. These tests will be used to identify our most skilled members and provide them exclusive access to higher paying tasks. These skill tests will also hopefully allow Contributors the ability to more easily identify tasks that are most suited to their interests.

    3) Much like Mturk, CrowdFlower also has many requestors that use our software to create and release tasks. We know that some of these requestors have been posting some problematic tasks as of late. As a result, we're soon going to add some transparency into which tasks have been verified by CrowdFlower and those which have not reached that distinction. This will help us and the community to quickly identify any problematic requestors using our platform.

    4) We're also about to begin a complete overhaul of our Test Question QA system. Our efforts should result in a MUCH improved QA system that doesn't needlessly reject good work and will hopefully lessen the frustration you often encounter in CrowdFlower tasks. I can talk more about that in the near future.

    5) This month we plan to have our first ever Contributor Question & Answer Webinar. We want to provide a venue to answer Contributor questions directly as well as provide more transparency as to our process. If you've ever been interested in any of the ins and outs of our platform, this will be a great way to have your questions answered. If you'd like to submit a question for us to answer, please visit the link below:

    Click Here to submit a question to CrowdFlower!

    In short, we know there are issues with CrowdFlower tasks, but we really are making strides to improve the Contributor experience. And we certainly do listen to our Contributors and value all the feedback we receive.

    Thanks! And it's good to be here.

    Brian @ CrowdFlower
     
  2. Andy

    Andy Administrator
    Staff Member

    Joined:
    Jan 25, 2009
    Messages:
    2,219
    Likes Received:
    21
    Welcome to MTF Brian!

    Thank you for choosing to spend some time here representing CF.
     
  3. RWStein

    RWStein User

    Joined:
    Aug 17, 2012
    Messages:
    1,815
    Likes Received:
    0
    Brian:

    Thanks so much for coming!

    I think this will both give you better work and relieve a lot of the tensions workers have had!

    All my appreciation!

    Rob

     
  4. Meg9

    Meg9 User

    Joined:
    Nov 21, 2012
    Messages:
    245
    Likes Received:
    0
    Thanks for coming here, and thanks very much for discussing point #3. The good number of crowdflower hits I've clicked into in the last few months have been the spammy ones that violate Mturk's rules and it gets very frustrating. It is nice to hear that CF also knows that they are a problem.
    The skills tests sound like a great idea.
     
  5. BeerMe

    BeerMe User

    Joined:
    Nov 2, 2012
    Messages:
    406
    Likes Received:
    0
    Thanks for coming to the forum. I think communication between workers and requesters is a win-win for all of us and I wish more of it took place. To be honest, I've stayed away from crowdflower for a while now because I've had more bad experiences than good. My biggest headache has been accepting HITs only to be told by the crowdflower interface that there are no more HITs available and I'm forced to return it. I look forward to rediscovering your work after the redesign you talked about is implemented. You want work done, we want to do the work, let's work together and get it done.
     
    #5 BeerMe, Feb 8, 2013
    Last edited by a moderator: Feb 8, 2013
  6. PavelDatsyuk

    Joined:
    Dec 28, 2012
    Messages:
    6,235
    Likes Received:
    0
    Woohoo! Sometimes I don't do CF HITs because I'm confused about the instructions and by the time I would receive any reply the batch would be gone already. Hopefully having you here will help clear things up a lot faster.
     
  7. bobbie01

    bobbie01 User

    Joined:
    Jan 29, 2013
    Messages:
    11,348
    Likes Received:
    62
    This is my concern, as well. I have written, to CF, twice. Both times I was referred to your Forum but didn't know what to do once I got there. I am really happy to "see" your presence on this Forum!!
    Welcome!
     
  8. bobbie01

    bobbie01 User

    Joined:
    Jan 29, 2013
    Messages:
    11,348
    Likes Received:
    62
    Can you give us an insight as to why this happens? I accept a Hit for which I am qualified, do 1 Hit out of hundreds or thousands available and this is what comes up.

    Thanks for your hard work!

    You have completed all of the work that you can for this task. Please check back again in the future. We are always adding new jobs and would love to have your help.

    You can find more of our tasks by searching for "crowdflower" on Mechanical Turk.
     
  9. BeerMe

    BeerMe User

    Joined:
    Nov 2, 2012
    Messages:
    406
    Likes Received:
    0
    I'm less interested in explanations and more interested in solutions. Crowdflower's interface is completely separate from Mturks. Their clients create HITs in CFs interface and it then gets inputted into Mturk most times without any control or review from CF. I think this has alot to do with the problems we all experience with their HITs. I hope we all benefit from these new changes that Brian mentioned, the quality workers on Mturk are on boards like these and they want to work on Crowdflower HITs.
     
    #9 BeerMe, Feb 8, 2013
    Last edited by a moderator: Feb 8, 2013
  10. janissary

    janissary User

    Joined:
    Nov 14, 2012
    Messages:
    340
    Likes Received:
    0
    Brian, thank you for taking the time out to interact with some of your workforce, it really shows your company is focused on making better efforts.

    I'm assuming this will be brought up in one of the Q&A sessions, but I recommend worker reward being high on the agenda. Many times with CrowdFlower HITs, batches in particular, the pay is far less than an acceptable minimum hourly wage, and I know this is something that can be addressed and worked on in a positive way for both parties.

    Since you yourself have clients that I'm sure specify exactly what they want to pay workers, perhaps some direct transparency with them on this issue could be tasked.

    Again, thanks Brian, and looking forward to hearing more from you.
     
  11. ewd76

    ewd76 User

    Joined:
    Oct 8, 2012
    Messages:
    1,133
    Likes Received:
    0
    Because they limit the number of people who can work on that particular job. Usually because a large sample is needed. Several requesters that I know of do that. Probably more. Nothing to be alarmed over.
     
  12. bumblepup

    bumblepup User

    Joined:
    Jan 24, 2013
    Messages:
    89
    Likes Received:
    0
    Brian,

    I just wanted to say thank you for introducing yourself in this forum. The increased communication can only be beneficial!
     
  13. Shego

    Shego User

    Joined:
    Feb 4, 2009
    Messages:
    475
    Likes Received:
    0
    I rarely do CF hits for a few reasons. A big reason is the pay; which is always touted as "great" but when I do the math, it's about 0.60 to $1.20 an hour. Another issue I have is when I'm doing a hit where the answer is basically subjective and I get a message saying, "One of your answers was not what we expected." How can any answer be "expected" when it depends on the vision and experience of the individual giving it?

    At any rate, I'm hoping to see some big improvements so I have more options when choosing hits to work
    on. Thanks Brian for introducing the company and yourself to the workers here and for giving us a heads up on the direction CF is taking.
     
  14. NeoPhoenix

    NeoPhoenix User

    Joined:
    Jan 26, 2013
    Messages:
    129
    Likes Received:
    0
    This 100%. There is absolutely nothing more frustrating about doing CF hits than this right here. Hopefully things get better in the future.
     
  15. Neona

    Neona User

    Joined:
    Oct 20, 2012
    Messages:
    220
    Likes Received:
    0
    I have been staying away from CF only because of the QA, last one really frustrated me since it asked me for my opinion and then told me my opinion was wrong.

    I look forward to seeing the new changes once they are implemented. Also I agree, raise the pay. Some of those hits take several minutes to get through if you want the worker to read everything and follow the instructions, so for some of us, its not worth spending the time on it and we move to better paying hits that require less of our time.
     
  16. brian_cf

    brian_cf Verified Requester

    Joined:
    Feb 5, 2013
    Messages:
    13
    Likes Received:
    0
    Yes, we definitely acknowledge those HITs as a problem. We're making efforts to monitor the HITs created by 3rd parties better as well as provide you with more transparency as to which HITs are made by 3rd parties. We hope to create a situation where a) those HITs don't even get posted but b) if they do, people feel compelled to alert CrowdFlower, as opposed to blame CrowdFlower. Building trust is a big part of that. We're not there yet, but we're going to be doing all we can to get there.
     
  17. brian_cf

    brian_cf Verified Requester

    Joined:
    Feb 5, 2013
    Messages:
    13
    Likes Received:
    0
    @Neona, @NeoPhoenix, @Shego -

    Yes, we definitely know that the QA has been a problem. It has unfortunately been a challenge to have consistency in the quality of our QA with so many different requestors using our platform. In the next few months, however, we're going to drive a lot of change. We have high hopes.

    And our new skills tests is going to be a mechanism for getting higher paying tasks to the right people. We hope that helps a lot.
     
  18. brian_cf

    brian_cf Verified Requester

    Joined:
    Feb 5, 2013
    Messages:
    13
    Likes Received:
    0
    @bobbie01,

    ewd76 is exactly correct. I know it can be frustrating sometimes and our specific messaging could be better. We're working on revamping that as well.
     
  19. Believe

    Believe User

    Joined:
    Sep 22, 2012
    Messages:
    203
    Likes Received:
    0
    I look forward to the changes.

    Thank you to liladybugz26, RWStein, and everyone working together along with Brian to make this possible!

    Welcome to the forum Brian and Thank You for the efforts you are making!
     
  20. RWStein

    RWStein User

    Joined:
    Aug 17, 2012
    Messages:
    1,815
    Likes Received:
    0
    Thanks!

    We are very glad to have Brian here!

    This will be a mutually beneficial situation. :D

    Rob
     

Share This Page