I'm working on the crowdflower natural language categorization HITs. I'd like to do a lot of them to get my number of HITs up but I'm worried about ruining my approved percentage rate if a lot of them rejected. Within the HIT crowdflower lists the assignments completed and accuracy. If it says a 100% accuracy does that mean all of the HITs will be approved?
CloudFlower will approve even if your percentage is not 100%. They do some bizarre calculation to figure out that number. I have submitted HITs with an "accuracy" of less than 80 and been approved. I was no longer allowed to work on that batch of HITs, but there are much better requesters, so no big deal.
OK. Thank you! I get the impression crowdflower isn't well liked around here. In this instance though the hits are quite easy and there are plenty of them. Do you have any recommendations for a better requester that offer lots of multiple hits?
There are many requesters I do work for. Lots of them are one timers. The good repeat requesters that I can recall off the top of my head are p9r, Joe Elliot, Kirsten Howe and something labs. There are lots of threads to check out for ideas.
I don't mind Crowdflower, but their system often seems kind of screwy. The thing that irks me most about Crowdflower HITs (like this batch) is that if your answer doesn't match what's expected, they give you the chance to explain -- but that doesn't seem to do much for your accuracy rating, even if their expectations are obviously wrong. I've been kicked out of some CF batches before for that when I was pretty sure that their answers were wrong. (About that time is when I yell at my screen, "If you already know what you were expecting for all the answers, why are we doing this?!" ) But yes, like others have said above, they are really good about approving.
I forgot to mention Buzz.com in my earlier post. They are good for HITs on a daily basis. I have seen negative comments on them, but the only problem I have is the slow loading of the HIT.
If yu go to CF's forum, and post - they will look into the answers. I have said quite a few times the classify tweets on some were subjective. I.e. "I just went to XXX store to get food! " - was considered neutral. but "She is taking me to XXX store to get food!! yum!" was positive. CF support, while slow to respond at times, are great to deal with.