What I PERSONALLY want to know from the requestor is pretty simple. Would he rather fight 100 duck sized horses or one horse sized duck!?
Hello all, Franklin here. First of all, I was surprised to find some...interesting...dialog in these threads, so let me be perfectly clear – the previous thread regarding rejecting all HITs was not posted by me, and shame on the user who began spreading chaos and false information. Secondly, I will pose my dilemma to the group. There was 1 control value in each of the HITs, one that was intended to be very clear and easy to answer and should have resulted in a 100% match to the known value (the information was either spelled out or repeated by both sides somewhere in the 20s snippet). A 100% match for this value would result in approval for the overall HIT. If I programmatically test your answers against my known values, then I would automatically reject 267 (~18%) out of the 1500 email HITs. This was a disappointing finding on my part – some were close, but others were WAY off. But I have decided to be more lenient this time and accept email answers that are within 1 character -- I realize some of the controls might not have been as clear as initially thought, and that will be corrected before posting more HITs. Herein lies the dilemma -- what shall I do with those that are 2 or more off (that’s more rhetorical than an actual question for the group, but I want to be very up front with what I am thinking). And some of your peers submitted answers that look like he lay down on the keyboard and then clicked submit (my action here is inherently obvious). I don't know any of you – though I feel like somewhat of a relationship has been started with many of you who emailed me directly – and I have to have a way to determine whether I can implicitly trust you and the data you are providing to me. Your results are then used by my company to match against leases for billing and in many cases, we have to have a 100% match (or pretty darn close) in order to bill our client. Is it unrealistic to expect a 100% match from you guys on a control value if it is very clearly spoken and spelled out / confirmed by both speakers? Discuss. --Franklin
Good morning and welcome to the forum For the calls that were 100% spelled out and confirmed by both speakers then it should not be a problem. I stuck with mainly e-mail and name calls so I can't speak for the phone number calls. Some situations I ran into were the following: 1. The caller spelled out one thing and the rep spelled it differently and the caller agreed. Which do we go by? 2. The caller and rep were both going back and forth to verify the info and the call ended before final confirmation. 3. The name ones were a bit more difficult since many times neither the caller or the rep spelled out the hard names or the ones with strong accents. Are the e-mail and phone number hits more useful than the names? If so I would stick with those to avoid low accuracy. It may help to add an extra 5 or 10 seconds of the call to the hit, that way if the rep/caller is verifying we can hear everything and have more accurate results. Another thing you could do is come up with a qualification test for your hits that each worker must pass before doing the work. Open communication is also a great thing
Nope - that's what you are paying for isn't it? I wouldn't expect you to pay for shoddy work - but! I would like it if you had some method of warning someone that their work wasn't up to par rather than just blocking someone outright. That gives them the opportunity to back off and not submit any more, or step up their accuracy rate. I haven't emailed you directly and I only had time to do about 3 hits prior to going to my day job, but I thought the ones I did were very straightforward. PS, Love your pay rate :bats eyes:
Thats another good point, to warn of accuracy and not issue blocks. Or do it like CS where you are just not able to do the task but don't receive a block.
Anyone who actually did HIT yesterday, can you please post the requester ID here. I was a bit hesitant to submit the HIT yesterday because every HIT I accepted, I ran into one or the other situation mentioned by footballchick29 and I don't wanted to mess it up for the requester. But since he came to this forum, I think he's genuine and a solid requester.
I should be a bit more clear.. for those who are blatant in spamming and poor after being warned should absolutely be blocked. But I do agree with giving them some kind of warning or reprieve first.
I also did mainly the email ones but I agree with the rest here, if it's spelled out there's no excuse. A qualification test is also a good idea to help thin out the group.
Yep, that was the one that confused me, too and after I heard it a few times it became clear that that particular call WAS a control. The first couple of times I came across it I left a note and marked it as not being clear. Once I realized that it was a control question I returned every hit that call was on and accepted a different one. I'm not sure how many times that particular snippet was used as a control in the email HITs but if it was 150 or more, I wouldn't be shocked OR disappointed that your results weren't what you expected, Franklin (also, welcome to the board and good morning!). Assuming the control is actually clear, no, it isn't unrealistic to expect a 100% match from us on a control value if it is very clearly spoken and spelled out/confirmed by both speakers.
I just finished reviewing the email HITs, and I am rejecting about 10 of them -- ones that were completely blank with no comments or no email communication from you yesterday. Question -- when a requester rejects your work, do you also receive the comments as to why it was rejected? I have also been thinking about creating a qualification test in addition to adding more clearly spoken and spelled/confirmed control values. I really appreciate everyone's comments -- I want to avoid many of the same problems the next time around and hopefully spend less time manually going through literally thousands of results. Many of you are concerned about being blocked, so I will set the record straight right now. You will only be blocked if you repeatedly submit sub-par work, and this will be determined only after we have reviewed your work by listening to each of the snippets that you accepted. If I have a way of contacting you beforehand, I will make every attempt to do so. Some of you have emailed me offline, but do I have a way of contacting others for whom I do not have an email address? Maybe directly through M Turk? And yes, names are more difficult for us to match on and believe it or not these are the ones our clients require a near 100% match for billing. Realizing this, and also realizing that spelling is a big problem, I don't expect to continue uploading name HITs -- will have to see how this first production run went. Thanks again, everyone.
I had a few problems with the Hits where we needed to input the name. 1) Names weren't always given, conversation in 20 sec was about something else, nothing to do with names. 2) Sometimes it was hard to differ customer rep from customer. 3) A lot of times the names weren't spelled out. For the phone number, there were no instructions about missing area codes. It would have helped if the time allotted was a bit longer. Five calls 20 seconds each, a minute is gone, at times with the heavy accents some calls had to be replayed. I think more time would allow for better quality work.
Agreed -- I will definitely extend the time allowed to work on each HIT. Some users accepted a HIT, submitted work before the time ran out, but nothing happened and time eventually ran out. Also, one user worked on 5 HITs and sent me comments saying audio was unavailable in all of the HITs. I reviewed the audio files myself and didn't have any trouble. All I have for this user is a customer ID and can't find a way to contact him/her through the M Turk interface. I hate to do it, but my only choice here is to reject the work.
Five Hits without audio..maybe a problem with their system. Sorry if anyone gets upset but if the requester checked files and they were audible then that is obviously work that should be rejected. I didn't work on as many as some, cough, cough, footballchick, but with all the ones that I did work on, I had no problem with the audio. I agree with Franklin Harrell and footballchick, scammers should be blocked, there is no point in having the requester, F. Harrell, having to go through all of this auditing of the work. I'm not a fan of blocks, not everyone who has a bad work day is a scammer, but I agree with some of the other turkers about doing soft blocks, like redwood, crowdsource.
I had around 5+ instances where the audio file didn't load, even after trying to download it to my computer and trying a different browser. But they weren't on the same HIT of course, and I did way more than 5 so I doubt it's me. But a few of them really didn't work. Franklin, you're supposed to have an option through MTurk to send an email to those who did your tasks. No idea how to go about it though.
granit, I know you guys did a lot more than me, so I know it's possible. I guess the requester has to use his judgement and compare those non audio Hits he referred to with the rest of the work that particular worker submitted.
10 really is not bad out of 1500. Yes, if you put comments with the rejection we do see them. Keep us updated on the name and phone number hits Thanks