We've been working on getting started as a requester on mTurk. I was hoping there would be some folks on these forums that could give us some feedback on our Hits to make sure they make sense and are easy to use. What's the best way to do that other than to publish some Hits to the main mTurk site and see what happens?
Task Our first tasks will be to locate the URL of newspapers' online sports sections. Given the main URL of the website, you'd have to use the site's navigation to find the appropriate page and then copy/paste the URL. Right now we have each site loading in an iframe in the HIT itself. Is it best to just publish some HITs and have a comment field on the form for providing feedback? Can I also post a link to the hits here to get feedback from this forum?
I prefer to have a link to the site rather than an iframe in the HIT itself. Other may like it this way, so maybe do both. Yes, keep the comment field.
And do think twice before rejecting a hit. A rejection will kill a person's approval rating and limit the hits available to him/her. A lot of people depend on mturk as main source of income. Rejections can't be reversed, please bear that in mind. Respond to mails sent by workers, so that you can get work done the way you want. If you are starting out, its a good idea to increase the time allotted. Say, allot 24 hrs for the hit, so if a worker accepts the hit and mails you, you will have tiem to respond to the query and have the work done. Too less a time will result in your hit expiring and people bitching about you in this very forum and turkopticon
Thanks for the feedback. We've already made some adjustments to the Hit and included some extra checkboxes so that even if the hit can't be completed in exactly the way we have in mind, we will still get enough valuable information so that we don't have to reject any hit. I have published 25 of the hits so that I can get a little more feedback before publishing all 3700 of them. Please take a look. https://www.mturk.com/mturk/preview?groupId=2YICGJ2D21O3DPA4QVGPZI2KYFAC6U
I'll submit 3, to see if they get properly judged and approved, if they do, it's the kind of hit i'd enjoy doing. Edit: Oh, they got approved quickly, that's great, I'll definitely work on these, thanks.
Thanks, acustic. Please provide any feedback you might have so that we can improve our Hits. We are willing to do anything that might improve worker speed and accuracy. We also aim to be fair in approving hits, so please don't hesitate to bring anything to our attention.
The hit is very clear, locate the highschool sports page or the general sports page if there's no highschool one, so no problem there, I would suggest though the part that displays the newspaper on the HIT to be somehow hidden or collapsed by default and only load when and if the user wants to, it causes the page to load hard.
acustic, thanks for the feedback. We have made that adjustment on the iFrame. riazam, I will be publishing about 1500 of these HITs shortly under the same group link that I provided earlier. https://www.mturk.com/mturk/preview?groupId=2YICGJ2D21O3DPA4QVGPZI2KYFAC6U
Also note that the down arrow which is supposed to paste the URL of the website in the submission box actually copies the name of it.
Few situational questions regarding things i encountered til now: -If a page does not have any sport news but links to another sports website, should I go on the linked site and submit it? -If the page does have a sports section but links to another site about high-school sports in particular, should i post the original news page or go to the new site with high-school news? -If the site has no sports category but has scanned pictures of sports pages from real newspapers, do I mark it as no sport section? -If the link automatically redirects to another news site from the start(different link) should I continue to identify the news page and post it?
Another quick suggestion, seems if you click the link via left click, the page loads over the hit instead of opening in a new window, it should open in a new window.
It would never hurt to go ahead and provide the link with a little note (ie. links to different site) or use the checkboxes (ie. no high school sports section found). That would work in all four cases. We won't complain if there's a valid link and a note or checkbox. The HIT is designed so that your effort to look at a site is not wasted and so that we have no reason to not approve the HIT. I have already seen a several workers just loading HITs and clicking Submit without doing anything and a couple that copied the original URL and/or left short "reviews" in the comment box. Those results are not usable. I'll get the link target fixed, thanks for the feedback.