This is rewrite of MTurk Extended HIT Search with some extra features. It now uses IndexedDB (http://en.wikipedia.org/wiki/Indexed_Database_API) to store the HITs. It also implements some ideas from 'HIT history Check'-thread that use of IndexedDB made possible. See screenshots on userscripts page: MTurk HIT DataBase (http://userscripts.org/scripts/show/149548). Other new features: pending/projected earnings (with progress bar), CSV export/import, requester notes and requester blocking (you can block specific HITs). Current version is: 1.2 (2012-12-11) Works on Firefox and latest Chrome (tested on 23.0.1271.64)
Hi ThirdClassIntMasterTurker, so I've just loaded this up this week and I like the potential of it, but had a quick question about the initial load in of information. When I first update the database do I have to go to each individual day? The output I'm looking at has a lot of my previous requesters, but also is definitely missing a bunch. Out of roughly 1,000 HITs done, it's pulling info for about 390. How do I make sure it grabs everything? Thanks!
It's actually not working for me at all in Chrome. The monkeys say they are working, but no data is pulled. Darned simians! Also, it apparently played havoc with the auto-approval script. I finally had to disable it and will probably go back to the Extended HIT Search script (not sure right now whether that's the official name) that I was using before. Loved that.
You don't need to go to individual days. At first update it should get all the data that is available from amazon. That is last 45 Days. After that it only refetches status pages that have changed. You still have to update it manually every now and then, it doesn't get new HITs automatically. Have you done more than 390 HITs within last 45 days? You can find that from bottom of the submitted column on your status page (https://www.mturk.com/mturk/status). Thats the number of HITs there should be in DB after first update. Requester Overview button should then list all the requesters and show how many HITs there are in DB.
It works on chrome 23 beta. That I have tested. I hope they get it stable soon. Sometimes it helps if you change the running order of scripts. But of course it they both want to change same part of the page that doesn't help. Yes, Extended HIT Search is earlier version. I made this MTurk HIT DataBase as a separate script because current chrome still have problems with it. You can keep using earlier version and try HIT DataBase again later. But DataBase version is definitely the better one. It saves all HITs and should be more scalable. Extended HIT Search just searches last 45 days, HIT DataBase potentially all the HITs you have ever done!
Thank you. You're always so helpful! I think I just finished my first 45 days (like, I'm on day 46 or 47), so that means that days 1 and 2 are forever lost to me, right, because HIT DataBase is not working and Extended HIT Search can no longer pull it? In other words, am I correct that if you don't start using HIT DataBase in your first 45 days, you will never be able to get the data that is any earlier than 45 days before the day that you started using it?
Well, Extended HIT Search doesn't currently remove any data so technically extra days are still there in its cache. And there will (almost certainly) be import function in MTurk HIT DataBase that gets old Extended HIT Search data and puts it in the new database. But yes, mturk only shows last 45 days. It used to be only 30 days if I remember correctly. If you want to make sure that you have almost all HIT data available in some form. You could temporarily install Firefox and HIT DataBase and then export HITs in CSV-format. Or just manually copy them to spredsheet from Extended HIT Search.
Yes, definitely. My status reports 1056 approved HITs, but the database only has 425. I am only just out of 45 days doing it. When I first did the update yesterday it took a couple minutes and then I did a couple updates during the day. It's useful in that it's a quick visual way to see recent ones I've done and that's basically what I wanted. I'm just not sure why it's not picking up the majority of HITs, so still leaves me questioning if I've done certain HITs before or not.
Thank you! Unfortunately, I clean my cache out all the time, so I guess maybe that's why my monkeys are always working so hard...sounds like my first few days are lost, then. I'll see what I can do from here!
I have not downloaded a single app of yours yet. Just wanted to say thanks from the community for developing apps for us Look forward to trying out your hard work
Update to 0.17 and see if it helps. It has some bug fixes and error checks. It should at least give some error message if something goes wrong.
YES! now we're talking, numbers are virtually the same. Great job! Really appreciate the work you've put into this. I haven't fully played around with everything (notes, blocks, etc.), but will report back when I do. Turktastic!
MTurk HIT DataBase 0.19 Main interface is now on dashboard. I also added pending earnings to dashboard and summary of all requesters with pending HITs. And previous version added a button to import HIT data from Extended HIT Search.
Hi ThirdClassIntMasterTurker, quick question - does the database only keep information for the last 45 days or will it keep all data going forward? So, say I started recording HITs into the database in August, would those HITs now fall out of the database, or remain there indefinitely? Also thanks again for the script - it works brilliant and I use it every day.
No 45 day limit, it keeps all HITs until somebody/something deletes the database. IndexedDB shouldn't have any size limits either. (When database size grows to 50MB browser may ask extra confirmation.) If you are using Firefox you can make backups by copying indexedDB/https+++www.mturk.com from Firefox configuration directory to somewhere safe.
MTurk HIT DataBase should now work on stable Chrome! I just tested it quickly on Chrome 23.0.1271.64 (on Linux) and after few minor GUI updates it looks OK. Some features work better on Firefox, like progress bar while updating database, but some features seem to be faster on Chrome. Get version 0.32 from here: http://userscripts.org/scripts/show/149548.