MTurk HIT DataBase (replaces MTurk Extended HIT Search)

Discussion in 'Scripts' started by ThirdClassIntMasterTurker, Oct 3, 2012.

  1. cldamw11211

    Joined:
    Sep 14, 2012
    Messages:
    448
    Likes Received:
    0
    I need help, I have pressed the update button and left it for 30 mins and it was still updating, I have 12000 hits over the last 45 days, is that why it would take so long?
     
  2. ThirdClassIntMasterTurker

    Joined:
    Sep 8, 2012
    Messages:
    86
    Likes Received:
    0
    What browser and which browser version? If not latest version, update and try again. It requires quite recent version because it uses IndexedDB. If latest version, press Ctrl+Shift+J (on firefox/chrome) to open error console and see it there are any errors.

    12000 HITs means about 500 status pages. That should take about 10 minutes maybe (first update).
     
  3. ThirdClassIntMasterTurker

    Joined:
    Sep 8, 2012
    Messages:
    86
    Likes Received:
    0
    HIT DataBase 0.41

    New version adds R and T button to HIT preview page (and Auto-Approval time).

    I hope nobody had problems with date conversion that I added in previous version. Had to change date in database from MMDDYYYY to YYYY-MM-DD, otherwise it doesn't sort very well when year changes.

    And remember that you can make backups by copying indexedDB/https+++www.mturk.com to somewhere safe from your Firefox configuration directory. (After 45 days HIT data is no longer available from MTurk page, backup at least once per month is a Good Idea)
     
  4. www

    www Guest

    After blocking a requester how do you unblock them?
     
  5. ThirdClassIntMasterTurker

    Joined:
    Sep 8, 2012
    Messages:
    86
    Likes Received:
    0
    Click the title of blocked HIT.
     
  6. ThirdClassIntMasterTurker

    Joined:
    Sep 8, 2012
    Messages:
    86
    Likes Received:
    0
    MTurk HIT DataBase 0.42

    Looks like Amazon has made a small change on MTurk pages. Status page is now on one page (if there is over 30 days). Update to 0.42 before you update your database. If you already clicked update database, script will complain that number of HITs do not match. You can just click cancel and update the script.
     
  7. Kris

    Kris User

    Joined:
    Sep 21, 2012
    Messages:
    801
    Likes Received:
    0
    Aha! So that's why it was grumpy this morning. Thanks for the update!
     
  8. pcereola

    pcereola User

    Joined:
    Oct 8, 2012
    Messages:
    955
    Likes Received:
    0
    This completely explains my problem. Stupid question, how do I update the script?



    Nevermind I figured it out:)
     
    #28 pcereola, Nov 30, 2012
    Last edited by a moderator: Nov 30, 2012
  9. brandonjones

    brandonjones Active Member

    Joined:
    Nov 20, 2012
    Messages:
    27
    Likes Received:
    0
    I have been using the MTurk HIT DataBase script and I love it. How ever it is not working now. I have updated to to version 0.42 and have also updated Firefox to the latest version. I opened the error console as suggested by a previous post. Is there any way I can get the errors to the developer?

    Thanks.
     
  10. Jenn27

    Jenn27 User

    Joined:
    Oct 18, 2012
    Messages:
    650
    Likes Received:
    0
    How do you update to 0.42?
     
  11. ThirdClassIntMasterTurker

    Joined:
    Sep 8, 2012
    Messages:
    86
    Likes Received:
    0
    Check that greasemonkey is enabled, clicking that monkey face on toolbar will disable/enable greasemonkey. Also check that there is checkmark next to HIT DataBase on that greasemonkey menu.

    If there is any error that says HIT_DataBase or 149548.user.js on error console, that is related to HIT DataBase. You can copy paste it here.
     
  12. ThirdClassIntMasterTurker

    Joined:
    Sep 8, 2012
    Messages:
    86
    Likes Received:
    0
    You can just install it again, it should overwrite the old version.

    But now that you mention it, I should probably add @updateURL to the scripts metadata to make it sure that greasemonkeys automatic update works. I update my scripts with editor ;)
     
  13. talz13

    talz13 User

    Joined:
    Oct 8, 2012
    Messages:
    89
    Likes Received:
    0
    I have been using this since around 0.24, and I started turkin' at the beginning of October... Now I have a new PC, and can't get it to update the HIT DB to be current across the various browsers I run it on.

    I did try deleting and updating fresh on my new PC, but it seems like it doesn't grab EVERYTHING, just back to a certain point. Does it only go back to the 45 days listed on the status page? Can it be updated to take a "begin date" parameter so I could put in 10/2/2012 and have it always pull everything past that date?

    I'm running across more and more HITs that I'm pretty sure I've done before, but they're not showing up green on the HIT DB anymore.
     
  14. ThirdClassIntMasterTurker

    Joined:
    Sep 8, 2012
    Messages:
    86
    Likes Received:
    0
    It gets all HITs that are available. Unfortunately that means max 45 days. :(

    But if you can copy the database files from from you old computer, it should work. Just copy the indexedDB/https+++www.mturk.com directory from your Firefox configuration directory. Same should work with Chrome. (Just don't mix Chrome and Firefox files. They have different IndexedDB implementation so it doesn't work.)
     
  15. talz13

    talz13 User

    Joined:
    Oct 8, 2012
    Messages:
    89
    Likes Received:
    0
    I did try copying once from FF16/17 on Windows to FF10 extended support version on Linux, and it seemed to hang when updating after that. I even tried renaming the file to match what was newly created for the new install, to no avail.

    I asked about the prior dates because I have been able to access dates beyond what is listed on the status screen. For example, on my status screen, the oldest available date is 10/22. If I click through that date, it takes me to the page:

    https://www.mturk.com/mturk/statusdetail?encodedDate=10222012

    If I modify the "encodedDate" parameter, I can go back further than what is available on the status page. It will let me go back to 10/7 as the oldest date... That's right around when I started, but not exactly...

    https://www.mturk.com/mturk/statusdetail?encodedDate=10072012

    So, it may not be able to go all the way back, but it CAN search for at least a couple weeks more data than it currently does by scraping the status page.
     
  16. ThirdClassIntMasterTurker

    Joined:
    Sep 8, 2012
    Messages:
    86
    Likes Received:
    0
    Have you tried copying whole Firefox profile? It's possible to share same profile with Linux and Win Firefox, so it should work. Also that FF10 might be the problem. I would try with latest Firefox, IndexedDB is still experimental.

    Yes, looks like about two weeks. I could just try to fetch next 15 days on first update. Should be quite easy to do.
     
  17. ThirdClassIntMasterTurker

    Joined:
    Sep 8, 2012
    Messages:
    86
    Likes Received:
    0
    MTurk HIT DataBase 1.1

    Ok, it should now try to fetch some extra status pages at first update.

    Also added requester details page (small + buttons on pending/requester overview pages). This is still work in progress, but it will show some detailed information about requester.
     
  18. pcereola

    pcereola User

    Joined:
    Oct 8, 2012
    Messages:
    955
    Likes Received:
    0
    I updated to Mturk Hit database 1.1 because I was getting the message "Number of HITs in DataBase does not match number of HITs available" and the last time this happened I updated and it solved the problem. This time however, it doesn't seem to have helped and I am using Firefox and I have the latest version, so I am not sure why it's not working???
     
  19. ThirdClassIntMasterTurker

    Joined:
    Sep 8, 2012
    Messages:
    86
    Likes Received:
    0
    Second number is number of available HITs from status pages, first number is number of HITs in database within same time frame.

    There has probably been some error while it updated database and now these numbers don't match. Remember that you should not load any other MTurk pages while this script is updating its database. That's because of MTurk page load limit and because number of submitted HITs must not change. Although then there should usually be "Script monkeys expected X bananas, but got Y" error. Just trying update again should fix that.

    Clicking OK and letting the script to refetch all status pages should fix this one. (This part needs improvements. It should check each day separately. That would be much faster.) But if you have made any backups recently, using them is probably faster.

    When amazon changed status page, the script got same status page two times and counted every available HIT twice. That time there wasn't actually any HITs missing, so refetch didn't help.
     
  20. pcereola

    pcereola User

    Joined:
    Oct 8, 2012
    Messages:
    955
    Likes Received:
    0
    I did load other mturk pages while updating, so that must be what happened, but now when I try to update it just says "script monkeys are preparing to work" and nothing happens. I left it all night last night updating and when I woke up this morning it was still saying the same thing.
     

Share This Page