
An open platform for science on Amazon Mechanical Turk.
Interested in psiTurk? Try out this quick start guide to running a simple experiment online! It steps you from installing to paying participants and should work for most people using updated versions of Linux or Mac OS X. The complete set of docs is available here.
This guide uses a simple example experiment provided in the defaul psiTurk installation, but can be used to run any psiTurk experiment.
psiTurk can be installed easily on any system that has the
python package manager pip
. More information on
installation options is available here.
$ pip install psiturk
psiTurk includes a simple example project which you can use to get started.
$ psiturk-setup-example
Output:
Creating new folder `psiturk-example` in the current working directory ... Creating default configuration file (config.txt) No '.psiturkconfig' file found in your home directory. Creating default '~/.psiturkconfig' file.
In order to get access to all the psiTurk features you need to enter credentials for accessing Amazon Web Services and psiturk.org. Both of these can be added to ~/.psiturkconfig.
To access Amazon Mechanical Turk and other Amazon Web Services features
you needs to enter your AWS Credentials (see these instructions for details). You can leave the aws_region
at the default value.
To access psiTurk online features such as the Ad Server you need to create an account on psiturk.org. Please visit http://psiturk.org/register to sign up or http://psiturk.org/login to obtain your crediations. On your psiTurk dashboard click "API Keys" and enter them into your file.
$ cat ~/.psiturkconfig
Output:
[AWS Access] aws_access_key_id = YourAccessKeyId aws_secret_access_key = YourSecretAccessKey aws_region = us-east-1 [psiTurk Access] psiturk_access_key_id = YourAccessKeyId psiturk_secret_access_id = YourSecretAccessKey
psiTurk should be run in the top level folder of your project. You should be greeted with a welcome screen and command prompt.
There is also an extensive help system in the command line.
Type help
to see a list of available commands.
Type help <cmd>
to get more information about
a particular command (e.g., help server
).
$ cd psiturk-example
$ psiturk
Output:
welcome... psiTurk version 2.1.1 Type "help" for more information. [psiTurk server:off mode:sdbx #HITs:0]$
The psiTurk server is the web server which responds
to external requests (no need for Apache!). To start or
stop the server type server on
, server off
,
or server restart
.
[psiTurk server:off mode:sdbx #HITs:0]$ server on
Output:
Experiment server launching... Now serving on http://localhost:22362 [psiTurk server:on mode:sdbx #HITs:0]$
To debug or test the experiment, simply type debug
.
This will launch the default web browser on your system and
point it at your experiment in a method which is helpful
for testing.
Hint: If you are running on a remote server and want to disable
launching the browser type debug -p
(print only)
which will print the debugging URL but not launch the browser.
Altering the experiment code is beyond the scope of the quick start, but see this guide for details on how to modify and extend the stroop example.
[psiTurk server:on mode:sdbx #HITs:0]$ debug
Output:
Launching browser pointed at your randomized debug link, feel free to request another. http://localhost:22362/ad?assignmentId=debugX12JJ8&hitId=debugA7NP2T&workerId=debugS9K039 [psiTurk server:on mode:sdbx #HITs:0]$
In order to make the experiment available to workers on Amazon Mechanical Turk you need to:
config.txt
and set the host
field under [Server Parameters]
to 0.0.0.0
.In the example on the right you will be using the Amazon "sandbox" which is a place for testing your task without actually offering it live to real paid workers.
Your HIT should be visible on http://workersandbox.mturk.com if you search for your requester account name or the word "Stroop" (set in config.txt
).
Statistics about page views and accepts for this HIT/Ad should appear in the psiTurk web dashboard. This is a great place to test your code and make sure everything looks ok.
To create a HIT with one assignent (i.e., room for one subject) which pays $1.00 as a base rate and is available for the next 24 hours:
Input:[psiTurk server:on mode:sdbx #HITs:0]$ hit create
number of participants? 10
reward per HIT? 1.00
duration of hit (in hours)? 24
Output:
***************************** Creating sandbox HIT HITid: 28K4SME3ZZ2MZI004SETTTXTTAG44LT Max workers: 10 Reward: $1.00 Duration: 24 hours Fee: $1.00 ________________________ Total: $11.00 Ad URL: https://sandbox.ad.psiturk.org/view/XXXXXXX?assignmentId=debugC1R1G1&hitId=debugFM1NWB Sandbox URL: https://workersandbox.mturk.com/mturk/searchbar?selectedSearchType=hitgroups&searchWords=XXXX
By default psiTurk saves your data to a SQLLite database
participants.db
in your base project folder. You
can check that everything is being recorded properly by opening
that file in a SQLLite tool like Base.
More information about databases can be found here.
One simple way to monitor how many people have actually accepted your
HIT is with the hit list --active
or hit list --reviewable
commands.
This shows the HITid for each task, how many have completed, and how many are pending.
[psiTurk server:on mode:sdbx #HITs:0]$ hit list --active
Output:
Stroop task Status: Reviewable HITid: 28K4SME3ZZ2MZI004SETTTXTTAG44LT max:10/pending:1/complete:0/remain:9 Created:2014-04-02T19:45:42Z Expires:2014-04-02T19:48:48Z
psiTurk provides many tools for approving workers, assigning
bonuses, etc... Try help hit
and help worker
.
One simple approach is to approve all the workers associated with a
particular HIT (one all the assignments are complete). To do this use
the worker approve --hit HITID
command.
[psiTurk server:on mode:sdbx #HITs:0]$ worker approve --hit 28K4SME3ZZ2MZI004SETTTXTTAG44LT
Output:
Approving....
In order to create public hits on the "live" AMT site you need to switch
modes in the command shell using the mode
command. To get
back to the sandbox just type mode
again.
To avoid mistakes, psiTurk defaults to sandbox mode (this behavior can be changed in config.txt
)
From here, everything is exactly the same as described for sandbox hits above.
[psiTurk server:on mode:sdbx #HITs:1]$ mode
Output:
Entered live mode [psiTurk server:on mode:live #HITs:0]$