gamelab

Experiment participants are identified by an e-mail address, an opaque identifier in captive configurations, or a worker identifier in Mechanical Turk configurations (this is effectively a captive scenario). An experiments can have as many participants as the server (or the experimenter!) can handle.

Participants are assigned by the experimenter, assign themselves in a captive configuration, or join from Mechanical Turk console. In the first case, participants are e-mailed a password and login credentials when the experimenter begins the experiment. (If no e-mail server has been configured, they are not e-mailed.) These participants can log in and log out over the course of the experiment. In the latter cases, participants are not e-mailed, and are assumed to be captive (i.e., they cannot log out or back in) for the duration of the experiment.

Participants may be added (or add themselves) before an experiment begins, so that participation is static; or be added (or add themselves) both before and after the experiment has begun. In this dynamic case, experimenters can limit the number of simultaneous participants, allowing for a constant flow of participants joining and leaving the system as the experiment progresses. Participants that have logged in but cannot yet play due to the concurrent-participant limit are placed in a lobby until they join the experiment. (The same lobby is also used for the Questionnaire.)

When the experiment begins or when participants otherwise join, participants are randomly assigned a player role (row or column) used for all games throughout the experiment. Experiments always have an equal assignment of row and column player participants unless the total number of participants is odd, in which case the extra participant is assigned a random role. If participants join after the experiment has started, they are assigned to a player role that balances role counts at the time of assignment.

The experimenter may stipulate that participants step through a questionnaire before being allowed to play. In this treatment, participants are initially dropped into a lobby where they must answer questions before beginning. (This lobby is also used to handle an overflow of participants as defined in the Participants section.)

The questionnaire is designed to ensure that participants know how to play with minimum bias toward a given configuration. Results are recorded (duration of answering, number of tries, etc.) so experimenters may correlate performance with outcome. Experimenters may customise their questionnaire manually—please contact us if you need guidance in doing so.

A game in gamelab is simply a payoff bi-matrix. Payoffs are rationally-valued numbers. (Note: negative participant tickets will cause the lottery computation to fail.) There is no limit to the number of strategies for this matrix; however, in practise, only so many columns fit neatly on a small screen. Games larger than four or five strategies wide will look funny. Each experiment can have as many games as desired.

Once the participant joins an experiment, participants can submit game-plays, one per game per round. A game-play consists of a strategy mixture: a set of probabilities assigned to each of the player role's actions. The submitted probabilities are non-negative and rationally-valued, and always sum to one. Participants submit these probabilites by clicking on rows in the game matrix (participants appear to play the row player regardless of their player role) and entering in a decimal or rational number corresponding to the action.

Participants can only play one game at a time, with each game played once per round. If they log out and log back in (an action available only to non-captive participants), then the last game to play is displayed, assuming the round has not advanced in the meantime. Game matrices have their rows and columns randomly shuffled between participants, making it slightly more difficult to compare actions. Moreover, the order in which games are displayed is randomised between participants.

At the conclusion of each round, participants who have submitted mixtures for all games have their payoffs computed as points. (Rounds with incomplete games yield zero points.) Points are computed by playing the participant's mixture against the average mixture of participants in the opposing player role. Only opposing role participants who have played all games are considered. The points are then computed from the game payoffs. All of these calculations are over rational numbers and are unaffected by rounding. If not enough participants have played in a given player role, no points are tallied for the round.

After the first round, participants can review the full history of play. This includes the history of their own strategy mixture, the average opponent role mixture, and the participant role's average mixture. Moreover, it includes the participant's accumulation of points, points per round, and the hypothetical points per round of playing pure strategies. Experimenters may also provide a seed history that is prepended to the actual history of play. This allows the results of prior experiments to be incorporated into the current one.

When a participant has exceeded her maximum allowed rounds, she is no longer able to submit plays to the experiment. When all rounds have concluded, the experiment halts. Participants may still log in and review the history, as well as see the total number of points awarded and their lottery ticket allotment.

The experimenter may then trigger the lottery from the console. (Experiments may also run without a lottery.) This consists of seeding a deterministic random number generator with a given value. Each of these values is mapped into the lottery tickets to choose a winner. A participant is only chosen once for winning: if the random number re-selects a participant, a new number is drawn (rejection sampling). If any participants have a negative total score, the lottery is disallowed.

Upon triggering winner selection, winner participants are notified on their login screen, along with the lottery ticket causing the win. This is for transparency of the winner selection. Non-winning participants are also notified.

The user interface to gamelab is via a web browser. In general, the user interface involves an HTML5 web-page styled with CSS. Both HTML and CSS have been carefully validated using the W3C Markup Validation Service and CSS Validation Service, respectively. These web-page uses JavaScript to request data from the gamelab server via a series of JSON objects. The JavaScript used by gamelab is fairly recent, so it's unlikely that older browser will work properly. (It has, however, been checked over with JSHint and other related services.) Check that your browser supports ClassList, FormData, and of course XMLHttpRequest (AJAX).

Graphs play a significant role in gamelab. The interface includes a local copy of flotr2 for drawing line and bar graphs. These graphs will display properly on any modern web browser.

The interface is styled with the Font Awesome icon set. The path to the icon configuration may be provided during compilation, and defaults to the on-line repository.

A copy of HumanizeDuration.js is also bundled with the system for displaying readable time durations.

The user interface also makes use of cookies to store your session materials. Gamelab cookies themselves don't store any identifiable material: they store only the server session identifier and a random number to prevent sharing of session identifiers. The system will also operate with session information embedded in URLs, but this is only enabled for Mechanical Turk play.

The backend of gamelab is a BCHS system consisting of two compiled CGI programs: one for the experimenter, one for participants. These are small ISO C applications linking to a number of open source libraries. Consult the Deployment section for how to install these libraries.

Gamelab has been run on OpenBSD with httpd and the historical Apache, Mac OS X with Apache2, and Linux with Apache. System administrators are not encouraged to use Apache2 since it disallows long-running child processes of the CGI scripts. Such processes are used in several circumstances.

Internally, gamelab stores all non-integer numbers (strategy mixtures, payoffs, etc.) as rational numbers encoded as strings. These rational numbers are manipulated by GMP routines. The reason behind using rational numbers is precision: while the numbers themselves may grow quite large in size (if they compound), the library ensures that no roundoff occurs when tallying points.

No floating-point numbers are used for internal tallies, although these are distributed to the web frontend for more intuitive representation.

Matrix ordering and player role, among other subsystems, require a non-deterministic random number generator. To accomplish this, gamelab uses the BSD arc4random system interface to provide high-quality randomness. This is emulated with libbsd on GNU/Linux. The lottery computation uses a deterministic random number generator using an experimenter-defined seed: this allows lottery draws to be repeatable.

Note: if you're running on GNU/Linux, you may not benefit from the high quality randomness, as well as the general security and reliability, of BSD systems. We strongly encourage BSD as a platform of choice, especially OpenBSD, which is the main development platform.

Gamelab is designed with security in mind, but inherits most of its security from the operating system. In the recommended OpenBSD deployment configuration, gamelab binaries are statically compiled and run within a file-system jail. You can effect the same conditions on any well-configured web server configuration, however. On any supported system, the library used to communicate with the web server, kcgi, is heavily invested in security.

The logical security of the application itself is proportionate to the data security. Passwords are stored in clear-text because participants cannot actually change the passwords: they are opaque tokens (cookies). The only security-sensitive information on the server is the e-mail server password and the AWS secret key if provided for Mechanical Turk configurations.

To protect against misbehaving linked libraries that can access the network, the gamelab executables do enforce a strict process separation when invoking any e-mail or network-access functions. The rational-number and database libraries are used within the main processing context, however.

To operate a gamelab server, you'll need a modern UNIX system with a web server accessable to your participants. You are encouraged to use OpenBSD, which is the default testing and development environment. I don't discuss web server deployment in this manual: please consult your operating system documentation on how to do so. Specifically, you'll want to research on how to enable CGI scripts. FastCGI is also possible, but not the default installation.

To date, gamelab has been deployed on Mac OS X, GNU/Linux, and a number of BSD systems. It has been run under the Apache web server, nginx via slowcgi(8), and OpenBSD's httpd(8) via slowcgi(8). The system is designed to run in a jail as is the default with OpenBSD httpd(8). You are encouraged to do the same in any environment for security reasons; however, the system will also run fine in a non-jail context.

As mentioned in Backend, many configurations have been tested: only Apache2 is known to have issues, and these generally are harmless.

With your environment prepared, you'll need to download the gamelab source code as well as its dependencies. These are compile-time dependencies, so if the system compiles, it will run just fine. There are no additional run-time dependencies. The dependencies consist of the following:

• kcgi (the CGI framework),
• GMP (rational number library),
• SQLite (database), and
• libbsd (Linux compatibility, if applicable),
• libCURL (e-mailing),
• libexpat (parsing Mechanical Turk responses), and
• json-c (testing system).

If you're running without a connection to the Internet, you'll also need to download the Font Awesome icon set and adjust the Compilation flags to point to your local copy.

Each of these systems has its own installation procedure beyond the scope of this document. After you've installed the dependencies, download the source archive and the checksum.

You can download the latest gamelab source code from the snapshots directory, or browse the source on the readonly GitHub repository. This also contains all previous releases.

Move the gamelab.tgz source code into a source directory (we suggest ~/Source or ~/src and unpack. First, however, verify the checksum. These instructions are valid for new installations and for upgrades: the source archive will unpack as gamelab-MAJOR-MINOR-BUILD, so you won't overwrite previous installations on upgrading.

openssl dgst -sha512 gamelab.tgz | cmp - gamelab.tgz.sha512
tar zxf gamelab.tgz && rm -f gamelab.tgz

Once you've unpacked the source code, you need to compile the server. Begin by editing the Makefile to your specifications: these will hard-code a configuration that will define run-time operation, e.g., paths and filenames.

ADMINURI

This is the administrative CGI URI relative to the server root. It will correspond to the mapped location of CGIBIN plus the script name. This is often /cgi-bin/admin.

CGIBIN

This is the path used for installing compiled (executable) CGI binaries. This is usually the cgibin directory for your web server. In most web servers, this defaults to /var/www/cgi-bin. It must be accessable (i.e., mapped) by the web server process.

DATADIR

This is the path used for installing read-only data files. This doesn't have a standard location. We recommend using /var/www/data. It must not be accessable by the web server process—only the running CGI processes.

HTDOCS

This is a path used for installing read-only static pages accessable by the web server process like CSS, JavaScript, and HTML. This is usually the htdocs directory for your web server. In most web servers, this defaults to /var/www/htdocs. It must be accessable (i.e., mapped) by the web server process.

HTURI

This is the mapped location of the HTDOCS variable. In most web servers, this defaults to / (the root).

LABURI

This is the player (laboratory) CGI URI relative to the server root. It will correspond to the mapped location of CGIBIN plus the script name. This is often /cgi-bin/lab.

LIBS

If you're going to statically link (see STATIC) then you may need to define extra libraries to satisfy your compiler. This really depends on the operating system. For OpenBSD, you'll need to define -lintl -liconv. For non-static compilations, you probably don't need anything here.

PREFIX

Convenience variable for defining the prefix of other variables. This is not used during operation.

RDATADIR

This is the path used by the running CGI process to find DATADIR. This is because the running process may be jailed in a file-system (such as /var/www) and the absolute path of DATADIR may not be available.

STATIC

If you're going to run in a file-system jail and don't want to import dynamic libraries, you can statically compile the gamelab executable by specifying -static.

Sample variables are set for a local Mac OS X installation (non-static in one's ~/Sites directory) and OpenBSD production (static in a named subdirectory of the usual installation locations).

There are some miscellaneous components that one may tune.

DSYMUTIL

Only used for testing on systems supporting valgrind. This should point to the command required to run dsymutil, which is usually sudo dsymutil, to generate debugging information for valgrind processes.

FONTURI

Specify an alternative path for the Fort Awesome icon set. This allows administrators to deploy LAN-only experiments, as otherwise they'd need to contact the given server for font icons.

Once you've defined these variables, compile the software in the usual way by executing make. The software should have no errors or warnings. If it does, please contact Kristaps with details.

Once you've compiled the software and configured its parameters, you can install it by simply running make installcgi. (You'll probably need to do sudo make installcgi, however.)

Warning: this will overwrite any existing database! To update an existing installation (assuming the database hasn't changed—the release notes for each version will tell you), use make updatecgi or sudo make updatecgi. I do not recommend this, as it's easy to miss a small database change. Gamelab does not provide any facility for modifying the database in-flight.

Both of these steps will install the following files.

• Administrative CGI executables are installed into CGIBIN. These files are identical but for the suffix. These must resolve to ADMINURI.
• Laboratory CGI executables are installed into CGIBIN. These files are identical but for the suffix. These must resolve to LABURI.
• Statically-served, read-only, web-server accessible files are installed into HTDOCS.
• Read-only data files are installed into DATADIR. They are used by the CGI processes to serve content to data requests. For example, there are a number of e-mail templates that are used by gamelab to programmatically send mails.

If you run make installcgi (instead of make updatecgi), the following will also be installed.

• The writable database file itself, installed into DATADIR. This is initialised to the defaults described in the Experimenters section.

There are a number of file-system requirements to which you must be mindful.

• The SQLite database system usually needs writable access to a temporary directory. If you're running in a file-system jail (e.g., OpenBSD), you'll need to make sure the temporary directory exists in the root of the jai. On default installations, this consists of a tmp directory in the document root, e.g., /var/www/tmp, with writable permissions.
• The database directory itself needs file-creation permissions to allow for journalling (it uses write-ahead logging).
• You will need the /dev/null device in your file-system. The file-system of the device directory must not be nodev in its mount options (see mount(8)).
• If you're running in a file-system jail, the security measures of kcgi on OpenBSD with systrace(4) (i.e., OpenBSD <5.9) require a systrace device in the dev directory of the document root, e.g., /var/www/dev. Moreover, the file-system of this directory must not be nodev in its mount options (see mount(8)).

Once you've deployed your system, you can test it by going to the experimenters site.

Begin by accessing the experimenter URL with your browser. This will usually look like http://foo.com/cgi-bin/admin, but it depends on how your system administrator has configured the system. You will be taken to a page prompting for an e-mail and password. Note: you'll need to configure your browser to accept cookies and run JavaScript for the domain upon which gamelab is running.

The default e-mail address is foo@example.com and the default password is xyzzy. If the system has been properly configured, you'll be logged in. Note: you can log out at any time by clicking the navigation bars in the upper left-hand corner, then selecting Logout.

The first thing you should do upon logging in is change the experimenter e-mail address and password using the Change Password and Change E-mail sections of the experimenter console. Set the experimenter e-mail to be your own e-mail address and the password to something fairly complicated. You'll be logged out—and must log back in—after each step. Make sure that no participants will be able to guess (or know) your credentials!

The next step is optional e-mail configuration in the E-mail Server section. You can run some experiments (e.g., captive-only) without e-mail support, but you won't be able to e-mail yourself the database. So it's best to set this. You may need your system administrator's help, but you should be able to enter the same information as when you configure your e-mail reader: the from address the e-mails will use (usually your own so that they can respond to you); the outgoing (SMTP) e-mail server in the form smtp://foo.bar:587; and the user and password for accessing that server. Gamelab won't work for non-secure (i.e., non-TLS) e-mail access. When you're finished entering this information, test the configuration. You should receive an e-mail in a few minutes from the gamelab server.

Next, the experiment itself. Start with a set of normal form bimatrix games in the Games section. These can have any number of strategies and either positive integer or rational payoffs. Enter these games into the Games section. Bimatrix games are laid out top-left to bottom-right. So let's say you have the following game shown first as a symmetric game then with the implied payoffs to the opponent player.

These would decompose into 1 -1 0 0 -1 1 0 0 -1 1 1 -1 -1 1 1 -1 0 0, where opponent payoffs are coloured.

Next, specify a set of participants identified by e-mail address in the Participants section. Identifiers may be separated by white-space (newlines, tabs, spaces). You'll see valid identifiers noted after submitting the form. If you specify Allow captive participants, participants can register themselves by accessing the playerautoadd.html URL from the moment you submit the form, so you may see participants appear as they add themselves. They are captive in the sense that their logout button is disabled and that they are never e-mailed (their identifiers are free-form, not e-mail addresses), nor are they retained if the database is wiped. Captive mode is disabled when you start an experiment: this is a security measure so that participants can't join after you've started. You can opt to allow participants to continually join by specifying that captive mode should be preserved when the experiment begins.

Once you've added your games and your participants, you're ready for the experiment configuration as assigned in Experiment Parameters. Start with the simple options (don't click on Show Advanced Options yet.) Begin with the time and date when the experiment begins, the number of rounds, duration of each round, and a fraction of participants that will cause the round to advance before the maximum time has elapsed. (You can set this to zero to force the maximum time to elapse before the round advances.) For example, if set to 50%, then 50% or more participants playing all games in both roles will cause the round to advance regardless the amount of time left. If there are zero participants in either player role, then it's as if nobody plays, making the round run until the maximum time elapses. The minimum time per round is one moment; the minimum number of rounds is one round.

The advanced options allow for finer control—be careful in using these! See the Mechanical Turk section for adding Mechanical Turk workers to your experiment.

The login URL is used when sending non-captive participants the welcome e-mail. Their query string credentials will be appended to the given URL. You don't want to change this unless, for example, the laboratory CGI script is sitting in another URL path.

You can customise the instructions shown to participants by choosing from a set of pre-made instructions or providing your own. If you provide your own, it's up to you to make sure that the instructions are a well-formatted HTML5 fragment. Review the existing instructions on how to customise.

You can customise round advancement in several ways. The first is limiting the total number of rounds per participant. Thus, if one specifies 10 total rounds and 5 rounds per participant, each participant will play at most five rounds. This is useful for running with captive participants that can join and participant over an extended period of play time. The minimum-time (in minutes) establishes a grace time for play. This is especially useful when <100% of participants play before the round automatically advances to prevent fast participants from having an unfair advantage. Lastly, the maximum limit per round bounds the number of simultaneous participants.

There are a number of option toggles in the last subsection. You can disable the lottery computation, which is useful for Mechanical Turk scenarios, so that participants are only shown their payoffs and not ranked and assigned lottery numbers. The questionnaire is a more significant option: if specified, participants are required to step through a series of questions before playing. Note: if you specify a questionnaire, participants will not automatically be joined to the experiment when it begins. In other words, until they finish the experiment, they will not be included in the count of current participants when looking at the fraction of completed game-plays per round. They join the round following the current one upon finishing the questionnaire. Lastly, you may specify that round advancement will trigger an e-mail to the participants.

Once you've deployed your experiment, you must now administer it. You can enable and disable participants, reset participant passwords, and reconfigure your experimenter credentials. You can also review some statistics on the current experiment status.

If a participant forgets her password, you can reset it by clicking on the participant's email address and selecting the Reset Password option. (Captive participants are not affected.) You can reset all non-captive participants' passwords with the Reset All Passwords option. If a participant is misbehaving, you can disable her login by toggling the button next to her e-mail address. Once toggled, she'll no longer be able to log in. If she's already logged in, she'll be logged out; no plays are accepted.

If you misconfigured the e-mail server and no e-mails were sent out, you can retry the failed e-mail attempts by re-configuring (correctly!) the server and selecting Resend Error Mails.

When an experiment has finished and you have provided a lottery amount, you can compute its lottery by specifying a seed for the random number generator and a number of lottery winners.

Once the winner has been chosen and notified, and the game wound down, you can wipe and start afresh by selecting the Wipe Experiment option. This will first back up the database (e-mailing it to you in the process), then wipe all by the experimenter credentials, listed games, and participant identities (though not their play, of course). Note that captive participants are not retained if the database is wiped.

Setting up a Mechanical Turk experiment is very easy. First, have your AWS access key identifier and secret key handy. Fill these in when configuring your experiment, then set the number of workers (corresponding to the max assignments in the API reference), the reward amount (reward in the API), and currency conversion (used when computing bonuses). You can use either the Developer Sandbox or production mode—I recommend starting with the first. You can also enter a name and a description, which will be shown to potential workers, as well as the keywords which are used when searching for HITs. Lastly, you can set some worker requirements. You can specify an assignment approval percent, number of accepted HITs, and locale. By default, there are no requirements.

You'll probably also want to use the Mechanical Turk instructions, and you should enable the questionnaire to control for workers' understanding of game-play.

When you start your experiment, the request will be sent to either the sandbox or production server. We strongly suggest that you use the sandbox server a few times to make sure that your HIT posts as you wish it to. If there are errors, you'll be notified of them.

Note: when Mechanical Turk workers join the system, their session information is embedded in the query string. This is because most browsers disallow third-party cookies by default. So be warned that this is not a secure method of operation!

You can optionally assign bonuses to Mechanical Turk players in your experiment after the experiment has concluded. To do so, you'll need to click the Assign bonus button that appears beneath the high-scorer table on the experimenter page. This button is only shown for Mechanical Turk experiments. It's ok to click this twice: the Mechanical Turk system will keep track of who has been awarded bonuses and not award the same bonus twice.

Bonuses are only awarded to Mechanical Turk players who have indicated completion. You can see which players have done so by looking whether they have checkmarks next to their identifiers in the player listing.

The bonus assigned is the number of tickets times the conversion rate.

The database schema is fairly straightforward. It consists of the following tables. In all of these, note that Unique Identifier fields are non-zero SQLite rowid fields. Rational numbers are stored as text, such as 1/2 or just 2, starting with 0/1.

admin

Experimenter (a.k.a. administrator) credentials. There is always one (and only one) row that exists, initialised to default values.

email

The experimenter e-mail address. This is used as a predefined template value for the instructions. It is also used as the destination for backups of the database. Initialises to foo@example.com.

hash

The experimenter password. Note: this is stored as cleartext, so it is not really a hash (yet). Initialises to xyzzy.

isset

Bit-field on whether the email and hash have been set by the user. Contains 0x01 if the e-mail has been set, 0x02 for the password.

id

Unique identifier.

choice

A strategy mixture created when a player plays a round in a game.

round

The round number (starting at zero).

sessid

The sess used when submitting this choice mixture.

created

The epoch timestamp when this choice was made.

strats

A text (space-separated) list of rational numbers of the strategy mixture ordered from the top if a row-playing role or left if a column-player.

stratsz

Number of entries in strats. This obviously equals the number of strategies available to the player in that game, given the player role.

playerid

A foreign key reference to the player id.

gameid

A foreign key reference to the game id.

id

Unique identifier.

experiment

This describes an experiment configured by the experimenter. There is always one (and only one) row in this table. It is initialised to default values.

autoadd

The experiment is currently accepting (or did accept) auto-added players.

autoaddpreserve

Preserve the autoadd bit during and after the experiment start. This is useful for rolling experiments so as not to have a break between starting the experiment and reenabling captive mode.

conversion

A positive real number used in computing a conversion rate between points and currency.

currency

The string currency (e.g., US dollars, Euro) used when displaying currency information.

end

An epoch-encoded (seconds) time and date when the experiment when the last round of the experiment concludes. This also corresponds to the starting time plus the number of rounds times round duration.

flags

Bit-field. Contains 0x01 if the history is not to be transmitted to participants and/or 0x02 if games and rows shouldn't be randomised prior to display.

id

Unique identifier. (There's only one experiment row in the table.)

instr

The instructions shown to players. This must contain valid HTML5. It may contain @@gamelab-admin-email@@, which is filled in with the experimenter's configured e-mail address; @@gamelab-games@@ for the number of games; @@gamelab-rounds@@ for the number of rounds; and @@gamelab-round-time@@, a decimal number of the number of hours per round.

loginuri

Set when the experiment begins (i.e., when state >0), this is the URL given to players in their initial e-mail for when they log in. This is suffixed by ?ident=EMAIL&password=PASSWORD.

minutes

The number of minutes per round. Minimum of one.

nolottery

If non-zero, do not show the lottery facility to players or the administrator.

playermax

The maximum number of players per player role at any given time. If zero, the number of players is unbound.

prounds

The number of rounds playable by each player. If zero, players will play until the end of the experiment. If >0, players will not be allowed to play more than the given number of rounds within the experiment.

questionnaire

If non-zero, new players are not directly assigned a join round and must use the lobby facility. There, they will be asked questions and cannot proceed until all questions have been answered.

round

The current round of the experiment, or -1 if the rounds have not begun incrementing. This will be set to rounds when the experiment concludes.

roundbegan

If round is non-negative, then the timestamp (epoch) when the round was incremented. Otherwise, this is zero.

roundmin

If roundpct is non-zero, this is the grace time (in minutes) before which the round will automatically advance giving the percentage.

roundpid

If specified when configuring the experiment, a daemon will be started that periodically checks to see if the round has advanced. Its process identifier is stored in this field. Otherwise, this is zero.

roundpct

If >0, this represents the fraction of players per role who play all games and determine that the round automatically advances. In other words, if set to 0.5 (50%), then from both player roles, if 50% or more players have played all games, the round advances in advance of the set round termination time.

rounds

The number of rounds that will be played. Minimum of one.

start

An epoch-encoded (seconds) time and date when the experiment when the first round of the experiment begins.

state

The running state of the experiment, being either 0, for new (still in the configuration stage); 1, for started (players can log in, though the experiment itself may not be accepting plays yet); 2, where the experiment has expired, but the winner has not been chosen by the experimenter; or 3, where the experiment has expired and the winner has been chosen.

total

When the experiment finishes (i.e., when state is set to 2), this is filled in with the total number of lottery tickets (the finalscore) awarded to all players.

game

A game is a payoff bimatrix configured by the experimenter. There may be any non-zero number of games in a running experiment.

p1

The number of strategies for the row player.

p2

The number of strategies for the column player.

payoffs

A list of payoffs from the top-left to the bottom-right of the payoff matrix, ordered row-player payoff, column-player payoff. See First Experiment configuration for more.

name

The name of the experiment. This is shown only to the experimenter.

id

Unique identifier.

gameplay

During a given round, this records a player's status in terms of number of choice rows (plays) made.

round

The round number (starting at zero).

choices

The number of games this player has played, i.e., the count of choice rows.

playerid

A foreign key reference to the player id. When this equals the number of games, the player has played all games for a round.

id

Unique identifier.

lottery

A lottery is created for an individual player when that player has been granted payoff for all game rows in a round.

aggrpayoff

The player's aggregate payoff (as a rational number) computing by adding the previous round's aggregate payoff to the current curpayoff.

aggrtickets

The value of aggrpayoff represented as a natural number of tickets.

curpayoff

The player's current payoff (as a rational number) computing by accumulating her payoff payoff for all games in the experiment. This is set to 0/1 if the player has not played all games.

id

Unique identifier.

playerid

A foreign key reference to the player id.

round

The round number (starting at zero).

past

Records the average strategy mixture of a given player role for a given round and game when a round has concluded.

currentsp1

For the row player, the average strategy mixture of the current round. Strategy mixtures are only considered for choice rows where the player played all games for the round, so this will be a set of zeroes if no players completed all games. This is recorded as a space-separated sequence of rational numbers, one per strategy.

currentsp2

Like currentsp1 but for the column player.

gameid

A foreign key reference to the game id.

id

Unique identifier.

plays

The total number of plays that were submitted (over all games) in this round if and only if the player submitted for all games in that round.

round

The round starting at zero.

roundcount

The accumulated count of rounds not skipping, i.e., the count of zero-valued skip rounds.

skip

If zero row or column players played all games, this is set to 1, else it is 0 (sufficient players played).

payoff

When the given round has completed, this consists of the payoff of the player's choice strategy mix for a given game when played against the average strategy of the opposing player role.

round

The round number (starting at zero).

playerid

A foreign key reference to the player id. When this equals the number of games, the player has played all games for a round.

gameid

A foreign key reference to the game id.

payoff

A rational number of the payoff.

id

Unique identifier.

player

The central player table consists of all participants. The hash variable is redacted when the experiment is wiped.

answer

If the questionnaire facility has been enabled (via the experiment questionnaire), this is non-zero if the questionnaire has been answered.

assignmentid

If the player joined from Mechanical Turk, the assignment identifier. Otherwise set to the empty string.

autoadd

Player was auto-added (i.e., auto-added herself). These players are never e-mailed.

email

Player e-mail address or identifier, in the event of a captive player. This is unique in the set of all players.

enabled

Whether a player is allowed to login during an experiment.

finalrank

The minimum of the slot of the player's tickets among all players' tickets. For example, given 100 players with roughly 10 tickets each, this might be 543 to indicate that slot 543 to 543 plus finalscore are this player's slots in the lottery.

finalscore

Set to the accumulated payoffs from the lottery aggrpayoff rounded up to the nearest integer.

hash

The player's password set when the experiment is started (i.e., when the experiment state is >0) or when reset, or when a captive player registeres. Note: this is stored in the clear: it is not a hash.

hitid

If the player joined from Mechanical Turk, the HIT identifier. Otherwise set to the empty string.

id

Unique identifier.

instr

Whether the player should be shown instructions when she logs in (versus being taken directly to the game-play tab).

mturkdone

Whether the Mechanical Turk player, if applicable, has indicated that they've finished the experiment. Note that this is unprotected and can be set at any time: it is only used to display a message to report to the Mechanical Turk server.

questionnaire

The number of questions the player has entered, if applicable. See questionnaire.

rseed

A non-zero cryptographically random number given to the player when created. This is used for many purposes, one of which being the bonus identifier for Mechanical Turk participants.

role

The player role (0 for row player, 1 for column player) set when the experiment state is >0 or when the player joins.

state

The state of a player can be 0, meaning the player is newly-added and has no password; 1 when the player has been mailed her password; 2, the player has logged in; and 3, an error occured when the password e-mail was attempted.

version

A number from zero that indicates the number of time the player field has been updated. This is used to make the player object cachable.

questionnaire

The questionnaire is an optional set of questions given to each player. These players must submit a set number of correct answers before joining the game. See questionnaire. These rows are unique per player-rank: only a player's first answer counts.

first

The epoch time when the row is created.

id

Unique identifier.

playerid

The associated player.

rank

The question number starting at zero.

tries

The number of tries, correct or not. This is zero only in the race after initially creating the row and noting an attempt. It should not be used and will be removed.

sess

A session. Sessions are the usual browser session used when players (or the experimenter) are interacting with the system.

created

An epoch timestamp of when the session was originally created.

cookie

A cryptographically random cookie assigned to each session. This is used as a security feature so that people can't guess others' session identifiers and re-use them. When a session has expired or the user is logged out, this is set to zero. (Session entries are never deleted.)

playerid

A foreign key reference to the player id. For experimenter logins, this value is null.

useragent

The user-agent as reported by the player's (or experimenter's) browser. Remember: these are easy to fake, so they shouldn't be taken as authoritative, but suggestive. Also note that this is set when the player first creates the session (cookie), and may not reflect later plays. (But usually does.)

id

Unique identifier.

smtp

This consists of the SMTP server information used in sending e-mails. There is always only one row set, which defaults to empty values (see isset).

user

The SMTP server username (used for logging in).

email

The e-mail address used as the From address in all communication from the server to players and the experimenter. It is usually the same as the experimenter email set in the admin table, but may be set as a standard No-Reply. (This is discouraged, as your players should be able to reply to you if things go wrong.)

pass

The SMTP username's password (used for logging in). Note that this is stored in cleartext, so make sure that your password isn't used elsewhere.

server

The SMTP server in smtp://server:port format.

isset

Whether these entries have been set.

id

Unique identifier.

winner

The winner table consists of rows corresponding to a player and her winner status. This table does not exist until the status field of the experiment is set to 3. This table is deleted when the database is wiped.

playerid

A foreign key reference to the player table.

winner

Boolean value as to whether the player is a winner. If this is false, then the winrank and rnum columns are undefined.

winrank

If the player is a winner, the rank (first, second, third draw...) of their winning.

rnum

The random number modulo the total number of tickts used for this winning draw. In other words, this is the winning lottery ticket.

id

Unique identifier.

The general sequence is as follows:

1. Get your username and password, or optionally, have your utility register with doautoadd.json as a captive player and get a randomly-generated password.
2. Wait for the experiment to accept players by polling dologin.json in order to login.
3. Log in to dologin.json with your username and password, accepting your session identifier as cookies.
4. Wait for the experiment to begin by polling doloadexpr.json. Use your session identifier as an identifier cookie.
5. Accept the experiment data from doloadexpr.json. Use your session identifier as an identifier cookie.
6. Using the experiment data from doloadexpr.json, construct and submit a series of game plays to doplay.json. Use your session identifier as an identifier cookie.
7. Wait for the next round, or end of game, by polling polling doloadexpr.json. Use your session identifier as an identifier cookie.

Do not flood the system with requests: be considerate! Unless specifically stipulated by the experimenter, you are not rewarded for fast play. Avoid re-connecting with anything less than 30–60 seconds between attempts, unless specifically allowed by an experimenter, or your bot may be banned from the experiment.

This section describes the resources (URLs) that may be accessed for programmatic play. Each of these resources is the immediate path following the laboratory CGI script name, for example, if you're running as localhost/lab.cgi, these would be mapped to localhost/lab.cgi/doautoadd.json.

doautoadd.json (POST)

Post your e-mail address using the ident field name. Returns HTTP error code 400 (invalid or missing e-mail field), 403 (player by that e-mail already exists), 404 (captive players disallowed), or 200 (success). Upon success, the return body is a JSON object consisting of the ident and password string fields.

dologin.json (POST)

Post your e-mail address (or identifier) and password using the ident and password field names. Returns HTTP error code 400 (invalid or missing input fields), 409 (experiment not started yet), 200 (success). If you receive HTTP error code 409, wait and continue polling: the experiment is not yet accepting player logins. Upon success, the return body is empty, but the cookie field consists of sessid, your session identifiers (a number); and sesscookie, a magic number attached to your identifier.

doloadexpr.json (GET)

Get a doloadexpr object describing the full experiment. You will need to pass the sessid and sesscookie cookie fields as described in dologin.json. Returns HTTP error code 304 (not changed since last request), 409 (experiment not started yet), 429 (player accepted, but not yet joined for play), 200 (success). If you receive HTTP error code 409 or 429, wait and continue polling: the experiment is not yet started or has started, but you are not allowed to join yet. Upon success, returns the JSON experiment object.

doplay.json (POST)

Post a probability mixture for a game's actions. You will need to pass the sessid and sesscookie cookie fields as described in dologin.json. The game identifier and round must be encoded in the gid and round fields, respectively. Each action with a requested probability must be posted as field indexNNN, where N is the action index (starting at zero), e.g., index0, index1, and so on. Probabilities may be decimal or fractional strings. Missing actions are assigned zero probability. Actions greater than the maximum action are ignored. Returns HTTP error code 400 (invalid game, round, or probability, or probability sum doesn't equal one), 409 (round has passed or is being played again, experiment hasn't started or has finished, or player hasn't joined), or 200 (success). Note: the error codes for this resource are likely to become more specific in future releases.

When using the Programmable API via the web Resources, you'll need to examine some JSON objects to ascertain the experiment status. The following defines the objects available to you by resource. The type of each field is listed alongside the field name: object for dictionaries (i.e., things that contain other things), strings being UTF-8 strings, integers being 64-bit signed integers, doubles being double-precision floating points, and times being seconds since UNIX epoch.

There are many undocumented fields that are transmitted while playing: these are to be considered unstable and will not be included in subsequent releases.

experiment

Obtained from a successful doloadexpr.json request, i.e., with an HTTP error code 200 and a parsed JSON object. The resulting dictionary consists of the following top-level fields.

expr (object)

Object describing the experiment status. This object is always non-null. This contains more or less all the information you'll need.

history (object)

Additional history. See the history object.

instr (string)

Instructions for participating in the experiment. This is an HTML5 fragment describing how to use the experiment. It is generally formatted in the web interface, so the styling may make the data non-obvious.

minutes (integer)

The maximum number of minutes per round (all games). This is the maximum: in reality, rounds may range from roundmin to this value.

prounds (integer)

The total number of rounds played by the logged-in participant. The joined round plus this is the participation span for any participant.

round (integer)

The current round number starting at zero. (If the round is less than zero, the experiment has not yet begun.)

roundbegan (time)

When the current round began.

roundmin (integer)

The minimum amount of time, in minutes, per round. This is also called the grace time. If this is non-zero, players are given this much time to play freely until the round will advance based upon the player fraction (see roundpct).

rounds (integer)

Total number of rounds in the experiment. Note: you may not actually play this number of rounds. See prounds.

start (time)

Time when the experiment started (or will start).

player (object)

Information about the current participant. This object is always non-null.

joined (integer)

The round when the current participant may first play. If this is less than the current round, the participant has been accepted into the system, but is not yet playing.

role (integer)

Whether the participant is playing the row (zero) or column (non-zero) role in the game bimatrices.

history (array)

The game history and information about individual games. This is an array of objects with the following fields.

id (integer)

The identifier of this game.

p1 (integer)

Number of actions for the row player.

p2 (integer)

Number of actions for the column player.

payoffs (array)

Payoff matrix. This is an array (of length p1 of arrays (each of length p2) of strings. Each string is a rational number formatted as a fraction (e.g., 1/2, -1/2, 0, 1, etc.). Clients should use the GMP library to work with these rational numbers.

roundups (array)

An array of prior play for this game, indexed by round. If this is set to an empty-length array, the experimenter might be prohibiting the display of history, or there may not yet be a history to display given the current round number.

skip (integer)

Non-zero if there weren't enough plays in the given round to compute payoffs. Zero otherwise.

navgp1 (array)

An array of doubles representing the average weight of this action for the row player role. The array if of length p1.

navgp2 (array)

An array of doubles representing the average weight of this action for the column player role. The array if of length p2.

navgs (array)

An array of arrays consisting of the multiples of the average fraction for the given row and column player cell.