This paper gives advice for using micro-task markets for user studies, to get quick (and yet reliable) feedback from users. The way a task is defined makes a significant difference in the results, and good design can reduce the number of users "gaming the system". They conclude that micro-task markets may be useful for user studies that combine objective and subjective information gathering, and provide specific advice (below).

This paper defines a "micro-task market" -- where short tasks (which take minutes or seconds) are entered into a shared system where users select them and complete them for some reward (generally money or reputation). The advantages of micro-task markets for user studies are that they are global and diverse, with very quick turnaround times (responses within 24-48 hours) at inexpensive rates (e.g. 5 cents per rating). The disadvantages are the lack of demographic inforation, lack of verifiable credentials, and limited experimenter contact.