Behavioral Economics with Mechanical Turk
01 Nov 2009 – New York
To find out just how predictable our apparently irrational decisions are, and partly to make sure these researchers aren't just making stuff up, I set out to reproduce the results of a study in behavioral economics.
Behavioral economics is a field full of surprising, and often entertaining, results. In his excellent TED talk, Are we in control of our own decisions?, Dan Ariely talks about a number of studies where adding a choice that nobody wants has a dramatic effect on which of the remaining options people ultimately choose. In one example, Dan asked students to choose a subscription package for the Economist. Offered a choice between an online subscription for $59, a print subscription for $125, and an online + print subscription for $125, 16% of the students chose online only, and 84% chose online + print. As expected, no one took the print only option for $125; it's clearly a bad deal. But when Dan removed the print only option, the results were inverted: 68% chose the online only option while only 32% chose the online + print option.
I'm sure I’m not the only one who finds these kinds of counter intuitive results entertaining. They're surprising in a kind of satisfying way, like you've just been let in on a secret. At the same time, results like these disconcerting; they reveal serious vulnerabilities in our basic thought processes. Vulnerabilities which are readily exploitable, because as the title of Dan's book, Predictably Irrational, suggests: our missteps in the decision-making process are so, well...predictable.
The Plan
To find out just how predictable our apparently irrational decisions are, and partly to make sure these researchers aren't just making stuff up, I set out to reproduce the results of a study in behavioral economics.
After a bit of googling, I came across Neil Stewart's paper, The Cost of Anchoring on Credit-Card Minimum Repayment's (you can have a full version emailed to you here). The short of it is that Neil finds that removing the minimum payment information from a credit card statement has no effect on the amount of people who pay their bill in full, but for the group which makes only partial payments, the omission increases the average payment size by 70%. He estimates that, thanks to the power of compound interest, for a typical credit card holder ($4000 in debt, 20% APR), the change in payment size associated with leaving out the minimum payment info could halve the total amount of interest paid over the life of the debt.
When you'd like to ask 200 random strangers a question and have the answers in a few hours, Amazon’s Mechanical Turk is a good place to start. Now since this experiment is more for fun than for science, I'll be ignoring plenty of scientific rigor. But, it turns out that the demographics of MTurk users are reasonably well representative of the overall US population. Details are available in this overview: Mechanical Turk: The Demographics.
The Experiment
The experimental setup is simple.
- Create two mock credit card statements: one with a minimum payment required, one without.
- Show each of the two statements to two groups of 100 MTurk users (200 total).
- Ask them how much they would pay.
Here are the mock statements, can you spot the difference?
They're identical, save for the "Minimum Payment Due: $15," line in the top right. If they look familiar, it's because they're based off the top section of a chase credit card bill.
Participants were restricted to US based MTurk users, and were offered $0.08 for completing the survey, slightly above the default setting. The survey was conducted in two blocks of 100 (both variations were shown 50 times in each block), one in the evening and one the next afternoon. Each of the two blocks was completed in under 2 hours. The survey also asked participants for their age and gender.
The Results
I found that the number of people who payed the entire balance was unaffected by the inclusion or omission of minimum payment information (like in Neil's study). But that when the minimum payment information was left out, the average partial payment rose by 25.8%, from $65.87 (30.1% of the balance) to $82.84 (37.9% of the balance), a Mann-Whitney-Wilcoxon test gave p = 0.0606. In other words, there is about a 6% probability we observed this result by chance. If you're interested, click here for the full dataset (you can change the "output" query param to csv or txt).
This chart shows the distribution of partial payments in $25 bins (except for the last bin, marked "200+", which represents those who payed more than $200 but less than $218). As you can see by just sort of eyeballing, omitting the minimum payment info shifted the distribution of partial payments upward.
A number of participants payed either $218 or $219 (instead of the exact total $218.65). I think this may have been a mistaking of the wording, "in US dollars," for "in whole US dollars." One participant responded $318 which was counted as a full payment.
Discussion
It worked! More or less. The simple omission of a single line on the statement increased partial payments by just over 25%. A substantial amount for such a small change, but considerably less than Neil's finding of a 70% increase. The difference in the size of the increase might be explained by the difference in account balance between our two studies. Neil's study used a credit card bill with a balance of £435.76, or about $721.05 (based on today's exchange rate); my mock bill had a balance of only $218.65. Though it's worth noting that this difference seemed to have had no effect on the proportion of participants that payed the full amount: Neil saw about 55% overall, while I saw 54%. It would be interesting to rerun the experiment over a range of bill sizes to see what happens.
Conclusion
There's something to be said for the lead time of this experiment. The whole thing, creating the mock credit card bills and MTurk task, running the experiment, analyzing the results and writing this article took just under 3 evenings worth of free time. I'd imagine that someone more familiar with MTurk and R (used to make the graph) could have done it in a single work day.
Does this have some kind of broad implications for science? No, I don't think so. If anything, the quick turnaround makes mining for answers you're already looking for easier. MTurk is definitely cool though. And providing the ability to quickly A/B test ideas and assumptions makes it a potentially powerful market research tool.