Gamification is one of those tech trends that exploded on the scene. TED talks, mass market books, love letters in esteemed publications, and all that jazz. But scientific evidence? Not so much. The problem with gamification is the huge gaping hole where empirical support should be. See, there’s not a lot of actual scientific research on the effects of gamification, and what’s there isn’t exactly a ringing endorsement. (Check out the meta-analysis by Hamari, Koivisto, & Sarsa, 2014).

I’ve worked with a couple of grad students on some gamification-related projects, focusing specifically on the more popular applications of leaderboards and badges. Kate Christy and I were curious whether leaderboards may trigger stereotype threat or social comparison. We placed women in a virtual classroom, where they saw a leaderboard dominated by women’s names, a leaderboard dominated by men’s names, or no leaderboard. Stereotype threat would suggest women would perform worse after viewing the male-dominated board; social comparison approaches would suggest that women would perform worse after viewing the female-dominated boards. After viewing the board, they took a math quiz.

Our study revealed that women performed worse on the math quiz after seeing the female-dominated board compared to the male-dominated board. Yet, the female-dominated board promoted higher levels of academic identification than the male-dominated board or seeing no board at all. In terms of application, these findings are a little frustrating as they suggest female-dominated leaderboards are bad for performance, yet good for academic identification, and the opposite is true for male-dominated leaderboards. Clearly more research is needed than this to determine the effectiveness of leaderboards among different populations and in different contexts, but it does provide a cautionary note.

You can check out the full article for free here until September 30.

This isn’t to say gamification doesn’t hold promise; it is to suggest that dropping proven educational methods in the name of fun–without any evidence of its effectiveness–is a bad idea. It’s also to suggest that gamification is a really broad term, and some forms may be better than others. And just like any treatment or method–especially with educational outcomes hanging in the balance–scientific evidence should precede practice.