Museum of Science and Industry Chicago
Formative Evaluation of Future Energy Simulation

Between 2012 and 2013, RK&A conducted four rounds of formative evaluation of the Future Energy Chicago simulation for the Museum of Science and Industry (MSI) in collaboration with the design firm Potion.  In the simulation, up to five teams of middle school youth compete when playing five media-driven games: Future House, Future Car, Future Neighborhood, Future Power, and Future Transportation.  In the games, players have to make decisions that challenge them to think about energy production and usage, and they receive badges as rewards for selecting energy-efficient choices. 

How did we approach this study?

Primarily middle-school youth (home-school groups, etc.) tested the simulation, as they are the target audience for Future Energy Chicago.  Each round of evaluation explored unique issues relevant to a particular design phase.  In the first round of evaluation, RK&A tested three-dimensional paper prototypes of each game to explore middle-school youth’s understanding of the concepts presented.  In the next two rounds (alpha and alpha prime), RK&A tested the games on touch-screen monitors to explore each game’s functionality as well as youth’s motivations and learning, including a badge system aimed at rewarding youth’s energy-efficient choices.  In the last round of evaluation, RK&A tested the games using a combination of multi-touch and projection technology that closely mirrored the final simulation environment.  For each round of evaluation, RK&A staff conducted observations and interviews.

What did we learn?

Each round of evaluation revealed successes and challenges of the Future Energy Chicago games that MSI staff and Potion designers used to improve the games’ functionality and messaging.  Throughout testing, findings revealed three key characteristics of the game that were compelling to middle-school youth—variety of energy choices, opportunities to design aspects of their energy environment, and challenging energy problems to solve.  Findings also revealed that youth’s prior knowledge and experiences with energy choices highly influenced the choices they made and the messages they took away from each game.  A consistent challenge throughout testing was helping youth understand the idea of trade-offs in energy choices (comfort or cost versus saving energy).  A badge system was implemented to address this issue, as well as to incentivize youth to select energy-efficient choices.   

What are the implications?

This study underscores the importance of iterative testing when evaluating a complex digital learning environment.  Not only did MSI staff and Potion designers need to learn the barriers to effectively using the games, including the intuitiveness of the technology, the museum needed to understand what about the simulation motivated youth’s game play and empowered them to make smart energy choices as future residents of Chicago.  Further, RK&A facilitated reflective discussions between rounds of testing that enabled MSI staff and designers to apply the study findings and recommendations to the next round of testing, ultimately improving the overall functionality and effectiveness of Future Energy Chicago.

See all Museum of Science and Industry projects.

Digital Projects Museum of Science