Description

In this step the end users as well as the management of the company are integrated. Depending on the iteration of the development process, there will be a formative or summative evaluation based on the actual outputs of the prototyping and implementation process.

In this step, the whole range of evaluation methods is possible. Thus, there are numerous methods to choose from. For early iterations -after the bug testing phase-, we recommend in-depth interviews and focus groups, to consider the users' requirements and preferences right from the beginning. Further iterations may include expert evaluations, lab tests with questionnaires, empirical evaluation, etc. Possible instruments are mentioned below.

After the evaluating step, the analysis of current differences in the objectives and weaknesses of current activities (based on the current prototype and evaluation results) is performed again as well as the following steps of the cycle.

From a literature review we identified a number of criteria measured in evaluations of gameful enterprise applications. These criteria serve as the elements of the Impact Layer of the Gamification Design Space by Schering (2014):

Tools, Methods and Links

Overall Gamification Questionnaires

• Internal Gamification Quesionnaire (IGQ)

Questionnaires for measurement of behavior

• Dimension Scales of Social Interaction
• Two-Dimensional Social Interaction Scale (2DSIS)
• Affective benefits and costs of mediated awareness questionnaire (ABC-Questionnaire)

Questionnaires for measurement of binding

• Loyalty to Supervisor Scale (LS)
• Organizational Commitment Questionnaire (OCQ)
• Organizational Identification Scale (OIS)

Questionnaires for measurement of motivation

• The Work Preference Inventory (WPI)

Questionnaires for measurement of experience

• Flow State Scale (FSS-2), Core Flow State Scale (C FSS-2) and Short Flow State Scale (S FSS-2) by Jackson, Eklund, and Martin
• Flow Short Scale by Rheinberg, Vollmeyer, and Engeser (2003)
• GameFlow heuristics by Sweetser and Wyeth (2005)
• Organizational Commitment Questionnaire (OCQ)
• Utrecht Work Engagement Scale (UWES)
• The Usability Metric for User Experience (UMUX)
• The Usability Metric for User Experience LITE (UMUX-LITE)
• User Experience Questionnaire (UEQ)
• Gameful Experience Questionnaire (GEQ)
• Overview about lots of UX evaluation methods

Readings

Csikszentmihalyi, M. Das Flow-Erlebnis, H. Aebli, Ed.: Klett-Cotta, 1985.

Brooke, J. "SUS: a "quick and dirty" usability scale," in Usability Evaluation in Industry. London, Great Britain: Taylor and Francis, 1996.

IJsselsteijn, W.A., de Kort, Y.A.W. and Poels, K. "The Game Experience Questionnaire: Development of a self-report measure to assess the psychological impact of digital games. ," Manuscript in preparation.

IJsselsteijn, W., van Baren, J., Markopoulos, P., Romero, N., De Ruyter, B. "Measuring affective benefits and costs of mediated awareness: Development and validation of the ABC-questionnaire," in Awareness Systems. London: Springer, 2009, pp. 473-488.

Ilardi, B. C., Leone, D., Kasser, T., Ryan, R. M. "Employee and Supervisor Ratings of Motivation: Main Effects and Discrepancies Associated with Job Satisfaction and Adjustment in a Factory Setting," Journal of Applied Social Psychology, vol. 23, no. 21, pp. 1789-1805, 1993.

Jackson, S. A., Martin, A. J., and Eklund, R. C. (2008). Long and short measures of flow: The construct validity of the FSS-2, DFS-2, and new brief counterparts. Journal of sport & exercise psychology, 30(5), 561.

Laugwitz, B., Held, M., Schrepp, T. "Konstruktion eines Fragebogens zur Messung der User Experience von Softwareprodukten [Construction of a questionnaire to measure the user experience of software products]," in Mensch und Computer 2006 – Mensch und Computer im Strukturwandel, A.M. Heinecke and H. Paul, Ed.: Oldenbourg Verlag, 2006, pp. 125 – 134.

Poels, K., de Kort, Y., Ijsselsteijn, W. "It is always a lot of fun!: exploring dimensions of digital game experience using focus group methodology.," Proceedings of the 2007 conference on Future Play, pp. 83-89, 2007.

Sweetser, P., Wyeth, P. "GameFlow: a model for evaluating player enjoyment in games," Computers in Entertainment (CIE), vol. 3, no. 3, p. 3.