The power of role-based e-learning: selected content, tables and figures
Chapter 9 Evaluating and researching online role play
- Focus of Evaluation
- Focus Groups
- User Statistics and Tracking
- Participant Survey
- Analysis of Learning Outcomes
- Researching Role-based e-Learning
- Research Agenda for Role-based e-Learning
- Summary
Evaluating the implementation of online role play is important. As it can be complex, it may be deferred or even abandoned as “too hard”. The most direct and quickest way is, of course, collecting feedback from participants during and after a role play. Such participant surveys will determine immediate participant satisfaction with the learning experience but may not contribute anything to knowledge about flaws in the design or provide guidance for future versions or adaptations.
In fact there is a wide range of other useful evaluation strategies (Reeves, 1997; Reeves & Hedberg, 2003). The participant feedback option is usually quite basic and more sophisticated evaluation can involve such things as consideration of links to “graduate attributes” (in a tertiary education setting) and “at work” performance improvements. Evaluation can be an on-going process (formative) or can be conducted after it is all over to review the effectiveness of the process (summative). It can focus on such things as fidelity of design and continuing relevance for emergent learning needs, and the moderator’s insights and performance as well as that of participants.
This chapter introduces strategies for effective evaluation of role-based e-learning designs and – for those who are interested in the scholarship of teaching and learning – outlines possible research agendas.
Research Agenda for Role-based e-Learning
Adoption of role-based e-learning is not without its hurdles. Although the authors have been working in the field of simulation and role play for over 20 years, it is still a strategy in its infancy. There is a lot yet to be understood and disseminated. Project EnROLE in its BLUE report (Wills et al., 2009) has identified some issues for further research including the following:
Design Issues: Developing design capabilities and improving educators’ awareness of the design factors implicit in creating engaging and informative learning environments
The discipline of “design” has a long-standing tradition with respect to creating more tactile and physical items. In regard to online role play this book contains three chapters on this issue, with the hope this will contribute substantially to a wider understanding of the discipline of creating well-designed role-based e-learning.
Cross-disciplinary Issues: Supporting cross-disciplinary research in the development, evaluation and implementation of role-based e-learning to enhance learning experiences
Online role play involves expert knowledge of both the subject matter and design principles to turn that content into interactive experiences. This often requires cross-disciplinary knowledge to ensure the creation of an effective learning environment. A cross-disciplinary link with drama educators is a particular example that might yield useful insights. Research on how best to establish, support and sustain such activity is essential to the future of active learning environments including online role play.
Technology Issues: Improving platforms for role based e-learning environments
Another potential hurdle is the lack of support in the ICT environment. The requirement to deal with new information technologies can be a significant inhibitor to the adoption of online role play in mainstream university education. The currently available implementations are very much unconnected to the mainstream learning management systems (or VLEs). The underlying teaching/learning strategy of most learning management systems is based on traditional classroom teaching – delivery of lectures, providing quizzes and tracking participant activities. Research is needed on the development of new, easy-to-use environments for conducting role-based e-learning to address concerns of educators that some aspects of technology may detract from learning and teaching outcomes.
While Project EnROLE has generated some knowledge of criteria for selecting online role play environments and tools (Russell & Shepherd, 2010), there is a need for more work on assessing the potential of newer tools and environments, such as Web 2.0 tools and virtual worlds. Further investigation into the challenges and opportunities associated with integrating role play technologies into enterprise-level learning management systems is also warranted. Ip and Canale (2003) and Canale and Ip (2004) have proposed standardizable mechanisms to integrate an LMS or VLE with other learning engines and/or content repository and dynamic customization of look and feel for SCORM compliant content. Such work is yet to be accepted into technical learning standards because of the lack of “demand”. The IMS learning sequence and the Sakai project’s user alias are a beginning, however they are not yet a convincing standard for wider adoption. Yet new emerging technologies will find a way to be part of the technology mix for online role play.
Community of Practice Issues: Understanding more about communities of interest becoming communities of purpose
Isolated innovators are not sustainable. To establish and spread use of new learning designs, there has to be a viable community of existing practitioners that can attract and support new adopters. Eraut (2002) notes that the ideal self-organizing community of practice is rare, and suggests that only a few of its characteristics are necessary to build a learning community. If this is so, it allows more scope for national and institutional initiatives that mix formal and informal support. Further research could examine which characteristics of successful communities of practice have contributed most to the adoption and spread of new learning designs such as online role play. This would show how future funding for educational design initiatives might be best directed.
Finding and Sharing Issues: Encouraging academics to contribute towards repositories of role play designs and improving accessibility of repository information
On one side of the sharing equation, find-ability is important, i.e. enabling a role-based e-learning design to be discovered by others. There is continuing effort on design of repositories of teaching practice, including interface and navigation enhancements, however classification systems also need to keep up. As demonstrated via the Simulation Triad in Chapter 1, there are nuances to the keyword “simulation” that require further research and development if find-ability is to be improved.
Designing an effective online role play is a non-trivial exercise so selecting someone else’s role play is a good strategy for getting started in the field. However in order for this to happen, educators need avenues for sharing what they develop. Sharing can occur at several different levels:
• effective practice
• assessment techniques
• evaluation templates and strategies
• the role play design
• the separate role play components such as the scenario and role descriptions
• cross-institutional participant collaboration in existing role plays.
Effective sharing is made more complicated by complex issues involving institutional policy and practices about intellectual property. How to resolve intellectual property when adaptation is layered on top of other adaptations? There are also practical issues, for example different curricula at different institutions: how does a design meet a particular need and how flexible is the design? How does an educator determine the effectiveness of someone else’s implementation? The field needs to develop richer ethnographic evaluation techniques in order to better communicate the practicalities of reusing role-based e-learning designs.
We also know from experience that establishing repositories of information is a necessary but not sufficient step for encouraging sharing of teaching practice. Project EnROLE recommended that a community of practice around sharing role-based e-learning is also important.
Summary
Choices about evaluation strategies and processes depend on what is needed to be known and understood. In many instances – as long as the learning assessment process demonstrates appropriate acquisition of knowledge and understanding – no further evaluation of the process may be necessary.
Participant satisfaction and engagement measures may complement learning assessment outcomes and support (or challenge) an educator’s perceptions of success of their use of online role play. These measures may also generate ideas for improving the design and/or implementation process. Consideration of participant learning outcomes – as recorded in assessment results – can also provide insights into an online role play’s effectiveness as a design for learning.
More extensive evaluation and further research will contribute evidence for improving, growing and supporting this newly emerging field of role-based e-learning. When this occurs as an element of implementation, evaluation has become an integral part of a continuous improvement approach to learning and the entire process is modelling its principles in action.