UNIVERSITY OF ILLINOIS URBANA-CHAMPAIGN

Simulation Training Evaluation

A growing movement has developed to provide experiential learning to professionals in child welfare and other human services to provide a more life-like learning experience. CFRC’s evaluation research has played an important role in the development of these methods in child welfare in Illinois.

[The image above is of an actor portraying a parent at a simulation training.]

Work in human services presents both mental and emotional challenges for professionals in the field. Classroom training alone is insufficient to prepare professionals to work with clients under stress who may have suffered significant trauma. A growing movement has developed to provide experiential learning to professionals in child welfare and other human services to provide a more life-like learning experience and enhance what they gain from training. In February 2016, the Illinois Department of Children and Family Services (DCFS) collaborated with a team from the University of Illinois at Springfield (UIS) to implement the Child Protection Training Academy that provided simulation training in which child welfare professionals work with a mock family in a mock residence and mock courtroom, to simulate child protection investigations and interventions. Every new child protection investigator in Illinois participates in a week-long simulation training program enacting the investigation and court involvement of a mock family suspected of child maltreatment. CPTA has also used a method called problem-based learning to provide online experiential learning as well. The UIS team is now expanding experiential learning to a wide array of different child protection and human service professionals.

The Children and Family Research Center has been evaluating the CPTA program since 2016 using an array of quantitative and qualitative methods. The simulation training evaluation team has produced an array of reports, research briefs, presentations and articles about the CPTA experiential learning program. CFRC’s evaluation research has played an important role in the development of experiential methods in child welfare in Illinois. As of this writing (August 2020), CFRC is also preparing to study experiential learning in other programs and sites.

Video: CFRC's Dr. Ted Cross Talks With Dr. Betsy Goulet About the Child Protection Training Academy's Simulation Training

Recorded on October 6th, 2020 via Zoom
Jan 2022 / Journal Publication / Simulation Training Evaluation    
Development and Application of a Self-Report Measure for Measuring Change During Simulation Training in Child Protection
Yu-Ling Chiu, Theodore Cross, Amy Wheeler, Susan Evans & Betsy Goulet

A national movement has been developed in the last decade to use simulation training to train child welfare workers. This article reports on a practical measurement method, Daily Experience of Simulation Training (DEST), that the authors have used since 2018 to evaluate every cohort in a week-long statewide simulation training program for new child protection investigators. The DEST measures daily changes in trainees’ self-report confidence, solicits feedback on the training team, and offers trainees the opportunity to reflect on their experience. Trainees report substantially increased confidence in 13 child protection skills, provided positive feedback to the training team, and offered a number of suggestions for improvement. We discuss methods for implementing a measurement system like the DEST, even for programs with limited resources. DEST results demonstrated considerable consistency across 27 training cohorts. Copies of the article are available from the first author at chiu22@illinois.edu.

Chiu, Y. L., Cross, T. P., Wheeler, A. B., Evans, S. M., & Goulet, B. P.(2021). Development and Application of a Self-Report Measure for Measuring Change During Simulation Training in Child Protection. Journal of Public Child Welfare. Advance online publication. https://doi.org/10.1080/15548732.2021.2016546

Dec 2021 / Journal Publication / Simulation Training Evaluation    
Evaluation of a Simulation Training Program for New Child Protection Investigators: A Survey of Investigators in the Field
Ted Cross, Yu-Ling Chiu, Kirsten Havig, Laura Lee, and Steve Tran

A new movement has developed to provide simulation training to child protection professionals to prepare them to work with families around child safety and well-being. This article reports on a survey of child protection investigators in Illinois that was conducted as part of a program evaluation of a prominent simulation training program, the Child Protection Training Academy. Simulation-trained investigators continued to value their simulation training months to years later, rated their certification training more highly than investigators without simulation training, and reported less difficulty developing the skills of evidence-based documentation and testifying in court.

Cross, T. P., Chiu, Y. L., Havig, K., Lee, L., & Tran, S. P. (2021). Evaluation of a simulation training program for new child protection investigators: A survey of investigators in the field. Children and Youth Services Review, 131. Advance online publication. https://doi.org/10.1016/j.childyouth.2021.106295

Sep 2021 / Report / Simulation Training Evaluation    
Ted Cross, Yu-Ling Chiu, Shufen Wang, Laura Lee, Steve Tran, and Kirsten Havig

In FY2021, the Children and Family Research Center’s (CFRC) evaluation team again used multiple sub-studies to examine the implementation and outcomes of simulation training for new child protection investigators in the Illinois Department of Children and Family Service. This is an important time historically to study simulation training because of the effect of COVID-19 on trainees, their work and the training itself. Chapter 1 summarizes CTPA’s implementation in 2021: adapting to the COVID-19 pandemic through virtual methods, training supervisors in problem-based learning, and re-formatting investigator training. Chapter 2 presents results from the Daily Experience of Simulation Training (DEST) measure. The measure was designed to examine change in trainees’ confidence over the course of simulation training. This is an important time to assess DEST results, because of changes in simulation training during FY2021, as discussed in the Introduction. Chapter 3 offers updated results from a post-training satisfaction survey. The chapter reports trainees’ satisfaction ratings for simulation training over this time period. It also provides qualitative results from the analysis of open-ended items in the post-training satisfaction survey. Chapter 4 examines whether simulation training is related to employee turnover. Using two different analytic methods, it asks whether investigators trained using simulation training have stayed in their jobs longer than investigators who were not provided simulation training. Chapter 5 examines the relationship of simulation training to child safety. We compared sim-trained and non-sim-trained investigators on the likelihood that children in their investigations were involved in re-reports to DCFS. The last chapter provides the conclusion of this year’s evaluation and recommendations for improving the program.


Sep 2020 / Report / Simulation Training Evaluation    
Yu-Ling Chiu, Laura Lee, and Theodore P. Cross

The FY2020 evaluation report of the Child Protection Training Academy presents an array of data on implementation and impacts. Chapter 1 focuses on the implementation of a second simulation laboratory for new investigators that opened in Chicago in April 2019. The results suggest that the Chicago laboratory is a modest re-invention of the Springfield laboratory, using Rogers’ terminology on diffusion of innovations. The combined work of DCFS, UIUC and UIS aimed at producing programs that were comparable clearly had an effect, and the extra work and travel of UIS trainers to help make this happen are noteworthy. Chapter 2 presents results from the Daily Experience of Simulation Training (DEST) measure. As in previous years, the DEST in FY2020 shows that the confidence that trainees report increased substantially from the beginning to the end of the simulation training week. Chapter 3 offers new quantitative results from a post-training satisfaction survey. Results corroborate the satisfaction that trainees reported on the DEST. Across eight satisfaction items, trainees gave simulation training positive to very positive ratings. Chapter 4 presents a content analysis of responses on the post-training satisfaction survey in which trainees describe their simulation training experience and offer suggestions for program improvement. The survey respondents reported that the training provided realistic simulation and increased their knowledge about what they will face in the field. They described positive emotional effects on increasing confidence and decreasing self-doubt. Chapter 5 discusses the implications for understanding and developing the simulation training program and recommendations for program improvement.


Sep 2020 / Journal Publication / Simulation Training Evaluation    
How a Training Team Delivers Simulation Training of Child Protection InvestigatorsWEB
Yu-Ling Chiu and Theodore P. Cross

This article presents results of a qualitative study regarding how a training team delivers simulation training for child protection investigators. Since 2016, a team from the University of Illinois at Springfield (UIS) has collaborated with the Illinois Department of Children and Family Services (DCFS) to implement the Child Protection Training Academy (CPTA) that provides full-scale simulations has been implemented for training all new child protection investigators. Using key informant interviews and focus groups, we explored how the training team, including the simulation trainer, the standardized patients playing the role of the family in a mock family house, and the professionals playing roles in a mock courtroom, collaborate to shape the simulation training. The qualitative data point to the central role of the trainer’s blend of skills, the dedication of the standardized patients staying in character, the interest of role-playing professionals in correcting misconceptions about court, and the teamwork involved in implementing simulation training.

Chiu, Y. & Cross, T. P. (2020). How a training team delivers simulation training of child protection investigators. Children and Youth Services Review, 118. In press. https://doi.org/10.1016/j.childyouth.2020.105390

Jul 2020 / Journal Publication / Simulation Training Evaluation    
Moving from Procedure to Practice: A Statewide Child Protection Simulation Training ModelWEB
Betsy Goulet , Ted Cross , Yu-Ling Chiu, and Susan Evans

In FY 2015 the Illinois Department of Children and Family Services partnered with the University of Illinois Springfield to develop the Child Protection Training Academy in order to redesign the six-week classroom training for new investigators and create an experiential component. This paper chronicles the goals of the partnership and the planning and implementation of the Academy.

Betsy P. Goulet , Theodore P. Cross , Yu-Ling Chiu & Susan Evans (2020): Moving from procedure to practice: a statewide child protection simulation training model. Journal of Public Child Welfare, 15(5), 597-616. DOI: 10.1080/15548732.2020.1777247

Sep 2019 / Report / Simulation Training Evaluation    
Yu-Ling Chiu and Theodore P. Cross

The FY2019 evaluation report of CPTA's simulation training included the following sections: 1) description of CPTA's updated training model; 2) daily experience of simulation training (DEST) that measured trainees’ daily changes in confidence on thirteen child protection work skills over the course of the simulation training week; 3) post-training satisfaction survey regarding trainees’ experience of the certificate training ; 4) simulation training and investigator turnover using DCFS employment data. The findings show that the positive results over three years of the program evaluation support the value of CPTA and suggest the potential of its current expansion. It is encouraging that investigators hired since February 2016 are staying on the job longer than investigators hired prior to that date. Data can be used both to advocate for the value of CPTA and to inform efforts at program improvement.


Aug 2019 / Presentation / Simulation Training Evaluation    
Yu-Ling Chiu, Theodore P. Cross, Betsy P. Goulet, Susan Oppegard Evans, and Monico Whittington-Eskridge

The CFRC evaluation team collaborated with the CPTA at the University of Illinois Springfield and the DCFS Office of Learning & Professional Development on a presentation at the 2019 National Child Welfare Evaluation Summit on August 20th. The presentation included discussion of the CPTA simulation training model, the main evaluation findings between 2017 and 2019, and the simulation training expansion at the Chicago site.


Apr 2019 / Presentation / Simulation Training Evaluation    
Betsy Goulet, Theodore Cross, Yu-Ling Chiu

Presentation at the One Loud Voice Conference in Biloxi Mississippi, April 16, 2019


Jan 2019 / Presentation / Simulation Training Evaluation    
Yu-Ling Chiu, Theodore Cross, Betsy Goulet, Susan Oppegard Evans, Monico Whittington-Eskridge, and Amy Wheeler

The investigator survey results in the second year evaluation of the Child Protection Training Academy’s Simulation Program were presented on January 20th, 2019 at the Society for Social Work and Research (SSWR) Annual Conference in San Francisco.


Oct 2018 / Report / Simulation Training Evaluation    
Ted Cross and Yu-Ling Chiu

The current report presents results from program evaluation activities conducted in FY2018. We used interviews and focus groups to explore the processes through which CPTA has an impact. In addition, we assessed the impact of CPTA on the experience of working DCFS investigators. We surveyed DCFS investigators and compared those investigators who have received simulation training with a group of investigators who had not received simulation training.


Feb 2018 / Research Brief / Simulation Training Evaluation  
Yu-Ling Chiu and Ted Cross

Since 2016, the Child Protection Training Academy (CPTA) at the University of Illinois at Springfield (UIS) has collaborated with the Illinois Department of Children and Family Services (DCFS) to implement the Child Protection Training Academy, which adds an innovative experiential component to the training of new DCFS investigators. This brief reports results of a preliminary program evaluation that the Children and Family Research Center has conducted of the program. We describe what simulation training is, why it could be valuable, what simulation training has been implemented, and what the preliminary data on its implementation and impact suggests about its value for enhancing worker preparation to serve children and families.


Jan 2018 / Report / Safety and Risk, Simulation Training Evaluation    
Ted Cross, Gail Tittle, and Yu-Ling Chiu

Investigating child abuse and neglect is a difficult job and investigators need all the preparation they can get. Given the demands of working with families in child protections, transferring theory to practice is particularly essential. Ideally, the initial training that new child protection workers receive should give them opportunities to practice the skills they need such as engaging families and assessment and critical thinking skills for protecting child safety. The Child Protection Training Academy at the University of Illinois at Springfield (UIS) has collaborated with the Illinois Department of Children and Family Services (DCFS) to add an innovative experiential component to the training of new DCFS investigators. All new investigators come to Child Protection Training Academy at UIS for a week at the end of their initial training to participate in simulations of real life situations that every DCFS investigator encounters. This program evaluation is consistent with the formative state of knowledge in the field and the fact that the Child Protection Training Academy is in a comparatively early stage of development. At this early stage, the program evaluation has focused on gathering data to describe the program’s objectives, methods and training theory, and examining trainees’ and other stakeholders perception of the impact of the simulation training. The goal is to inform program development and improvement, provide evidence of the program’s immediate impact on trainees, and help prepare for more rigorous program evaluation of the CPTA’s simulation training in the future.


Oct 2017 / Presentation / Simulation Training Evaluation    
Ted Cross, Betsy Goulet, Susan Evans, Gail Tittle, and Yu-Ling Chiu

Given the demands of investigating child abuse and neglect, transferring knowledge gained in training into practice to bolster child protection investigators’ skills and confidence is essential. Yet studies of transfer of learning across different domains of employment have shown that only 10 to 15% of training content is transferred to the workplace. Simulation Training Laboratories at the University of Illinois at Springfield is helping to change that with an experiential training program it provides to all new child protection investigators hired by the Illinois Department of Children and Family Services (DCFS). New investigators are trained at a Residential Simulation Laboratory in a mock house designed to simulate a family environment and a Courtroom Simulation Laboratory designed to resemble family court. This Children and Family Research Center is conducting the program evaluation of simulation training. This presentation at an international conference gives a brief overview of the program and presents initial program evaluation results.


Jun 2017 / Presentation / Sexual Abuse and Assault, Simulation Training Evaluation    
Cross, T., Goulet, B., Evans, S. & Tittle, G.

The Child Protection Training Academy at the University of Illinois at Springfield (UIS) is collaborating with the Illinois Department of Children and Family Services (DCFS) to add an innovative experiential component to the training of new DCFS investigators. The program supplements the classroom-based Foundations Training that every new investigator receives with experiential training days in a Residential Simulation Laboratory and a Courtroom Simulation Laboratory. The Children and Family Research at University of Illinois at Urbana-Champaign is conducting the program evaluation for simulation training. This presentation describes the program, discusses its first year of development, and presents preliminary program evaluation results.