Evaluating afterschool: The evaluation basics, part I

The Afterschool Alliance is pleased to present the eighth installment of our "Evaluating afterschool" blog series, which answers some of the common questions asked about program evaluation and highlights program evaluation best practices. Be sure to take a look at earlier posts of the series, including blogs on how to collect data and how to build your own evaluation advisory board.

Regino Chávez has worked for LA’s BEST since 2007, designing and implementing formative research for program improvement and internal evaluations, as well as monitoring external evaluations of the program. LA’s BEST serves approximately 25,000 children ages 5 to 12 in close to 200 Los Angeles Unified School District elementary schools. A recent evaluation of LA’s BEST by the Center for Research and Evaluation of Student Standards and Testing (CRESST) at University of California, Los Angeles, found that LA’s BEST students with high levels of attendance in the program were 5 percent less likely to drop out of school and 6 percent more likely to graduate from high school on time compared to their peers who did not participate in the program.

During the first year implementation of a financial literacy program in LA’s BEST, we carried out a formative evaluation. Using surveys and focus groups with staff and students, the evaluation sought to identify how to make its implementation more effective. Findings from the assessment led us to revamp the training and scale down the number of activities staff needed to implement to match our program structure with no corresponding loss in the learning of crucial concepts among students and staff.

Why evaluate?

Jennie Chow, a native speaker of Mandarin, was a young girl new to Tom Sawyer Elementary. Jennie spoke little English and, as a newcomer had few friends in the school. Her mom enrolled her in LA’s BEST, as she needed to be at work until 5:30.

By the second semester, no one had ever heard Jennie speak during day school. Afterschool staff would talk with Jennie and invite her to join structured activities so she could engage with other students. In one particular activity, students drew themselves as superheroes and identified how their powers helped the community. Jennie proudly held up her drawing and loudly said, “Superhero” while smiling at those around her. This was the first time that anyone had heard her speak in English.

Most of us who work in afterschool see successes like Jennie's daily – the perceived disruptive child who is now a leader in sports; or the students in nutrition classes who now ask parents to buy healthier foods. However, one needs more than anecdotal evidence to demonstrate the difference we make in students’ lives. Evaluations will provide helpful information to identify what might need to change.

Are we ready to be evaluated?

In prepping for an evaluation, the first thing to ask is “Is my program ready to be evaluated?” If this is its first year, do not contemplate an impact evaluation as your program and its elements may not yet be fully developed, clearly understood or fully implemented.

A program model consists of activities that staff members implement to meet the goal(s) along with identified resources and assumptions made about what needs to be in place to support goal achievement. The goals define how the program is intentional in what it seeks to do. For example, the goals may focus on developing 21st century learners (a child who asks questions, gathers information, analyzes it, and uses it to problem-solve along with others); enhancing performance in literacy and math; or developing responsible decision-makers. The goals, activities and assumptions one makes in goal achievement are elements of the logic model – the “what” and the “how” one will meet the goals and corresponding outcomes.

Logic models are simply a road map that describe the relationship among inputs, outputs, outcomes and impact. Inputs are the resources you will put in to get your outputs or products of the effort; outcomes are desired results while impact refers to long-term change. Below is the beginning of a logic model for a nutrition and physical activity program designed to support youth in adopting healthy lifestyles.

Logic models can also drafted for organization-wide initiatives. For an example of a logic model more applicable to an organization-wide SEL effort in LA’s BEST, see Figure 2.

Stay tuned for Part II and Part III of this blog. Part II will focus on the two basic types of evaluations programs can perform, how to determine which type is the best fit for a program, and how to determine which research questions to prioritize. Part III delves into the decision-making process of selecting who can conduct the program evaluation and how to find the resources to conduct an evaluation.