Evaluating afterschool: Find your best data-collection strategy

By Y-USA Achievement Gap Programs Evaluation Team.

The Afterschool Alliance is pleased to present the fourth installment of our "Evaluating afterschool" blog series, which turns to program providers in the field to answer some of the common questions asked about program evaluation. Be sure to take a look at the first, second, and third posts of the series.

Photo courtesy of Lori Humphreys, VP of Child Care, YMCA Of East Tennessee.

At YMCA of the USA, our Achievement Gap Programs evaluation team provides a comprehensive evaluation strategy, training, tools, and support to hundreds of local Ys doing the important work of delivering Achievement Gap Signature Programs to thousands of children. The Achievement Gap Afterschool Program has expanded to over 130 sites since it began in the 2012-2013 school year and is currently serving over 7,000 children across the nation.

Organization leadership, funders, and community partners are often eager to see the data that comes out of program evaluation, and it is not uncommon for organizations to need additional guidance and resources to start the data collection process. We'd like to share what we think are data collection essentials for this important and possibly overwhelming part of the evaluation process.

The first step is for the program’s primary stakeholders to define program goals and benchmarks. Identifying the questions that should be answered about the program helps to focus on what matters most for the program.

Before you start

• Be organized: Develop a plan from start to finish before diving into the data collection process. How will data be collected? What tools will be used to collect data? Who will do the data collection, entry, and analyses? How will the data be reported? Include a general timeline for each activity in the plan.
• Be realistic: As a data collection plan is developed, be realistic about the resources you can dedicate to the project and plan accordingly.  When you can, simplify.
• Be clear: All data collection processes should include a clear explanation for why the data is being collected and how the data will be protected and reported. Clarity of purpose ensures that staff, parents, and participants are fully informed on the program’s data collection practices.
• Be concise: When developing tools to collect data, stay focused on only gathering essential information that relates to goals or the questions stakeholders have agreed upon. Collecting information that you don’t plan to use takes up precious time when creating data collection tools, when users fill out the tools, and when the data is analyzed.

How should we collect data?

Surveys are a popular way to gather program input from parents, staff, and even participants. They are a great tool for gauging satisfaction and outcomes, but beware they can become incredibly time consuming! From developing the survey, administering it to the recipients, following up, completing data entry, and running analyses (especially if asking open-ended questions)—there can be a lot to do!

Using a free, on-line survey tool can increase the response rate, decrease data entry, and provide analyses, but be sure to consider if the people filling out your survey have easy access to an internet-connected device.

Observation protocols are a great way to understand if the program expectations are being met. When compared to a survey, observation protocols can be easier tools to manage since the process is usually internal. Using the questions and goals that stakeholders have identified, develop measurable items to put on an observation tool. For example, if one of the goals is ensuring that students are engaged in the program, an item may look like this:

It’s ideal to have multiple observers and include leadership, frontline staff, and parents. Averages can be reported across these observers for each item or even across observers for multiple items in a category (e.g. engagement, satisfaction, quality, etc.).

Focus groups and interviews provide an opportunity to gauge program outcomes and gather ideas for program improvement. Use a prepared list of open-ended questions that align with program goals. It is ideal for focus groups to have no more than six to eight people to maximize the potential for everyone to share their thoughts. Have multiple focus groups to identify common themes that may arise. If you’re running a focus group with children, be sure to have a facilitator with which the group is familiar and comfortable. Have someone ready to take notes inconspicuously.

Data that already exists can be an easy way to minimize the data collection effort. When developing your evaluation questions, do some research to see if some of your questions already have answers. Examples of common data that you may want to access as part of an afterschool program evaluation are school attendance, school discipline, and assessment/achievement results. Be sure the data you are gathering will inform the already defined program goals. Note that if you're not a staff member of the organization that houses the data, you may need to sign a data agreement before gaining access.

Plan for continuous improvement

Be sure to create your data collection plan with ample time prior to your program launching. As you execute your plan, analyze your data, and report your findings, take note of lessons learned and speak with your team to improve data collection tools and processes for the following program year.