Evaluating afterschool: Tips for getting started from Dallas Afterschool

Program evaluation can be an overwhelming and intimidating undertaking for afterschool program providers. There are questions ranging from where to start to what to do with evaluation results and everything in-between that program providers need to think about. To answer some of the common questions raised by afterschool program providers about evaluations and to help make evaluations more approachable, the Afterschool Alliance has started a new blog series,"Evaluating afterschool," on program evaluation best practices. For this blog series, the Afterschool Alliance turns to program providers in the field who can offer tips and lessons learned on their evaluation journey.

The first blog of this series is written by Rachel Johns, the research and evaluation manager at Dallas Afterschool in Dallas, Texas. Dallas Afterschool promotes, expands and improves the quality of afterschool and summer programs in low income neighborhoods in our community.

This spring, Dallas Afterschool released findings from the 2014-2015 school year as part of an ongoing, engaged evaluation process. Our dynamic partnership with the Center on Research and Evaluation at Southern Methodist University has allowed us to explore questions about how to improve the quality of afterschool programs effectively and efficiently, and how the quality of an afterschool program might affect students in our context. As we enter our fourth year of this evaluation, we'd like to share some of what we have learned in the process.         

Considerations for practitioners

While an evaluation as extensive as Dallas Afterschool’s may not be practical for all organizations due to financial or human capacity restraints, there are many ways to enhance your benefit from any evaluation process.

1. Clearly define the questions you want answered and circle back to them often. These questions are the guidepost for your evaluation and can help keep you focused on the pieces of data and the analyses that matter most. Evaluation becomes less useful when it lacks direction or tries to address too many questions.
2. Plan for more time than you think you need. If you know what questions your evaluation is asking and what data needs to be collected to answer those questions, then you have a great start. Collecting your own data can make scheduling simple, but if you rely on colleagues to collect some of it, plan for an extra week buffer. Competing priorities can make data collection fall to the back burner, but good data collection is essential for a useful evaluation. Additionally, the amount of time it takes to clean that data to make it ready for analysis can be hard to estimate. When data is derived from many different sources or is collected inconsistently, you never know what you might find or need to correct.
3. Regularly monitor your data to save a headache in the end. Especially if several people are collecting and entering data, regular monitoring of the data can give you the opportunity to retrain before a lot of time is wasted on data “cleaning” and correcting work that has already been done.
4. Provide more support than you expect people will need. Some people may not need training or support, but you never know who will. You may need to document protocols for data collection, provide periodic trainings, or help staff and stakeholders to understand the process and the results.

Leveraging a university partner

Dallas Afterschool partners with a local university to access expertise in evaluation design and analysis, as well as to enhance our self-reflection with external perspectives. Though choosing a university partner and engaging with them throughout the evaluation process may be daunting or even confusing, consider the following to maximize your organization’s benefit and enjoyment of the process.

1. Know what you want. Do you simply need a report for a specific grant requirement, or are you looking for a thought partner to challenge your assumptions about your program and help you make it even better? Many evaluators jump at the chance to help a program that truly desires to improve and is willing to engage with them throughout the entire process.
2. Develop a symbiotic relationship. Find out what research the university is interested in doing that your organization might be able to help with. Are they working on anything that might benefit your field or an issue related to your population? By opening your program to engage in research or evaluations that align with your mission but extend beyond your own evaluation, you can develop a relationship with your University partner that is beneficial to both entities and potentially addresses systemic issues that your program could not affect on its own.
3. Trust their academic expertise but challenge the practical application of results. University partners can provide excellent direction on the design and methods of your evaluation, but you know your population best. If they propose an angle for the evaluation that doesn’t seem especially useful to your program or its participants, push back and work together to find an angle that does. Evaluators want their work to be used to help programs and the people they serve, so don’t be shy.

Applying results

Most programs that receive an evaluation hope to find that the program is working well and their participants are better off than they were before attending the program. We want to hear something like, “keep doing the great work you’re doing! Don’t change a thing.” Often, however, the results are a little more complicated than that.

1. What if our results are inconclusive? Don’t worry. Evaluations require a lot of moving pieces to fall in line, like the way a program is implemented and how consistently the data is collected. An inconclusive evaluation is just that; it does not mean that your program is not doing good work. Try to refine your evaluation for next year and in the meantime, discuss the process and results with your staff. What knowledge do they have that isn’t reflected in the evaluation? Can your staff identify issues with the program that weren’t covered by the evaluation? What other factors should be considered in the future? Are there trends or hints of success, regardless of statistical significance, that you can explore further?
2. What if the results indicate that we need to make a lot of changes? Great! Take the opportunity to strengthen your program for the people you serve. This is exactly the point of engaging in evaluation. Your staff may need time to process the results and to be comfortable with implementing new methods. They may need support and training in the process. You may need to focus on one or two changes at a time. Change is hard but the improvements to your program will be worth it in the end.