Archives:

Select:

Evaluating afterschool: The evaluation basics, part I

READ FULL STORY

Evaluating afterschool: The evaluation basics, part I

The Afterschool Alliance is pleased to present the eighth installment of our "Evaluating afterschool" blog series, which answers some of the common questions asked about program evaluation and highlights program evaluation best practices. Be sure to take a look at earlier posts of the series, including blogs on how to collect data and how to build your own evaluation advisory board.

Regino Chávez has worked for LA’s BEST since 2007, designing and implementing formative research for program improvement and internal evaluations, as well as monitoring external evaluations of the program. LA’s BEST serves approximately 25,000 children ages 5 to 12 in close to 200 Los Angeles Unified School District elementary schools. A recent evaluation of LA’s BEST by the Center for Research and Evaluation of Student Standards and Testing (CRESST) at University of California, Los Angeles, found that LA’s BEST students with high levels of attendance in the program were 5 percent less likely to drop out of school and 6 percent more likely to graduate from high school on time compared to their peers who did not participate in the program.

During the first year implementation of a financial literacy program in LA’s BEST, we carried out a formative evaluation. Using surveys and focus groups with staff and students, the evaluation sought to identify how to make its implementation more effective. Findings from the assessment led us to revamp the training and scale down the number of activities staff needed to implement to match our program structure with no corresponding loss in the learning of crucial concepts among students and staff.

Why evaluate?

Jennie Chow, a native speaker of Mandarin, was a young girl new to Tom Sawyer Elementary. Jennie spoke little English and, as a newcomer had few friends in the school. Her mom enrolled her in LA’s BEST, as she needed to be at work until 5:30.

By the second semester, no one had ever heard Jennie speak during day school. Afterschool staff would talk with Jennie and invite her to join structured activities so she could engage with other students. In one particular activity, students drew themselves as superheroes and identified how their powers helped the community. Jennie proudly held up her drawing and loudly said, “Superhero” while smiling at those around her. This was the first time that anyone had heard her speak in English.

Most of us who work in afterschool see successes like Jennie's daily – the perceived disruptive child who is now a leader in sports; or the students in nutrition classes who now ask parents to buy healthier foods. However, one needs more than anecdotal evidence to demonstrate the difference we make in students’ lives. Evaluations will provide helpful information to identify what might need to change.

Are we ready to be evaluated?

In prepping for an evaluation, the first thing to ask is “Is my program ready to be evaluated?” If this is its first year, do not contemplate an impact evaluation as your program and its elements may not yet be fully developed, clearly understood or fully implemented.

A program model consists of activities that staff members implement to meet the goal(s) along with identified resources and assumptions made about what needs to be in place to support goal achievement. The goals define how the program is intentional in what it seeks to do. For example, the goals may focus on developing 21st century learners (a child who asks questions, gathers information, analyzes it, and uses it to problem-solve along with others); enhancing performance in literacy and math; or developing responsible decision-makers. The goals, activities and assumptions one makes in goal achievement are elements of the logic model – the “what” and the “how” one will meet the goals and corresponding outcomes.

Logic models are simply a road map that describe the relationship among inputs, outputs, outcomes and impact. Inputs are the resources you will put in to get your outputs or products of the effort; outcomes are desired results while impact refers to long-term change. Below is the beginning of a logic model for a nutrition and physical activity program designed to support youth in adopting healthy lifestyles.

 Goal: Kindergarten and 1st grade youth will develop healthy eating and physical activity habits from exposure to My Plate, the 5 basic food groups, how to put together a balanced meal, and physical activity

Inputs –Resources

Activities

Outputs - Products

Outcomes

Desired Results

Impact – Long term change

My Plate

 

Happy Healthy Me curriculum

 

Books tailored to each food group

 

BEST Fit Fitness Activity Packet

6-week lessons

 

Each lesson has activities to meet a specific objective such as developing familiarity with My Plate

 

Each lesson will have a cooking activity, a reading activity, and an indoor/outdoor physical activity

4,000 Kindergarten and 1st grade students in the LA’s BEST program are exposed to the My Plate program 

 

4,000 youth taste healthy dishes they make, learn at least 3 breathing exercises, engage in at least 3 indoor/outdoor games, and read 1 book per activity.

Students demonstrate knowledge of what constitutes a healthy meal

 

Students gain knowledge of how to prepare a healthy meal from items at home.

Students adopt healthier lifestyles

 

Students become agents for healthy eating and being physically active

Training of Staff

 

4 Hour training for 200 LA’s BEST staff to support them in implementing the My Plate program

A 4-hour training implemented

 

200 staff training in the program

Staff gain knowledge of food groups

 

Staff gain knowledge of how to prepare healthy snacks tailored to youth

 

Staff demonstrate knowledge of how to prepare heathy meals familiar to students

Staff gain confidence to eat healthier

 

Staff adopt healthier lifestyles

 

Logic models can also drafted for organization-wide initiatives. For an example of a logic model more applicable to an organization-wide SEL effort in LA’s BEST, see Figure 2.

Stay tuned for Part II and Part III of this blog. Part II will focus on the two basic types of evaluations programs can perform, how to determine which type is the best fit for a program, and how to determine which research questions to prioritize. Part III delves into the decision-making process of selecting who can conduct the program evaluation and how to find the resources to conduct an evaluation.

Data sharing with afterschool leads to student success

With more than 10 million children participating in afterschool programming, the hours between 3 and 6 p.m. hold crucial potential for academic, social, and emotional development that extends well beyond the school day. So how can we ensure that more students have access to high quality afterschool...

BY: Rina Moss      11/13/18

Mapping afterschool in New Mexico

By May Sagbakken, executive director of the New Mexico Out-of-School Time Network All children and youth deserve access to high-quality afterschool and summer learning programs that prepare them for school, college, careers, and life. While many students across the country are receiving valuable...

BY: Guest Blogger      06/05/18

New resources! National and state-level fact sheets on afterschool

As spring finally arrives in Washington, D.C., so does a suite of new resources highlighting key facts and stats on afterschool!   The first resource is a new fact sheet on afterschool, which summarizes positive outcomes for students who participate in programs, as well as the supports...

BY: Nikki Yamashiro      04/13/18

Experts weigh in on the value of afterschool

Last month, we saw afterschool programs across the country open their doors to host Lights On Afterschool events, providing a firsthand look at the broad array of fun, enriching, and engaging activities students take part in at the program and the vital role programs play in their community. A...

BY: Nikki Yamashiro      11/27/17

Looking for evidence on the impact of afterschool?

Look no further! Adding to the research conversation discussing the value of afterschool and summer learning programs, the Afterschool Alliance has released two new fact sheets that provide a sampling of evaluation findings demonstrating the positive impact programs have on students, with subjects...

BY: Nikki Yamashiro      11/09/17

Evaluating afterschool: What my toddler taught me about evaluation

By Allison Riley, PhD, MSW, Senior Vice President, Programming and Evaluation at Girls on the Run International. Girls on the Run is a physical activity-based positive youth development program that inspires girls to be joyful, healthy, and confident using a fun, experience-based curriculum that...

BY: Guest Blogger      09/08/17

New poll finds 9 in 10 parents support schools providing afterschool programs

Last night, the Institute for Educational Leadership (IEL) released the 2017 Phi Delta Kappa International (PDK) Poll of Public's Attitudes Towards Public Schools. The overall takeaway from this report, which is PDK’s 49th annual report on Americans’ views toward public...

BY: Nikki Yamashiro      08/29/17

Guest blog: Q&A with an afterschool researcher, part II

Welcome to part II of our Q&A with Neil Naftzger, American Institutes for Research (AIR), about his evaluation work related to 21st CCLC programs specifically and the afterschool field broadly. Below are answers to oneof the questions we asked, with our emphasis added in bold, which...

BY: Guest Blogger      08/18/17

Guest blog: Q&A with an afterschool researcher

In May, the proposed FY2018 budget eliminated funding for the 21st Century Community Learning Centers (21st CCLC), the only federal funding stream dedicated to before-school, afterschool, and summer learning programs. In the budget, a justification given for the elimination of funding was that...

BY: Guest Blogger      07/28/17

Evaluating afterschool: Building an evaluation advisory board

By Jason Spector, Senior Research & Evaluation Manager at After-School All-Stars The Afterschool Alliance is pleased to present the sixth installment of our "Evaluating afterschool" blog series, which answers some of the common questions asked about program evaluation and...

BY: Guest Blogger      07/05/17

5 steps to effectively implement SEL initiatives in your community

Awareness of the importance of social and emotional learning is beginning to be reflected in policy measures. Practitioners and policymakers alike are continually recognizing that a whole-child approach is necessary for children to be well-positioned to succeed in their academic, personal, and...

BY: Erin Hegarty      02/08/19

Afterschool, community partners featured in Aspen Institute’s "Nation At Hope" report

Today’s youth must navigate a complex, economically competitive, and globally connected world. In efforts to help young people thrive, educators, parents, and leaders have historically focused on academic improvement as the key target for future success. But that’s not the whole...

BY: Dan Gilbert      01/15/19

New issue brief: A big-picture approach to wellness

From empowering students to take charge of their health to training staff to model healthy behaviors, afterschool and summer learning programs across the country are offering safe and supportive environments that promote young people’s healthy eating and physical activity, as well as build...

BY: Nikki Yamashiro      09/07/18

An Ideal Opportunity: The Role of Afterschool in Social and Emotional Learning

High school students in The Possibility Project, New York, N.Y., tackle issues close to their heart, taking on leadership roles to enact change in their communities through performing arts and community action. At Girls on the Run councils located in all 50 states, plus D.C., girls run with their...

BY: Nikki Yamashiro      05/04/18

New research on easy, affordable SEL practices

As the prominence of social and emotional learning (SEL) to support students’ development in school and beyond continues to grow in education circles, challenges implementing SEL programming have also arisen. The latest issue brief in a series supported by The Wallace Foundation, Kernels of...

BY: Nikki Yamashiro      02/20/18

Research shows the benefits of early childhood programs

Over the years, research surrounding participation in early childhood programs—such as preschool, home visiting programs, and parent education programs—has continuously shown improved outcomes for children and families. A research report...

BY: Leah Silverberg      01/25/18

"Building Workforce Skills in Afterschool" highlights promising practices for all ages

The next generation of the American workforce is growing up right now and afterschool programs are vital partners in helping young people discover new passions and work towards their dreams. As in so many other subjects, the variety and versatility of afterschool programming offers opportunities...

BY: Nikki Yamashiro      11/14/17

New report from the Wallace Foundation: Strategies to scale up

The question of how to scale up—taking a successful program, project, or policy and growing it to expand its reach and therefore its impact—has been an important one when thinking about systems change. It is a key component in efforts to make sustainable, positive social gains; a...

BY: Nikki Yamashiro      10/06/17

New issue brief showcasing afterschool promoting civic engagement

Earlier this week on my way into work, a father and his daughter, who couldn’t have been older than an elementary schooler, were at the bottom of the metro escalator greeting metro riders with a huge sign that read “VOTE!” That sight brought a huge smile to my face; it was...

BY: Nikki Yamashiro      11/01/18

New issue brief: A big-picture approach to wellness

From empowering students to take charge of their health to training staff to model healthy behaviors, afterschool and summer learning programs across the country are offering safe and supportive environments that promote young people’s healthy eating and physical activity, as well as build...

BY: Nikki Yamashiro      09/07/18

Afterschool after 30 years: Q&A with Dr. Milbrey McLaughlin, part II

By Sam Piha, founder and principal of Temescal Associates, a consulting group dedicated to building the capacity of leaders and organizations in education and youth development who are serious about improving the lives of young people.  Milbrey McLaughlin has been a leading...

BY: Guest Blogger      06/26/18

"Building Workforce Skills in Afterschool" highlights promising practices for all ages

The next generation of the American workforce is growing up right now and afterschool programs are vital partners in helping young people discover new passions and work towards their dreams. As in so many other subjects, the variety and versatility of afterschool programming offers opportunities...

BY: Nikki Yamashiro      11/14/17

How does afterschool contribute to military readiness?

In 2016, the Council for a Strong America released America Unprepared, showing data that more than 70 percent of young adults in the United States would not qualify for military service due to obesity and other health issues, poor academic performance, drug abuse, or involvement in crime. As a...

BY: Leah Silverberg      08/14/17

5 ways afterschool prepares kids to succeed in the workplace

Cultivating tomorrow’s workforce remains a central part of the discussion about America’s economic future. As today’s children begin to develop the skills they will need in the workplace, experts in the education and afterschool fields are turning their attention to the ways that...

BY: Charlotte Steinecke      05/04/17

A researcher's view: 21st CCLC programs "play an invaluable role"

Students and staff at a 21st Century Community Learning Centers program in Oakland, Calif. Yesterday, the administration’s FY2020 budget proposed elimination of funding for the 21st Century Community Learning Centers (21st CCLC), the only federal funding stream dedicated to...

BY: Guest Blogger      03/12/19

New polling shows America is for afterschool

Just in time for Lights On Afterschool, the only national rally celebrating afterschool, a new national poll is out that shows strong public support for afterschool. Nearly 9 in 10 adults (89 percent) agree that afterschool programs are important to their community, with bipartisan and nationwide...

BY: Nikki Yamashiro      10/22/18

Looking for evidence on the impact of afterschool?

Look no further! Adding to the research conversation discussing the value of afterschool and summer learning programs, the Afterschool Alliance has released two new fact sheets that provide a sampling of evaluation findings demonstrating the positive impact programs have on students, with subjects...

BY: Nikki Yamashiro      11/09/17

Guest blog: Q&A with an afterschool researcher, part II

Welcome to part II of our Q&A with Neil Naftzger, American Institutes for Research (AIR), about his evaluation work related to 21st CCLC programs specifically and the afterschool field broadly. Below are answers to oneof the questions we asked, with our emphasis added in bold, which...

BY: Guest Blogger      08/18/17

AFT poll shows opposition to federal funding cuts to education

The clear message coming out of a recent national poll on attitudes toward federal education spending is that voters are overwhelmingly opposed to the federal government cutting funds for public education. In the poll, conducted by Hart Research Associates for the American Federation of...

BY: Nikki Yamashiro      08/02/17

Guest blog: Q&A with an afterschool researcher

In May, the proposed FY2018 budget eliminated funding for the 21st Century Community Learning Centers (21st CCLC), the only federal funding stream dedicated to before-school, afterschool, and summer learning programs. In the budget, a justification given for the elimination of funding was that...

BY: Guest Blogger      07/28/17

New polling shows America is for afterschool

Just in time for Lights On Afterschool, the only national rally celebrating afterschool, a new national poll is out that shows strong public support for afterschool. Nearly 9 in 10 adults (89 percent) agree that afterschool programs are important to their community, with bipartisan and nationwide...

BY: Nikki Yamashiro      10/22/18

Why invest public funds in afterschool?

A new infographic released by RAND draws our attention to the key benefits of public funding for afterschool and summer learning programs. Based on their 2017 research report The Value of Out-of-School Programs, which we covered here last year, the infographic reflects the evident need for...

BY: Rina Moss      10/05/18

Mapping afterschool in New Mexico

By May Sagbakken, executive director of the New Mexico Out-of-School Time Network All children and youth deserve access to high-quality afterschool and summer learning programs that prepare them for school, college, careers, and life. While many students across the country are receiving valuable...

BY: Guest Blogger      06/05/18

New resources! National and state-level fact sheets on afterschool

As spring finally arrives in Washington, D.C., so does a suite of new resources highlighting key facts and stats on afterschool!   The first resource is a new fact sheet on afterschool, which summarizes positive outcomes for students who participate in programs, as well as the supports...

BY: Nikki Yamashiro      04/13/18

AFT poll shows opposition to federal funding cuts to education

The clear message coming out of a recent national poll on attitudes toward federal education spending is that voters are overwhelmingly opposed to the federal government cutting funds for public education. In the poll, conducted by Hart Research Associates for the American Federation of...

BY: Nikki Yamashiro      08/02/17

Guest blog: Q&A with an afterschool researcher

In May, the proposed FY2018 budget eliminated funding for the 21st Century Community Learning Centers (21st CCLC), the only federal funding stream dedicated to before-school, afterschool, and summer learning programs. In the budget, a justification given for the elimination of funding was that...

BY: Guest Blogger      07/28/17

Afterschool, community partners featured in Aspen Institute’s "Nation At Hope" report

Today’s youth must navigate a complex, economically competitive, and globally connected world. In efforts to help young people thrive, educators, parents, and leaders have historically focused on academic improvement as the key target for future success. But that’s not the whole...

BY: Dan Gilbert      01/15/19

Park and recreation afterschool programs make a real difference in communities

A new survey study released by the National Recreation and Park Association (NRPA), 2018 Out-of-School Time Report, highlights the multitude of ways that local park and recreation before school, afterschool, and summer programs are positively impacting the lives of children and their families. With...

BY: Rina Moss      01/11/19

Creating high-quality arts programs in national youth-serving organizations

While research has shown that participation in the arts promotes positive youth outcomes, providing quality arts programming can seem like an unobtainable goal to many programs, especially those that mix various art disciplines into daily programming but do not have an arts focus. But a new report...

BY: Leah Silverberg      10/27/17

New report from the Wallace Foundation: Strategies to scale up

The question of how to scale up—taking a successful program, project, or policy and growing it to expand its reach and therefore its impact—has been an important one when thinking about systems change. It is a key component in efforts to make sustainable, positive social gains; a...

BY: Nikki Yamashiro      10/06/17

New resources for STEM in afterschool from the Research + Practice Collaboratory

Check it out: the Research + Practice Collaboratory has some new and updated resources for the afterschool field! If you are not familiar, the Research + Practice Collaboratory works to bridge the gap between education research and STEM education implementation. The Collaboratory’s goal is to...

BY: Leah Silverberg      07/10/17

New issue brief: A big-picture approach to wellness

From empowering students to take charge of their health to training staff to model healthy behaviors, afterschool and summer learning programs across the country are offering safe and supportive environments that promote young people’s healthy eating and physical activity, as well as build...

BY: Nikki Yamashiro      09/07/18

An Ideal Opportunity: The Role of Afterschool in Social and Emotional Learning

High school students in The Possibility Project, New York, N.Y., tackle issues close to their heart, taking on leadership roles to enact change in their communities through performing arts and community action. At Girls on the Run councils located in all 50 states, plus D.C., girls run with their...

BY: Nikki Yamashiro      05/04/18

The prevalence of adverse childhood experiences

Close to half of children (45 percent) in the U.S. have experienced at least one adverse childhood experience (ACE)—an experience that could have negative and lasting effects on one’s health and wellbeing, such as depression, drug abuse, and poor physical health. One in 9 children...

BY: Nikki Yamashiro      03/09/18

"Building Workforce Skills in Afterschool" highlights promising practices for all ages

The next generation of the American workforce is growing up right now and afterschool programs are vital partners in helping young people discover new passions and work towards their dreams. As in so many other subjects, the variety and versatility of afterschool programming offers opportunities...

BY: Nikki Yamashiro      11/14/17