Guest blog: Q&A with an afterschool researcher, part II

READ FULL STORY

Guest blog: Q&A with an afterschool researcher, part II

Welcome to part II of our Q&A with Neil Naftzger, American Institutes for Research (AIR), about his evaluation work related to 21st CCLC programs specifically and the afterschool field broadly. Below are answers to oneof the questions we asked, with our emphasis added in bold, which establish that there is in fact clear evidence demonstrating that 21st CCLC work for students. To read part I, click here.

What changes would you like to see in terms of 21st CCLC data collection and evaluation?

This is a big question. First, I think we need to be clear around the purposes we’re trying to support through data collection and evaluation. Normally, we think about this work as falling within three primary categories:

  1. Data to support program staff in learning about quality practice and effective implementation
  2. Data to monitor the participation and progress of enrolled youth
  3. Data to assess the impact of the program on youth that participate regularly in the program

States have done an amazing job over the span of the past decade to develop quality improvement systems predicated on using quality data to improve practice (purpose #1). Effective afterschool quality improvement systems start with a shared definition of quality. In recent years, state 21st CCLC systems have come to rely upon formal assessment tools like the Youth Program Quality Assessment (YPQA) and the Assessment of Program Practices Tool (APT-O) to provide that definition, allowing 21st CCLC grantees to assess how well they are meeting these criteria and crafting action plans to intentionally improve the quality of programming. Use of these tools typically involves assigning a score to various program practices in order to quantify the program’s performance and establish a baseline against which to evaluate growth. A recent report completed by AIR indicates approximately 70 percent of states have adopted a quality assessment tool for use by their 21st CCLC grantees. Our sense is that these systems have been critical to enhancing the quality of 21st CCLC programs, and any efforts to modify the 21st CCLC data collection landscape should ensure program staff have the support and time necessary to participate in these important processes.

Secondly, additional work needs to be done to define key indicators for the program to support efforts to monitor the participation and progress of enrolled youth and inform efforts to make targeted refinements to programming to enhance quality and effectiveness (purpose #2). For example, our sense is that indicators could be crafted to answer the following three questions:

  1. To what extent are centers retaining youth in 21st CCLC programming, both during the span of the school year and across school years? From our statewide evaluation work, we have plenty of evidence to suggest that youth benefit more from 21st CCLC programming the more they participate. Keeping youth enrolled in programming is linked both to the underlying quality of a center’s activities and ensuring that youth have access to developmentally appropriate activities as they get older that keep them interested and engaged in program activities.
  2. To what extent are youth reporting having positive experiences in 21st CCLC programs? We consider it vital for programs to understand the subjective experiences youth have while participating in programming and use this information to enhance program offerings to ensure a “goodness of fit” between where youth are and what learning supports and opportunities the program is providing. Youth surveys are primarily used to obtain these types of data.
  3. To what extent are youth demonstrating improvement on those outcomes a center is specifically attempting to impact through the provision of intentional programming aligned with those specific outcomes? This question is predicated on the belief that all centers should not be expected to positively impact the full array of school-related outcomes we typically examine when attempting to assess program impact. Rather, federal, state, and local stakeholders and funders should be looking for improvement in those specific areas targeted explicitly by center goals and objectives and the manner in which center staff go about designing and delivering aligned activities meant to support the positive development of youth.

We also would advocate for the collection and use of these data using a quality improvement, as opposed to an accountability, framework. The focus here should be on using data to enhance program implementation, as opposed to making summative judgments about efficacy or impact.

Assessing program impact represents the final way of looking at 21st CCLC data (purpose #3). In this instance, the focus should be on using rigorous quasi-experimental designs to support making causal inferences about how participation in 21st CCLC programming may be having a positive impact on participating youth. Here, we would like to see improvement in two areas: (1) enhancing the rigor of evaluation efforts and (2) improving on the measurement of outcomes that are especially likely to be impacted through 21st CCLC participation.

In terms of rigor, if we are trying to make causal inferences on the impact of 21st CCLC, we need to make sure we are using research designs that will support these inferences. In all the statewide 21st CCLC evaluations completed by AIR, each of the analyses undertaken to assess the relationship between regular program participation and youth outcomes compared youth participating in programming for 60 days or more during the school year with a similar group of youth from the same schools who did not participate in programming. In order to ensure the participating and non-participating groups were as similar as possible, we used an approach called propensity score matching to create the non-participant comparison group. Given that youth were not randomly assigned to participate in programming or not, there is always the concern that youth who did opt to participate differed in important ways from youth who did not enroll in programming. That is, potential program effects could be driven more by existing differences between the groups at baseline than a true relationship between program participation and youth outcomes. The goal in using propensity score matching was to mitigate this selection bias when estimating potential program effects by accounting for preexisting differences between youth who attended the program regularly and those who did not. As a result, these analyses helped us to isolate the potential effect participation in 21st CCLC had on youth outcomes.

Efforts to evaluate the impact of 21st CCLC should increasingly try to employ these types of designs, particularly at the state-level where there additional evaluation resources are available to support these types of analyses. In addition, very little work has been done to understand the longitudinal effects of 21st CCLC participation on the educational and career trajectories of participating youth. More work needs to be done in this area as well.

Finally, there is a great opportunity to improve upon how we are measuring youth outcomes supported by 21st CCLC participation. When we talk with center coordinators about how youth benefit from participating in the program, quite often their responses can be classified as falling in one of the following categories:

  • Belonging/Mattering
    • Creating a sense of belonging and connectedness for participating youth through positive and supportive relationships
    • Supporting the development of positive interpersonal skills
  • Promoting Youth Agency
    • Supporting the development of positive mindsets and beliefs that participating youth can succeed through effort , including promoting confidence and a sense of self-efficacy
    • Cultivating new cognitive tools, like strategic thinking, by being afforded the opportunity to take ownership in the learning process through approaches like project-based and inquiry-based learning
  • Interest Development
    • Providing opportunities to participating youth to experience and do new things
    • Discovering new areas of interest and passion (e.g., STEM, the Arts, etc.)
  • Self-Management
    • Developing skills to manage cognition, behavior, and emotion
  • Sense of Purpose
    • Supporting the understanding of why what youth do with their lives matters

Unfortunately, many of these areas are currently going unmeasured, leading to a gaping hole in our understanding of how 21st CCLC programming is truly impacting participating youth. Our sense is that if we really want to understand how 21st CCLC may be impacting youth, we need to dedicate some additional effort to examining these types of outcomes. However, before states and grantees widely pursue efforts to measure these types of program outcomes, they should wait for the research community to provide more concrete recommendations regarding which skills, beliefs, and attitudes can be reliably measured (and under what conditions), and what protections need to be in place to ensure youth are not adversely impacted by participating in these measurement efforts.

Mapping afterschool in New Mexico

By May Sagbakken, executive director of the New Mexico Out-of-School Time Network All children and youth deserve access to high-quality afterschool and summer learning programs that prepare them for school, college, careers, and life. While many students across the country are receiving valuable...

BY: Guest Blogger      06/05/18

New resources! National and state-level fact sheets on afterschool

As spring finally arrives in Washington, D.C., so does a suite of new resources highlighting key facts and stats on afterschool!   The first resource is a new fact sheet on afterschool, which summarizes positive outcomes for students who participate in programs, as well as the supports...

BY: Nikki Yamashiro      04/13/18

Experts weigh in on the value of afterschool

Last month, we saw afterschool programs across the country open their doors to host Lights On Afterschool events, providing a firsthand look at the broad array of fun, enriching, and engaging activities students take part in at the program and the vital role programs play in their community. A...

BY: Nikki Yamashiro      11/27/17

Looking for evidence on the impact of afterschool?

Look no further! Adding to the research conversation discussing the value of afterschool and summer learning programs, the Afterschool Alliance has released two new fact sheets that provide a sampling of evaluation findings demonstrating the positive impact programs have on students, with subjects...

BY: Nikki Yamashiro      11/09/17

Evaluating afterschool: What my toddler taught me about evaluation

By Allison Riley, PhD, MSW, Senior Vice President, Programming and Evaluation at Girls on the Run International. Girls on the Run is a physical activity-based positive youth development program that inspires girls to be joyful, healthy, and confident using a fun, experience-based curriculum that...

BY: Guest Blogger      09/08/17

New poll finds 9 in 10 parents support schools providing afterschool programs

Last night, the Institute for Educational Leadership (IEL) released the 2017 Phi Delta Kappa International (PDK) Poll of Public's Attitudes Towards Public Schools. The overall takeaway from this report, which is PDK’s 49th annual report on Americans’ views toward public...

BY: Nikki Yamashiro      08/29/17

Guest blog: Q&A with an afterschool researcher, part II

Welcome to part II of our Q&A with Neil Naftzger, American Institutes for Research (AIR), about his evaluation work related to 21st CCLC programs specifically and the afterschool field broadly. Below are answers to oneof the questions we asked, with our emphasis added in bold, which...

BY: Guest Blogger      08/18/17

Guest blog: Q&A with an afterschool researcher

In May, the proposed FY2018 budget eliminated funding for the 21st Century Community Learning Centers (21st CCLC), the only federal funding stream dedicated to before-school, afterschool, and summer learning programs. In the budget, a justification given for the elimination of funding was that...

BY: Guest Blogger      07/28/17

Evaluating afterschool: Building an evaluation advisory board

By Jason Spector, Senior Research & Evaluation Manager at After-School All-Stars The Afterschool Alliance is pleased to present the sixth installment of our "Evaluating afterschool" blog series, which answers some of the common questions asked about program evaluation and...

BY: Guest Blogger      07/05/17

Evaluating afterschool: Answering questions about quality

By Jocelyn Michelsen, Senior Research Associate at Public Profit, an Oakland, California-based evaluation consultancy focused on helping high-performing organizations do their best, data-driven work with children, youth, and families. The Afterschool Alliance is pleased to present the fifth...

BY: Charlotte Steinecke      06/06/17

Afterschool after 30 years: Q&A with Dr. Milbrey McLaughlin, part II

By Sam Piha, founder and principal of Temescal Associates, a consulting group dedicated to building the capacity of leaders and organizations in education and youth development who are serious about improving the lives of young people.  Milbrey McLaughlin has been a leading...

BY: Guest Blogger      06/26/18

"Building Workforce Skills in Afterschool" highlights promising practices for all ages

The next generation of the American workforce is growing up right now and afterschool programs are vital partners in helping young people discover new passions and work towards their dreams. As in so many other subjects, the variety and versatility of afterschool programming offers opportunities...

BY: Nikki Yamashiro      11/14/17

How does afterschool contribute to military readiness?

In 2016, the Council for a Strong America released America Unprepared, showing data that more than 70 percent of young adults in the United States would not qualify for military service due to obesity and other health issues, poor academic performance, drug abuse, or involvement in crime. As a...

BY: Leah Silverberg      08/14/17

5 ways afterschool prepares kids to succeed in the workplace

Cultivating tomorrow’s workforce remains a central part of the discussion about America’s economic future. As today’s children begin to develop the skills they will need in the workplace, experts in the education and afterschool fields are turning their attention to the ways that...

BY: Charlotte Steinecke      05/04/17

What is "The State of Girls" in 2017?

Since the Great Recession, the United States has experienced many demographic, social, economic, and technological changes. A new report from the Girl Scout Research Institute, "The State of Girls 2017: Emerging Truths and Troubling Trends," tracks resulting trends in...

BY: Leah Silverberg      02/17/17

When talking about social and emotional learning, what language works best?

We all know that high quality afterschool and summer learning programs provide kids with the skills that they need to succeed in school and life. While the concept sounds simple, finding the best way to describe these skills and their development is anything but. Some people use phrases such...

BY: Dan Gilbert      01/05/17

New report explores gap between school hours and work schedules

The misalignment between parents’ working hours and kids’ school hours is widely recognized by the afterschool community and working parents everywhere. Years of public polling have highlighted that this issue is top-of-mind with parents and voters. That’s why advocates...

BY: Jen Rinehart      10/25/16

New report: Participation in summer learning programs yields positive outcomes

A new report shows that high levels of participation in summer learning programs can provide positive benefits for low-income students’ math and language arts performance and social-emotional skills. Last week, The Wallace Foundation released Learning from Summer:...

BY: Erin Murphy      09/15/16

New report underscores the high cost of child care

Affordable and accessible high-quality child care is a critical issue for working families across the U.S. Although the benefits of quality child care for both children and their parents are numerous, many families struggle to afford and find child care that meets their needs. The 10th edition of...

BY: Nikki Yamashiro      12/21/16

New report explores gap between school hours and work schedules

The misalignment between parents’ working hours and kids’ school hours is widely recognized by the afterschool community and working parents everywhere. Years of public polling have highlighted that this issue is top-of-mind with parents and voters. That’s why advocates...

BY: Jen Rinehart      10/25/16

Poll: In public education, Americans want more than academics

Phi Delta Kappa International (PDK) recently released the results of their 48th Annual PDK Poll of the Public’s Attitudes Toward the Public Schools. This report, Why school? Americans speak out on education goals, standards, priorities and funding, identifies what...

BY: Erin Murphy      10/13/16

Updated interactive dashboard with data on high-poverty communities

Following the release of our latest America After 3PM report, Afterschool in Communities of Concentrated Poverty, which looks at the role of afterschool programs in areas where there is a high concentration of families living below the poverty line, our interactive web dashboard has been...

BY: Nikki Yamashiro      09/26/16

New report reveals how afterschool aids communities of concentrated poverty

Where you live has direct and indirect impacts on the fundamental resources and opportunities you count on, and which many people may take for granted. Your location affects the quality of schools available to you, your access to healthy and affordable food, and your overall wellbeing and future...

BY: Nikki Yamashiro      08/30/16

An Ideal Opportunity: The Role of Afterschool in Social and Emotional Learning

High school students in The Possibility Project, New York, N.Y., tackle issues close to their heart, taking on leadership roles to enact change in their communities through performing arts and community action. At Girls on the Run councils located in all 50 states, plus D.C., girls run with their...

BY: Nikki Yamashiro      05/04/18

New research on easy, affordable SEL practices

As the prominence of social and emotional learning (SEL) to support students’ development in school and beyond continues to grow in education circles, challenges implementing SEL programming have also arisen. The latest issue brief in a series supported by The Wallace Foundation, Kernels of...

BY: Nikki Yamashiro      02/20/18

Research shows the benefits of early childhood programs

Over the years, research surrounding participation in early childhood programs—such as preschool, home visiting programs, and parent education programs—has continuously shown improved outcomes for children and families. A research report...

BY: Leah Silverberg      01/25/18

"Building Workforce Skills in Afterschool" highlights promising practices for all ages

The next generation of the American workforce is growing up right now and afterschool programs are vital partners in helping young people discover new passions and work towards their dreams. As in so many other subjects, the variety and versatility of afterschool programming offers opportunities...

BY: Nikki Yamashiro      11/14/17

New report from the Wallace Foundation: Strategies to scale up

The question of how to scale up—taking a successful program, project, or policy and growing it to expand its reach and therefore its impact—has been an important one when thinking about systems change. It is a key component in efforts to make sustainable, positive social gains; a...

BY: Nikki Yamashiro      10/06/17

Looking for evidence on the impact of afterschool?

Look no further! Adding to the research conversation discussing the value of afterschool and summer learning programs, the Afterschool Alliance has released two new fact sheets that provide a sampling of evaluation findings demonstrating the positive impact programs have on students, with subjects...

BY: Nikki Yamashiro      11/09/17

Guest blog: Q&A with an afterschool researcher, part II

Welcome to part II of our Q&A with Neil Naftzger, American Institutes for Research (AIR), about his evaluation work related to 21st CCLC programs specifically and the afterschool field broadly. Below are answers to oneof the questions we asked, with our emphasis added in bold, which...

BY: Guest Blogger      08/18/17

AFT poll shows opposition to federal funding cuts to education

The clear message coming out of a recent national poll on attitudes toward federal education spending is that voters are overwhelmingly opposed to the federal government cutting funds for public education. In the poll, conducted by Hart Research Associates for the American Federation of...

BY: Nikki Yamashiro      08/02/17

Guest blog: Q&A with an afterschool researcher

In May, the proposed FY2018 budget eliminated funding for the 21st Century Community Learning Centers (21st CCLC), the only federal funding stream dedicated to before-school, afterschool, and summer learning programs. In the budget, a justification given for the elimination of funding was that...

BY: Guest Blogger      07/28/17

Mapping afterschool in New Mexico

By May Sagbakken, executive director of the New Mexico Out-of-School Time Network All children and youth deserve access to high-quality afterschool and summer learning programs that prepare them for school, college, careers, and life. While many students across the country are receiving valuable...

BY: Guest Blogger      06/05/18

New resources! National and state-level fact sheets on afterschool

As spring finally arrives in Washington, D.C., so does a suite of new resources highlighting key facts and stats on afterschool!   The first resource is a new fact sheet on afterschool, which summarizes positive outcomes for students who participate in programs, as well as the supports...

BY: Nikki Yamashiro      04/13/18

AFT poll shows opposition to federal funding cuts to education

The clear message coming out of a recent national poll on attitudes toward federal education spending is that voters are overwhelmingly opposed to the federal government cutting funds for public education. In the poll, conducted by Hart Research Associates for the American Federation of...

BY: Nikki Yamashiro      08/02/17

Guest blog: Q&A with an afterschool researcher

In May, the proposed FY2018 budget eliminated funding for the 21st Century Community Learning Centers (21st CCLC), the only federal funding stream dedicated to before-school, afterschool, and summer learning programs. In the budget, a justification given for the elimination of funding was that...

BY: Guest Blogger      07/28/17

Creating high-quality arts programs in national youth-serving organizations

While research has shown that participation in the arts promotes positive youth outcomes, providing quality arts programming can seem like an unobtainable goal to many programs, especially those that mix various art disciplines into daily programming but do not have an arts focus. But a new report...

BY: Leah Silverberg      10/27/17

New report from the Wallace Foundation: Strategies to scale up

The question of how to scale up—taking a successful program, project, or policy and growing it to expand its reach and therefore its impact—has been an important one when thinking about systems change. It is a key component in efforts to make sustainable, positive social gains; a...

BY: Nikki Yamashiro      10/06/17

New resources for STEM in afterschool from the Research + Practice Collaboratory

Check it out: the Research + Practice Collaboratory has some new and updated resources for the afterschool field! If you are not familiar, the Research + Practice Collaboratory works to bridge the gap between education research and STEM education implementation. The Collaboratory’s goal is to...

BY: Leah Silverberg      07/10/17

New report explores gap between school hours and work schedules

The misalignment between parents’ working hours and kids’ school hours is widely recognized by the afterschool community and working parents everywhere. Years of public polling have highlighted that this issue is top-of-mind with parents and voters. That’s why advocates...

BY: Jen Rinehart      10/25/16