Null Impacts of Education and Information Interventions on Children’s Cognitive and Noncognitive Skills: Experimental Evidence from Poor Urban Households in the Philippines
Abstract
We use a randomized-control trial (RCT) to assess the impact on children’s cognitive and noncognitive skills of several novel educational (drawing, storytelling, and computer aided learning) and parental information (children’s skill development, savings, and returns to educational investments) interventions in a low-income urban community in the Philippines. We found no impacts across a range of outcomes for either the educational intervention or the educational plus parental information intervention. We discuss potential reasons for the null result and directions for future research and policy.
Introduction
While education has played a critical role in global poverty reduction (Gethin, 2025), poor parents and guardians in developing countries still often underinvest in their children due to a lack of financial resources. Incomplete credit markets, riskiness of such investments, irreversibility, and limited information about returns can also all lower potentially efficient investments in human capital (Aghion et al., 2025; Becker et al., 2018). In principle, governments in developing countries can address some of these market failures. However, one of government’s main policy tools, schools, faces its own set of challenges in effectively producing human capital (Evans and Popova, 2016; Glewwe et al., 2021; Glewwe and Muralidharan, 2016; Kremer et al., 2013; Snilstveit et al., 2015). Financial constraints, infrastructure limitations, large enrollments, high learner-teacher ratios, and teachers’ elite bias can all make teaching and remediation difficult1. Although communities can sometimes step in to address market and government failures in education service delivery (Sawada et al., 2022), community-driven school-based management is not necessarily a silver bullet to such pervasive and intractable issues.
Learners also frequently fall behind early in school, which can affect investment decisions by children, parents, and schools (Cunha et al., 2010, 2006; Cunha and Heckman, 2007). This can rationalize dropout decisions and a lack of motivation as well as lead to issues with remediation and negative spillovers within the classroom. Poverty is also extremely challenging for mental health (Barker et al., 2022; Haushofer and Fehr, 2014; Jin et al., 2025; Ridley et al., 2020) and economists have been increasingly recognizing the importance of cognitive and noncognitive development not only for economic outcomes but also for the production of skills themselves (Almlund et al., 2011; Heckman et al., 2006; World Health Organization, 1999). Research suggests that personality traits themselves are not necessarily fixed (Roberts et al., 2017) and meta-analyses show that some interventions can indeed change such traits (Cipriano et al., 2023). Finally, urban poverty in developing countries is often quite concentrated, which can create strong neighborhood and peer effects that present both a challenge and an opportunity for policymakers (Marx et al., 2013). Taken together, these issues all help to reinforce the “Great Gatsby Curve”: a pernicious entrenchment of inequality and intergenerational poverty (Durlauf et al., 2022; Krueger, 2012).
To understand some potential policy approaches, we investigated whether supplementary educational interventions interacted with a parental information intervention might change the trajectory of children’s skill development in a poor urban community located in Rizal Province in the Philippines. In our study we worked with two elementary schools and one high school. Learners were randomly assigned to one of three groups: educational intervention only, educational plus parental information intervention, or control. The two treated groups received the same one hour per week educational intervention of either drawing, storytelling, or computer aided learning (CAL) that depended on the learner’s grade level2. The parental intervention provided an initial pre-intervention information session (with periodic follow-ups) on children’s skill development, the returns to schooling, and the importance of savings and investment for education. Given the limited resources of both schools and families in the context, we tried to design these interventions to be feasible but also sustainable.
Disappointingly, we found no impacts of our interventions on a range of human capital outcomes for children even for several years of follow-up data. We hope this paper can be useful to the literature in avoiding publication bias (Abadie, 2020; Andrews and Kasy, 2019; Chopra et al., 2024) and to give a more complete picture of when and why such interventions do not appear to be effective. Particularly when viewing development as an engineering problem, null results from RCTs are quite instructive (Karlan and Appel, 2017). For the CAL intervention, our findings are in contrast to the literature, which typically finds positive impacts on mathematics test scores in developing countries3. Information about the returns to education have been demonstrated to produce positive impacts in some contexts (Jensen, 2010; Nguyen, 2008)4. RCTs for storytelling and drawing interventions appear to be completely novel in the literature5. CAL has been much more studied but generalization and context of its effectiveness are not well understood (Abbey et al., 2024; Bulman and Fairlie, 2016; Kremer et al., 2013). We are also not aware of research examining interactions between parental information and child educational intervention. We provide more details below when discussing the inventions.
Background and context
We conducted our intervention in Kasiglahan Village (KV), which is located in Rizal Province the Philippines. Although initially the village was a resettlement area for victims of the Payatas dumpsite landslide, the area has subsequently grown significantly6. Our intervention was designed based on consultation with school district officials in the Department of Education in Rizal Province and with an NPO called Salt Payatas that has been working with women and children in Payatas and KV since 19957
In 2014, we conducted preliminary interviews with parents, teachers, volunteers, and the children themselves that provided background information about the challenges that the children faced in their lives and helped to design the intervention. Our observations were as follows. First, some children already stop attending school as early as G1 and G28. Second, many learners do not show much interest in reading or learning at school. Even at the beginning of their school careers, lack of motivation and interest were already salient. Third, some learners lacked very basic skills such as ability to spell their own names or to perform simple arithmetic calculations. These literacy and numeracy issues obviously compound and make learning in higher grades difficult. Furthermore, given the extremely large classes of approximately 60 learners per class, it is often difficult for teachers to provide individualized remediation or to motivate those who are not interested in school. Trying to increase motivation and the use of technology for large-scale remediation were identified early on as potential areas for intervention.
Given these facts on the ground and the intensive discussions with various stakeholders, we worked with subject-matter experts to design three types of extracurricular educational interventions: drawing, story telling, and CAL. The assigned intervention depended on the grade of the child at the time of random assignment. Drawing was for learners initially in G1-G2, storytelling for those in G3-G4, and CAL for G5 and above. Learners would continue to receive the same type of intervention for the duration of the study. Most of the interventions were for one hour per week, were extracurricular, and were around lunch time (either after the morning school session or before the afternoon school session). There was some variation in the number of intervention days depending on the year and the type of intervention. We provide details below. Children were encouraged, but not required, to attend and because the interventions occurred near the children’s lunch hour we also provided lunch for the treated children9. The drawing and storytelling interventions ran for two consecutive school years (2016-17 and 2017-18) whereas the CAL intervention ran for three consecutive school years (2016-17, 2017-18, 2018-19). This was partly because during the first intervention year (2016-17) there were issues with very weak internet access and the CAL intervention could not be successfully implemented so we extended it for an additional year through the 2018-19 school year.
Child educational interventions
Drawing
The youngest learners (initially in G1 and G2) were provided with a drawing class. The class always began with an open-ended story and the learners were then encouraged to create their own endings to the story through their illustrations. This was designed to stimulate their creativity and cognition (Okumura, 2010). This program was created by our co-author Dr. Takaaki Okumura who is an art scientist that formerly worked for Japan’s Ministry of Education designing art education at primary and secondary schools in Japan, which aims exactly to develop such cognitive and noncognitive skills. Staff were also given training about how to interact with the learners in order to encourage (but not distort) their creative processes. Although few studies have rigorously measured the impacts of art education on skill development, existing research finds positive effects on psychological traits and behavior (McDonald and Drey, 2018) particularly for those with learning disorders (Regev and Guttmann, 2005). Art therapy has also been shown to reduce anxiety in children with leukemia who were undergoing painful procedures (Favara-Scacco et al., 2001). We hypothesized this might have some parallels to a stressful high-poverty environment (Bossuroy et al., 2022; Haushofer and Fehr, 2014; Ridley et al., 2020). We are not aware of any other RCTs that specifically focus on the impacts of art education.
Storytelling
For learners initially in G3 and G4, we worked with Mr. Rey Buhi and Ms. Jennica Gan, experts from a Ginebra Ako award-winning volunteer organization called The Storytelling Project, to design a storytelling intervention10. They provided training to staff on how to conduct the storytelling sessions in order to engage and manage learners. The goal was to improve learners’ imaginations and to instill a love of reading with the hope that this would translate into increased literacy and interest in school. The intervention involved a group reading activity in which the staff member would show the book to the learners and read out loud. Learners were again encouraged to craft their own endings to the story in order to help stimulate their imaginations. In addition to the story, the intervention also included singing and dancing at the beginning of each storytelling session to help the learners focus during the story. A meta-analysis of meta-analyses of student learning in developing countries suggests pedagogical interventions, teacher training, and improving accountability are effective (Evans and Popova, 2016). Early reading interventions in developing countries can also work (Graham and Kelly, 2018) and our intervention shared some aspects of successful interventions such as staff training and scripted pedagogy. However, to the best of our knowledge, there are not any RCTs specifically related to storytelling as a way to promote reading skills.
Computer Aided Learning
For learners intially in G5 and above, we implemented an adaptive CAL software for mathematics. In comparison to the drawing and storytelling that focused on broader skill development, this intervention was focused more on remediating specific mathematics deficits that might be holding the learners back. Learners were in a classroom together and each learner had their own computer to use. The CAL software would identify and drill learner weaknesses while any issues with learner focus and motivation were addressed by trained staff.
During the first year of intervention (2016-17), a Japanese edtech company called Quipper allowed us to use their CAL product11. However, we could not implement the program properly because of limited internet access, which is one among many difficulties that can arise in implementing RCTs in developing countries (Karlan and Appel, 2017). From the middle of 2017-18 school year, we received generous support from a Japanese CAL company called Surala Net including full access to their online materials and intensive training of our staff members to serve as coordinators. Internet access had also improved. Because the first-year intervention could not be implemented properly, the Surala CAL program was extended an additional year through the 2018-19 school year. We also encouraged learners to attend classes twice a week for a total of two hours, rather than just once a week12.
In general, the literature on computer aided instruction shows mixed evidence of effectiveness, except for a clear pattern of some impacts on mathematics scores in developing countries (Bulman and Fairlie, 2016; Escueta et al., 2020; McEwan, 2015). However, even this evidence produces some variation in impact depending on whether the CAL is integrated into the curriculum or held outside of school as well as on the initial ability level of the learners (Bai et al., 2016; Lai et al., 2015, 2013; Linden, 2008; Mo et al., 2015, 2014b, 2014a). There have been calls for more research to help isolate mechanisms and to understand generalizability of these CAL programs (Escueta et al., 2020).
Parent information intervention
For one of the treatment arms, in addition to the educational intervention, we also provided an information intervention for parents. Some research has found that providing information can substantially change the investment behavior of households in developing countries (Jensen, 2010; Nguyen, 2008). This is particularly intriguing for policymakers because information interventions can in principle be done cost-effectively, possibly at scale, and information can also spread across social networks, which can lead to seeding strategies (Banerjee et al., 2013). Although the impacts of information interventions tend to be small (Escueta et al., 2020; McEwan, 2015), their low cost implies that even modest impacts can pass cost-benefit analyses.
Our information intervention was held for one hour in groups of 20-40 parents, which was so they would feel comfortable to ask questions. This information intervention was conducted before the start of the educational interventions. In the intervention, we stressed the importance of education to parents by providing information on how graduating from college could increase monthly earnings, on children’s skill formation and the importance of early investments, some basic information about the costs of college and the availability of scholarships, some calculations on how not spending on gambling and smoking regularly could compound over time. Finally, we encouraged savings for educational investment by giving parents an alkansya13. We also distributed a pamphlet biweekly for the duration of the treatment to remind the parents about the information. The education only intervention group also received an explanation of the importance of cognitive and noncognitive skills. All groups (including control) received an introduction to the project team, an explanation of the survey, assistance in filling out survey/consent/waiver forms, and an introduction to the JICA-funded child library in KV run by Salt Payatas.
According to our pre-intervention survey, parents seemed to drastically underestimate the monetary returns to schooling. We asked parents about their expectations for their child’s income at various hypothetical levels of education. We then compared the averages of these expected incomes with the actual incomes collected in our baseline survey14. Parents typically perceived that incomes were 20 - 60% lower than actual incomes15.
In addition, preliminary calculations suggested that some simple financial adjustments could easily put college within reach for many of the households, which is consistent with some of the nudge and behavioral economics literature (Duflo and Banerjee, 2011; Kremer et al., 2019). Although information treatments tend to have smaller effect sizes for self-reported attitudes and behavioral measures than for belief updating (Haaland et al., 2023), our hypothesis was that there might be a complementarity between the parental information intervention and the educational interventions that could be used to leverage the impact. To our knowledge there are no studies looking at such interactions.
Data
Sampling, randomization, and stratification
We did our initial sample selection and survey in 2015-16 in the year before the intervention, which began during the 2016-17 academic year among learners in G1-G7. We sampled at the classroom level by randomly choosing one classroom from each grade level at the three schools. This sampling was done at the classroom level to minimize disruption to teachers and administrators16. The randomization then occurred at the individual learner level so that some learners within the same classroom would be assigned to one of the two treatments or to control. Although such a within-classroom design can increase power, there is some potential spillovers from treated to control learners (or vice versa). We discuss such possibilities below when we interepret the experimental results.
Summary statistics
Our data come from three sources. First, children were given direct assessments or asked survey questions. Second, we also interviewed parents or guardians of the children. Third, we received administrative data from teachers and principals on the children’s school attendance, performance in school, and health status.
Table 1 shows mean and standard deviation for the outcome variables in our analysis. We collected data on a wide variety of educational inputs and outcomes: study time, reading habits and attitude, college aspirations, self-esteem, grit, impulsivity, family environment, IQ, school outcomes, discounting, and mathematics. These data are pooled across time and grouped by theme. At baseline in 2016, we collected data from 1103 learners and this gradually decreased to 789 for the final round in 202017. This population is somewhat transient with families often relocating because of labor market opportunities. However, the attrition does not look different by treatment status. In Table 1, the sample size across outcomes shows the extent of missingness and attrition by year and outcome. School outcomes were only available through 2018 and in general have the most missing data mainly because of difficulty obtaining records. Draw-a-person was not collected in 2019 due to a funding issue, and we also added Raven’s Progressive Matrices beginning in 2018. A “marshmallow test”, which we used to back out estimates of a discount factor \(\delta\) and a present bias parameter \(\beta\) (Laibson, 1997), was not collected at baseline.
The data indicate that the children spend more than an hour studying on both weekdays (80.88 minutes) and weekends (66.11 minutes). They also report reading 3.66 books last week and 6.39 books last month, which suggests some recall bias. 68% of children report reading after class. Among the children, 82% report they are likely to go to college. This is much higher than the tertiary education enrollment rate in the Philippines, which was 34% in 201718. Furthermore, the college completion rate was only 16%19. These unconditional statistics would likely be lower for children from impoverished households. Although enrollment has increased to 47% in the latest data from 2024, the statistics still suggest a substantial gap between the children’s beliefs (and aspirations) and the reality of opportunities for higher education in the Philippines.
The next five measures used scales common in the psychology literature. Most of the measurement items are Likert scales and we re-orient the item responses to have the same direction before either averaging or summing the items depending on which statistics is used in the literature. We also try to give a qualitative interpretation and compare to measurements reported in the literature.
Reading motivation is captured by the 9-item reading subscale of the Elementary School Motivation Scale (Guay et al., 2010) and the average of 3.03 corresponds to a “sometimes yes” interest in reading. Impulsivity is the 8-item Domain-Specific Impulsivity Scale for Children that aggregates a child self-report about the frequency of different types of behavior the child engages in at school and home. Higher scores correspond to better behavior. The mean of 3.36 indicates that on average the children engage in various types of misbehavior 2-3 times a month. Family environment is the 18-item questionnaire Family Environment Scale (FES) which measures the social-environmental characteristics of the child’s family (Moos and Moos, 1976). Each item has a yes or no response, which we reoriented, coded as 0/1, and averaged. Values closer to 1 indicate a better family environment. The mean was 0.77. Self-esteem is the 10-item Rosenberg self-esteem scale that asks the children about their feelings towards themselves (Rosenberg, 1965). The average of 2.92 indicates that on average the children “agree” with items describing high self-esteem. Grit is the 8-item Short Grit Scale (Duckworth and Quinn, 2009). Higher values correspond to more grit. Interestingly the average 3.50 in our sample is actually quite similar to samples from the US (Duckworth and Quinn, 2009).
We used two IQ related measures. The first is the “draw-a-person” assessment which asks children to draw the figure of a person. The measure has 50 items related to the complexity of the figure drawn and it was designed to be a systematic way to measure the “nature and organization of the child’s mental processes” (Goodenough, 1926). Our sample averaged 29.03, which is remarkably similar to the averages reported by Goodenough (1926) for American school children in the 1920s. From 2018, we also collected a more standard IQ measure in the form of Raven’s progressive matrices. We used the 12-item short form and the mean of the sum score is 6.79, which is smaller but similar to statistics reported in the literature (Arthur Jr and Day, 1994). The difference is likely accounted for by the younger age of our study participants. Interestingly the correlation between our Raven’s and draw-a-person measures was 0.15, which is consistent with other research questioning the validity of draw-a-person as a measure of IQ (Imuta et al., 2013).
The school outcomes (school attendance, BMI, and GPA) come from the school records and not from surveys. These data are more likely to be missing. While school attendance is quite high at 97.72%, children tend to be underweight with an average BMI of only 15.32. GPA is measured on a 100 point scale with a mean of 81.51 in our sample. Below 75 is considered failing. However, failing is somewhat rare, as only 3% of learners in our data appear to be failing.
We estimated present bias and discount factor using a “marshmallow test” except with chocolate20. Using real rewards, we asked children whether they preferred 5 chocolates today or 6 chocolates tomorrow, 5 today or 8 tomorrow, and 5 today or 10 tomorrow21. The responses were used to construct the midpoint of a present bias x discount factor that would rationalize their choices between today and tomorrow. This gave an estimate of \(\beta \delta\) of 0.81. We also asked the children the same questions except in one week or in 8 days, which we used to get an estimated \(\delta\) of 0.79. The two estimates taken together imply that \(\beta\) is close to 1 and is inconsistent with hyperbolic discounting. In fact, using individual specific estimates, only 11.2% of children had an estimated present bias term less than 122.
Finally, we implemented two mathematics assessments. The first we call CEM after the company that administered the assessment (Center for Educational Measurement). This was collected at baseline, at each midline, and at endline. The second assessment (Surala) was only given in 2020 and was specifically designed to be aligned with the CAL program from Surala Net. For both assessments we report the percentage correct, which averaged 43.16% for CEM and 79.03% for Surala23.
We also conducted a parental survey given that our intervention specifically targeted the parents’ information set. While some of the children’s items captured the household environment or the children’s skills, we were also quite interested in the savings behavior of households given the encouragement for the parents to save. However, there was substantial attrition in the parental survey and item non-response. The respondents are largely unbanked and interviewers informed us that parents were reluctant to provide information about any savings they kept in the house for security reasons. In addition, parents usually work 6 days per week and are often away from the village so it was often much more difficult to survey parents compared to children.
Table 2 shows baseline balance for outcome variables measured at baseline as well as selected parental characteristics. We do not find any difference in the mean of the outcome variables at baseline by random assignment.
Table 3 shows the number of intervention days and the attendance rate. The number of treatment days offered ranged between 11 and 29 and varied both by intervention and by year. Among treated learners the overall attendance rate was only 29% of the offered intervention days so there was a substantial amount of “never-takers” at the intensive margin. At the extensive margin, only 18.8% of children never attended any days across all years, which shows that there is some cycling in and of out the intervention among treated children. There were no crossovers as attendance among the control group was zero. Attendance increased substantially in the second year of the intervention and was higher for drawing and storytelling24 CAL was extended through 2018-19 while the other two interventions ended. All interventions ended by the 2019-20 school year and we collected endline data in 2020.
Empirical strategy
We estimate the impact of our interventions using the following regression model: \[\begin{equation} Y_{isct} = \alpha_0^{t} + \alpha_1^{t} T_{i}^{e} + \alpha_2^{t} T_{i}^{ep} + \epsilon_{isct} \end{equation}\] where \(Y_{isct}\) is the outcome for child \(i\) in school \(s\) in class \(c\) at time \(t \in \{2017, ... , 2020\}\), \(T_i^e\) is an indicator for whether the child was assigned to the educational intervention, and \(T_i^{ep}\) is an indicator for whether the child was assigned to the combined educational and parental information intervention. The parameters are superscripted by \(t\) to allow the impact of the program to differ by time period. In robustness checks, we also experimented with time invariant child/family controls \(X_i\) and baseline outcome \(Y_{iscb}\). We also estimated the model on different subsets of the data. Testing \(\alpha_1^{t} = 0\) can tell us about the impact of the educational intervention and testing \(\alpha_1^t = \alpha_2^t\) can tell us about any differential impact of the parental information intervention. We can also examine dynamic treatment impacts over time.
The intent to treat (ITT) parameter helps us to understand the effect of the program as implemented. However, given the attendance rate, it is also useful to look at a local average treatment effect (LATE) that identifies a weighted average along a “causal response function” (Angrist and Imbens, 1995). One way to model this is to examine the impact of the attendance rate. We look at a cumulative attendance rate through period \(t\) for educational treatment (\(A_{it}^e\)) and for the combined education and parental treatment (\(A_{i}^{ep}\)). This variable equals 0 for control children and no-shows and ranges up to 1 for children with perfect attendance so it can also be intrepreted as a dosage effect. To deal with the endogeneity of attendance, we can instrument for these two variables using random assignment \(T_{i}^{c}\) and \(T_{i}^{cp}\). This gives the impact of attendance on children who are induced to change their attendance rate by assignment to the treatment. The model is given by: \[\begin{equation} Y_{isct} = \alpha_0 + \alpha_1 A_{it}^{e} + \alpha_2 A_{i}^{ep} + \epsilon_{isct} \end{equation}\] In this model, we again experimented with adding control variables and baseline outcomes as well as estimating the model on different subsets of the data.
Results
Our main results are shown in Table 4 (ITT) and Table 5 (LATE). Overall we do not find any consistent pattern of impact estimates. There are sporadic statistically significant estimates but are often the “wrong” sign, do not persist over time, and would likely disappear with adjustments for multiple comparisons25 One exception is that there appears to be some decrease in learners’ study time. This is an example of behavioral response to an exogenous intervention (Pop-Eleches and Urquiola, 2013; Todd and Wolpin, 2003) and is consistent with some work that finds school inputs can crowd-out private inputs (Houtenville and Conway, 2008). However, the magnitude of the decrease in study time is outweighed by the 60-120 minutes of intervention, which shows that some of the transfer indeed “sticks” to the child (Jacoby, 2002).
We tried several robustness checks including adding controls for baseline household characteristics, baseline learner characteristics, and baseline outcome variables. We also examined impact heterogeneity by intervention type (drawing, storytelling, or CAL), child gender, maternal education, and star section. Figure 1 summarizes the various ITT estimates using a Forest plot that is faceted by year and treatment. The dark lines represent 95% confidence intervals centered around the impact estimates reported in Table 3. Each light gray line represents an alternative specification or subgroup analysis detailed above. As can be seen, we did not find any consistent pattern of impacts across any of the analyses. In Figure 2, we repeat the same procedure using our LATE estimates and again find no discernible pattern of impacts.
Discussion
Identifying the precise driver of these null results is difficult because many potential factors could explain the lack of impact. Although the simplest explanation is that the interventions were just not effective, there are other contextual explanations that we can broadly categorize into three dimensions: time inputs, behavioral responses, and complementarities. Such contextual explanations can be useful for future meta-analyses given the increasing number of RCTs in development economics.
First, an obvious possibility is the time input or the program attendance was simply insufficient. Issues of program duration and measurement timing have been underexplored in the literature (King and Behrman, 2009). Our program was not particularly time intensive and the attendance rate was somewhat low, especially initially. Scaling the hours or increasing attendance might have produced an impact26. A counterpoint is the LATE parameters for compliers who increased attendance as a result of exposure to the treatment also did not show any statistically significant impacts. We also did not see impacts in second year of the intervention that had more time inputs and higher attendance, which would be expected if the intensity of the inputs was a factor. The extracurricular aspect meant that treated learners who decided to attend were not able to immediately go play after school or needed to come to school early. Both factors could obviously have contributed to reduced attendance so an intervention that was incorporated more directly in to the curriculum might have been more effective through a participation channel. This would be especially true if there was a negative selection into treatment based on impacts in which case mandatory participation through a curriculum might be expected to be more effective.
Second, the randomization was done within classrooms. Although this is often done in educational evaluations for reasons of statistical power, such a design increases the likelihood of spillovers to or from peers. One potential test is whether there are different impacts for afternoon session learners (who would immediately interact with their peers after the intervention) compared to morning session learners (who would leave to go home after the intervention). However, we did not see any differential impacts by morning vs. afternoon session students, which is inconsistent with the peer effects story. The intervention may also produced differential behavioral responses in other ways as well. For example, control learners may have wondered why some peers were attending a special extracurricular class and this could have motivated them to work harder (attenuating any potential impact) or been demotivating and led to reduced effort (which would be inconsistent with our null results). Treated learners could also have reduced their own study inputs (a crowding-out effect, which there was some evidence for) or potentially transfered their newly acquired skills to control learners, which both have attenuated the impacts of the program itself. These types of behavioral responses cause difficulty in estimating production function parameters as well as comparing estimates across studies (Todd and Wolpin, 2003). Learners also had the opportunity to participate in other extracurricular programs, which may have introduced substitution bias into our experimental evaluation (Heckman and Smith, 1995; Kline and Walters, 2016). Analogous information spillovers between parents could have occured within the community. For the parents we initially tried collecting data on social networks within the village but the networks were too sparse to produce any meaningful data (i.e., people really did not seem to know each other). We hypothesized that this was related to KV being a resettlement community, which would mean there were less long-standing community connections. The somewhat transitory nature of the population both for migration and for work would also potentially lessen community ties.
A third possibility is that some aspect of the production function for skills in our context (either other inputs or its shape) caused the null impact. For example, our extracurricular interventions were not necessarily well integrated into the broader school curriculum. Disconnected educational interventions do not necessarily leverage existing school inputs so it would have perhaps been more effective to incorporate the activities directly into the classrooms. However, curricula are much more difficult to change, especially in a randomized way (for both ethical and bureaucratic reasons), and so offer less discretion for experimentation. Yet another possibility is there may have been poor alignment between the intervention and the initial skills of the learners. For example, the first-year mathematics assessment was too difficult for the learners, which seemed to intimidate some of them and suggested the measurement was not aligned with their existing skills. However, we also did not see impacts among the advanced “star section” learners, which one might expect if initial ability was a limiting factor. A related possibility is that strong complementaries in the education production function might create an O-ring scenario (Kremer, 1993). For example, with 60 learners per class, perhaps any additional variation in inputs would be completely ineffective until class sizes are smaller. So existing school inputs might simply be limiting. That would also imply that locally searching over the policy space via RCTs would be unlikely to uncover any evidence of impacts. Instead, a “big push” policy might be needed to transition from a bad to a good equilibrium (Matsuyama, 1998; Murphy et al., 1989; Rosenstein-Rodan, 1943). That is, everything needs to be fixed at once. This view of the world is consistent with the small effect sizes often reported in education research (Evans and Yuan, 2022) and with evidence that some innovative charter schools or complete school redesigns are capable of producing extremely large impacts (Angrist et al., 2010; Eble et al., 2021). At least in the first year, we encountered some technical issues with the CAL technology and the internet. Such infrastructure bottlenecks are also an issue and are likely particularly salient in developing countries in a way that would prevent technology adoption and experimentation by schools.
Conclusion
Overall we do not find any impact of our interventions on the cognitive or noncognitive skills of the learners. For CAL and mathematics in developing countries, this contrasts with the existing literature (Bulman and Fairlie, 2016; Escueta et al., 2020). We also did not find any impact of the information intervention on investment behavior, skill development, or expectations. The fact that the parental information intervention did not alter behavior suggests that parents’ underinvestment is not driven by a simple information deficit and insteads points to credit constraints, uncertainty, or risk aversion as potential explanations. We also did not find any complementarity between the educational and information interventions. Previous research on information interventions in developing countries is actually quite limited and successes might depend heavily on context. Research also suggests that nudges do not necessarily scale or generalize well (DellaVigna and Linos, 2022). The novel drawing and storytelling interventions designed to stimulate creativity and motivation also did not show any effects even on measures highly aligned with the intervention. Rather than a mere reporting of statistically insignificant coefficients, these precisely estimated null results offer an important data point for when educational interventions in low-resource environments do not have impacts. From a cost-effectiveness perspective, interventions with low marginal costs, such as informational nudges or CAL, are often viewed as promising. However, our findings show such interventions may be limited by important contextual factors like participation constraints, spillovers, low baseline skill levels, infrastructure limitations, and integration failures with existing curricula. Spillovers can be viewed positively in implementation while still causing problems in program evaluation. Human capital accumulation in developing countries cannot necessarily be achieved by marginal changes or by programs superimposed on the existing educational infrastructure. By reporting these results we hope to avoid publication bias on educational and information interventions in developing countries, which can distort the literature (Brodeur et al., 2020; Chopra et al., 2024). Rigorously conducted RCTs, regardless of findings, should be reported to give a complete picture of the state of knowledge. Finally, a positive aspect of our research is that we have continued our collaboration with the school district and with the NPO Salt Payatas as we work to identify areas where the research team can complement the efforts of local stakeholders through data analysis and capacity building.
References
Footnotes
The two elementary schools in our study have such large enrollments that they split the day in half with some learners attending during a morning session typically from 5:40 - 12:00 and other learners attending an afternoon session typically from 12:00 to 18:20 (these times vary slightly by grade level). In addition, class sizes are often as high as 60 learners per teacher. In the Philippines, they use the term learner in place of student so we adhere to their preferred nomenclature.↩︎
This type of intervention is variously called e-learning, computer aided instruction, or computer aided learning in the literature (Bulman and Fairlie, 2016).↩︎
Programs that simply give technology such as computers or tablets tend not to find any impacts (Barrera-Osorio and Linden, 2009) whereas CAL software seems capable of producing large impacts (Bai et al., 2023, 2016; Banerjee et al., 2007; Lai et al., 2015, 2013; Linden, 2008; Mo et al., 2015, 2014a; Muralidharan et al., 2019)↩︎
Instead of a very general information treatment, more papers seem to examine individualized information treatment such as providing parents clearer information about their child’s performance in school. Interestingly, it seems that lower income parents are less responsive to such information interventions (Boneva and Rauh, 2018; Dizon-Ross, 2019)↩︎
There are claims that these types of arts interventions will provide spillovers to other types of skill development (Simpson Steele, 2016). Despite the evidence base for such claims being fairly weak (Goldstein et al., 2013), policies, including those in Japan, are predicated on exactly such mechanisms. The storytelling RCTs we found in the literature are more oriented around promoting health behaviors of adults and teenagers, which is qualitatively quite different than our intervention.↩︎
On July 10, 2000 there was a devastating landslide of a garbage dump that had been used for scavenging by squatters living in a nearby area called Payatas. Although officially 218 people died, the total was perhaps much higher and many people’s homes were either destroyed or declared at-risk and uninhabitable. A national outcry followed, and the efforts of activists and children affected by the tragedy, such as the Bangkang papel boys, led to resettlement efforts.↩︎
Salt Payatas provides educational scholarships to children and empowers women through production of craft goods for sale. These activities are partly financed by offering study tours of local areas. However, Salt Payatas felt that these scholarships were often insufficient to stop children from dropping out or performing well in school. According to the World Bank DataBank, in the Philippines in 2016 11% of primary school age children had already dropped out of school. This led to efforts to find other solutions and members of Salt Payatas made contact with our research group in Japan to begin thinking about what studies or interventions could be done to help local children. This collaboration led to the current project. The Japanese government through Japan International Cooperation Agency (JICA) has also contributed to the community by helping to establish a children’s library in KV in 2015 that was subsequently managed by Salt Payatas.↩︎
Grades in the Philippines are referred to by G1 (1st grade), G2 (2nd grade), etc. We again follow local nomenclature.↩︎
Nutrition has been shown to be an important component of school productivity (Glewwe et al., 2001). In that sense our intervention is a compound intervention consisting of both educational and nutrition intervention so it would be impossible to identify them separately.↩︎
The Ginebra Ako awards honor individuals or organizations that have made exceptional contributions in the Philippines.↩︎
Quipper saw huge early-stage growth in the Philippines during the pandemic (Ignacio, 2022).↩︎
We had difficulty increasing the attendance rate particularly among learners in higher grades. Many had anxiety about mathematics, which made them avoid the CAL program. But those who tried it often became strongly attached and continued to attend.↩︎
Alkansya is the Tagalog word for piggy bank. This population is largely unbanked and lacks access to various financial services. We also noticed other type of savings commitment devices in other contexts in the Philippines like for tricycle drivers who sometimes have a locked alkansya where they can deposit fares for savings purposes.↩︎
Because there are few college graduates in our sample of parents, we extrapolated the income for high school graduates using the returns to tertiary education reported by Montenegro and Patrinos (2014).↩︎
It is somewhat unclear how such large misperceptions can persist. Financial illiteracy may play a role. In addition, respondents are largely unbanked so financial information sharing may be limited due to security concerns.↩︎
We also stratified in order to sample a “star” section, which consisted of more advanced learners. In practice, there was usually only one star section per grade so this “sampling” was primarily nonstochastic.↩︎
During the 2015-16 school year prior to the RCT, we originally collected data from 1441 learners in our survey. However, at the time of randomization, just prior to the 2016-17 school year, we could only locate 1103 learners. The randomization was done with this smaller population of learners. As always there is some selection into participation in the RCT. However, none of the baseline covariates seem to predict this participation decision so at least based on observables it appears to be random. Also, the child self-assessment at baseline was deliberately restricted to older children due to concerns that the questions were too challenging for younger children. This led to the smaller sample size at baseline for most of the child assessment outcomes.↩︎
School enrollment, tertiary (% gross) - Philippines reported in the World Bank DataBank based on data from the UNESCO Institute for Statistics.↩︎
Authors’ calculations for age 25+, 2010 Philippines census (IPUMS, 2020).↩︎
Our local team members suggested using chocolates instead of marshmallows because of learner preferences.↩︎
Children were told they would receive chocolates according to their response to one of the randomly drawn questions. This of course raises standard issues about experimenter commitment. In addition, the fact that the children experienced a lottery over their responses make the interpretation of our discount factor and present bias estimates not necessarily straightforward.↩︎
Other research reports a much larger percentage of hyperbolic discounters (27%) among Filipino adults (Ashraf et al., 2006). It is unclear what drives this difference although our sample of children is obviously much younger, which might be expected to show a larger percentage of hyperbolic discounters. The Ashraf et al. (2006) sample came from bank customers, which intriguingly hints at a selection into using banking services among hyperbolic discounters relative to the general population.↩︎
For the first midline CEM mathematics assessment in 2017, learners only got 36% of answers correct. From this we deduced that the assessment was too difficult for the learners and unlikely to capture any potential impacts of our interventions because of floor effects. After asking the company to make the test easier, the average increased to 50 in 2018. However, the test also seemed qualitatively misaligned with our intervention so in the final year we also added an assessment designed specifically by Surala to be aligned with their CAL software.↩︎
The days offered of CAL in 2017-18 was lower because the Surala CAL software was only available from November while the drawing and storytelling started from the beginning of the school year in July.↩︎
We have colored positive statistically significant impacts in blue and negative ones in orange.↩︎
Our reading of the literature is that other CAL interventions offered similar duration of sessions and interventions. However, some offered substantially more time inputs. This would be a good issue to explore in a meta-analysis.↩︎