How Much Does this Operation Make Kids Smile?
Recently my coauthors and I finished a CEGA working paper for a research project that fell into the category of one of those “dying to know if this popular thing really works” studies. Previous questions that have fallen in this category for me have included the impacts of international child sponsorship, animal donation, and TOMS Shoes. This new project was with Operation Smile. Almost certainly you have seen the advertisements on the web or in magazines with the before-and-after pictures of kids born with cleft lip and/or palate (often abbreviated CLP) like the one below. Those are either from Operation Smile or its 1998 offshoot, SmileTrain.
When you donate to Operation Smile, they use your contribution to provide cleft surgeries to overseas children born with CLP. Now it seems obvious that something good should happen from repairing cleft lips and palates. Especially in low and middle-income countries, anecdotal evidence suggests CLP children face myriad forms of discrimination, teasing, and bullying (adeptly captured in the 2008 Academy Award-winning short film Smile Pinky, a production you shouldn’t see if you don’t like crying.) CLP kids have a tough time in school, finding jobs, and forming relationships. In ancient Rome, cleft children were often killed because they were believed to be possessed by evil spirits. In some places today, this is still believed to be true.
I found the existing research on the impact of unrepaired CLP and the effect of the reparative surgery to be unsatisfactory in the statistical sense. Most of the previous research simply compared the outcomes of CLP kids to non-CLP kids and, at best, compared the life outcomes of CLP kids with and without the surgery. Approaches like these are unsatisfying because to the extent that science understands the causes of CLP, it tells us that CLP results from a combination of genetics, maternal behaviors, and environmental factors. And all of these might also cause differences in outcomes between children anyway, cleft or no cleft. What was needed was an attempt at a quasi-experimental study to estimate real causal effects.
So when you decide to click on the Operation Smile ad and donate the $240 needed to fund a cleft surgery, how much does it help? My graduate students and I decided to try to find out.
In my experience it has often been tough to get well-known organizations to participate in research projects. In some cases they may feel they have less to gain than the have to lose if the results don’t indicate that their program works very well. For example, when we initially invited a number of child sponsorship organizations to participate in a research study with us, you’d think we had sent an invitation for experimental root canal surgery. Not so with Operation Smile. They said they had been considering a study on their work anyway, and almost immediately they responded with an eagerness to subject their program to the scrutiny of outside researchers.
It made for a nice match, and they even offered to front some of the funding for the project, but we declined: better for researchers to avoid the perception of bias emanating from funding sources. Instead, we funded the research with a pair of generous grants from the Kellogg Institute for International Studies at Notre Dame and some graduate research funding from the University of San Francisco. The research team included five of our master’s students from USF: Mustafa Zahid, Sam Manning, Jeremiah Maller, Kira Evsanaa, and Susann Skjoldhorne along with Notre Dame social psychologist Matthew Bloom, and two physicians, Abhishek Das and Gaurav Deshpande at the Mahatma Gandhi Medical College who specialize in CLP surgery in India, the site of our fieldwork.
We decided to look at the impact of CLP itself and CLP reparative surgeries on an array of life outcomes of teenagers, on their speech, physical health, psychology, academics, and social integration—as much of the whole picture we thought we could measure.
Our survey in India took much longer than most similar-sized surveys. The prevalence of CLP is only about 1 in 700 births, and while most field surveys are able to reach many households in a day, our enumerators were often tracking down individual CLP children in remote villages. The survey work began in May of 2017 and lasted until July of 2019, by which time we had gathered firsthand data on 1,118 children consisting of CLP children and their nearest-age sibling, as well as data from some non-CLP families in the same villages. Our survey took place across a number of Indian states, with a primary focus West Bengal, Chhattisgarh, and Karnataka (see below).
What we quickly learned upon beginning the study was that CLP is not a 0/1 condition; it varies substantially not only by cleft lip or palate (or both), but by severity (unilateral or bilateral) and whether it is complete (cleft connected to the nose) or incomplete. Since our unit of analysis was the single CLP surgery, the physicians on our medical team evaluated each CLP child in the study by how many surgeries would be required to restore the child’s appearance and functionality in daily life to “near normalcy.” This was the number of surgeries needed to repair the cleft lip and/or palate to being physically comparable to someone born without CLP—plus a few small scars. The average number of surgeries needed to do this was about 4.5 in our sample.
The quasi-experimental design in our study harnessed the fact that given the environmental and genetic makeup of a household, CLP status of children occurs essentially randomly within the family. Our “household fixed-effects” approach essentially holds all familial factors constant, allowing us to compare the life outcomes of CLP teenagers (with varying numbers of surgeries, including zero) to those of their nearest-age sibling (accounting for age, gender, and birth order). In this way we could control for family genetics as well as factors like maternal smoking and nutrition that could affect all of the siblings in the family in different ways.
So what did we find? First of all, our results are able to sharply measure the distressing effects of CLP on the lives of teenagers. Not surprisingly, we found that unoperated CLP teenagers ranked lower in a speech quality index, but perhaps even lower than we might have expected—on average 1.55 standard deviations lower than their nearest-age sibling. In plain English, this means that the average CLP teen in India has speech quality lying at about the 6th percentile in the normal distribution of non-CLP teenagers. Physical outcomes on average were at the 37th percentile, academic and cognitive ability was at the 33rd percentile, and social inclusion was at about the 36th percentile. CLP clearly saddles kids with some big disadvantages.
What did Operation Smile’s surgeries do to restore the different life outcomes of these CLP teens? Our data show—with one big caveat—that CLP surgeries essentially fully restore speech quality to what would be normal for a non-CLP child. But the big caveat is key: the first surgery has to occur at an early age, ideally when the child is around a year old. In fact, we find that if the first surgery occurs later than age five, it doesn’t have any statistically detectable effect on speech quality at all. This doesn’t mean there aren’t other reasons to perform CLP surgeries at a later age, but it does strongly suggest that by far the biggest benefits are realized with early surgeries.
We separated the effects of Operation Smile surgeries from surgeries performed by other organizations and find the effects of CLP surgery on psychological and academic outcomes to be statistically insignificant for other surgeries. They are bigger and statistically significant for Operation Smile surgeries, although it is difficult to say exactly why. (There is some thought at OS that it may be attributed to aspects of their program that involve follow-up counseling and accompanying services, but we cannot rule out some factor that may cause a difference in CLP child selection between OS and other surgeries.) Indian teenagers with the OS surgeries have psychological outcomes that move them forward toward the average for non-CLP teens, primarily through lower levels of depression.
To me, one of our most interesting results is that CLP surgeries appear to fully restore social inclusion. There are indications that this particularly seems to be true for girls. Although the results are somewhat less precise than in some of our other findings, this seems to happen mainly through a reduction in bullying. CLP teenagers lie in the 37th percentile of a “freedom from bullying” index we created (meaning they are teased and bullied significantly more than the average kid).
Our point estimates suggest that the requisite number of CLP surgeries seem to virtually eradicate bullying of CLP teens, although the estimates of the surgery impacts are only statistically significant at the 80% level of confidence, lower than the standard 95% significance level. In exploring the channels through which bullying and social exclusion occur, somewhat surprisingly we find it appears to happen through poor speech quality of CLP teenagers rather than visual appearance, which we indicate through having an unoperated cleft lip (as opposed to cleft palate, which is visually hidden.)
So to prevent bullying and social exclusion, it appears from our results that it is important to restore speech, and to restore speech, surgeries need to occur when children are young. In my recent book, Shrewd Samaritan, I give reparative surgeries, such as for CLP, cataracts, and clubfoot, a high intervention rating for effectiveness. The results from this study on Operation Smile would seem to reinforce this view. CLP surgeries are obviously not as cheap as health interventions like vaccines and deworming medicine. But the results of this new study suggest that CLP surgeries have meaningful impacts on holistic measures of human well-being.
Follow AcrossTwoWorlds on Twitter @BruceWydick.
- Snap-Back and Gone-Forever Goods: Understanding the COVID Recession’s Economic Winners and Losers
- The Impact of COVID on the American Church