Whitney Espich: Hi, I'm Whitney Espich, the CEO of the MIT Alumni Association and I hope you enjoy this digital production created for alumni and friends like you. Thanks very much for inviting me to speak at tech day, so that I don't think I need now, right? I'm thrilled to have a chance to share a few of my econometric insights with you. Today's lessons come from my quarter century of teaching and the MIT Department of economics. And I know that many of you recall your college economics and statistics courses very fondly. So I hope that this will be as much fun for you as it is for me.
I am very lucky to teach at MIT, as I'm sure most of you can appreciate. Every day here is like a scene from Good Will Hunting. Well, maybe a little different in reality. I'm using my iPad.
I see, I understand. OK. Got it. There's Good Will Hunting.
Yeah, OK. So the iPad is not working, Glenn? Got it. Let me move it down here. Anyway, the real thing is maybe a little different of course, but my students, no matter where they come from, they work very hard to get here, and every semester, I get to greet a fresh crop of smart young people who have studied thousands of hours in an effort to earn a coveted place at MIT, at MIT's entering class.
If you see the latest Spider-man movie, I have a scene here, you'll know the extraordinary lengths people go to get in. Spells cast and portals opened and villains unleashed. That's MIT applicant Peter Parker, upper left there. He saves our Dean of Admissions from a rampaging Doctor Octopus on the right. So while that bit is a little bit exaggerated, of course very few of our students have actually done that. But many are justifiably proud of their achievement in getting into MIT.
Getting in to MIT is a moment of triumph. And MIT, of course, is proud to graduate students who accomplish wonderful things. But this evidence of success notwithstanding, I've long pondered a secret question that I'll discreetly share with you today, privately. Are our alumni successful because they went to MIT? On the first day of my econometrics class most of my students would likely say yes to that. An MIT education is surely the key to a lifetime of success. After all, why else would they have worked so hard to get here? Yet on the last day of my class, and that's an MIT subject 1432, for those thinking of taking it, I think, or I hope, that their answers will have changed.
What sounds like an obvious conclusion is in fact likely to be wrong. The belief that highly selective schools like MIT are the key to success is an illusion of sorts. And my colleagues and I have a name for this pervasive misreading of the data. We call it the [INAUDIBLE] illusion. I've spent my career untangling questions of cause and effect. Messy cause and effect questions.
Many related to education and work. My domain as a labor economist and applied econometrician covers questions like, would you have earned more if you'd waited a little later to start your family? Would you have learned more if you'd gone to an elementary school with smaller classes? Would you have seen less career success if you'd enrolled at a less selective school than MIT? These questions are hard to answer, because they force us to imagine what would have been in a counterfactual scenario, in an alternate reality, where we make different choices. The idea of alternate realities is expressed nicely in Robert Frost poem, "The Road Not Taken." I'm sure you know it.
"Two roads diverged in a wood, and I, I took the one less traveled by, and that has made all the difference." For Frost it's a question of whether taking one road-- there's an image of two roads-- and my grasshopper student. It's a question of whether going-- for example for my students it's a question of whether going to MIT changes their lives in some fundamental way, or whether they're just smart, driven kids, likely to succeed whether they enroll in an elite institution like MIT, or perhaps attend a less selective school, and considerably more affordable school. Maybe their local public University in their home state or country.
We all wonder about questions like this. Questions of what might have been had we taken a different path. But the other road isn't as mysterious as you might think. Although we can't usually say what lies at the end of the road not taken for a particular person, we can often map what would have been for groups of people. And we don't need Doctor Strange's magical powers for this either.
Rather the road not taken is revealed through something just as cool, econometrics. Well, it's just as cool to me. And it is Nobel Prize worthy.
That's the 2021 Nobel Prize poster. And my co-laureates in the bottom right, [INAUDIBLE] and Dave Card. You can see I'm the only one that couldn't find a tie for the purposes of that poster. So what is econometrics? Econometrics is the science that aims to turn abstract economic models into information that people can actually use. The econometric tool kit contains powerful tools designed to answer causal questions involving counterfactual outcomes. In other words, the outcomes that lie at the end of a road not taken.
These econometric insights change how my students think, and maybe they'll change how you think too. Suppose we'd like to map the road not taken by MIT students. Specifically, I want to know if they are more successful by virtue of their choice to attend MIT over a lesser, lower performing school, say Harvard or Stanford. In an ideal world, at least as far as research goes, we run a randomized experiment to find out. Of course, behind the scenes, medical researchers have been doing this for decades in the form of randomized controlled trials, depicted here.
As you've likely noticed, RCTs are nowadays visible as never before. In the spring of 2020, biotech startup Moderna quickly synthesized the candidate COVID vaccine using mRNA technology. Before distributing this they had to establish safety and efficacy of the new vaccine. This they did in an RCT.
First they found some volunteers, 30,000 in fact, including me. That's my investigational drug label. Next they tossed a virtual coin.
Half got the experimental vaccine, and half got a placebo, a fake vaccine. Finally they waited a few months and compared COVID infection rates in the vaccinated treatment group to COVID infection rates in the unvaccinated control group. A colleague of mine married to a physician was shocked to learn that I had volunteered for the trial. Dangerous stuff, she said. At the time Moderna had been working with mRNA technology for over a decade with nothing to show for it. But as we all know, the trial turned out rather well.
Hardly anyone in the vaccinated group got sick. The random assignment of treatment ensures that the experimental and control groups are truly comparable, so the only possible explanation for sharply lower COVID infection rates among the treated is the fact that the treated got vaccinated. This single rigorous study helped change the course of the pandemic and all of our lives. Well, I've long dreamed of an MIT admissions RCT. But so far our admissions office resists my proposal to reject half of our top applicants at random, and then survey everybody years later to see how their lives turned out, and well, that's not so unusual for economists.
We often face the fact that the ideal experiments that we imagine don't square with real people's lives. Sometimes the interventions we want to study are just too disruptive to be doable, so we must be creative instead. When we can't conduct a real RCT we must go out and find one. That may sound fanciful. Randomized trials are highly engineered social experiments.
Why would such a thing be found naturally? Well, as it turns out, natural experiments happen all around us. You just have to know where to look. So you want to know whether military service affects your earnings later in life. For economists like me who study the labor market, that's a pretty important question. The Pentagon is one of the largest employers in the world, and the consequences of military service matter to millions of our veterans and to the many high school students considering military service instead of college. If only we could randomize military service.
Well, remarkably the US military kept very close to this in the 1970s, a period of transition from conscription, a military draft, to today's all-volunteer force. From 1970 to '73, the order in which men were called up was determined by draft lottery numbers from 1 to 365 assigned to birthday. This slide has a picture of the drawing. Men with low draft lottery numbers were called to their local draft boards for possible service. Those with high draft lottery numbers needn't have served if they didn't want to. This natural experiment research design suggests we compare the health of and earnings later in life of men with low draft lottery numbers to these outcomes for men with high numbers.
But it isn't as simple as that. Many called for induction managed to avoid service. Some potential conscripts are disqualified for health reasons. Some are deferred while in college, as you've seen in the movie Woodstock. Meanwhile 20% of men with high draft lottery numbers volunteered for service. So the draft lottery is not a real RCT.
It's messy, and we need an econometric framework to account for that. In 1989, I was thinking about this econometric mess while an assistant professor in the Harvard econ department where I had taken my first academic job. As a grad student I'd made progress on the problem of how to use the draft lottery to estimate causal effects of military service. That was my thesis. But aspects of my thesis work remained open and unsatisfying.
That's when I fortuitously welcomed a new neighbor. [INAUDIBLE] arrived at Harvard in 1990, a young economist like me. Neighbors in Harvard faculty housing, we did our laundry together in the basement of my building. Watching our clothes spin we got to talking about the problem of how to use the Vietnam era draft lottery to estimate average causal effects of military service. Those laundry room conversations and the lifetime of collaboration that followed led to our shared Nobel Prize.
The crux of the matter is that a natural experiment like the draft lottery doesn't reveal causal effects for everyone. The lottery doesn't teach us about service effects on those with deferments that help them avoid the draft. Likewise the lottery teaches nothing about effects on men ineligible for the draft who nevertheless volunteer. The draft lottery reveals causal effects on veterans who were conscripted as a result of having a low lottery number, but wouldn't have served otherwise. [? Pedro ?] and I came up with a mathematical model that formalizes this intuitive point, and can be used to describe the affected group, which we called the group of suppliers.
The basic idea is described on this slide. We'll have a test on this later. The term compliers, the people affected by the lottery, comes from the world of randomized clinical trials.
That's our North Star in the pursuit of causal effects. In many clinical trials some of the subjects randomly assigned to treatment don't end up getting treated. The group that's assigned to treatment and treated as planned is said to comply with the trial's protocol. The average difference in potential outcomes for compliers is called a local average treatment effect, or LATE for short.
[INAUDIBLE] and I debuted LATE and a short theoretical paper published in 1994. The LATE paper, as it's now known, takes the form of a few mathematical theorems, but it applies very concretely to the sort of natural experiments that are my bread and butter. For instance, the LATE theorem explains an important finding in research on the effects of military service.
Other studies, including one by me, have shown that volunteering for service ultimately increases earnings. The LATE theorem explains why estimates based on the draft lottery actually go the other way. Being forced into the military as a conscript reduces earnings by as much as 15% 10 years later. The Theoretical work that [INAUDIBLE] and I did hones the econometric theory behind the instrumental variables or IV method of estimating causal effects. An instrumental variable is a naturally occurring and often indirect source of variation that creates a natural experiment. In the case of the Vietnam study, the instrument is draft eligibility.
IV allows us to capture causal effects of intervention, like military service, that cannot be cleanly randomized in an RCT. And my work with [INAUDIBLE] allows us to look at IV estimates in a new way. Take the effects of child bearing on women's wages. That's another important issue in the labor market.
Does having more children lower women's wages below what their wages would have been otherwise? And if so, for what sorts of women and for how long after birth? We can imagine an RCT that answers such questions. But this is one trial that's likely to remain fanciful. We might instead find an instrumental variable. A mechanism that randomly assigns babies to some families, but not to others. Let's compare families with two children to families with three. Three used to be the norm for American married couples, now it's two.
Where's the randomness here? Of course, parents of two children can choose to have a third, but a few wanting two get third randomly, surprised, though not necessarily delighted, when their second birth is to twins. Twin second births randomly drive sib ship size from two to three, buy one get one free. There's our first instrument for family size. Here's another. You've probably heard that in some countries parents prefer sons to daughters. Not so in the US.
Here are many parents are interested in having a diversified sibling sex portfolio. This is visible in American census data. Mothers of two girls or two boys are more likely to have a third child than are the mothers of a mixed sex sibs ship. Like my mother. After having me, the oldest, and my second born brother Misha, she really wanted a daughter, and that led to the birth of our youngest sibling, my brother Ezra. At which point my mother retired from childbearing.
Since sex-- child sex is randomly assigned at birth, the event of having a same sex sib ship is a natural experiment that changes family size, and this is where it gets really interesting. Conventional wisdom says that having more children damages women's earnings potential but IV estimates show more nuance. Having another child does decrease women's earnings, but the effect of a quasi experimentally delivered birth is much smaller and shorter lived than you might think. That brings us back to the elite illusion. Much of what parents, policymakers, and scientists believe about the world is disproven by a good natural experiment, and the results of these experiments have consequences.
Better information leads to better decisions. That's especially apparent in education. Economists like me believe that education matters greatly for most people.
We're not surprised by the intense public controversy over who gets to attend the best schools. This debate burns hottest today in the context of selected public schools. That is high schools where students take an admissions test to get in.
Bostonians, for example, focus on exam schools like Boston Latin, pictured here. New Yorkers pined for a seat at one of the city's legendary specialized high schools, like Stuyvesant or Bronx Science. Exam schools like these are considered the pinnacle of each city school system. The competition to get in is fierce.
Last year over 23,000 students took the test for New York City specialized schools. Less than one in five were offered a seat. And although anyone can take the test, Black and Hispanic students are much less likely to gain a coveted exam school seat than are Asian and whites. Even though one third of New York City students are Black, last year only eight Black students were offered seats at stratospherically selective Stuyvesant. Not surprisingly, given these results, exam school admissions are contentious.
Yet arguments over who gets an exam school seat ignore a fundamental question. Are exam school seats really worth fighting for? Research my colleagues and I have done answers this question with a surprising but resounding no. The exam school admissions exam gives us another opportunity to design a natural experiment. We zoom in on the group of students with application scores near the cutoff for admission. Some are offered a seat, some aren't.
But on the whole the students just making and just missing the cut off look similar. And the results of these comparisons are striking. Elite high schools are terrific at selecting smart students, but exam schools boost neither learning nor college enrollment for those who attend them.
The average causal effects of exam school attendance are basically zero. That's the elite illusion. Everyone is scrambling for a seat at one of these prestigious schools, but the students who managed to go aren't necessarily better off as a result, and you see that in this figure, which shows that there's no jump in outcomes at the cutoff. So what's the point? All of us, scholars, policymakers, exam school applicants, and parents make too many decisions based on theory rather than empirical evidence.
In this, decision makers are much like the academic economists of yore who valued appealing theory over messy empirical analysis. But the evidence is there. And we don't need an RCT to interpret it. We can instead use the econometric tools that are my stock in trade.
And in domains like education, where we have few opportunities to deploy an RCT, that's the best way to test our assumptions and escape our illusions. Most everyone I meet has a theory about school quality. In the course of my work as a director at MIT's Blueprint Labs, I often meet donors anxious to see us explore their view of what high performing schools should do.
The best I can offer our funders and friends is a proposal. Perhaps we can study that. Conviction, good intentions, aren't enough, not when we have more information and more powerful tools to make sense of it than ever before.
Where does this leave our old friend Robert Frost? The poet at the crossroads? Maybe my students would do well to simply sidestep Frost's dilemma, hitting instead the advice of legendary Major League Baseball slugger, Yogi Berra. "When you come to a fork in the road, take it." For the other path might not be as they imagined it.
The fork might not matter much at all. For my MIT students, this lesson begins in my class, econometric data science. Starting with the most elementary of econometric tools I show my students that, yes, those who attended elite private schools like MIT earn more later in life than those who took a humbler road.
Yet when we start using more sophisticated instruments, the kind that [INAUDIBLE] and I spent our careers [? refining, ?] a surprising new picture comes into focus. Once we look only at students who applied to and were offered seats at elite schools, those in the group who chose cheaper and much less selective public universities earn as much later on as those who opted for the elites. Here too, the elite illusion-- is the elite advantage is illusory. The good news for bright students everywhere is that hard work and ambition trumped the name of the school on your diploma. Your fate is in your hands. You needn't call on Dr. Octopus to pry open
the gates of your dream school. By the end of our semester together, I hope my students will have learned to question conventional wisdom, both as it applies to their own lives and more widely. And I hope you'll be questioners, too. Our paths forward, yours and mine, are illuminated by tough questions and by our openness to surprising answers. That's what will make all the difference. Thank you.
[APPLAUSE] I guess I don't need the iPad. No. Thank you so very much. Albeit with a slightly anti-MIT message there at the end that we're not helping.
It's subtle, it's subtle. Thank you. So I wanted to ask just, before as I it takes me a minute for the questions to roll in, can I just ask you to say a little bit about Blueprint Labs and the work that you guys are doing with Blueprint and you gave us a few of the questions you've looked at, but you start to tell us a little bit more about Blueprint. So that's what I was alluding to. Donors and funders. We run a lab, a social science lab called Blueprint Labs, and we study things related to education and human capital in the labor market.
We do have a lot of research on school reform, alternative types of schools, charter schools, vouchers, different educational approaches, and our work is distinguished by our ability to exploit quasi experimental variation and come up with very convincing estimates of causal effects, like the exam school effects I was discussing. OK, so can I ask questions about you were showing us some of your results on using your randomized controlled trials, or your regression discontinuity to sort of say what is the effect of the Vietnam-- getting draft in the Vietnam War on your earnings. Can you say more about, is earnings the only thing you can get? Or can we talk more about broader contributions of the Vietnam lottery on people's lives and their contributions to society? Well, of course I'm a labor economist, and so I'm interested in earnings. And many people are interested in their earnings.
But scholars, including me, but also others, have used the Vietnam era draft lottery to study the effects of military service on health, on mortality, and on education. You might think education is affected by military service, for example, through the GI Bill, which is one of the most important federal roles in education. OK, and on the elite illusion, sort of say what are the measures that you're showing are not improved by attendance at an elite school, and do you have any-- then what are your policy recommendations given if you're finding that this sort of Boston Latin does not do what people think Boston Latin does, what are the policy implications? So we look-- we've looked at a series of schools, not only Boston Latin, though that's a great example. It's the oldest high school in the United States.
And it's famous for the success of its graduates, including many successful economists. So we look at achievement as measured by test scores, SAT scores, effects on college, whether you go to college, whether you go to a four-year college, how selective a college you go to. And what we see over and over again is that exam schools pick students who are likely to have good outcomes in those domains, but they aren't actually improving outcomes.
It's an example of what econometricians and economists call selection bias. There's-- an elite illusion is a version of selection bias. That there's a process that chooses people who are likely to be successful, and then looking at the data you see that they are successful, but it's not really a causal effect. OK and another I guess advice to parents or grandparents thinking about what they should be doing, do your results-- do you have other randomized trials in education you can talk about to sort of give insights on public schools, versus elite public schools, versus private schools, versus charter schools, or religious schools, or other types of schools? Well, we have a lot of results in all of those areas, though not necessarily time to talk about them today. Generally I tried-- I don't think of myself as a clinician giving individual advice.
I'm interested in overall causal effects on average. It's hard to say what's good for a particular individual. But I do tell parents and friends that they shouldn't worry too much about where their kids go to college. OK. Can I ask, I have some questions coming in about your COVID vaccines and randomized controlled trials. First of all, just one the most voted up question right now is, what did you get? The placebo or the real vaccine? So I got the placebo.
And I didn't get COVID. And-- [INAUDIBLE] Exactly. And the moral there is that there's very little information in the outcomes for an individual. And I try to encourage people to my students and people who read our work not to think about individual stories. The individual stories are often misleading. I got placebo, I didn't get COVID, but I'm-- I believe in vaccines.
OK. And can I ask a follow up question? On COVID vaccine trials are some of the largest trials have been run recently, and they are also imperfect trials in the people-- there are noncompliers and the world was changing as these vaccines were being delivered. Are there any of the insights that you get from-- Well, the Moderna trial is pretty clean, actually. So since I got to be in it, and I was thrilled to be in it, it's sort of a lifetime ambition.
I mean, I wouldn't say the pandemic is worth it just for that. But I was very happy to be part of it. It's the equivalent for our generation of the polio trials in a previous generation. So I like to share that with my students. But the Moderna trial I happen to know is fairly straightforward. There's not a lot of noncompliance.
But there are other therapies that were being piloted in 2020 and '21 that have the compliance issues that our tools are designed to address. So I've questions coming up here about property taxes and educational inequity at the K to 12 level in the United States. Do you have any insights there on do school-- do you believe the school quality differs across high and low tax districts? And what effects those may be having? Well, that's an interesting question. That's very specific about a line of research. There is a line of research about resources and school spending, and economists have contributed greatly to that over the years.
I think mostly that research suggests that resources do matter. That there's a payoff to spending more on public education. So that's a sort of-- I want to distinguish, that's a distinct question from does going to Boston Latin matter.
That's a very different question than, does spending another $1,000 per student matter. OK. And can you-- do in your [? initial ?] work, do you find that these schools have no effect, or do you say are there detrimental effects, or counterproductive effects of going to elite schools? Sometimes we see negative effects, and that's an interesting outcome. The-- so a feature that drives the exam school effects is what's the counterfactual school. So on average, going to an exam school doesn't boost outcomes.
But for a subset of students, and this is in the work of one of our PhD students, [INAUDIBLE],, do for a subset of students whose public school alternative is not very good, exam schools are actually better. On the other hand, we have a set of results for the city of Chicago where a leading public provider is a high performing charter network called the Noble Network, and kids who are diverted from Noble into exam schools by an exam school offer actually see their achievement fall. So it's a negative effect. OK. One more kind of detailed question on your elite skills. Your data suggests that these high performing students, school choice may not matter much.
Do you have a sense of whether elite schools matter for-- what could matter for lower performing or underperforming students? Well, that's related to my earlier comment. So for somebody whose public school counterfactual to an exam school is not very high quality school, it might matter. And that might be correlated with family background.
So I'm kind of giving you on average zero. There would be a sub sample that might benefit from some exam schools. OK, and another question just on education and human capital development. Your findings provide any insight on whether a talent is inherent or a product of societal effects? That's not really in our domain.
OK. And what about-- I mean, let me qualify that, Glenn. So school quality matters. So your outcomes are not entirely in your genes. We see that.
School quality matters, going to a better school boosts achievement, increases the likelihood of going to college, and other scholars have shown that it also increases earnings. OK. And can I just sort of ask, because I guess one final question. Sure. [INAUDIBLE] to talk about other variables other than income and academic achievement, you sort of would think we would look at and effects of schools? Like can we model things like intellectual enjoyment or other broader effects on people? Well, that's hard to measure, enjoyment. A big outcome for us is health.
Education seems to improve health. And of course, that's very important. And I would say being healthy is necessary for all the other good things in life.
So is not worrying about where your next meal is coming from. There are very few poor college graduates. OK. I think we're out of time, so thank you very, very much. Thank you. Thank you all for your questions.
[APPLAUSE] Whitney Espich: Thanks for joining us and for more information on how to connect with the MIT Alumni Association please visit our website.
2022-06-24