How to Overcome Survivorship Bias

A few months ago I shared a story and post on Facebook about survivorship bias and was amazed how often it was liked and shared. It also highlights the risk of survivorship bias in the gaming and gambling space. The image and blurb told the story how the navy analyzed aircraft that had been damaged and based future armament decisions on where they had received battle damage, thus they were going to increase the armor on the wingtips, central body and elevators. These were the areas that showed the most bullet holes.

One statistician, Abraham Wald, the founder of statistical sequential analysis, however fortuitously stopped this misguided effort. According to Wikipedia, “ Wald made the assumption that damage must be more uniformly distributed and that the aircraft that did return or show up in the samples were hit in the less vulnerable parts. Wald noted that the study only considered the aircraft that had survived their missions—the bombers that had been shot down were not present for the damage assessment. The holes in the returning aircraft, then, represented areas where a bomber could take damage and still return home safely. Wald proposed that the Navy instead reinforce the areas where the returning aircraft were unscathed, since those were the areas that, if hit, would cause the plane to be lost.”

Survivorship bias is universal

Survivorship bias occurs everywhere. If you are a poker player, you may have a hand of three of clubs, eight of clubs, eight of diamonds, queen of hearts and ace of spades. The odds of that particular configuration are about three million to one, but as economist Gary Smith writes in Standard Deviations, “after I look at the cards, the probability of having these five cards is 1, not 1 in 3 million.”

Another example would be professional basketball. If you look at the best professional basketball players, a high percentage never went to university for more than one year. From this information, you (or your teen son) may infer the best path to the NBA is going to university for one year or less. The reality is that there are millions (if not billions) of people who went to university for less than a year and never played in the NBA (or even the G League). The LeBron Jameses and DeAndre Aytons are likely in the NBA despite playing less than a year in college due to their great skill, not because they did not go to university for more than a year.

As an investor, survivorship bias is the tendency to view the fund performance of existing funds in the market as a representative comprehensive sample. Survivorship bias can result in the overestimation of historical performance and general attributes of a fund.

In the business world, you may go to a Crossfit gym that is packed with the owner making a great living. You decide to leave your day job and replicate his success. What you did not see is the hundreds of Crossfit gyms that are not profitable and have closed.

The problem exists in gaming

You often see survivorship bias in the gaming and gambling space. People will look at a successful product and select a couple of features or mechanics they believe have driven the success. They then try to replicate it and fail miserably, only to then wonder why the strategy did not work for them. What they fail to analyze is the many failed games (for every success there are at least 8-10 failures) because they do not even know they exist. The failed games may have had more of the feature you are replicating. Getting a star like Kim Kardashian is a great idea if you only look at Kim Kardashian: Hollywood, but if you look at the hundreds of other IPs that have failed your course of action might be very different.

Survivorship bias can also lend its ugly head when building a VIP program. You talk to your VIPs and analyze their behavior, thus building a program that reinforces what they like about the game. What you neglect, however, is that other non-existent features might have created even more VIPs.

In the gambling space, you may look at a new blackjack variant that is doing great and build a strategy around creating new variants of classic games. What you did not see is all the games based on new variants that have failed.

Avoiding survivorship bias

Looking simply at successes, or even failures, leads to bad decision making. When looking at examples in your industry or other industries, you need to seek out both the successes and failures. With the failures, you need to make sure they are the failures (not the airplanes that returned shot up but the ones that were destroyed). You also should not use others successes or failures as a short cut to robust strategy decisions. You need to analyze the market, understand your strengths-weaknesses-opportunities-threats (SWOT) and do a blue ocean analysis. Only then will you build a strategy that optimizes your likelihood for success.

Key takeaways

  • In WW2, by analyzing surviving aircraft the US navy almost made a critical mistake in adding armor to future airplanes. The planes that returned were actually survivors, while it was the planes that were destroyed that showed where on the plane was the greatest need for new armor. This phenomenon is called survivorship bias.
  • This bias extends into the gaming and gambling space, as companies analyze what has worked in successful games but do not know if it also failed (perhaps to a greater degree) in products that no longer exist.
  • Rather than just looking at survivors or winners to drive your strategy, you should do a full SWOT and Blue Ocean analysis, that is the strongest long-term recipe to optimize your odds of success.

Univ of Illinois COVID Saliva Test Could Change the Game for K-12 Schools

Inside a tent outside the National Center for Supercomputing Applications on the University of Illinois campus, Uni High teacher Joel Beesley swiped his university ID card and was handed a small vial.

He drooled into the vial, and later in the day, he found out he had tested negative for COVID-19. The ability to take a quick, cheap test just blocks away from their school might just be the reason Uni High students will be able to return to the building sometime this fall.

Uni High will begin the year with remote learning, but it may eventually move to a hybrid model of in-person and online learning. To put it simply, the test developed down the street would be the only reason that’s possible.

“There’s no way we would even try to have hybrid school without the capacity to have weekly testing,” Uni High Director Elizabeth Majerus said. “The ease of testing is super valuable, not having to go far away. The fact that it’s very simple to do is very helpful.”

After the UI accomplishes its gargantuan goal of testing 10,000 students and faculty on campus every day this fall, its saliva test could theoretically offer a lifeline to K-12 schools in Champaign-Urbana and beyond.

Preliminary conversations about such a process began a few weeks ago. At the Champaign school board’s July 13 meeting, Superintendent Susan Zola said that UI chemistry Professor Martin Burke, one of the test’s originators, had reached out to her earlier that week to talk about a collaboration.

“They reached out, and we have some cabinet-level staff that are working with them,” Zola said the next day. “With a community that’s rich in resources and expertise and technology, we’re excited that we might be able to partner with the university on a very quick and efficient test that would give more data than we would normally have around teachers and staff if we were actually able to administer that kind of a test.”

To get to that point, though, won’t be simple.

To Anita Ung, who studied biostatistics and epidemiology at Johns Hopkins University, testing in elementary schools could be a game-changer for multiple reasons.

For one, it’s a way to democratize testing. Currently, test-positivity rates in Champaign County are much higher for Black and Hispanic residents than White ones. As of Thursday, 13.6 percent of tests taken by Hispanic residents and 4.2 percent of tests taken by Black residents came back positive. Test-positivity among White residents was 1.3 percent.

“If they had primary, free, easy access through schools, it would be a way to equalize the pandemic in our town,” Ung told the Champaign school board at that July 13 meeting. “By having widespread testing throughout schools, you guys will have infinitely more choices.” Teachers wary of going back to the classroom would suddenly have another line of defense.

“I think it would give (teachers and staff) confidence in lock-step with other safety measures, with other safety protocols being explicit,” said Mike Sitch, vice president of the Champaign Federation of Teachers union. “If we’re going to get a grip on the pandemic, we have to know where the disease is. And the only way to know where the disease is, is to test and monitor.

Implementing widespread testing in schools, though, likely won’t happen quickly.

UI spokeswoman Robin Kaler said it’s too early to answer questions as to when the test might be available for broader community use. She did say, though, that the UI’s committee in charge of implementing testing is “working tirelessly to make it possible to share the test beyond our campus community as quickly as humanly possible.”

The test offers advantages that could make it viable for schools. Because it bypasses one of the most time-intensive steps in the normal testing process and doesn’t require the use of nasal swabs, it’s quicker and easier to administer. It’s also cheaper. A paper published by UI researchers estimated the cost at $10 per test, and that price could go down if pool testing is used.

Even if enough tests were manufactured, they need further regulatory approval before being used by the broader community. During an appearance July 20 in Urbana, Gov. J.B. Pritzker said that’s something he and his staff are working on.

“We’re working very closely with the U of I to get that fast-track authorization from the FDA,” Pritzker said. “There’s still work being done on it, and they have to manufacture enough of these tests so that they’re widely available. I’ve talked to some of the best epidemiologists in the world, some of them right here in Illinois, and they say that these saliva tests have real promise. So I’m relying upon the science that U of I is doing along with epidemiologists, and that if we can do that, it helps us expand testing across the state even more than we’ve done.

“I have to say, so far, the evidence (on the efficacy of the test) seems pretty good, but there’s still a little bit to go.”

Of course, a situation like Champaign’s is far different from Uni High.

Tests would theoretically have to be delivered back and forth between all of its schools and a testing center if it plans to test students. And regular testing isn’t a reason to throw vigilance out the window cautioned Champaign-Urbana Public Health District Administrator Julie Pryde.

“I think U of I has been very amenable to all situations, but the logistics of testing every single schoolkid in the entire county, I don’t think anyone has addressed that yet,” Pryde said. “There are costs associated with that and there would be the logistics of getting it done and getting the samples out to the school.

“There’s certainly widespread testing available in our community, but these tests are only a point in time. You’re only saying that you’re not positive, and again, (nasal-swab tests) have a 20 percent false-negative rate, so you’re not even 100 percent sure saying you’re negative.”

For the beginning of the year, Champaign and Urbana schools will likely be left without in-school testing, although free testing is still available around town.

How the saliva test plays out on campus when students return to class on Aug. 24 will likely have a large hand in when and if the test is more widely distributed. While Champaign school board president Amy Armstrong said that collaborating with the UI is “a bridge to cross when we come to it” and still seemed far down the road, the upside could be tremendous.

“We could be a model,” Armstrong said. “We could provide data to the rest of the country.”

❖ ❖ ❖ ❖

Beesley hadn’t taken the nasal-swab test that’s offered around town, but he’s heard about the discomfort of sticking a swab deep into each nostril for 15 seconds.

The saliva test was painless.

“It was very simple and very well prepared,” he said, adding that he’s wary of false-negative results. “I would love to be able to be back in the classroom with the kids, and I hope this helps facilitate it.”

For children, the lack of a swab alone may make it possible to self-administer.

“It would be logistically impossible to swab (10,000) kids,” Ung said. “However, the cheaper tests, and one where kids can literally drool into a test tube, it’s way, way easier and totally in the realm of possibility.”

While it wouldn’t mean a return to packed hallways and interactive classrooms, a widespread, quick, accurate test for students could change the frame of mind for teachers and administrators. It could also allow more kids to return to school and more parents to get back to work.

For now, though, it’s still a theoretical solution as society struggles to inch back to normal.

Creativity may be what drives you to be innovative, but you’ll need to be brave as well.

This image is a graphic recording (or sketchnotes) I created from a brief talk by Trish Martinelli about the personal journey of starting to work in creative ways that leads to being a full-time innovator. It doesn’t matter what field you’re in, or the kind of work you do, it matters that you overcome both the “self-talk” of doubt that comes when you go against the tide and, perhaps even more importantly, when you can stand your ground to face un-tested opposition to your innovative work. Bravery is a key strength needed for innovators to use and for innovation to happen, particularly to counter resistance to change in business and in life. Design thinking may sound like a classroom exercise where cool heads prevail, but corporate politics can poison the smallest step towards something different. And shifting a culture through Change Management usually doesn’t sound like there are going to be more fun office parties, corporate swag or the latest portable device to make life wonderful for those below the C-Suite.

Whether or not your kind of innovation work isn’t a product or service, but involves changing a process, re-organizing a business, or changing the business culture, you most certainly will be challenged with questions as direct as, “Who authorized you to have ideas?” or other ways of creating an “US vs THEM” situation. Will you be pitted in the middle? Reframe the concept of being innovative as something that everyone has input and a stake in, so the ownership (and celebration of success) becomes, “We all though the world was flat, but look, together, we discovered it’s round!”.
Find your champions, allies and support. Be wary of bad blood you may cause by not considering the politics and culture you are working in and use emotional intelligence to address these situations, and stay brave, even if your idea fails. After all, the process of creativity doesn’t produce a masterpiece every time, but repeated efforts will exercise your strengths, test your bravery, and, with the right audience, right time and right venue, will change the game for all!