The Danger of Survivorship Bias By: Gabriel PotterMBA, AIFA® 2019.03.18

A World War II analogy

Allied bomber planes in World War II would often return to base after a bombing run riddled with holes from enemy aircraft fire.  The Navy’s researchers took thorough measurements of returning aircraft and where they had been hit.  The researchers determined that bombers were returning to base most frequently with damage along the rear wings, midsection, and wingtips.  Therefore, these researchers reasoned that additional armor ought to be placed on those broad sections of the aircraft, as they seemed to attract the most hits.

Columbia statistician Abraham Wald disagreed with this initial assessment.  He argued that armor needed to be placed over the engines and cockpit of allied bombers.  He correctly deduced that planes which were hit in the engines and cockpit never got the chance to return to base whereas damage to the wingtips or midsection were perfectly survivable. 

This is a famous, and rather literal, case of survivorship bias.  Everybody in the previous example wanted the same outcome – greater protection for their planes.  However, the sample set used (i.e. surviving planes) was not complete.  The full data-set (including both surviving and downed planes) might have yielded better analysis from the researchers, but it wasn’t available, and therefore wasn’t considered. 

Survivorship bias in consulting

Survivorship bias is the basis for errors of logic across a variety of fields, but we are surprised how often it is used - both unintentionally and as a willfully exploited tactic - in the fields of investments and consulting.

Let’s create a hypothetical situation.  Imagine you are considering investing your personal money at a large wirehouse consulting firm with local offices.  You walk into the office of John Smith Consulting, an office of nationally known ABC Bank.  John Smith takes you into his office and gives you his best pitch: 

“If you become a client of ABC bank, I want to assure you that you’ll get access to the best investment managers to manage your money.  How do we know this?  Well, at ABC Bank, we have only the best investment managers on our platform.  Our analysts do all the hard work for you when selecting an investment manager.  Here, let me show you which managers are currently on our Premier-Investments list.”

John then passes you a list of the current investment managers.  You immediately notice some of these investment managers are underperforming on a short-term, 1-5 year, basis.  “That is to be expected as investment styles move in-and-out of favor,” John helpfully points out.  However, over the long term, you notice with satisfaction that every investment manager on ABC Bank’s platform does indeed have an excellent long term track record.  Every investment manager, in the long term, has excellent metrics, beating both indexes and peers in returns and risk-adjusted returns. 

The unspoken, implicit promise is that, while you may see some short term underperformance, all you have to do is stick with ABC Bank and their vetted Premier-Investments over the long term and you too will achieve superior results.  It’s a compelling illusion, but it is just that – an illusion.  You might have some follow up questions.  Have you had all of these managers on your platform for the duration of their outperformance, or did ABC add them recently?  Did ABC bank, quite naturally, fire the investment managers who were underperforming in the long term to make room on the list for recent winners?  In fact, ABC could pick a group of random investment managers – with no vetting process at all – and periodically rotate out long-term underperformers to achieve the same result.  The list is a product of survivorship bias and other selection fallacies.  It is quite possible John Smith himself is not aware of the logical error he is making; he may dutifully toe the company line, swapping investments as recommended funds move on-and-off the Premier Investments list.

Survivorship bias in investments

Individual investments – like a mutual fund or ETF – are also targets for manipulation via survivorship bias.  Let’s reconsider the large wirehouse, ABC bank, but this time focus on their underlying mutual fund division, ABC Funds.  ABC funds creates two similar products:  ABC Large Cap Value fund and ABC High Dividend fund.  Over time, it becomes clear that ABC Large Cap Value has a more successful, consistent track record of outperformance while ABC High Dividend fund lags the index and peers.  ABC Funds might like to assert that each of their funds are beating their indices and peers, but they can’t do that while the High Dividend fund remains in existence.  Thus, ABC opts to merge the High Dividend Fund with the Large Cap Value fund.

This action creates several advantages for ABC bank.  First, ABC may retain all of their customers instead of simply closing the product and losing that revenue.  Second, some unsophisticated investors who had previously been enrolled in the ABC High Dividend fund are now receiving fact sheets with more encouraging performance numbers, despite the fact they had not been invested in that product to yield any of these superior results. 

It seems like a potentially deceptive practice.  However, given the proper disclosures, it’s perfectly legal and it happens all the time. 

Final thoughts

Survivorship bias is particularly tricky because, as we have seen, it is entirely possible that the offenders of this logical fallacy may not know they are committing an offense.  Be careful, double-check the analysis, and be willing to ask an expert to validate your conclusions.

  

Disclosures & Disclaimers

 

 

The information contained herein has been obtained from sources that we believe to be reliable, but its accuracy and completeness are not guaranteed.  Westminster Consulting, LLC reserves the right at any time and without notice to change, amend, or cease publishing the information.  It has been prepared solely for informative purposes.  It is made available on an "as is" basis.  Westminster Consulting, LLC does not make any warranty or representation regarding the information.  Without prior written permission from Westminster Consulting, LLC, it may not be reproduced, in whole or in part, in any form. The information in this document is confidential and proprietary to Westminster Consulting, LLC including its business units and may be legally privileged. Any unauthorized review, printing, copying, use or distribution of this document by anyone else is prohibited and may be a criminal offense. Indices mentioned are unmanaged and cannot be invested into directly.  Past Performance does not guarantee future results.

 

Gabriel Potter

Gabriel is a Senior Investment Research Associate at Westminster Consulting, where he is responsible for designing strategic asset allocations and conducts proprietary market research.

An avid writer, Gabriel manages the firm’s blog and has been published in the Journal of Compensation and Benefits,...

More about Gabriel Potter
Sign up for our Newsletter
Sign up for our Newsletter