Stop it! Statistically, an airplane is still the safest way to travel (Yes, even Boeing aircraft).
Base Rate Fallacy: When you hear sensational headlines, ask yourself, “Compared to what?” A VERY DEEP DIVE into our aviation safety record (Hint: It's quite good)
Para una versión en Español oprima aqui(for the spanish version, click here).
For weeks, I have watched as headlines about Boeing aircraft flooded my social media feed. Soon, friends began asking whether they should switch their flights to Airbus and incur penalties. It was an expected reaction. Following the flawed design issues with the MAX 737, which contributed to two catastrophic crashes several years ago, it was only a matter of time before Boeing returned to the news for any mundane issue. We all saw it coming.
However, the reality is much more nuanced, too complex to be adequately addressed by most journalists who prioritize sensationalism to drive traffic to their platforms. In truth, much of the reporting on these events lacks context, often neglecting incidents involving Airbus, the other major commercial aircraft manufacturer, and failing to consider the denominator: the 45,000 flights per day in the US alone. I would call it misleading if it weren't simply egregious.
We call this the base rate fallacy. The base rate fallacy is a cognitive bias that occurs when people ignore or underestimate the base rate (or prior probability) of an event and instead focus on specific information that may not be representative of the overall situation. In simpler terms, it's the tendency to focus on individual case details while disregarding the statistical likelihood of an event happening in the broader context.
Despite the statistics, many people continue to experience a profound sense of anxiety and apprehension that can be difficult to align with the numerical data.
Aviation Safety Is Unparalleled
The friendly skies are extremely safe. In fact, commercial aircraft carriers have close to ZERO fatal incidents, taking into consideration the 34 million flights recorded every year around the world. In 2023, there were only six fatal accidents globally, resulting in 115 deaths on commercial airline flights, according to Flight International's definition. 2022 saw 12 accidents resulting in 229 fatalities. According to Flight Global, no jet airliners were involved in fatal accidents, with the largest aircraft lost being an ATR 72-500 twin-turboprop. Other accidents included smaller commuter planes such as the Beechcraft C99 and Embraer EMB-110, as well as single-engine Cessna Caravans and an Antonov An-26. The most significant loss was the Yeti Airlines ATR 72 crash on January 15, which claimed the lives of 68 passengers and four crew members. 2023 only saw 2 fatal commercial aircraft incidents. Given these numbers one would have to fly daily for 342,000 years in a commercial aircraft to have a 99.99% chance of being involved in a fatal accident.
While Boeing and United Airlines selectively made the news, pilot stories made it to my direct messages filling in the missing parts of the narrative. The same week aircraft headlines peaked in the news, Airbus reported incidents with hydraulic failures, engine oil warnings, flight control malfunction on a Delta flight, and even an American Airlines Airbus landing with the cargo door open. None of those made headlines.
This piece is NOT in defense of Boeing (their company culture HAS eroded) but rather as an offense to fear-mongering tactics that keep us trapped in cycles of misinformation and using our emotional response center rather than our prefrontal cortex, in our decision-making processes. It is also the perfect excuse to discuss statistics or, in this case, their misuse.
Understanding the Base Rate in Transportation
So just how safe are commercial aircraft?
Well, we can look at data provided by the Bureau of Transportation Statistics (BTS). Since transportation safety changes over time, I selected data from roughly the last decade: 2010-2021, excluding anything beyond that as the data is not all available for aviation statistics.
To enhance the visual representation, the data is displayed using a logarithmic scale, where the bar sizes change by factors of 10 rather than linearly.
Examining fatalities alone reveals that land vehicles, namely highway and railroad incidents, account for over 95% of fatalities throughout the 12-year period. The statistic of only 16 deaths in commercial aviation underscores the safety of this mode of transport. The FAA oversees more than 15 million flights (commercial and private) every year, with roughly one-fifth being private flights, meaning that over the last 12 years, there have been over 140+ million commercial flights.
Notably, private planes exhibit a stark contrast with commercial or commuter planes, with fatalities reaching more than 100 times higher with respect to either. Trains and water passenger vessels have significantly lower numbers by comparison.
In reality, while one might be tempted to look at the total number of fatal accidents or fatalities per transportation mode (flight, train ride, car ride), normalizing by number of hours or miles traveled for all passengers provides a more accurate depiction, yet is a tad more complex.
Let’s do the exercise anyway (the analysis continues past this section). We can extract the passenger miles (given in millions) traveled for most modes of transport once again from the Bureau of Transportation Statistics (BTS) since 2010.
The vehicle-miles information wasn’t directly available for private airplanes. However, we do know there have been roughly 3.5 million private flights per year (that is a substantial amount considering each year there are 15 million flights total) and on average 4 passengers per flight (excluding crew). Assuming most private flights travel for roughly 3-5 hours before refueling (about 1000 miles) we can get a rough estimate for the total vehicle miles for the 12-year period. (42.65 million miles X 4 passengers/flight X 1000 miles/flight), resulting in about 170 billion miles traveled, more or less.
If we normalize fatalities by billion passenger miles, we would see once again how commercial aviation safety is vastly superior to any other mode of transportation. Let’s put the values into perspective as the numbers are quite compelling: over the last 12 years (2010-2021) there were over 140 million commercial flights in the U.S., totaling over 7000 billion passenger miles traveled, and despite this staggering volume, there were only 16 fatalities total. Taking into consideration the distance traveled, it is over 1600X riskier to travel by car and notably 10,000X riskier to travel on a motorcycle.
I want to note that due to the heterogeneity of the data across the databases for the Bureau of Transportation Statistics, the normalized estimates for train fatalities per billion passenger miles may not be exact - the data is imperfect, but this is a crude approximation. Also, I wasn’t able to obtain fatalities for ferryboats, so I extracted them from here (I repeated their analysis in this section, but did download all the data to verify).
How did Boeing get here?
The Boeing 737 MAX aircraft experienced two fatal crashes within five months of each other:
Lion Air Flight 610: On October 29, 2018, Lion Air Flight 610, a Boeing 737 MAX 8, crashed into the Java Sea shortly after takeoff from Jakarta, Indonesia. All 189 people on board were killed. The investigation revealed that a faulty sensor, known as the angle of attack (AoA) sensor, provided erroneous data to the aircraft's Maneuvering Characteristics Augmentation System (MCAS), which automatically pushed the aircraft's nose down repeatedly. The pilots struggled to control the aircraft, ultimately leading to the crash.
Ethiopian Airlines Flight 302: On March 10, 2019, Ethiopian Airlines Flight 302, also a Boeing 737 MAX 8, crashed shortly after takeoff from Addis Ababa, Ethiopia, killing all 157 people on board. The flight's trajectory and behavior were similar to that of the Lion Air crash, leading investigators to focus on the MCAS system again. Like in the Lion Air crash, it was determined that erroneous sensor data triggered the MCAS, causing the aircraft's nose to pitch downward uncontrollably.
Following these crashes, the Boeing 737 MAX fleet was grounded worldwide. Investigations into the crashes revealed concerns not only about the MCAS system but also about the aircraft's certification process, pilot training, and oversight by regulatory authorities, highlighting just how complex and vast the failure modes are in catastrophic events.
Headlines versus the Nuanced Truth
While headlines have pivoted around Boeing’s shifting company culture (from engineering-centric to corporate-centric) or around the foregoing narrative of budget cuts in favor of corporate profits, what happened with Boeing is far more complicated. Buckle up (pun not intended).
The Maneuvering Characteristics Augmentation System (MCAS) was introduced on the Boeing 737 MAX to address a specific aerodynamic characteristic of the aircraft related to its larger engines. The 737 MAX features larger and more fuel-efficient engines compared to previous 737 models. These new engines were positioned further forward and higher on the wings to optimize fuel efficiency.
By substituting a larger engine, the new design altered the aircraft's aerodynamic behavior, particularly at high angles of attack (AOA), which could potentially lead to stall conditions. An aircraft's angle of attack (AoA) refers to the angle between the chord line of its wing (an imaginary line connecting the leading and trailing edges of the wing) and the direction of the oncoming airflow.
A stall occurs when the airflow over the wings becomes disrupted to the point where the wing can no longer produce enough lift to support the aircraft's weight. This disruption typically happens when the angle of attack exceeds the wing's critical angle of attack. As the airflow over the wings separates from the surface of the wing, the smooth flow of air is disrupted and reduces the lift generated by the wing. High angles of attack also increase drag. Drag is the force that opposes the aircraft's forward motion. As drag increases, the aircraft's speed decreases, exacerbating the stall condition. It's important to note that stalls can occur at any airspeed and are not solely dependent on the aircraft's altitude.
Early tests conducted in 2012 subjecting the model to high-speed airflow to simulate extreme maneuvers revealed a concerning tendency for the plane's nose to pitch upward during certain maneuvers. Despite unsuccessful attempts to rectify the issue, engineers devised a solution: the Maneuvering Characteristics Augmentation System (MCAS) to mitigate this aerodynamic challenge. This software would activate a powerful control surface at the tail to counteract the nose-up tendency, pushing the nose down by a certain amount.
MCAS was designed to automatically adjust the aircraft's horizontal stabilizer to lower the nose in certain situations where the angle of attack becomes too high, thereby reducing the risk of stalling without direct input from pilots and providing an additional layer of safety.
However, issues arose with the implementation and operation of the MCAS system, particularly related to the reliance on data from a single angle of attack sensor. In both the Lion Air Flight 610 and Ethiopian Airlines Flight 302 crashes, erroneous data from the angle of attack sensor triggered the MCAS to repeatedly push the nose of the aircraft downward, contributing to the accidents.
If you were to add a room to your house, would you tear down the whole thing?
The introduction of MCAS highlights the complex challenges involved in incorporating new technologies and design changes into aircraft. While the system was intended to enhance safety by addressing specific aerodynamic characteristics of the 737 MAX, its implementation and interaction with other systems became a focal point of investigation following the two fatal crashes.
You will hear me repeat the same thing often: catastrophic failures are rarely attributable to a single point of failure. Catastrophic failures result from the intersection of failure modes across different and rather diverse systems, ranging from design and regulatory to behavioral.
Regardless of this, the engineering approach was indeed correct. While many have argued that the whole aircraft should have been redesigned, we only know this in hindsight.
Imagine wanting to add an extra room to your existing home. Would you tear down the whole thing? Unless you had structural stability issues, likely not. Adding a new room without demolishing reduces risks such as compromising structural stability, foundation integrity, exposure to environmental hazards, construction delays, and cost overruns associated with demolition and rebuilding projects. Overall, adding a new room without demolishing offers a practical, cost-effective, and less risky option for expanding living space.
The same can be said for new cars, aircraft, and even off-label use of some drugs. Starting from scratch would introduce many more unknowns and more likely than not, increase risks, costs, and time delays. Thus it would not actually guarantee a safer system.
I summarize the thorough investigation by the Seattle Times (one of the best out there) below:
The original version of MCAS was designed to activate only if two specific sensors detected an extreme maneuver: a high angle of attack and significant G-force. The angle of attack measures the angle between the wing and the oncoming airflow, while G-force is a measure of the plane's acceleration in the vertical direction. The degree of tail movement triggered by MCAS was determined by the angle of attack and the jet's speed.
During the certification process for the 737 MAX, Boeing outlined its MCAS plan in documents submitted to the FAA. According to Boeing's proposal, MCAS would only activate under specific conditions, aiming to counteract potential nose-up pitching moments during high-angle-of-attack situations at high speeds. Boeing limited MCAS's nose-down command to 0.6 degrees from the trimmed position, as specified to safety regulators.
Boeing's goal was to minimize MCAS's impact, with the system's authority restricted to 0.6 degrees. The company incorporated this limit into its FAA system safety review. Boeing's submission included an analysis of possible MCAS failures, categorized as minor, major, or hazardous, based on severity and redundancy requirements.
Boeing's safety assessment concluded that virtually all equipment, including sensors, met the reliability threshold for a major failure, meaning they are reliable enough to meet the “major failure” requirement, which is that the probability of a failure must be less than 1 in 100,000.
A "major failure" is anticipated to result in increased involvement of the cockpit crew rather than serious injuries. Therefore, systems classified as major failures often depend on a single input sensor for operation and don't necessarily call for redundancy.
Boeing evaluated various failure scenarios, such as inadvertent MCAS activation and prolonged operation during a wind-up turn. These were classified as major or hazardous, with the latter considered highly improbable due to the unique flight conditions involved. Despite the original requirement for two triggering factors, Boeing's safety analysis suggested that a single sensor would suffice in all scenarios.
During flight testing in 2016, Boeing made significant alterations to MCAS, prompted by issues specific to low-speed flight conditions. At low speed, a control surface must be deflected more to have the same effect. Engineers expanded the scope and power of MCAS, boosting the MCAS system's low-speed stabilizer nose-down deflection from 0.6 to 2.5 degrees per activation. This allowed just two activations to reach the maximum allowable 4.7-degree deflection. Due to the absence of excessive G-forces at low speeds, the G-force factor was removed as a trigger and they relied on a single angle-of-attack sensor. Since the hazard assessment classified an MCAS failure as "major," redundancy wasn’t required.
The rationale for not utilizing both angle-of-attack sensors despite discarding G-force was to avoid dependence on two sensors, as a failure in either would render the system non-functional. This decision, while affecting safety, also considered airline costs, as system downtime could lead to operational disruptions and financial losses.
Redundancy was minimized to ensure reliability and operational continuity. Boeing's adherence to safety regulations led to the adoption of a single sensor, as deemed acceptable by their safety analysis.
Additional analysis on the new low-speed, higher-authority changes concluded that potential failures at low speed posed a minimal added risk and were not deemed critical to the most crucial phase of flight, higher cruise speeds, thus not requiring revisions to the documents submitted to the FAA.
While the adjustments were substantial, discussions during flight testing about the new MCAS design appeared limited, likely reflective of a company culture that pressured limiting flight testing to avoid project delays.
Boeing's failure analysis did not anticipate the possibility of repeated MCAS activations, effectively granting unlimited authority if pilots did not intervene, moving multiple times in increments of 0.6 or 2.5 depending on the speed. Notably, Boeing never conducted flight tests involving a scenario where a faulty angle-of-attack sensor triggered MCAS independently, relying instead on simulator analysis.
What role did the FAA have in all this?
The FAA (The Federal Aviation Administration) mandates a comprehensive System Safety Assessment (SSA) for each aircraft system, aiming to identify and mitigate functional hazards to meet specified safety targets. Equipment failures, their combinations, exposure times, criticality and independence of failure modes are all considered in the analysis.
In this scenario (as I learned from reddit), the SSA (System Safety Assessment) relied on the longstanding assumption, dating back to the original 737s of the late 1960s, that flight crews would deactivate malfunctioning stabilizer trim actuators, as per their training. Consequently, the impact of a faulty AoA (Angle of Attack) sensor was deemed "minor," and redundancy was deemed unnecessary. As systems grow more complex, the process becomes more challenging due to the introduction of additional functional hazards. Therefore, unless redundancy is essential, its addition may escalate costs and potentially compromise safety too.
Boeing engineers also deemed the absence of redundancy acceptable, as revealed by confidential data reviewed by The Seattle Times, because they assessed the likelihood of a "hazardous" MCAS malfunction to be exceedingly improbable. However, many argue against this, advocating for the inclusion of additional AoA sensors.
Two tragic accidents demonstrated that this assumption was no longer valid. As a result, the system underwent several modifications to ensure safety even if the crew fails to deactivate a malfunctioning stabilizer trim actuator.
Certainly, we benefit from hindsight that the designers lacked. They relied on the FAA's acceptance of the assumption, backed by decades of validation, including simulator tests by test pilots. However, many failures demand prompt crew action for aircraft safety, underscoring the importance of pilot training. The US Airways Flight 1549 incident exemplifies this, where the crew's adherence to training saved lives. Despite this, the software algorithm granting pitch authority to override the crew should have raised concerns for the designers, even then.
The Role of Pilot Training
Boeing's evaluation of the MCAS system's safety assumed almost immediate pilot intervention in response to any issues. However, the pilots' difficulties in controlling their aircraft before the MAX crashes indicate that reassessment of the FAA's three-second guideline for expected pilot response time is necessary.
The adequacy of pilot training for MCAS has become a contentious issue, with some advocating for more rigorous simulator training. Boeing's promise to Southwest Airlines of substantial rebates contingent on the absence of simulator training requirements highlights internal pressures to avoid design changes that could prompt such mandates. Pressure to expedite the certification process, as evidenced by Boeing executives' confidence in avoiding simulator training, as well as the suggestion to the FAA not to include MCAS in the pilot manual underscores the debate over transparency and pilot awareness of critical system functionalities.
Despite these concerns, Boeing secured FAA approval for minimal training on the MAX, omitting MCAS from the training material and manual. After internal discussions, the FAA decided not to include MCAS in the manual, citing the rationale that MCAS functions as a background software code within the flight-control system.
Still others point to the limited training pilots received in Asia relative to the United States, and the lower safety certification standards. It is a fact that Asian carriers and airlines in the Asia-Pacific and Middle East regions experienced too rapid of an expansion to meet soaring passenger demand. In 2014, passenger numbers surged by nearly 6% in Asia-Pacific and a remarkable 13% in the Middle East, as reported by the International Air Transport Association (IATA). With the aviation industry experiencing rapid growth, a notable pilot shortage emerged. Despite the training of new pilots, an experience gap persisted. It is important to note the same is happening here.
Base Rate Fallacy is Hard At Work: Looking at Aviation Safety Data
I wanted to provide further insight into this whole debacle not just to highlight the multiple levels of failures that intersected with Boeing’s current situation, but also to elucidate how headlines without context distort statistics to trigger an emotional response.
I opted to download the safety data from the National Transportation Safety Board, specifically looking at occurrences, incidents and accidents across all aircraft carriers since 2010. I exclude non-scheduled flights in this analysis, as scheduled flights are most comparable to the aircraft making headlines. I made a Google Sheet here.
The quantity of Boeing accidents and incidents related to passenger flights this year appears consistent with those of previous years, spanning at least a decade. While number of incidents is higher with Boeing, there are a lot more commercial Boeing airliners in the U.S. than Airbus. As of 2018, out of the 7,356 aircraft within the U.S. commercial inventory, Boeing manufactured 3,226, accounting for 44 percent, followed by Airbus with 1,558 aircraft (21%). Bombardier produced 1,018 aircraft (13.8%), Embraer manufactured 829 (11.3%), and McDonnell Douglas, now under Boeing, contributed 333 aircraft (4.5 percent). Incidents should be normalized by flight hours or miles (see tables ahead), but just looking at the distribution of events over the share of the types of aircrafts is significant, and there is a clear correlation between number of events reported by the NTSB and the share of aircraft (And, yes the share has changed since then albeit not dramatically)
Of all aviation incidents, seldom are related to Boeing per se. Some issues like ground collisions and turbulence are not specific to Boeing aircraft and can occur across various manufacturers, including Airbus. Phenomena like striking a flock of birds occur irrespective of whether the plane is made by Boeing, Airbus, Embraer, etc. Yet, people often fail to discern between airlines and aircraft manufacturers.
In 2022 the accidents per 100,000 flight hours were roughly .117 (20 accidents for over 17 million flight hours, as seen in Table 1 below). If we look at the distribution of data over the last couple of decades we also see how accidents and accident rates (per million flight hours )have remained rather consistent as well (see Table 2 below and accompanying graph). I placed the raw data from the NTSB here.
In 2024, the average has slightly increased to about three incidents per month, according to NTSB data. Up to this point in 2024, a total of eight individuals have sustained serious injuries on six Boeing planes. These injuries occurred due to turbulence encountered during flight.
We often also see the base rate cognitive distortion when folks make arguments against vaccination (claiming sickness despite a vaccine to argue that vaccines do not work), or when people cherry-pick data without adjusting for the denominators, as is often done by those who drink unpasteurized milk to claim more events of food poisoning occur in those who drink pasteurized products, often omitting that consumers of unpasteurized products represent only 1.6% (cheese) to 3.2% (milk) of the population.
The base rate distortion is also at work when reporting on fatal plane crashes. While these events are almost non-existent, when they do occur they gain our attention, for a variety of reasons. For individuals who already have a fear of flying (aviophobia), hearing about plane crashes can reinforce their anxiety and make them feel unsafe about air travel. Air travel inherently involves placing trust in the pilots, crew, and aircraft systems. Plane crashes serve as a brutal reminder that we as passengers have very little to no control over our safety while flying, which can be quite distressing. Plane crashes also make us wildly confront the fragility of our own lives and the possibility of sudden, unexpected death, evoking existential dread. In light of what happened with Boeing, aviation incidents may further shake people's trust in the safety measures and regulations governing air travel, leading to feelings of uncertainty and insecurity.
However, what many people don't realize is that this distortion leads to an underestimation of the importance of prior probabilities or base rates in decision-making. Instead, we tend to focus too heavily on specific, anecdotal information or characteristics of a case, often overlooking the broader context provided by the base rate. This can result in flawed judgments and decisions, as important statistical information is disregarded or given less weight than it deserves. In essence, people may overlook the likelihood of an event occurring based on general probabilities, leading to biases, errors in reasoning, and unnecessary panic.
Many Systems Intersect for Failures, But Many Systems Also Intersect for Safety
The aviation industry is exceptionally safe due to a combination of stringent regulations, advanced technology, rigorous training, and a strong safety culture. Aviation authorities worldwide, such as the Federal Aviation Administration (FAA) in the United States and the European Union Aviation Safety Agency (EASA), enforce strict regulations governing aircraft design, maintenance, operations, and pilot training.
Aircraft are equipped with sophisticated and redundant (where needed) technology designed to enhance safety, while airlines and maintenance providers adhere to rigorous maintenance procedures outlined by regulatory authorities and aircraft manufacturers. Regular inspections, preventive maintenance, and component replacement schedules are followed to ensure the airworthiness of aircraft.
Pilots, air traffic controllers, maintenance technicians, and other aviation personnel undergo extensive training and recurrent proficiency checks to maintain high standards of competence and situational awareness. In the event of accidents or incidents, thorough investigations are conducted to determine the root causes and contributing factors. Lessons learned from these investigations lead to safety enhancements, including improvements in technology, procedures, and training.
Aviation stakeholders employ systematic risk management approaches, such as Safety Management Systems (SMS), to identify, assess, and mitigate risks proactively. This proactive approach helps prevent accidents and incidents before they occur. The aviation industry maintains a high level of safety through various rigorous processes, one of which is Failure Mode and Effects Analysis (FMEA). FMEA is a systematic approach used to identify and prioritize potential failure modes within a system, assessing their potential effects and implementing measures to mitigate risks.
Overall, the combination of robust regulatory oversight, technological advancements, comprehensive training, a strong safety culture, ongoing learning, international collaboration, and proactive risk management measures collectively contribute to the exceptional safety record of the aviation industry. The risk of a fatal event will never be ZERO (no such thing), but it is sure darn close!