Technical debt. It’s a term most people have never heard of. But over the holiday season of 2022, thousands of air travelers became personally acquainted, much to their dismay, with the disastrous impact a failure to modernize critical applications and eliminate technical debt can have on companies and their customers.
Starting on December 2, 2022, Southwest Airlines was forced to cancel almost 17,000 flights, shutting down two-thirds of its operations during one of the busiest travel seasons of the year. How could such a catastrophe happen?
The Southwest Airlines Shutdown Debacle
Year-end winter storms covered much of the country with snow, ice, bitter cold, and high winds, forcing all the nation’s major airlines to scramble to adjust their flight schedules and aircrew assignments. Most did so without inflicting severe inconveniences on their passengers. But the story at Southwest was different. The carrier had to cancel 59% of its flights while other major airlines canceled only 3% of theirs.
The difference was software. As Matt Ashare, a reporter writing for CIO Dive succinctly put it,
“Outdated technology was at the heart of the Southwest meltdown.”
Southwest manages its flight and crew scheduling using an application called SkySolver, which the company admits is nearing its end of life. This system first went into service in 2004, the same year that Facebook and Gmail were introduced.
But unlike those franchises, which are still going strong, Southwest failed to keep its SkySolver implementation up to date as airline flight volumes surged 69% between 2004 and 2019. Although Southwest’s version of SkySolver was specifically designed to handle aircrew scheduling issues, when severe weather caused nationwide flight disruptions the system was overwhelmed. As a result, when flight schedules were scrambled aircrew members were forced to resort to manual procedures to inform the airline of their whereabouts and get reassigned.
With 20,000 frontline employees trying to coordinate their activities through phone calls, text messages, and emails, Southwest found itself unable to track the whereabouts of its pilots and flight attendants and match them with planes. It had no choice but to shut down most of its operations.
How Technical Debt Shut Southwest Airlines Down
The Southwest shutdown provides a classic case study of the impact technical debt can have on a company that neglects to deal with it in a timely fashion.
What is technical debt? The term was coined by computer scientist Ward Cunningham in 1992 to describe what happens when software teams take shortcuts that seem expedient in the short term, but that result in higher remediation costs in the future.
That aptly describes what happened at Southwest. The company grew quickly and was considered a forward-looking innovator in the airline industry, particularly in the area of customer experience. And it wasn’t adverse to investing in its technology. Earlier in 2022 the company announced a plan to spend $2 billion on customer experience upgrades.
The problem was that Southwest focused so intently on investment priorities that seemed to promise immediate earnings rewards, such as customer experience improvements, that it failed to address the needs of the software that ran its own internal operations. Ironically, that choice resulted in what’s probably the worst customer experience fiasco in company history.
It’s not that Southwest was unaware that its crew scheduling application was in desperate need of attention. The pilots union had been loudly complaining about the issue since 2015. In fact, in 2016 the pilots and aircraft mechanics unions went so far as to proclaim votes of no confidence in then-CEO Gary Kelly due to his “inability to prioritize the expenditure of record-breaking revenues toward investments in critically outdated IT infrastructure and flight operations.”
Christopher Gilchrist, a Principle Analyst at Forrester sums up the company’s problem this way:
“Southwest managed its technology portfolio as a near-term cost instead of a long-running investment driving growth. This misguided approach resulted in an unmanageable level of technical debt exposing Southwest in the most public way possible — the failure to deliver an acceptable customer experience.”
Because of its failure to address technical debt in its crew scheduling software, Southwest took an immense hit to both its reputation and, with total losses from the disruption ranging upwards of $1 billion, to its bottom line. Even worse, it probably lost some customers it will never regain because they no longer trust Southwest’s ability to smoothly handle potentially disruptive events.
Even the FAA Isn’t Immune to Technical Debt-Related Glitches
The U.S. government agency that oversees airlines, the Federal Aviation Administration (FAA), experienced its own embarrassing shutdown of flights just weeks after the Southwest debacle.
Because of a failure in its 30-year-old Notice to Air Missions (NOTAM) system, which provides critical flight safety and other information to pilots, the FAA was forced to cancel departures nationwide for a short period of time. Geoff Freeman, president and chief executive of the U.S. Travel Association issued a statement saying,
“Today’s FAA catastrophic system failure is a clear sign that America’s transportation network desperately needs significant upgrades.”
Although the agency started modernizing NOTAM in 2019, that effort won’t be completed until 2030.
Related: The Cost of Technical Debt and What It Can Mean for Your Business
No Organization Is Safe As Long As It Ignores Technical Debt
These cases illustrate a fundamental reality that every business leader should be aware of: any organization that depends on software for its operations but neglects to deal with its technical debt makes itself vulnerable to being suddenly pitched into a crisis that can ruin both its reputation and its bottom line. If you want to shield your company from the potential of suffering similar disruptions, you need to be proactive about dealing with technical debt.
There’s an old saying that goes, “If it ain’t broke, don’t fix it.” That, apparently, was the attitude of executives at Southwest concerning their flight crew scheduling software. Although flight crews complained about the SkySolver system, the software’s deficiencies didn’t seem to be having any direct negative impact on customer experience. So dealing with the application’s all-too-evident technical debt issue remained low on the company’s priority list.
But failing to invest in eliminating technical debt from critical systems because those systems seem to be working acceptably at the moment is a high-stakes gamble. As CIO Dive’s Matt Ashare says,
“The bill for tech debt rarely arrives on a good day… Systems tend to fail when stressed, not when conditions are optimal. Waiting for a bad situation to pay down technical debt is a high-risk strategy.”
Business leaders should also consider that the longer technical debt is allowed to remain in their systems, the greater the cost of fixing it when that task finally becomes unavoidable.
For example, Southwest now says that it’s committed to upgrading SkySolver. But the software’s current vendor, GE Flight Efficiency Services, says there have been eight update releases in just the last year. That means that Southwest is presumably at least eight releases behind. And to make matters worse, SkySolver is an off-the-shelf package that each airline optimizes for its own operations. Integrating those eight or more upgrades with Southwest’s previous modifications is almost certain to be a time-consuming, costly, and risky endeavor.
Related: Eliminating Technical Debt: Where to Start?
What About Your Company’s Technical Debt?
Are your company’s systems burdened with a load of technical debt? Unless you don’t depend on software at all, or are already proactive in addressing and reducing technical debt, the answer to that question is very likely, “yes.”
Your biggest hindrance in dealing with technical debt may well be simple inertia. Remember that technical debt continues to grow as long as it’s in place, and so will the difficulty and cost of fixing it. If you wait until some sudden, urgent, and possibly very public crisis, such as the one Southwest had to suffer through, forces you to address your technical debt issue, both the costs and the risks of fixing the problem will multiply.
One thing to remember is that even recently created software may have technical debt issues if developers cut corners to get it released more quickly. As William Aimone, Managing Director at Trenegy explains,
“Technical debt is the result of issues that accrue over time due to previously rushed or poorly executed IT projects. Teams often implement an easy or quick IT solution to save money, get something released quickly, or meet deadlines, but it always comes back to bite.”
So, you shouldn’t take it for granted that because the software you depend on has all its updates installed, it must be free of technical debt.
Getting Started
Dealing proactively with technical debt needs to be a continuous best practice and become part of your development culture. To be successful, you’ll need both the right expertise and the right tools. Most businesses simply haven’t applied the time and resources to address technical debt and can benefit from new skills, tools, and guidance.
A good first step in dealing with your technical debt issue would be to consult with an expert partner that can come alongside you and help guide you on the journey toward freedom from technical debt.
vFunction not only provides experience and expertise in dealing with technical debt but an advanced application modernization platform that can help you assess where you stand with regard to technical debt. It can then substantially automate the process of refactoring your applications to modernize them and eliminate technical debt, propelling your organization towards a more efficient and competitive future.
If you’d like to know more about how you can deal with your organization’s technical debt, contact vFunction today to see how we can help.
