Tech Debt Isn't a Burden, It's a Strategic Lever for Success

How to Not Manage Tech Debt

The definition of tech debt is historical work that, when viewed through today's lens, creates problems with functionality, stability, or speed.

In this section, we cover 4 underlying assumptions that cause mismanagement of tech debt:

• ☠️ Assumption #1: Tech debt = bad
• 🕸️ Assumption #2: All tech debt = complicated work
• 📈 Assumption #3: Tech debt ≠ Product work
• 👨‍👨‍👧‍👦 Assumption #4: Individual pain = Organizational pain

☠️ Assumption #1: Tech Debt = Bad

There's no denying that tech debt has a bad reputation. One of the puzzles with tackling tech debt is that the type of tech debt you face can vary greatly — a small experiment rollout could be harmless in nature, but a product redesign could be hairy with dependencies on several services. Despite this, most tech debt is classified under a single bucket of 'bad' or 'burdensome' because of the unknowns. Some examples of these unknowns include the business impact could be missing/unclear/hard to articulate or the engineering time it will take is still undefined in size.

Now, to counter the tech debt = bad assumption:

• All tech debt should not be classified into a single bucket of 'bad.' Classifying all tech debt as the same makes it sound monotone to prioritize fixing because it's all (seemingly) the same problem. It would be like classifying all product work as "features." It's much better to break the tech debt bucket apart by naming and sizing the pain, so each project stands on its own and has an opportunity to be worked on. We'll talk about naming and sizing later in the article.
• Some tech debt is good to have and every team needs to have tech debt. This is necessary because it indicates the team is not blindly focusing on tackling tech debt, but also emphasizing core business areas (like new product development, experimentation, partnership support, etc). Every successful company has scaled their business while accumulating debt. Prioritizing technical decisions over business decisions often means the org won't live to see their own growth.

Specific example from Twitter on tech debt being good to have, from a strategic perspective:

• From their founding in 2006 to 2014, Twitter used publicly available storage systems to scale their product and platform
• Instead of building an in-house system early on, they relied and contributed to open source databases. This was likely because teams were moving so fast on other business-critical priorities (acquiring users, shipping features, monetization, IPO prep).
• Come 2014, Twitter announced Manhattan, their next gen distributed database to serve millions of queries per second with very low latency in a real time environment.
• By putting off the investment (and accumulating a variety of tech debt), Twitter made significant strides in their first 8 years of operating before reaching a point where they needed Manhattan to support the next stage of growth from a core storage perspective.

"On my team at Headspace, we ran experiments simultaneously or in quick succession. The devs and PMs would together make the trade off decision to pause cleaning up experiments. We essentially cut corners to launch experiments and would later set aside a week to clean up a bunch of dead code. I actually viewed this tech debt as positive because: [1] it allowed engineers to focus on cleaning up and maintenance, without context switching in and out of new development work, and [2] it gave the team optionality to change our rollout decision, if the latest experiment results came out that gave us some new info."



Keya Patel (OIR at Reforge, Ex-Director of Product Growth at Headspace)

Questions to ask to challenge Assumption #1 Tech debt = bad:

• What can we gain by accumulating this tech debt now and addressing it next month?
• Under what circumstances would the executive team be interested in this area of tech debt? What information could I pass to the CTO to help them pitch it effectively?
• How can this initiative ladder into the larger company vision or strategic direction?

Unlock a lesson from the Reforge Product Strategy program. In “Managing Debt”, gain a deeper understanding of how to think through, categorize, and manage the technical debt that accumulates in the production lifecycle.

🕸️ Assumption #2: All Tech Debt = Complicated Work

As with any piece of challenging work, not just tech debt, there are several ways to handle complexity. With tech debt specifically, there are ways to approach defined and undefined work.

DEFINED = the work clearly has a start and an end.

• The path to the end might be painless or arduous (depending on the type and size of tech debt), but there is an explicit conclusion and way to get there.
• Anytime tech debt can be defined, this assumption is overthrown.

UNDEFINED = the work has a start, but the end is unclear.

• This is harder to solve for and manage expectations around, compared to 'in the box' defined tech debt.
• As discussed in-depth in Reforge's Scaling Product Delivery program, in these situations it's quite possible to move to a more defined, less complex state by frontloading certain work
• Work that can be frontloaded includes: defining the problem, scoping all potential solutions at a high level, and figuring out how/when to actually make a fix or implement. Note that frontloading does not involve any execution as yet, but instead helps to make sense of potential ways forward.
• For instance, the team can set interim milestones to reduce tech debt complexity. Or, the team can move from undefined to defined with access to additional historical information or spending heads-down time creating a proposal to address the work.

To counter the tech debt = complicated assumption:

• Be clear about if the tech debt at hand is defined or undefined. If it's defined, understand the expected number of hours or days it would take to complete the work and give some buffer time. If it's undefined, list out as many unclear areas as possible to illustrate why the work is complex and won't have a definite end date — then communicate it to the stakeholders involved to get input on best ways to move forward.
• Make undefined tech debt less complex by breaking it into segments of easier-to-digest work. When you move from large swaths of complexity to smaller chunks of complicated but doable work, a team can be more motivated to tackle pieces of this tech debt over a defined period of a sprint or quarter.

"We rewrote the machine learning data systems at Credit Karma 3 times in 5 years. Every time it was rewritten, the tech lead was confident it was the last rewrite for a long time. In the moment, they made the best decision they could by treating the rewrite as defined work that needed to get done. But, if you think about it, the work was undefined because companies and data systems grow and change in ways you never expect. If we actually treated it as undefined and evolving work, then it may have been easier to come in with fresh ideas since there wasn't a solid finish line."



Matt Greenberg (CTO at Reforge, Ex-VP Engineering at Credit Karma)

Questions to ask to challenge Assumption #2 All tech debt = complicated work:

• What assumptions am I making about the system that might be incorrect?
• If I brought in someone with more experience or familiarity with this area, what are the top questions I should ask them?
• Does my team have the right people and knowledge needed to do this work? If not, how would I consider rearranging info and when might be the ideal time to address this?
• What's the best way to break down this piece of tech debt to someone who doesn't understand the complexities at all?

📈 Assumption #3: Tech Debt ≠ Product Work

Organizations are clear about how the product work they undertake will advance company metrics — they tie each experiment, feature, or rehaul to core business goals or OKR improvements. This level of clarity is usually missing for tech debt and it's therefore incorrectly decoupled from product work. Addressing tech debt at the right times can be mission-critical to advance growth, even if it's not immediately quantifiable. It can help improve user experience with software and/or help the company unlock future new features.

To counter the tech debt ≠ product work assumption:

• Even if a particular area of tech debt does not directly tie to metrics, think broadly about what user or product experiences it might aid down the road. Equally emphasizing these user and product benefits, instead of just spotlighting the tech benefits, will facilitate others to paint a picture of why the team needs to care about this work.
• Allocate time to verify business leads and tech leads are on the same page. This can take place as biweekly conversations across product and tech leads, a joint Slack channel to show areas of tech debt and implications, or simply inviting appropriate tech reps into quarterly roadmap previews. More cohesion between tech and product allows for opportunities to understand where there might be lulls in new experiments to address tech debt or how to incorporate the top 2 infrastructure-related asks from the VP of Eng.

"At Airbnb, a big reason why we initiated a massive refactoring of our entire tech stack (10M+ lines of code) around a Services Oriented Architecture was to enable a platform that could launch multiple other businesses. Our early conversations revolved around developer productivity, team morale, and other tech-centered reasons to invest. We realized the refactor was also significant to unlock business lines, and the work was ultimately listed as one of the top six company priorities in the S-1 filing before going public."



Zainab Ghadiyali (OIR at Reforge, Ex-Head of Product on platform at Airbnb)

Questions to ask to challenge Assumption #3 Tech debt ≠ Product work:

• What is the single most important reason for the work to happen now?
• Who all needs to understand the impact? Why should they care about this?
• What is it that I want to say? Does this match what stakeholders would want to hear? If not, how can I address their concerns?
• What do I / the team consider to be a reasonable versus a great outcome for this work?
• Am I over-promising the expected outcome? Can we at all split the outcome into good versus great, to level set expectations?

👨‍👨‍👧‍👦 Assumption #4: Individual Pain = Organizational Pain

Engineers, or others close to a piece of tech debt, will regularly reiterate that piece of tech debt as personally painful for them to deal with. They may struggle during QA around certain pieces of prehistoric code or altogether want to quit their job because the team is taking too long to fully adopt a new programming language. The employee might feel like it's starting to become the end of the world for them, while the rest of the organization probably doesn't feel the same pain (if they even notice at all). This is especially common for people early in their career, where their pain point seems like the most critical thing, due to a lack of broader strategic context.

To counter the individual pain = organization pain assumption:

• When a someone comes in with a high priority area of tech debt, take a moment to take stock on if this is an individual or org level pain point. Usually org level pain points will directly impact the customer or business in some way.
• Think through the lens of someone who has more org context than you do. You can even pitch the pain point to someone on a different team. If your individual perspective holds up in broader contexts, then continue to surface it with backing from other affected peers and teams.

“Early in my career, my team and I were preparing for a major customer demo that could enable a huge sale. We cut some corners and were working specifically to enable the demo, with time being of the essence. An engineer from another team made a massive change to our codebase the weekend before the demo. They enabled a new capability and they were sure it would be a moment of delight for us, but instead we were deeply stressed out and rolled it back. The moment taught me context matters and my top priority wasn't everyone else's top priority."



Matt Greenberg (CTO at Reforge, Ex-VP Engineering at Credit Karma)

Questions to ask to challenge Assumption #4 Individual pain = Organizational pain:

• How many surface areas or teams can benefit from classifying and resolving this tech debt?
• When has the company recognized or rewarded individuals who took on tech debt work? What type of tech debt was it and why was it needed at that time? Can I talk with that person about how they positioned the work?
• Does my engineering manager understand the value of tech debt work? Would they advocate for my contributions to this specific piece of work during performance reviews?

The 6 Types of Technical Debt

Tech debt often falls under one enormous umbrella term that includes everything from latency speeds, to security vulnerabilities, to refactoring, to much more. Instead of the umbrella term, you need to understand the different types of technical debt you might be taking on to use it is a strategy lever. This classification will help individuals across the organization better understand what type of tech debt is at hand and what's involved, as opposed to talking ambiguously about 'tech debt.'

The 6 key types of technical debt that teams encounter are:

• Maintenance debt
• Developer efficiency debt
• Stability debt
• Security debt
• Technical product debt
• Decision debt

Note that a piece of tech debt can fall across more than a single category, making it a potentially more critical piece of work to address.

Maintenance Debt

What this is = When teams don't keep up with updates to tech work. This includes not deleting dead code at the right time after an experiment launch / feature rollout / unshipping event, updating libraries, commenting on pieces of code for context, and documenting implementation decisions.

An example of maintenance debt...

• Not cleaning up months old code for a 'log in with Spotify' account creation experiment
• Leads a new engineer to incorporate Spotify login into his technical scoping and documentation for his first round of feature work, since he believed the experiment was launched
• In reality, the original experiment had negative results and the team didn't have time to clean up their code because they had to move on to a new experiment
• The new engineer is frustrated that (a) no one clued him in that it was maintenance debt via dead code when he started investigating and (b) there aren't team processes in place to clean up or at least note code that needs to be ripped out

Developer Efficiency Debt

What this is = When companies do not have the right tests, monitoring, and/or alerting in place on the product. This is a common type of debt, where engineering workflows are highly inefficient, deployment and build times can take several hours or days, and developers lack the tools that allow them to detect technical issues before they go live in production.

An example of developer efficiency debt...

• As an org grows from 15 to 50 engineers over a year, thorough tests and alerting were not put in place for a new purchasing experience, since there were so many experiments done on the surface area over the last six months.
• Therefore, there are at least 2-3 releases that go out where bugs are found in production and the team has to follow up and fix them quickly, since they're prioritized as high severity bugs.
• The team with ownership of the purchasing flow then carves out time to add in proper testing of the business critical flow and altering on staging.

Stability Debt

What this is = When an org builds up different types of technical debt, which then affects the stability of their infrastructure. This leads to scenarios where instead of proactively managing on-call, you have to reactively manage on-call by pulling in subject matter experts or, really, indirectly having the entire team on-call. This becomes quite an intense pain point for the engineers and on-call rotation team, but the rest of the company cannot grasp the issue to explain it well. Stability debt can also impact the reliability of the product, making it a customer-facing issue as well.

An example of stability debt...

• On a mobile (iOS/Android) app, the team prioritizes iOS far above the Android app since there are more iOS developers on the engineering team.
• As a result, the Android app is lacking fundamental product guidelines around business critical flows (and according tests that need to be in place) and also has kryptonite code around initial features that were developed by a third party vendor.
• This causes the Android app to crash for users trying to access older features. Eventually, the product receives bad reviews in the Google Play Store for its unreliable quality and frequent erroring out.

Security Debt

What this is = When there are issues in the tech stack that leaves the company open to security vulnerabilities like brute forcing password info, data breaches, or competitors knowing how to collect confidential information. Since humans have trouble planning and evaluating hypotheticals that may (or may not) happen, this results in most orgs having significant security debt.

An example of security debt...

• A data breach occurs when a company fails to patch a known vulnerability, due to failures in internal processes and not allocating the right time to updates. This results in the personal data of thousands to millions of customers being breached.
• This happens all the time through minor events with no name companies and major breaches like with Equifax in 2017 (impacting 140M+ people)

Technical Product Debt

What this is = When there is visible negative product impact. This work is easiest and most compelling to address, since it has evident impacts to users and is outwardly facing so any team at the organization can see the repercussions to customers and sales/revenue.

An example of technical product debt...

• When it visibly takes an extra few seconds for a user to load a specific action within a product.
• Although this might be caused by underlying developer efficiency debt, stability debt, or maintenance debt, it's apparent enough that it impacts the core product experience.

Decision Debt

What this is = When a past technical decision was made that was X% wrong or had some tradeoffs on scope, time, or resources, and the team is now paying for that decision. This is typically the most common form of tech debt.

An example of decision debt =

• A team originally decides to use a third party vendor for running experiments on their website. After a few years of stellar growth, the product now has millions of unique visitors a day to the website.
• As a result, the eng, data, and product teams are running into hiccups when running simultaneous complex experiments, since the vendor can't easily support how advanced the team wants to go with traffic, audience segmentation, and exclusions.
• In this situation, there's decision debt because the team made the tradeoff decision of using an experimentation vendor vs. building an in-house experimentation tool. This was likely the right call at the time, but now has repercussions in understanding the team's audience and traffic needs in relation to experimentation.

Sizing Tech Debt: Acute vs Systemic

Once you understand the type of tech debt you'd be taking on, you need to size the cost of it so you can compare it to the return you will get. When teams ask the valid question of 'When will we have time to work on tech debt?' it's hard to know if they mean something small or huge in terms of time, thought, and effort.

There's a spectrum of the size of technical debt, from acute to systemic:

Acute Tech Debt

• Relatively small sized tech debt
• Example: In order to get out a new feature, the team skipped some event instrumentation, monitoring, implementation on a lesser used platform/browser, etc. The team can easily add in the updates and the work can be done quickly (<1 day if no other blockers).

Systemic Tech Debt

• Relatively large to enormous sized tech debt
• Example: The CTO/founder made a product (and indirectly technology) decision 5 years ago and the team has struggled with its implications ever since. Changing it would mean it impacts tendrils across every decision they have made since. This could take the form of decisions around the core product setup, key feature investments, or user experience and resulting design/tech architecture. The team cannot easily address this tech debt and the work could take months to years to change.

Acute > Systemic Example: Developer Efficiency Debt

Note that each type of tech debt can be sized on the scale of acute to systemic. To illustrate, with developer efficiency debt...

Acute developer efficiency debt

• An engineer did not write Selenium tests for an MVP referral feature experiment that is only exposed to 10% of a product's users
• They didn't write the tests because the full team committed to doing manual regression of the feature before the experiment launched
• The engineer later takes a few hours to add in the tests as the next round of the referral experiment is being prepared

Systemic developer efficiency debt

• Because an early-stage engineering team didn't codify practices around testing and alerting, they're now feeling the pain a few years into exponential product growth. Each day they have at least a few production incidents that impact their users, and their NPS score is dropping month over month.
• Adding all tests and alerts cannot be done overnight while balancing new development. So, the tech leads sequence it out: start with creating testing and alerting guidelines, apply those guidelines to critical surface areas and features (P1) this quarter, followed by P2 areas next quarter, and P3 areas the following quarter.
• By the end of the year, or 7-9 months from now, all areas of the product should have uniform testing and alerting in place, based upon the newly established guidelines. Therefore, the number of daily/weekly incidents should decrease and customer sentiment should start to rise as P1 (and later P2, P3) areas are addressed.

Prioritizing Tech Debt Strategically

So far, we've gone through assumptions, classifications, and sizing in relation to tech debt. Once you understand those things you can make a strategic decision within the broader context of your product decisions.

Flexing In & Out Of Prioritizing Tech Debt

The key to seeing tech debt strategically revolves around understanding when to zoom in or out of prioritizing tech debt vs. other product development work. In order to properly manage tech debt, consider the following vectors:

• Confidence - Is there a high probability that this will lead to a significant problem for the team?
• Time - When will this become a problem?
• Impact to User - Does not doing this result in a speed or quality problem that hurts user experience?
• Sequence - Will this prevent the team from reaching important milestones?
• Accumulated debt - How much debt have you chosen to accumulate already?

From there, each vector should be classified on a scale of low priority to high priority:

• Confidence - Where on the low to high scale is the probability that this piece of tech debt will lead to a significant problem?
• Time - When on the not urgent to urgent scale will this piece of tech debt become a problem?
• Impact to User - Where on the low to high scale does not doing this piece of tech debt result in a speed or quality problem that hurts user experience?
• Sequence - Where on the minimal impact to blocking impact scale does this piece of tech debt prevent the team from reaching important milestones?
• Accumulated debt - Where on the minimal to significant scale does the amount of already accumulated tech debt fall?

Based on where the piece of tech debt falls across the comprehensive scale (illustrated in the visual below), it's then easier to understand how to leverage the tech debt strategically against other company initiatives:

• Is it necessary to solve for sooner rather than later, because it can harm other product and business initiatives if the tech debt festers?
• Is it better to put a temporary remedy into place, to work on even higher potential items for the time-being?
• Is it actually a piece of tech debt that can wait until a clean up period since the priority is low, especially when stacked against other company/team initiatives?

In the visual directly above, it's clear it's necessary to solve the systemic tech debt sooner rather than later since there's a high chance of a significant problem, the problem will become urgent soon, the debt will greatly impact reaching set milestones, and there's already a decent amount of accumulated debt.

• If some of the vectors - say impact to user and sequence - were smaller and to the left, then it could be fine to remedy the problem through a fast solution or workaround for next few months.
• If even more of the vectors - say impact to time, impact to user, and sequence were all smaller and to the left, then it's likely okay to backlog this area of tech debt to a time that makes sense in the future.

In each situation (solve, remedy, backlog) played out above, it's important to consider the piece of tech debt critically, across each vector and the the low to high priority scale. Evaluating tech debt in this manner allows for the team to make the best strategic decision on how to move forward vs. other company initiatives also in the picture.

Your Strategic Tech Debt Portfolio, Based On Company Growth Stage

Another way to approach tech debt strategically is crafting your tech debt portfolio based on tech debt type in relation to your organization's size. Here at Reforge, we classify companies as they go through four stages of growth on the S-curve:

• Traction = Bottom of the S-curve, pre-product market fit. Linear, unscalable efforts to get the growth engine going.
• Inflection = Bend in the S-curve, clear signals of product market fit on a small scale. Transitioning from unscalable to scalable growth loops.
• Scale = Hyper growth part of the S-curve. Optimizing core growth loops. Doubling down on what works.
• Expansion = Approaching top of the S-curve, saturation starts to hit. Need to think about PMF expansion and restarting the process all over again or layering on a new growth loop.

Related to the S-curve, each type of tech debt should be balanced appropriately based on the company's stage of growth. Represented visually below are guidelines to consider, for the distribution of tech debt (by type of tech debt) as an organization progresses over time.

Some key points to observe, in each stage of growth:

Traction

• Remember this is pre-product market fit
• At this point your team should be making engineering decisions for speed over accuracy, stability, process, etc - hence, the large volume of developer efficiency debt
• This generally means picking an opinionated full stack framework like Django, Rails, or PHP and developing fast, especially since the vast majority of early products are crude apps and need good integration to web and mobile

Inflection

• This is when there are signs of product market fit and the product is transitioning into scalable growth loops
• Where the team realizes some process is necessary (so developer efficiency debt starts to be resolved) and is still determining best ways to balance internal process and user experience, hence the increase in technical product debt

Scale

• This is the period of hyper-growth for the company
• In this phase, both technical product debt and developer efficiency debt start to decrease/stabilize as a result of finding better internal practices and balance in regards to process and user experience
• This is where security debt, maintenance debt, and decision debt is growing as a result of very rapid growth and not realistically being able to keep up with security updates, clean ups, and 'fixing' past decisions
• Scaling is also where there is the most changes and corrections in test automation, deployment systems, monitoring and alerting, logging and instrumentation, migrations, test and staging, and ETLs

Expansion

• This is where saturation starts to hit. At this point, the business is more mature.
• Maintenance debt and decision debt continue to increase as a result of the amount of historical code and decisions that took place
• Developer efficiency debt is starting to increase again as the team looks for new opportunities for growth (outside of the core growth to date and the existing product)
• Technical product debt, security debt, and stability debt are balancing out after the previous intense scaling period

Starter tips for managing a tech debt portfolio

As you manage a tech debt portfolio through company growth, there are a few key areas to pay particular attention to: process, tools, sprints, and roadmaps. Below are quick guidance points for each area, for when you're looking to reduce volume of one type of tech debt to balance it out by being able to take on a different type of tech debt as the company evolves over time.

Processes To Reduce Tech Debt From Being Created

Although it is strategic to accumulate tech debt, there are times where it would have made sense to stop tech debt from even being created in the first place through process implementations.

• This is especially the case in Inflection and Scale stages of company growth, where developer efficiency debt should decrease as more engineers join the team. The developer efficiency debt decreasing can then give way to necessary increases in decision debt and maintenance debt.
• Some examples of these processes include code/PR reviews, monitoring standards, QA sign-offs, and tech/design reviews.

Tools That Prevent Certain Types Of Debt From Forming

• Similar to the processes above, there are also times where investing in foundational tools will prevent some types of debt from being formed.
• This is particularly important in the Scale stage of org evolution to have more uniform methods in place to avoid security issues (security debt), prevent bugs that could impact user experience (technical product debt), and allow for code consistency (developer efficiency debt).
• Some examples of these tools include linters and CI/CD pipeline.

Sprint Work For Reactive & Acute Tech Debt

• As an org reaches the scale of having more established on-call responsibilities, the on-call sprint should be spent on fixing fires (the purpose of on-call) or on reactive work associated with tech debt (during downtime from on-call fires).
• This allows the organization to make progress on any acute tech debt items and actually take action from current/past incidents via the on-call team.
• Enabling those on-call to work on this reactive work is especially important in the Scale and Expansion phases of growth, since solving acute tech debt will then allow other teams to build new features/products and take on additional debt.

Roadmap Work For Proactive & Systemic Tech Debt

• Unlike using on-call time for reactive acute tech debt, putting tech debt into roadmaps should be used for debt that requires significant roadmap alignment and work across teams
• Examples of when to place tech debt onto roadmaps include during proposals for a major rewrite, rehauling the data system for the top product feature used by customers, defining and implementing alerting around critical pathways, migrating from one revenue/payment platform to another, and other areas that could take months to successfully implement.

How to transition tech debt from a burden to a strategic lever

• Recognize that accumulating tech debt allows your team to make holistic strategic decisions around what initiatives to take on.
• Think through common tech debt assumptions your team might be falling into. Are you up against tech debt = bad and tech debt ≠ product work? Or, are you hearing from a peer who thinks individual pain = organizational pain?
• Stop throwing around the umbrella term of 'tech debt.' Name what you're facing as maintenance debt, developer efficiency debt, stability debt, security debt, technical product debt, and/or decision debt.
• Size your tech debt to make it less ambiguous. Is it closer to acute (short time to solve) or systemic (long time to solve)?
• Prioritize tech debt strategically against other company areas. You can prioritize based on vectors like confidence, time, impact to user, and more.
• Balance a constantly evolving portfolio of tech debt, based on company growth changes and needs.

This Product Strategy lesson, “Managing Debt” provides a framework on how to categorize and manage technical debt so that managing debt becomes part of a repeatable process in the production lifecycle.