If you’re leading a profitable, cash-flow-positive business, you’ve probably watched technical debt pile up: those accumulated consequences of choosing quick fixes over well-designed, long-term solutions. If you’re not carefully managing it, it can become overwhelming.

Some executives think tech debt is like healthy corporate debt with low interest. It’s not. It’s more like credit card debt that compounds aggressively. That’s why tech debt isn’t just an “IT issue;” it’s a business strategy problem. And unless you know what to look for, how it manifests can be pretty mysterious.

In this episode of Velocity’s Edge, Huw Rogers frames the strategic challenges that lead to unhealthy levels of tech debt. He and host Nicko Goncharoff dig into key questions: Why does tech debt cause system fragility, and information security issues? Why does it delay product releases?

Addressing or even identifying these issues isn’t straightforward. Tech debt is a necessary byproduct of innovation, but it leaves a chain of consequences. You really have to dig deep to find the root problems.

There are different flavors, too.

• Conscious tech debt happens when teams deliberately build something they know they’ll need to rebuild later, taking documented shortcuts with explicitly accepted risks. This is the traditional definition of tech debt.
• Unconcious tech debt is inadvertent: the byproduct of bad practices - for example, setting unrealistic deadlines that force engineers to make their own choices. This gets particularly messy when multiple teams work in parallel on inconsistent components. Sometimes it’s just ad-hoc decisions where someone faces two paths forward without fully appreciating the risks and rewards of each.

Huw Rogers has spent two decades driving technology transformation in fintech, specializing in electronic markets technology, FX, equities, derivatives, and crypto. He has successfully led teams and projects across multiple regions, delivering scalable solutions that create tangible business impact.

As in all our episodes, we speak in plain, executive-summary business terms, framing complex business and technology strategic challenges in context, using language that makes them more accessible and actionable.

Listen here, download it from Apple Podcasts, Spotify, or find it wherever you get your podcasts.

Episode Information

Season 1, Episode 2
Length: 19 minutes, 42 seconds
Host: Nicko Goncharoff
Guest: Huw Rogers
Recorded: VOXPOD Podcast Studios, Parsons Green, London
Engineer: Dayna Ruka
Editor: Dayna Ruka, Jeet Vasani

Episode Transcript

Nicko Goncharoff (HOST): Hi, this is the Velocity’s Edge podcast. I’m Nicko Goncharoff.

Huw Rogers: And I’m Huw Rogers.

NG: And we’re here to talk about tech debt for scale-ups.

So Huw, you and I have both witnessed tech debt. And it’s something that actually has personally driven me crazy at almost every company I’ve been at. But I have found that it’s not that easy to identify. So maybe you can take us through some of your experiences with tech debt, and what threat it poses organizations and how they’ve managed to address it.

HR: Sure, it’s a super interesting topic because it’s really mysterious, I think, to a lot of people and especially to a lot of non-engineering folk. And there are symptoms, and a lot of those symptoms have a root cause in tech debt. And those symptoms are things like fragility outages, incidents, security vulnerabilities, lack of productivity, or difficulties pushing product into production. You know, all of these things can have a root cause in tech debt, and it’s especially prevalent in companies which are later-stage, which are either scaling up or maybe in the early stages of scaling up or have scaled up quite a bit and are relatively mature.

And when it sort of rears its head, I think people are generally slightly confused by it or slightly, sort of, like they don’t really understand why tech debt causes these symptoms, why it causes these problems. So it’s a little mysterious, is generally what we’ve seen.

There are a few different things you can do, to make it clearer for everyone, to try. And the first step towards solving any problem is to understand it and to acknowledge it. And there are a few different tactics that we’ve practiced with our customers to get on top of it. One of them is a tech debt survey where you can survey your engineering teams and discover their perspectives on tech debt and what’s encumbering them. And when you do that, you often find that there are recurring themes and recurring issues, and you suddenly realize that there’s a widespread, consistent set of problems. And that in turn then translates into a set of actions you can take.

NG: Some of our listeners might wonder at hearing: “So it’s just about doing a survey. Oh, my company does surveys all the time. Nobody listens to them.”

I think, and I’m not pretending I don’t know the answer to this, but I think it’s worth explaining that how you go about the survey, and with whose backing you implement the survey at the company, matters quite a bit. Right?

HR: Yeah, totally. There can be quite a bit of lack of candor among engineers talking about tech debt, particularly if there’s a history of engineers advocating for change and getting denied. This can then make them very leery of raising the same issues again, even if they’re being asked to.

So I think that you need to be very specific about what you’re looking to capture in your survey. And it’s different than an engagement survey. It’s different than other kinds of surveys you might do. And you have to be very explicit about wanting to solicit the opinions of those line developers, of those staff, software engineers, senior software engineers about what it is they think is holding them back. And that could be issues with the technology stack they’re working with. It could be issues with the software that they’re developing, because unless it’s totally new software, generally speaking, most engineers are working on improving software that already exists. Or it could be issues with the way that product and engineering work together.

All of these issues are kind of on the table there and need to be captured as part of this survey. So when you do this tech debt survey, you need to go out of your way to really focus on that specific set of issues to make people express their opinions with candor.

NG: Right. And I think one of the challenges there might be that they’ve expressed these opinions before, possibly, and possibly have not been listened to. Our colleague, Sarah Wells, talks about how respect is the currency of engineering, and it’s going to be challenging in certain organizations to reassure the engineers that if this time they have that candor, that they will be rewarded for it, or that the information will be acted upon.

HR: Yes. Yeah. And that’s a recurring theme that we’ve observed in many organizations, partly because tech debt is quite mysterious. You know, why does tech debt cause system fragility? Why does tech debt cause delayed product? It’s not straightforward to answer that question. There’s a chain of consequences. You have to really dive deep a lot of times to get to the root of the problems.

At the end of the day, tech debt is sometimes consciously taken, where people take shortcuts and consciously decide that they’re going to do something or build something in a way which is going to need to be rebuilt later. That’s quite a lot of tech debt, but there’s also quite a lot of tech debt which is unconscious, which is inadvertent. That can be because you have a lot of teams working in parallel doing different things which are inconsistent with each other and don’t mesh well with each other. And then those things create problems.

Or it can be because there are two paths forward and one of them has a pitfall and the other one doesn’t. Neither path is superior to the other in terms of the prevailing product requirements or in terms of engineering effort. They’re both basically the same amount of engineering effort. But what often happens is the path with the pitfall gets taken just because it’s the default, or because nobody really can forecast or foresee the pitfall. And those kinds of unconscious tech debt are also very prevalent, especially in scale-ups.

And the survey is really one of the only ways to get a sort of wide view, a broad view across the organization that allows you to capture these things. There’s few other ways of gaining that visibility, unless you’re a very small organization. If you’re an organization of any size with more than one or two teams working on more than one or two different areas of the technology stack, then you’re going to need to have some sort of systematic way of discovering this stuff.

NG: I would posit that there’s another type of tech debt, which is somewhere between unconscious and conscious, which is acquisitions. You have CEOs or senior leadership who are focused on growing revenue any way, and carry out a series of mergers and acquisitions without necessarily mapping out the technical integration that will be required to have all these products work together. This is something I’ve witnessed personally multiple times, and that is almost a conscious decision to take on tech debt without recognizing that you’re making that decision. The engineers may, but in my experience they’re not involved in that process.

HR: Absolutely, absolutely. You end up with essentially a factional situation inside your organization where you have different groups of people working on different, potentially inconsistent or incompatible technology stacks, and getting those things to work together and integrate can be very painful.

As with a lot of things in technology, what I call a “barbell strategy” works very well for this kind of thing, where basically you choose one of two extremes. Don’t try to do something in the middle. And one extreme is: leave these things completely separate, completely different; don’t bother trying to integrate them. Have them side by side. And if they need to talk to each other, have it happen at arm’s length with some almost public-facing API, right? That you put in place and they call each other, but other than that, you don’t try and integrate them. You just leave them as-is. That can work very well for integrations, and for post-acquisition integrations, because you don’t have to pay the cost of rewriting your software and integrating it deeply.

But on the other hand, if you want to have a unified platform, and you want to pay down that investment of integrating into the platform and integrating that software in, then yeah, you’re going to have to deal with that and you’re going to have to invest in that. And that’s at the other end. That’s the sort of other extreme where you really have to make that investment and you put a lot of effort into it and you make it happen.

Both of those things can succeed very well. I think anything other than those two things, in my experience, tends to fail and tends to cause a lot of trouble.

NG: Yeah, I completely agree. And actually, this leads us to, I think, the next topic in terms of effectively dealing with tech debt, which is organizational and cultural.

We’ve already identified that there’s a consistent gap in knowledge or perspective between engineers and senior leadership. And while that may itself not be any great shock, I think the key to effectively dealing with tech debt is making sure that the senior leadership understands the cost of tech debt to business outcomes, and also understands the importance of pushing through organizational and cultural change that is needed to address that.

You may have had a fundamental fix if you’re a feature factory. Basically, engineers are focused on getting things out, shipping out, and that’s where their incentives– that’s what the metrics are based on– and all of the sudden you’re able to walk in and convince the senior leadership that they need to deal with tech debt because you’ve diagrammed out their reduction in velocity, or they’re having a lot of outages, a lot of incidents. Then they really have to be bought into the solution, and they have to communicate that solution. Otherwise, when you go to carry out the survey, there may not be confidence that it will be listened to. I think one of the big challenges in tech debt is drawing a direct line between the perceptions of engineers and the understanding of the senior leadership team.

HR: Yeah. I think this goes back to the Sarah Wells quote, “Respect is the currency of engineers.” So one of the things that I think is very important to do is to respect the ability of the engineers at every level down to the just regular software engineers, the line developers in your teams. You have to respect their ability to be able to appreciate the goals of the organization from a business perspective. And you have to be willing to share that the organization is struggling, for example, with a delayed product, or struggling with perceptions of fragility by their customers, or struggling with too many incidents or too many outages– whatever it might be, right? Or struggling with a perception of security vulnerability undermining the product or undermining the company’s reputation. Right? Maybe there’s been a security incident.

Sharing that authentically with everybody in the organization and getting them on board with the goals of remediating those things and tackling those things, tackling those challenges, I think that sets the right context for them to be candid and to surface tech debt. And it’s important to make that link that tech debt could be– it may not be, but it could be– it could well be the underlying root cause of some of these challenges. And when you set that context, I think people will open up and they’ll have the the goals of the organization in mind, in their hearts, and they’ll work together to surface all the issues.

NG: So let’s just say that you are a CTO and you are aware of tech debt, and you’re trying to convince your fellow executive management team members about what the business impact is. How can one convey that? In what terms? What type of terminology or mindset would one need to adopt to be effective in communicating that? And I ask this because a lot of times we’ve seen technical problems rendered in technical speak, and some key decision makers turn off from that. I think it needs to be rendered in a way that is very, very clearly linked to business outcomes.

HR: Yeah. I mean, I think what I would advocate here really depends a little bit on the stage of maturity of the company, of the organization, and also on whether you’re focused on prevention or cure. Because if you’re in an early stage and you’re more focused on prevention, then the key is to get the engineers thinking about the future without over-engineering for it. So if you’re trying to prevent tech debt and you want to ship product fast– velocity is the most important thing in an early stage in a startup. Speed is like the most important thing. You’ve got to get your engineers delivering product fast. Maybe it’s right to take on some tech debt.

The right way of making that decision is to be very explicit about it, and to record that decision and to share that decision and what the trade-offs are of it. So if you’re, to use an analogy, working on a coffee cup, be very plain that you’re working on a single-use paper cup that will be disposed of after one cup of coffee. Or, you’re thinking ahead and the product is saying, “Hey, yeah, this is going to need to scale in the future. This is going to get a lot of use. It’s going to be at the heart of the platform. It needs to be engineered properly. We don’t want to have to go back and do this over again six months later, or a year later then.”

Yeah. Then you’re going to build a stainless steel mug, which is still going to be around in a hundred years’ time.

NG: With the lexicon activity. Yes.

HR: Yeah. So, I think being explicit about the decision, sharing the decision, recording the decision, sharing the implications of the decision… that’s kind of very key for engineers. They need to be coached to do that. Right? They need to be encouraged to do that and need to be coached to do that. They certainly shouldn’t take those decisions quietly in a sort of underground way without sharing them. And then creating pitfalls down the road where people are surprised that those decisions were taken. And that’s a lot of what happens that encumbers scale-ups. So that’s kind of prevention.

I think when it comes to cure, it’s uncovering the consequences of those decisions, or of a lack of decision making, effectively, the unconscious type of tech debt. It’s just uncovering that, and that’s a lot of where the survey comes in. Because people who are dealing with code that maybe they didn’t write, maybe especially if it’s a later-stage scale-up you’re often dealing with other people’s code, they’ve left the firm. They’re no longer with the firm.

But you know what you’re dealing with, right? You’re dealing with flaky tests. You’re dealing with all kinds of problems. It’s slowing you down. It’s causing all kinds of trouble. You can articulate that, but you’ve got limited avenues to share that out. And if it’s not shared out in the right framing, and in the right context, you’re not going to get the budget or the time that you need to actually solve the problem. So that’s more on the cure side of it.

NG: And I’ve witnessed both, because I’ve had a few startups, and I was at one organization where a certain number of clients just wanted a “select all” option, and we were told that would take six-to-nine months to implement. I had a hard time believing that, and then I delved into it and I saw why. That was about 15-to-20 years of accumulated tech debt, I think.

Now we have a CEO who we’ve communicated the impact of debt to them in terms of business outcomes. So what are some immediate steps that they can take to try and assess tech debt and then prioritize how to remediate it?

HR: We’ve talked about the survey a couple of times. Obviously that’s there. I do feel like the DORA metrics have a role here, because that will tell you whether or not you’re suffering from some of the engineering productivity consequences of tech debt. If your DORA metrics are not on par with the top of the industry, then that can often be associated with being encumbered by tech debt.

NG: Can you just take… in case some of our listeners are not familiar with DORA metrics?

HR: DORA became predominant in the industry for statistically proven correlation with engineering excellence and engineering productivity.

So, the DORA research group established that there were four hallmarks of high productivity, high performing engineering organizations, high deployment frequency, low change failure rates. (These are two very key ones.) And there are a couple of others, I don’t want to go into them all here in detail.

Reporting on those systematically– and there are tools like Keypup, which works well with Jira if those are tools that you use. But it’s fairly easy to get DORA metrics going in an engineering organization. And dashboarding those things, and keeping an eye on them, is a great way of assessing how your engineering organization is doing.

If you have poor DORA metrics versus industry standards– and when I say industry standards, the top-performing engineering organizations in the industry– deployment frequencies will be intraday. They’ll be deploying many times a day. Change failure rates will be relatively low. They’ll have very few failed changes hitting production and getting in front of customers.

So these DORA metrics are kind of a key indicator, a key set of indicators, for ELTs as to the health of the engineering organization.

The only thing I would add to the four DORA metrics is velocity overall, and you can measure velocity in a few different ways, but points are a reasonable way. If you’re doing Agile and Story Pointing and you have a consistent pointing methodology, then (which is a good thing to have, because it allows you to sort of report points across your engineering organization) that’s also another key sort of thing that gives you an idea of the speed with which your engineering organization is working. And that can go alongside the DORA metrics.

If you see that you’re suffering from poor metrics, that’s indicating unhealthy organization and unhealthy productivity, then that could well be due to tech debt. So that’s a great indicator to look at to see if you’re suffering from tech debt.

NG: Right. So a few takeaways:

One: Tech debt is often hidden.

Two: It’s not necessarily just a technical problem, but a business problem. And its origins can be cultural as well as technical.

Three: The DORA metrics may help you understand what level of tech debt you have.

And Four: A survey may help you assess in more detail where the problems lie and how you can address them.

HR: Yeah, that pretty much sums it up. On the survey, you’ve got to really go out of your way to encourage candor and shared goals with the engineering rank and file.

NG: Excellent. Thanks for your time, Huw.

HR: I appreciate it. It was a great conversation. Glad to be here.

NG: And I’m Nicko signing off for Velocity’s Edge.