Other Recommended Resources

Video Transcript

0:00:02.1 Bob Quillin (BQ): Welcome to the vFunction webinar. We’re gonna wait a couple minutes to get started, but welcome and look forward to a great discussion today. So for the moment, hold on.

[pause]

0:00:46.1 BQ: Welcome to those who have joined us. Our vFunction webinar will start in a couple minutes, waiting for people to join in. So looking forward to a lively discussion today. So hold on just another minute or so as the group forms, and we’ll talk to you soon.

[pause]

0:01:41.3 BQ: Okay. Well, welcome to the vFunction webinar on technical debt. It’s a couple minutes past the hour, so why don’t we get started here? Today’s discussion is about technical debt, the math behind the biggest obstacle to modernization, and look forward to a great discussion today. Just a little housekeeping. We’re gonna have a discussion that’s gonna last about 45 minutes. We have two major panelists, I’ll be the host. And we have the Q&A function on the button at the bottom, Q&A button at the bottom of the screen there, so please load up those questions. We’ll answer them as we get towards the end, and if there’s anything that pops up during that time, during our time today, we’ll also try to hit that as we go.

0:02:26.5 BQ: My name is Bob Quillin. I’m the Chief Ecosystem Officer here at vFunction, and I’m joined by Mike Amundsen, esteemed international author and speaker. I’ll let everyone introduce themselves in a second, and Amir Rapson, our CTO and co-founder of vFunction. Maybe over to you, Mike. Quick introduction, and then we can kind of get started.

0:02:46.8 Mike Amundsen (MA): Yeah, thanks, Bob. It’s great to join you here at vFunction for this conversation. I’ve spent the last 25 or 30-plus years somewhere around that working in the space of webs and APIs and networks and services. I spent the last 10 or 15 years focusing a lot on APIs themselves, and I tend to write about them as well as present on them, and I’m happy to join the discussion here with you and Amir.

0:03:19.8 BQ: Great. Welcome, Mike. It’s great to have you. And Amir, why don’t you introduce yourself also?

0:03:26.3 Amir Rapson (AR): Alright. So as Bob said, I’m one of the founders and the CTO of vFunction. I mean, this is one of the most interesting subjects I’ve ever worked upon. When we started this company, I’ve already had a vast experience in creating a lot of technical debt as an engineering manager, and I feel this is like a redemption, being able to help organizations reduce their technical debt with technology. So thank you all for coming, and thank you, Mike, for joining us today.

0:04:02.2 BQ: Yeah, it’s going to be basically a roundtable discussion. We’ll be talking a range of questions in this area, so let me stop sharing and kind of get the screen going here. Maybe the first place to start is as we talk about technical debt in general, maybe just a definition. What is technical debt? Why is it important? Maybe I’ll let… Mike, why don’t you start off and give us some of your background, some of your thoughts to get us warmed up? And Amir, you can jump in right after that.

0:04:29.5 MA: Yeah. I mean, I can speak from technically, sort of like Amir’s perspective a little while ago. I’ve inflicted a lot of technical debt over the decades, and I’ve also seen organizations work through different versions of technical debt. Typically, people think of technical debt as code that’s old or code we haven’t used, or a code that… Some people even say code that was written quickly or poorly because of feature velocity reasons. But I tend to think of, in my own experience, technical debt is a sort of a mystery. People are worried, we’ve got some stuff here, I can’t really see what it’s doing or how it’s operating. So often, I’m trying to help people figure out how they can observe what’s going on and figure out where the value is. I often talk about where value is hidden in all sorts of code and where there are things we might be able to trim. But that’s a very general sort of seat of the pants kind of understanding. And I know you at vFunction have actually gone well beyond that kind of suspicion and idea into some very specifics, and I’ll kind of turn it over to Amir for that.

0:05:41.1 AR: Thanks. So I think, well, we all know technical debt, and technical debt is kind of what I feel is the root of all of the reasons why we want to modernize software. So if you think about why we want to modernize software, because we’re developing slow, because releases are slow, because our developers are not happy with the quality of the code. So all of that, if you think about like, you ask yourself like the five whys, like why are the developers unhappy? Why is it taking us long? And you kind of ask yourself why and why and why, you get to the… The end result is technical debt. So whatever it is you answer at the bottom, that’s technical debt. I also like the term technical debt because it’s due to technical reasons, but you should really treat it as debt. So it’s really money. It’s money that you’re spending on the application. It’s money that you’re spending by having more developers work on less features over time. And I think that money is also a good way of measuring technical debt at the end of the day. So lots of examples and lots of similes are kind of more like from the money and debt kind of world. But I think it’s actually quite proper.

0:07:02.0 BQ: You kind of think of innovation is a big goal for many CIOs and technology teams, but there seems to be a direct relationship between innovation and technical debt ’cause you want to have innovation, but there’s a relationship between those.

0:07:19.4 AR: Absolutely.

0:07:22.0 BQ: Got you. Yeah, so there’s been a lot of, I guess, research behind technical debt on the academic side and a lot of kind of people looking at technical debt on a pretty broad set of terms. But I guess the academic research has kind of moved forward over the last 10 years. Amir, maybe you can start catching us up where the academic side is going and maybe how it becomes more practical and some of the things that we could do in the field.

0:07:53.3 AR: Yeah, so academic research has been done on technical debt. And of course, as being academic, it’s kind of defined a little bit more in a more structured way. So there are many types of technical debt and many types of decisions or the fact that, even product changes over time that may constitute as technical debt. But in academic work, we see a lot of work around architectural technical debt. Like, what is it within the architecture of the system that caused this technical debt to accumulate? And as you expect from academic work, they also kind of do simulations. They create specific applications to simulate certain levels of technical debt. They try to calculate based on specific examples, what is a technical debt within a specific application? And we also had quite a comprehensive research around how to calculate technical debt in a large number of applications. And we didn’t take like a very… It’s like, you know, I’m a physicist originally. So when you’re a physicist, you’d like study very, very small things and understand why they happen. So that’s like the regular research. And we did it more like a biology kind of statistical way, which we now call machine learning.

0:09:22.8 AR: And that’s like trying to figure out like we did it in a machine learning way to kind of figure out what are the features that are within these applications that contributed this technical debt. And kind of by asking people to connect these qualitative sides with the quantitative sides of is it a lot of debt? Is it little debt? Is it something that you can overcome? Is it something that you can’t overcome? So lots of these things. So definitely a lot of research being done. Much of it is interesting. And one thing that’s mostly interesting to me is like how much… It’s a very technical field, obviously. Software architecture, technical debt. And there is academic work on this. But usually as architects, we have very little technical tools to help us guide us with actually measuring this. So this is kind of a funny bit that something that I thought was a bit funny.

0:10:26.6 MA: Well, you mentioned a whole bunch of things, Amir, that again, from my own experience, these all kind of ring true. One of the things that I was curious about and I got a chance to look a little bit at the paper that you shared earlier, but I don’t quite understand all of the pieces. One of the things that comes up quite often is experienced architects or experienced craftspeople, one of the things from Christopher Alexander, the person who gives us the timeless way of building and the idea of patterns that Gang of Four uses and stuff. He always talked about this notion that a great teacher sort of knows how far to go and still be able to walk it back. You know what I mean? It’s like, I can go ahead and make this decision later. I think Mary Poppendieck has this idea of making the decision at the last responsible moment, designing a system so that I could change it a little bit later or that it can flex a little more. And I sometimes think that what happens is we build systems that don’t flex or aren’t easily changed. And that sort of increases the debt or in my way of thinking, sort of brings the debt on early.

0:11:37.3 MA: Does that sound like anything is… Again, this is just a thing that I use every day, but is that reflected anywhere in any kind of the material that you’ve worked on?

0:11:52.6 AR: I think you’re right. And it’s definitely what you need to think about in short timeframes. When it’s a short timeframes, like over a month, two months, one release, two releases, three releases, I think it’s worth being flexible on your earlier decisions and kind of flex. If you think about applications that have been around for, let’s call it five years, with, let’s say, a hundred man years within these five years, I don’t think it matters much anymore. ‘Cause all of these decisions, some of them are good, some of them are bad, some of them not here, not there. But because of the complexity, because there’s so many decisions, because as we plan these applications, we don’t have a lot of visibility into all the decisions that were made prior to the next one. So you said, like I was saying, that standing on the shoulders of giants. So the thing is that you’re standing on top of a lot of code written by a lot of smart people, but it’s like so far down, it’s very hard to see those decisions that were made. So I think that over time, it’s very, very hard to maintain a low level of technical data and a good architecture. So I don’t think it’s anyone’s fault if the application kind of goes sour.

0:13:23.5 MA: Exactly, right. Mel Conway once was quoted as saying, maintaining a large system is very much an act of archaeology. You have to kind of dig around, you find layer upon layer upon layer, and there’s really not much you can do except sort of wonder and discover, gee, I wonder how they did that. Because I’ve worked on systems that are 10, 15, 20 years old. I’ve worked on minis and mainframes where they’re still operating. So there’s a lot of archaeology involved in that kind of structure.

0:14:00.5 AR: Yeah. Strictly archeology.

0:14:00.5 BQ: One of the questions that came in as we talk about technical debt, maybe it’s also a good level set. We talked about technical debt, it mentions in terms of apps, in terms of architecture, but there’s also technical debt with legacy OS, with the aging platforms too. So kind of the, how do you separate that whole spectrum of technical debt? And how does it relate to modernization too? I guess those are the kind of the context. We’re getting some questions around that.

0:14:29.3 MA: Yeah, that’s a really interesting point too. There is a lot of times when you’re motivated for change, you’re motivated to make some change, to re-platform or something like that, simply because you’re sort of running at the end of a license. You’ve got some costs and commercial off-the-shelf software that isn’t going to be supported anymore. You don’t necessarily need new features or need new functionality, but I need to get off of this platform onto another one. So that’s definitely a kind of debt. As soon as you make a decision to, we’re going to use this OS or this platform or this set of frameworks, you’re sort of committing a debt. You’re sort of you’ve spent the money. Now you’re hoping it pays off for a certain amount of time, but then at some point, the value of maintaining that automobile is well above the actual value that it returns to you. So at some point you’re gonna have to make those kinds of changes. That’s a really interesting, so it isn’t just a design debt then, right Bob? It’s really sort of just inherited in some kind of way.

0:15:39.2 AR: Yeah. I’ll tell you what, I think it’s like certain frameworks, definitely when you choose the wrong framework, it’s really bad. It’s debt. I still remembered when we chose GWT as the greatest UI technology and then a year later Google said, “Well forget about it, it’s gone.” And so yeah, it’s definitely debt and there’s some of these decisions that we’ve started to take, like we’re kind of okay with, like when we choose UI framework, it’s like buying a new car. We know it’ll get old and it’ll look kind of funny in a few years. And when we look more at the core logic stuff, it’s like, okay, I’m going to buy a new car. I’m going to buy a new car. I’m going to buy a new car. And when we look more at the core logic stuff, it’s more like buying a house, even if it gets a little bit old, we’ll probably fix it. We won’t replace it.

0:16:30.4 MA: Yeah, so that’s a different sort of different pace of change, right? There’s the feature or the style pace of change, which goes relatively quickly like UIs or JavaScript frameworks or something like that. And we kind of protect ourselves a little bit from that. But then there’s the more deeper foundational pace of change that it’s gonna be difficult once we re-platform all our data into this system, it’s gonna be expensive and complicated to move it again. So we kind of need to think of them. There’s a little bit more investment, right? Is that right?

0:17:09.7 AR: Yeah. A little more investment, a little more careful with financial obligations surrounding it.

0:17:20.3 BQ: Like the Risk. Yeah. [chuckle]

0:17:20.4 AR: Risk. Exactly. Yeah.

0:17:24.0 MA: Yeah. Is technical really just a measurement of risk or is it reversed? Like what is the relationship there?

0:17:32.7 AR: That’s a good question. I don’t have a good answer to it. I think that’s kind of maybe… It goes back to the conversation we just had about microservices being like a sort of insurance from technical debt. So we’re talking about risk, then maybe using like an architecture that saves you from creating a big mess, from accumulating too much technical debt. Maybe you can see that as like an insurance on that debt that you’re about to accumulate.

0:18:06.2 MA: Yes. So does microservice sort of change the pace, right? So if I use a monolith, it gets larger and larger and more complicated, not just complex, but complicated to manage. That’s a bigger debt maybe than a lot of small services. I can change those individual services with lower risk, maybe lower cost, and get improvements without having to actually upset the Jenga that I’ve already built, maybe, I guess. Is that kind of what you mean?

0:18:38.0 AR: Yes. But on the other hand, a monolith has less constraints. When I want to develop a new feature, I can do it any way I want, really. And when it’s a small team, it’s a young monolith, I mean, that’s like the best architecture. I have no constraints. Nothing is constraining me to work… To do something in a specific way.

0:19:00.3 MA: Yeah, I get…

0:19:00.5 AR: But microservices…

0:19:01.8 MA: Go ahead, Amir.

0:19:02.0 AR: But microservices, it’s like the walls are already there. I have to go through the doors. I can’t just walk through.

0:19:13.3 BQ: No. And there’s lots of doors, right? That’s the other thing. With a monolith, especially you talk about startups, right? It’s the old joke. Nobody creates a startup with 150 people and 200 services. That’s not day one, right? We have a monolith and a small team. With a monolith, you have a lot of observability, especially the small monolith, the young monolith. I remember where this is. I know why we did this. I don’t even need ADRs. I don’t need architectural decision records. It’s the three of us that we did this six months ago. But year upon year, you talked earlier about five years on, maybe 10 years on. Now, it’s hard for me to see. And microservices really compounds that, though, doesn’t it?

0:19:54.8 AR: Yeah, it does.

0:19:58.4 MA: That ability to understand what the path is. I need some serious tooling for that.

0:20:03.0 AR: Yeah, definitely. That visibility or that limiting the complexity or seeing the complexity or calculating the complexity. Yeah.

0:20:15.7 MA: Yeah, exactly.

0:20:17.2 BQ: Yeah. Well, maybe it’s… If we think about all these things around technical debt, one of the hardest things to do is where do we start? What’s the starting point? And how do you get moving towards fixing the solution? And Mike, I know you’ve written a lot about this. What are your thoughts around how to get started?

0:20:37.6 MA: I used to tell people, if you’re replatforming or if you’re creating microservices, if you’re breaking up the monolith, whatever sort of meme it is over the years, find some non-risky, non-trivial thing that you can kind of modify. A lot of times, people start with some internal system. Well, let’s modify the jobs to be done list inside our team. And we’ll use the new framework of the new architecture, the new data model, and see if we can learn how to do that. Start with something that you can learn from and scope it correctly. That was really brought clear to me by Adrian Cockcroft, who was the cloud architect, I think, for Netflix. I don’t remember what title his exact title was. He said, “But at Netflix, what we did was we looked for the smallest possible thing we could change and learn the most,” which I thought was a really interesting kind of idea. Find some small thing you can change. Boy, if we can figure out how to migrate these objects into this store, we would learn so much about the rest of the system. So trying to figure out where you could learn the most without and inflict the least amount of damage, ’cause you’re going to make mistakes, you never do it right the first time. System one, system two, and all that.

0:22:01.8 MA: So again, that’s just sort of experiential seat of the pants things I’ve learned. But Amir, you probably, I know at vFunction, you’ve worked out how to identify where complexity or where possible debt is, right?

0:22:20.0 AR: Yeah. At vFunction, we’ve not only kind of been able to kind of measure the technical debt and allow you to kind of prioritize between applications, but we also kind of help you figure out how to decompose the applications, which does allow you to kind of focus in on maybe larger problems, the more pressing problems, ’cause modernizing is such a tough business and splitting into microservices is so tough that organizations, until they get budget for it, it’s only to solve like a major pain. And so, they want to solve that major pain. And one of the main problems with modernization is that if you do it manually, you don’t do it with tools, it costs so much to be wrong. So people, even if they kind of figure out that they’re wrong halfway, they kind of move on, they go straight on. So it’s like having a compass without a map, right? They see that there’s a mountain, but they’re like halfway up the mountain, they just have to go on.

0:23:28.1 MA: Yeah. And it’s a sort of a point of no return, right? One of the things that I counsel people against are making like an all in big bet, like if you play cards or something, Texas hold them, “I’m gonna bet all my money, we’re gonna be using this new framework by June.” You know, it’s like, you have no idea what’s ahead of you, right? I’d never heard that a compass without a map. I really like that idea. So what I need is the map, right? I need a better map. You need to figure out where things are and where I need to go and what I need to avoid, right?

0:24:07.3 AR: Yeah. And you need to somehow minimize the cost of failure, right? So if you’re wrong, let’s say you’re wrong, you make a bad decision, a bad architectural decision. Also try to minimize that. I mean, it’s figure out when I kind of figure out that I’m wrong, how to make sure to see that I’m wrong and to make it in such a way that it’s not so bad to be wrong and just revert back, start all over again.

0:24:33.8 MA: Some of that is cultural too, right? We’re worried about failing inside our organizations and so on and so forth. So you have to kind of get past that. But some of it is like you said, I need feedback. I need early feedback often and I need to have the courage to say, “You know what? All this feedback is telling me I’m working on the wrong problem or I’m headed in the wrong direction. So I need to be able to observe every change I make, right? And to know whether or not it’s getting me to the right place.”

0:25:03.7 AR: Yeah. Yes.

0:25:06.9 MA: Again, that’s tooling, right? That’s tooling.

0:25:09.0 AR: It’s tooling because when you say “Every tool that I make,” it’s not really every move that you make. You probably are an architect and you have 20 people on your team doing those moves for you. You maybe told them kind of slightly what direction you want them to go, but you don’t know.

0:25:27.4 MA: Yeah.

[chuckle]

0:25:31.7 MA: And I think that’s another one that we’re touching on is you’re always working on it. The system is operating. It’s actually the plane is flying. So if you poke it somewhere, it’s gonna come out somewhere else, right? I mean, it’s a… You used the biology earlier. It’s a living breathing system. So I’m not just making a change in isolation. When I change this component, there’s a ripple effect. I need to be able to see that ripple too.

0:25:58.8 AR: Yes. It’s very hard to stop development and just modernize. I haven’t seen too many organizations that actually did that. And if you’re gonna do that, you have to have a way of minimizing the length of this journey. Because I have seen companies that decided to do that. I thought it will take them a year to completely what we now call re-imagine, not modernize their application. And instead of one year, it took them two and a half years to complete development and another two years to migrate clients to their new platform, which meant four and a half years of no development for their existing clients, which meant a drop in their share value from, I think it was from $10 to like $4 during that modernization time. So it’s really, when I said technical debt is about money, it’s really about money.

0:27:00.4 MA: Yeah. And I think people… You’re right. I think often people minimize what they think the cost or the length of time is going to be. I’ll use Netflix again, because they’ve been so open about the way they operate. They moved from on-prem to the cloud, more than a decade ago from on-premise servers to the cloud. And I know that in their technical blog, they finally announced, “We’ve just moved our last on-premise server into the cloud. And it only took us… To do all of our servers, it only took us eight years.” And all through that time, they had to maintain the business and maintain the system. Now I have a relatively straightforward business of streaming, but it takes time to do these things. And of course, by the time they were done moving everything to the cloud, they were all EC2 instances. Now we’re talking about the idea of containers or the idea of Lambda servers and, it’s not done. You’re gonna have to be faced with other changes as well. So it’s really kind of continuous in a lot of ways, right?

0:28:04.7 AR: Yeah. Like painting the Golden Gate Bridge.

0:28:06.2 MA: Yeah.

0:28:08.5 BQ: Right?

0:28:08.7 MA: Yeah, exactly.

0:28:10.0 BQ: Yeah. Continuous problem. So if I’m going to calculate technical debt, I need to have components. I need to break it down into an equation or something. What are the components that are… The elements that make up technical debt? And maybe, Amir, you can kind of start with maybe your thoughts around how the research and how we’ve actually taken a look at beginning to calculate that on a data driven basis too.

0:28:37.2 AR: So the traditional formula says that the cost of refactoring a certain element is the cost of refactoring that element and the sum of all of the dependent elements of that feature. So, and of course, it’s turtles all the way down. So cost of this class, the cost of its dependent classes and cost of their dependent classes and so on and so forth. And then, you sum it up and that’s kind of the cost of technical debt. And you could do that for the entire application. And when you think about this, you figure out a few things. First of all, the more connected or the more complex my network is, the more cost I have to… The higher the cost that I have. The longer my dependency chains, the more debt that I have. So this was done in the research from, I think, 2012 by Robert Nord and Al. And we did and we took a somewhat different approach, but surprisingly got to the same intuitive results that it’s all about like the complexity and the length of the dependency chain. So we took about 100 different monolithic applications from various clients of various sizes of various verticals.

0:30:12.8 AR: And we took the feature extractions on the dependency graphs. And we also found… And then based on these features we fed them into a machine learning and we trained a network and then we did some research on what were the results. And we found again that based on these results, the two most contributing factors to technical debt were the entanglement of this graph of dependencies and the length of the dependency chains. We kind of coined it into something that’s even more intuitive a little bit. So the complex entanglement is really the complexity. So how complex your application is. And these dependency chains is really the risk of you of changing something in one place on the application and affecting a very, very distant place within the application. So what is like the chance of impacting the whole application? So really, if you manage to kind of quantify these two factors, your complexity and your risk or your entanglement and the number of dependents or the average number of dependents or the force root of the number of dependents, then based on that, you can get a really good sense of the level of technical debt within your application compared to other applications.

0:31:40.1 AR: We also kind of set it to a baseline of in dollar signs. So that’s another thing that we did. That’s not where all the technology went. All the technology went to actually figuring out all those features and the machine learning stuff. But we have managed to also give it kind of a dollar sign because that was also very important for us. And it is part of how we view technical debt. It’s something that should have a dollar sign at the end of it.

0:32:06.2 BQ: Gotcha. And that translates to a kind of financial impact too in terms of TCO and ROI ’cause people kind of were getting questions around how do I build business cases around this? We have the data around complexity and risk and that all kind of moves into how I began to judge how long it’s going to take. What is the potential ROI of doing this? And so it’s that, how do I translate that into some dollars too? That’s building business cases. That seems to be another, that’s the big first step. Is that knowing what to do and then how much it’s going to cost me? What are the benefits?

0:32:39.7 MA: And then…

0:32:39.9 AR: So understanding the technical… Sorry, Mike, go ahead.

0:32:45.6 MA: Well, I was just going to say, you mentioned TCO. I think one of the other ones is opportunity cost, right? So at some point, because I have to keep maintaining what’s here, I keep sort of like paying off this debt in some way. I may be missing opportunities as well, right? Yeah. Go ahead, Amir.

0:33:07.1 AR: So when you think about ROI, so obviously the investment is what will cost you to modernize that application. And so let’s put that aside, but we’re kind of more focusing on the technical debt. And your technical debt is the opportunity. So if this is the… Or the return, sorry. So that’s the opportunity, like Mike, you said, opportunity is also the return. If I invest the modernizing, this is the return that I get. I get back my technical debt. When you think you’re gonna… Let’s just say it in dollar signs, that’s you think you’re going to invest your R&D budget just to make it round and nice, like a hundred thousand dollars for the next year, the small application. Okay. Half of it, let’s say your technical debt is like 50%. So 50% of your budget goes to feed the technical debt or to repay the technical debt. So that actually means that you only paid 50k for developing features. And that means that your TCO is twice as much as you think it is. And that’s your kind of your business case. Okay. Now your budget is a million dollars and your technical… It’s a bigger application. Your technical debt is now somewhere around 70%. So $700,000 each year goes to feed the technical debt. How much will it cost you to modernize? One and a half million, 800,000, six million, I don’t know.

0:34:41.1 AR: But if it’s like a one and a half million, and you can return and the ROI could be within two years. And this is not counting the opportunity cost of actually developing more, innovating more, creating more client satisfaction, developer satisfaction, increasing your developer retention, right? Because of their satisfaction. These things mean a lot to organizations and definitely something that can help you build a business case to modernize.

0:35:15.0 MA: Yeah. Now, and you may have figures on this too. There’s another thing that I kind of experience along the way. And there’s a lot of this is non-linear too, right? So the thing is, these things add up not just in a simple mathematical plus, plus, plus kind of way, but they start as a sort of a hockey stick situation. At some point, your servicing on the debt becomes so overburdening that you can’t even service the debt, let alone figure out how much of this is actually going to be for innovation and how much is actually going to be for maintenance. Like we’re totally swamped. I’ve run into lots and lots of departments inside organizations where they’re completely outside of any notion of thinking about innovative change inside their sphere because they’re completely inundated with maintenance costs.

0:36:10.3 AR: You know, I have a funny story about this. When we first released our assessment piece, we had a bug and we stole this at the client and the client showed us that we showed them that they have 116% debt. And they were laughing because they said, the reason we’re laughing is because we actually think that it’s kind of real. Yeah, because we have felt for a long time that every dollar that we spend on developing this application, we actually need to spend more to fix it.

[laughter]

0:36:57.6 MA: Yes. I totally…

0:37:00.9 BQ: It was actually a feature, not a bug.

[chuckle]

0:37:01.0 MA: That’s right. It’s a feature. I think there are so many examples we can all kind of look back on in our own experience where we really have that feeling that I’m actually, I’m making this worse. This is actually not getting better. We have to kind of break out of this cycle. And a lot of that for me has to do with, and I’ve talked with a lot of companies about this, not being able to actually see or quantify what I’m actually dealing with, which is one of the things that I really like about what you’re talking about, which is creating that visibility point, figuring out what the dependency chain really is and what the real risks are. A lot of times it’s empowering, just like the 116% people. Like, well, at least now we know kind of experience.

0:37:52.0 MA: Yeah.

0:37:53.7 AR: Yeah.

0:37:53.8 BQ: Yeah. There’s a question live actually around enterprise observability and that observability, be able to see inside the application. Observability is typically performance, but a lot of it now we’re talking about is just being able to understand what’s going on inside the application, the complexity, the risk and all those components. I think observability, moving outside of the infrastructure side into the application side is definitely, is a theme going forward.

0:38:19.2 AR: Yeah. So I think as I said before, as I mentioned, what’s the return of in the ROI and what’s the investment? I think that observability allows you to reduce the investment. That’s partly, the return is still the same return. I mean, you have junk in your application, you have debt, you’re going to get rid of that. It’ll give you a great opportunity. The investment might still be big. You can reduce it by adding that visibility. Now, over time, that visibility will help you maintain that technical debt so that that continuous observability, continuous modernization will allow you to constantly make sure that the debt stays only in a certain level. They’ll never get to zero, but at least it stays there.

0:39:06.6 MA: Well, one of the things that we talk about in the book, continuous API management is this notion that just like any other product, you get to some point where it’s about as valuable as it’s going to be. And I don’t want to mess with this product anymore. I want to lower my maintenance costs. I don’t want to keep disrupting it and re-releasing it. Let’s get the most out of this investment for the longest period of time until we can replace it with something else. Having the information that tells me my feature curve is winding down. I don’t need to invest a lot more in this. I can spend my innovation dollars somewhere else and know that I’m observing a sort of a flat line on the maintenance. It’s not ramping way up. It’s just staying steady. That’s a really valuable tool to have at your disposal to help you sort of figure out what things really need to need focus. I think of Goldratt, Eliyahu Goldratt, The Goal. This is a book from the ’80s. He talks about you want to be relentlessly focused on that one thing that is the blocker for the rest of the business. And you need a lot of data. You need a lot of information for that. You need a lot of observability. And observability gets harder in global enterprise than just a single application ’cause it’s all the network pieces as well.

0:40:31.3 AR: And also because things are so complex, there’s so many different choices to make. Luckily, a computer can kind of do those, lots of decisions very, very quickly and actually calculate some result. So yeah.

0:40:47.4 BQ: Yeah. Amir, you were talking about looking at this on a continuous basis. I know Mike, you talk about your star, S-T-A-R, and the last R is repeat. But I like that structure in terms of maybe you can walk us through the star acronym and kind of the loop it creates in terms of being continuous and always looking forward.

0:41:06.2 MA: Yeah. The challenge in helping people reimagine, modernize, is everyone’s situation is different. But you can come up with some overall patterns that we see over and over again when you want to tackle a modernization. I need to stabilize what’s there. Often we see people create facades, like we would have encode, but now we’re doing them on the network level. We have sort of these API gateways or something like that. We stabilize what we have. The individuals talk through this stabilizing platform. That gives you a chance to transform existing services into like maybe I can just rewrite them or maybe I can replace this search tool with an off-the-shelf tool or maybe I can actually just come up with a new solution entirely. So I can stabilize and transform. And then once I have that pattern working, I can append or add new features and functionality. I can finally start to spend those innovation dollars because I’ve done the work of lowering the debt and now I’m innovating. And then of course, it’s rinse and repeat. It’s constantly doing that over and over again. And the actual details of that pattern are different for every organization.

0:42:19.0 MA: I will tell you that most of the organizations I work with are doing this in some sort of manual way. They have reviews and processes and observability. Being able to see what they need to stabilize and transform can be super challenging, so a lot of times they sort of take the easy way. What would help them a lot are better tooling to do all of those things, including that rinse and repeat part.

0:42:45.4 BQ: Great. I think we’re getting towards the end of our session, but we can probably start moving into some of the Q&A and talk a little bit about some of the questions that are coming up. One of the most recent ones is, is it better to create versus modernize? And there’s, when do I rewrite? When do I refactor? And this is always one of the big questions, which are to choose when you modernize. So I’ll throw that out to the panelists here.

0:43:14.3 AR: Yeah, I have a very strong opinion about this. It’s a very simple rule. And the answer is never rewrite. [chuckle] I think it’s, as a developer, I remember that it’s always, you look at really, really bad code and you see, and you kind of feel, “Now this just needs to be thrown away. I’ll rewrite it from scratch.” And then you rewrite it, but then you start debugging it and you say, “Oh, I forgot about this. And then, oh, I forgot about that, oh I forgot about this.” And then you send it off to testing, you get a few rejects back, you change a few more stuff, you look at your code, it looks exactly the same.

0:44:01.0 AR: It’s so hard to, I mean, it’s like a lovely idea to think about, oh, if I could just build it again. But the truth is you’ll probably make half of the same decisions you made originally. It’ll take you maybe three quarters of the time. So I would say that there are so many rare occasions where the right thing to do was to paint it blue and throw it into the ocean that… Just don’t. I think if you break it up into microservices and if you break a big application and then you have like little pieces, and then little pieces, you want to try and rewrite them, fair, right? Especially if they’re built using like old constructs, old frameworks, like people don’t do these things anymore. It doesn’t fit the business need, right? Throw it away and rewrite it. But as a whole for an enterprise application, never.

0:45:04.5 MA: Yeah. Yeah, that’s totally the same experience I have too. One of the patterns we talk about constantly is it’s called the strangler pattern or the deprecator pattern, right? It’s like a lot of times the debt that we see accumulated is code that we don’t use anymore. One of the challenges I find seeing a lot of people I work with is they have a sort of a whole library, like an API library, and there’s like 150 endpoints and they want to modernize all 150. Well, it turns out, you know what? About 50 of those are used. The other 100 are barely ever used. Getting rid of them and not changing the other 50 sometimes is a much better strategy. As long as you’ve got the observability to know what’s going on, often it’s just a matter of cleaning up what you have rather than rewriting it.

0:45:56.5 AR: Right. And you brought up another point, which is that dead code is also technical debt.

0:46:02.4 MA: Yes.

0:46:02.5 AR: Because even if it’s like fantastic code but it’s not used and you’re maintaining it, you’re actually spending money on maintaining something that’s not used and not helping for innovation and not providing business value. So that’s technical debt.

0:46:16.9 MA: Exactly. And that’s taking dollars into maintenance instead of innovation, right?

0:46:22.1 AR: Yeah. Exactly.

0:46:25.9 BQ: Yeah. There’s a few questions on this but here’s a good one on, if I had an old and flexible application portfolio, where would you start progress? Maybe you have a whole landscape of application. What metrics would you look at to measure progress? Maybe how to stack rank some of these and actually look at how I’m progressing through this. So a good question.

0:46:45.3 MA: That’s a question for Amir.

0:46:47.1 BQ: Yeah. Where to start and how to measure progress.

0:46:49.7 MA: Yeah.

0:46:52.1 AR: No, I think here I would give the same answer, Mike, you answered before. I mean, start with applications with less risk and learn how to modernize the smaller applications and not tackle your big behemoths. Still modernize, still deal with the core business problems of these applications and still handle the core of the technical debt. But the smaller applications. So start with, so if you have a portfolio and you have applications around 1000 classes, I think 1000 classes is a good place where it’s feasible, you can still learn for some stuff. It’s still going to be a difficult application, but it’ll teach you how to do it, how to make the right decisions around it and get some knowledge and expertise around this business of modernizing. ‘Cause modernizing is becoming a profession. So becoming a modernizing architect, like a modernization architect. I think a modernization architect, we see it as something that’s a new profession in our software architecture community. So start with a smaller application, build your way up to the massive applications that require a lot of work. And also, even with the right tools, it’s still going to be a heavy lift to kind of modernize them.

0:48:16.8 MA: Yeah. We talk a lot about Tiger teams. And when I’m talking to organizations, you find a team that has this skill that you, they kind of hone their skill as these, I never thought of thinking of them as modernization architects, but they can help come in, help you stabilize your existing system, help you analyze what you have, get some observability, figure out what are the good transformation techniques, whether it’s the deprecation pattern or some refactoring or replacement or whatever you want to do, and sort of help you go through those steps. And every time you do that, you learn a little bit more. And in at least one organization I’ve worked with, we sort of help them build a practice of teaching others how to handle this transformation, this modernization, and then teach the others, and then they handle it in their own way as well. So you sort of enable people. But I like this idea of this, you say modernization architect, is that what?

0:49:12.7 AR: Yeah. Yeah.

0:49:14.4 MA: Yeah, I like that. I like that.

0:49:18.2 AR: Yeah. I think we see that starting. I mean, up until like a year or two ago, the only thing that I’ve seen is that… I’ll tell you about one financial services company that unfortunately, I’m not a client ’cause it’s a very good logo, but they kind of had the notion of measuring the amount of technical debt, of… Sorry, the backlog for each application. And the backlog of each application, was there a way of kind of measuring technical debt versus innovation? Which I think is a good way of thinking out of it and like, not going into too much like data driven, heavy work. And then when the backlog got to a certain size, they put a team on it. So team of developers that they had available, or they brought them from different teams, and they put them along this application. So they had like a targeted team to modernize applications really, not calling them modernization architects. And that specific company didn’t really think of that being a specific skillset. Now, two years later, or 18 months later, we do see companies starting to put people with specific skills and specific experience and seeing that as really a profession and a new expertise.

0:50:35.6 BQ: Well, there’s a great set of questions around good best practices and limiting the building up of debt in the future. The risk of accumulating more debt again, another question around, will I ever get rid of debt altogether? Can I ever get to zero? Et cetera. So, that’s the range of questions there, but a good topic.

0:51:03.2 MA: The last question you asked is easy. You’ll never get rid of debt. There will always be some kind of accumulation. And that’s okay too. I mean, we talked about… Somebody was asking about portfolios. Every organization has a very rich portfolio of brand new things and things that are in that sort of maintenance mode that are doing really well. Things that are in that sort of declining years need to be deprecated in some kind of way. You’re always gonna be having to be able to do that in some kind of way. You’re always going to be having a lot of this inside and outside an organization. And the way you tackle each one of those, the way you innovate or how much innovation you do is gonna be different all the time. So having a sense of the landscape is not a monoculture either, not just one service, but all of those services, I think is a really good way to think about it as well. So you can reduce some debt here, but at some point it doesn’t pay off trying to, there’s a limited amount of returns, right? The ROI on maintenance here would be better spent on innovation somewhere else sometimes. That’s another kind of important role. How you make that mix, I think is kind of the leadership part of it, I think.

0:52:13.7 AR: Yeah. So I have a friend that used to be a developer that was way back in like the early 2000s, just around the dot-com era. And that friend was especially smart and he’s now a professor of math at NYU. So he was at the company, I forget the name, but they were writing an application and but they didn’t have any clients ’cause we’re talking about like the dot-com era, the clients and customers, wasn’t really a necessity. And so they were writing this application and then they kind of rewrote it and they rewrote it. And it got to a point where they came to the office and they sat and they looked at the code and they just admired it. That’s what they did. They admired their own code. So they got to like zero debt. But if you think about like getting to zero debt, I would tell you like the same thing that my finance professors taught me at my MBA. If you’re good at making money, take debt and invest it in order to make more money.

0:53:14.4 AR: So if you’re good at innovating, innovation creates business value. It makes sense to take some technical debt in order to innovate more in order to make more money. But you have to manage it. You have to measure it. You have to not let it go get out of hand. So that’s kind of the point.

0:53:34.2 BQ: Yeah, there’s always that tipping point where it starts to drag you down. So another set of questions has popped up too around just employee morale and retention and recruiting and kind of, I guess, especially over the last couple years we hear the same thing, which is technical debt and all these projects are… It makes it harder to recruit and retain and morale can actually be pulled down. So maybe we kind of talk a little bit about the people side of and how that debt affects people and engineering teams.

0:54:06.9 MA: Amir, you want to start on that one?

0:54:09.6 AR: Sure. I think developers like to feel creative and productive. When you want to feel creative and productive, you want to use things that can keep you creative and make sure that you can see your productivity come to fruition. When you’re working on a big monolithic application, sometimes we see developer ramp up times up to one year. So it takes one year before when you recruit a developer until they’re effectively contributing to the company. This is a very, very depressing first year. Yes. Okay. And that depressing first year will cause attrition. And even if it’s not attrition, and you’re just talking in terms of like an average three years spent in a company. So they were three years good, right? So they were productive two years out of three. It means that you also pay them 33% more every month to be productive. So it’s only also from an organization or perspective, you’re spending too much. And also from a developer’s perspective, man, it’s mean to just put someone on an application that takes them a whole year to learn and not really feel that they join like the right organization. And it’s very hard to get out of that hole, I think.

0:55:40.5 MA: Yeah. And I think that that pattern, I see that pattern a lot. And that pattern can be repeated across a single organization. I join a team. It takes me almost a year to figure out where all the moving parts are and be a valuable contributor. And then, suddenly I am assigned to another team or I wanna join another team somewhere else. Well, guess what happens? Now I have another big ramp up time so I can figure out how long it’s gonna be before I can be a valuable contributor for this team as well. Add to that cultures where the actual pace of change is quarterly or every four months or every two months. Now I don’t even get feedback on my contribution until two or three months later. Whereas if you look at organizations like Etsy where they completely change their culture to be around releasing to production. They have a little ritual that they, I think they still use this. The first… When you first joined the company, the first day of the company, you put something in production. You press the button that sends production code live to get you into that habit of either they’re small sprints and you get to contribute early and often.

0:56:56.2 MA: So they completely rearrange their culture to try to beat back that depressing thing that I see so many people experiencing that I experienced myself, which is, man, it’s going to take me a long time to figure out where all the levers are before I can actually contribute. So I think it’s a big, and as you begin this sort of re-imagination as a practice, what we would call continuous API management, now you’re gonna change the culture of the organization quite a bit because now everyone has to sort of get used to that idea that, “No, no, no, what did you change today? What did you change this week? What did you contribute now?” And suddenly it becomes a lot more vital inside the organization.

0:57:43.0 BQ: Yeah, that’s great, great conversation. Yeah. I think we’re starting to hit the top of the hour here and I wanted to say, yeah, thank you both. Great, great, great conversation, great dialogue. Just great material. Mike’s written some awesome books, hit his page on Amazon. I think you have a new book coming out on RESTful Web API patterns. Is that also true, Mike?

0:58:10.2 MA: Yeah, there’s a Patterns and Practices book that was just released this month. So it’s available on Amazon and wherever your eBooks and so on and so forth are sold. I’m actually very excited. It collects up a lot of these patterns and practices that I’ve learned over the last 20 years or so into one place. So I’m very excited about it.

0:58:27.0 BQ: Yeah. That’s great. Thanks. And thanks for joining us, Mike. I love it. I encourage everyone to take a look at Mike’s materials. And they’re very consistent with a lot of things we see, but we always learn something every time we talk and every time I read stuff. So thanks for joining us, Mike.

0:59:16.3 MA: Thanks.

0:59:16.8 AR: Thank you.