“AI-Driven Innovation” is an oxymoron to excuse over-relying on “legacy code as a service”. I don’t think there is any sort of paradox here in the way that “AI-Driven Innovation” as a strategy is in consequence risk adverse and not a strategy for innovation and creativity. I don’t see a need to bury any ledes here and I think this is awful on multiple levels. I worry it is going to take too many years for companies to realize the mistakes that they are making right now. It’s had me making too many jokes to hide the depression that I feel that I need to find a new career altogether.

“AI-Driven Innovation” is an investment in a rental economy of “legacy code as a service”. Legacy code is always an indicator of some of the most risk adverse pieces of a company’s software portfolio.

Legacy Code as a Service

I have spent so much of my career in “the legacy code mines”. It’s always been a skill set I’ve tried to downplay on my resume. It’s always been a skill set I’ve kept being asked to return to because I’ve earned a lot of experience in it, often the hard way. For decades, I’ve joked that my “retirement” plan was to find a cushy high paying, low effort COBOL job.

But now I don’t think those joked about jobs are going to be so easy to find, not because the LLMs are going to replace all that COBOL code but that the LLMs are going to help companies build so much more, equivalent or worse legacy code faster than ever. The LLMs are (unfortunately for my retirement fancies) commoditizing legacy code. They are making it plentiful. They are giving tomorrow’s legacy code experts more work than ever before at an unprecedented scale.

I’ve been accused about being overly unfair to LLMs by calling their output “legacy code as a service”, but think about it: which companies offered their truly innovative, proprietary, most modern, most domain-driven codebases to LLMs for training? LLMs were trained on open source and a lot of open source is just a funny place to stash “someone else’s legacy code” or at least in the most generous cases “some other business’ complement”.

If AI can’t train on the innovative software, how can it help you find innovation in your own software strategies?

There is No Creative Ghost in the Machine

Show me the part of the LLM family of algorithms that is the “creativity” and I can show you a slot machine and try to teach you about the Gambler’s Fallacy.

People are inherently bad at reasoning about probability-based algorithms (and probability and statistics math in general) and some of the current love affair of LLMs seems explicitly to be that people don’t understand the randomness of algorithms deeply associated with probability and are happy to anthropomorphize it or assign to it a creativity or “surprise” that is really just random variation on a theme (and eventual regression to the mean).

You could put a lot of creativity into the input of the system, so called “prompt engineering”, but what good is laundering that much creativity through a machine designed to produce the most statistically average outputs from it?

That of course entirely relies on people seeing “prompt engineering” as a creative tool for innovation. Most of the companies seeking “AI Innovation” wrote off software as “overhead” and “cost centers” decades ago. In a lot of those companies the creativity of the software engineer is a defect in what should be cogs in a well-oiled machine that takes business requirements and spits out software. “AI-Driven Innovation” only continues to further misplace and confuse the sources of creativity in software development. These companies aren’t going to learn anything from the mistakes they are about to make. (The mistakes they are already making.)

There is No Warranty Among Thieves

When you hire a software engineer, they have an ownership and a responsibility to the code they write. When working on legacy code, that person with any sort of ownership is already gone, and that is almost always an impediment. Some of the key skills on working on a legacy code codebase are essentially mind-reading, trying to read everything that person wrote around the time they wrote that software that you can get some semblance of feeling like you can at least temporarily think like they used to think in those days on that project.

LLMs are incredible in how much they are a (soylent) sausage factory for taking all those human pieces (the poetry hidden in the required poetry forms) and smearing them into a reconstituted meat slurry. The output code has no ownership and is divorced from any ownership it might have once had in a gelatinous puree. There’s no way to read everything “that developer” wrote at the time an LLM output what it did. We don’t have access to the full training sets. Even if we did, the major models themselves fully retrain at a roughly six month cadence and aren’t the same from “version to version”. Sometimes they aren’t the same from month to month due to partial retraining that the model makers don’t feel is worth a version bump. (An LLM is a Ship of Theseus by design.)

Even if you could somehow connect the code that the LLM has produced to specific open source code via the training set that you don’t have access to, one of the first safety gates in almost every open source license is a “No Warranty” clause. It’s one of the foundations that keeps open source a viable practice. (Just as software developers generally don’t keep ownership when they are severed from a company is a sort of a foundational creator of legacy code.) Many open source projects will offer services similar to a “warranty” or “support contract” for some material cost (sometimes even just as volunteer effort returned the open source community). Yet the LLM can’t provide even this disclosure, can’t sell you that service, can’t tell you what exactly you have “licensed” or how many “No Warranty” clauses are between you and whoever originally owned the code it just output. There’s not just “no owner” there is a multi-layer onion of “negative ownership”. Your EULA with the LLM declares “No Warranty”. Most of the code the LLM trained on declares “No Warranty”. Who really owns anything? It’s all just a rental to some landlord, as a service.

Then there is the added awful layer in that onion that much of the code the LLM trained on was also effectively stolen, according to the licenses and agreements in place around that code. A lot of the open source world is built on “copyleft” licenses designed to legally encourage “what you take from the open source community you volunteer to bring back and continue to enrich the community with your own contributions”. LLMs today are ignoring these legal requirements. LLMs are not attempting to enrich the community with their sausage factories, they are trying to claim that they don’t need to because the sausage is so different from the animals it was processed out of. In the process of reconstituting that meat slurry they claim that they’ve liberated it from such concerns as “this is an illegal slaughterhouse” and “this is a health code violation” and “this is just generally a violation of common sense human decency”.

Is that a great foundation for “AI-Driven Innovation”? Does a nasty-smelling onion of “no one owns this code”, “no one wants to know what the real ingredients in this code are”, and “if they did know all the ingredients in this it would be illegal” sound like “innovation”? There was an era in American economics when companies innovated on how much diseased rat meat they could get away with in their sausage. I can’t be alone in thinking this will all end poorly. I’m also sure I’m not alone in thinking this won’t end poorly fast enough to save us from so many of the worst mistakes.

Garbage In, Garbage Out

“AI-Driven Innovation” is a garbage plan of creating garbage inputs to software that produce nothing but statistically probable legacy code, as a service. It’s a simple oxymoron. There’s no real “innovation” to be found in the output. It’s going to keep checking boxes in PowerPoint slides to an investor class that has bought into “AI-Driven Innovation” as a shared delusion of their (inherent disconnect from) reality rather than accepting it to be a sad improbable dream that makes no sense when easily questioned. In the meantime a lot of software is going to regress to some ugly averages and real software innovation will take a huge hit.

In the mean time, the truly innovative, experienced software engineers are encouraged to accept layoffs if they can’t “align” on “AI-Driven Innovation”. (Leaving plenty of now immediately legacy code behind in so doing.) In the mean time, tomorrow’s truly innovative software engineers are told to avoid growing real experience and instead feed LLMs to generate so much mediocrity, if they have jobs at all. In the mean time, “AI-Driven Innovation” is crafting a mesmerizing illusion that software as a labor practice is over and companies don’t really need software developers at all.

Garbage in, garbage out, in a feedback loop threatening to spiral to destroy real innovation in software development.