Software development is not innovation

By Thomas van Rompaey - Ruim Baan Solutions

Summary

Labeling routine software development as "innovation" creates predictable business failures that consistently result in budget overruns and delivery delays.

In my experience architecting enterprise solutions, projects marketed internally as "innovative software development" consistently struggle with timeline and budget adherence. The culprit isn't technical complexity—it's misaligned expectations between stakeholders .

In the most positive interpretation, innovation can be seen as a way to think outside the box and provide the business with a tailored solution that is not stuck in legacy-thinking. When not managed properly, thinking outside the box for projects that have a tried and tested solution, results in increased risks for software projects. This risk translates into issues that impact time (planning), money (budget) because the scope (quality) is misaligned to the expectations.

It is important to improve and try new things in order to solve tomorrows business problems. My main point here is to make sure though, that there is a explicit and shared expectation on the risks. If that is not in place, just revert to what we should be doing most of the time: Deliver a working solution within budget and with the right quality. From my experience for most companies this is a large enough challenge in itself. It's not about choosing the 'easy' route, it's about choosing and aligning on the right approach.

Assuming that you made the conscious decision to innovate, my proposed way to manage innovation in your process is to implement a separate workstream (or even perhaps a separate department) that purely focuses on innovation and where projects are allowed -and maybe even expected- to fail. When you get a positive result, a viable solution, use this to hand over the implementation to a team that is good at implementing stable, production-ready solutions.

Introduction

With my 25+ years of experience in software development, I have worked in all relevant roles (including management roles) around software development and I have been a solution architect for the last 10 years. To say I have seen it all seems presumptuous and I hope that is indeed presumptuous because I would never say that I know it all. Also, seeing and learning new things will keep the rest of my career interesting! That said, I also have seen most organizations struggle to manage software development or implementation.

Having seen a few methodologies (remember Rapid Application Development?) in my time, each and every one tried solving the same problem: Bringing solutions to end users quickly, within budget and the right quality. Even though we've come a long way, I see businesses and companies still struggling to be predictable when it comes down to implementing bespoke software solutions successfully. This certainly applies to custom software solutions but just as much to configuration of COTS (Commercial Of The Shelf) solutions. A well-known example where customization can spiral out of control is ERP implementations.

Scope

Solving all issues in software implementation and development would be a fun challenge but for now I will focus on:

Why would it be a problem to see software implementation and development as "innovation", what issues does that mindset bring and what can be done to improve on this.

If you look at the textbook definition of Innovation it is quite broad:

Innovation
The act of innovating; the introduction of something new, in customs, rites, etc.
A change effected by innovating; a change in customs
Something new, and contrary to established customs, manners, or rites.
A newly formed shoot, or the annually produced addition to the stems of many mosses. (source)

💡

Because this is the first time you are doing this, it is innovation by definition. So there is the perspective of doing something new in your organization and the perspective of doing something the planet has never seen.

My focus here is on the majority of software development or implementation projects; the ones that optimize and improve business processes. I am aware a niche software startup or custom commercial website is something different than eg an ERP implementation. This article does not try to solve everything but rather share some key insights that may or may not apply to your situation.

💡

While optimizing business processes using software would not qualify as innovation in my book, experimenting and learning definitely are key in an organization that wants to be a differentiator in their field.

Context

Though there are many challenges in the world of IT/software implementation and development, the one I would like to focus on here is the challenge of managing expectations and making sure all parties involved have a holistic understanding of what is a likely outcome.

Having worked with a lot of people and teams who I would gauge as far smarter, more experienced and driven to excellence than I could ever be, one would wonder what the problem is to deliver what the business needs, within budget and on time.

Should we be predictable and dependable?

When implementing solutions to a business problem, being predictable in the fact that you can deliver a working solution goes a long way. Not every solution has to be innovative. In my experience, the problem you are trying to solve is in many cases an efficiency improvement of your business processes. Most likely you have an existing process like eg sales, order tracking, a, that is not as efficient as it could be, error prone and you want to improve on that.
Especially when improving key business processes (e.g., finance) your key metric of success should be whether you can pay your employees and the invoices in time. For most companies, having accurate financial statements or financial insight should always take precedence over "thinking outside the box".

Always deliver within a reasonable timeframe and at reasonable costs. Something new, and contrary to established customs, manners, or rites might sound inviting to some, but most of the time when talking to end users they simply expect to get a working solution that helps them solve their day to day problems.

‼️

Should we avoid risk, challenges or the opportunity to bring the organization to a new level while improving (automating) a key process?
No of course not. If you would do that then most likely you would never initiate improvements.
Just know that you have agreement and consensus on the goal you are aiming for and the risks are agreed and managed.

Be sure to deliver a solution. Perhaps not the one you dreamed of, but the one that works.

If you must : Fail quickly

Taking risks as a business is part of the game; nothing comes for free.

When doing something new, something for which the outcome is not guaranteed: fail as fast as possible, but do make sure to learn from the project. It could be that the business learns what they really want, that the idea doesn't bring the expected benefits in the real world, that the business case is not viable anymore eg due to a changing market, your team is learning and needs to mature before they can deliver the needed product, etcetera. All these things are valuable, never qualify as waste, and they will bring you closer to success the next time. Just remember to celebrate what you have achieved so your team will be just as eager to pick your next project, the one that might make it one step closer to business success.

💡

As one of the key principles in Agile software development, failing fast is in my opinion not embraced enough. (See also link) Even in the western world where losing face is interpreted differently from other parts of the world, your last "failure" is still often the thing people remember. So make sure your co-workers, managers, sponsors understand what "success" looks like. (See also link)

Defining clear goals and responsibilities

Bringing the software implementation process together in one overview might help you to understand from what perspective innovation might be wanted or unwanted.

💡

Note that I'm calling it here "software implementation process" instead of "software development or implementation process". the reason is that it is a generic process, the question whether you buy or build has not been answered.

To capture this process, I have created this overview. It aims to show the goals and responsibilities for teams involved in software delivery. And how these can work together to provide predictable services.

The way to read this overview is to use the yellow Inputs to guide you through the processes. So eg a Business Idea is input to decide on the next steps, in the same way as knowing that you have a Clear Solution is. The processes Operations, Innovation, Implementation are relevant only after you have identified the Inputs. So a process is never leading in guiding your next steps, the Inputs are.

Business idea, where it all starts?

Most models would say that a business idea, and more specifically a vision, is the starting point for any project. In the reality I've seen most of the time, there are definitely visions and ideas, but the process to get a project started is often a collaboration between people with ideas, people with skills in realizing solutions, and people finding each other to align these two. The vision is used to validate the idea to the business direction, but is often not the initial driver for a new project. Is that an odd statement? I would say not. Although the vision is essential, it is—and should be—too high-level to be the specific reason a software implementation or development project is started.

Getting conversation and cooperation between business and IT rolling is the way to sprout new ideas and the starting point for the things a project needs: skilled people, budget, and time. Before we get there, though, a lot of things need to happen. A common pitfall is to jump into the details and start 'solutionizing' the ideas; taking one step back allows us to define the result we are looking for in advance by defining requirements and key success factors.

Clear solutions. You know it when you have them.

Input Business idea directly to Process Implementation

When you have a good understanding of what problem is we are solving and what the Minimum Viable Product (MVP) would be (MVP terminology being a whole separate chapter, so I won't dive into that topic right now), are we sure we know what we want and what we need? Do we have a clear picture of where we are going? Let's go then, and implement it --no time to waste.

🧠

A simple measure for knowing whether your idea is ready for implementation is whether you can explain the goal in one paragraph (in writing!) and whether the key users of the new solution are happy if they would get exactly what is written down.

💡

One interesting way to bring across what you are trying to achieve is the High Level Operational Concept Graphic, a graphical image of what a scenario might look like when implemented. Although developed for the DoD, a business application might be just as valid. (see link)
The only pitfall is that an image likely is not unambiguous so use it with care.

Given the fact that defining the solution was apparently pretty straightforward and everyone seems happy with it, in many cases this solution will be an off-the-shelf tool. If it's that simple, someone else will likely have thought about it long before you did. This might be a broad statement that could be proven wrong by specific examples, in a broad sense for companies that are not in the software business, I would say this is true.
Yes, you might need to tweak or configure the off-the-shelf solution, but what if you find off-the-shelf options that almost fit? Then, likely, the key feature you are looking for is not software but process or organizational change. I would bet that the one specific thing can be solved by tweaking your processes instead of the software, and this would be far cheaper. Doesn't the software support that one specific reporting format the business is used to? I think you know the answer on what to do.

For simple processes, you can consider a low-code/no-code solution to automate what you are looking for. But be aware that you will need to maintain and support this solution, and it will likely grow in complexity over time. Still, it can serve as a stopgap and buy you time in the short term.

Viable solutions, world are you ready for us?

Process Innovation to Process Implementation

By going through the Innovation process, and repeating that until you have a solution that ticks the right boxes, you are ready to mature the solution. But do so only when you are confident the solution will bring the expected value and that it is worthwhile to invest in maturing the product into a supportable solution. This means you’ve hit the jackpot — your plan has come together into something that actually works.

Would you need a full-blown solution as output from your Innovation process? No. Does the solution need to be 100 % ready for go-live? No. If you have great confidence that honing this solution and smoothing over the rough edges will be sufficient to bring you a successful implementation, then go ahead.

Handover to Operations (Support & Maintenance)

Process Implementation to Process Operations

Implementing business solutions is a skill that requires a certain type of flexibility and a skillset that is different from supporting and stabilizing an environment. From a Support and Maintenance point of view the key thing to avoid is uncertainty and unpredictability. The show must go on and with as little tickets as possible. This capability drives the handover from the implementation team to the stabilizing team, if you could call it that. Make sure not to put the burden of support on the implementation team, as it would not be their strength.

💡

“You build it, you support it” is a driver for success for teams that work on ongoing continuous development projects. Most businesses and teams do not have that luxury, so any additional work to support the solution after go-live will be added to the workload, project after project, leaving no time for the team’s key skill: implementing sound solutions.

Remember that Operations doesn’t just mean IT operations. The business side is involved here as well, and they should be providing functional support through, e.g., key users. A proper handover from the implementation team should include a go-live, a successful hypercare period, and proficient end users.

Practical overview

Process	Description
Operations	IT or business day to day support and maintenance. Benefits from structured and predictable processes, including SLA (Service Level Agreement) to monitor performance.
Innovation	The other side of the spectrum; Works in iterative cycles to move from a vague or high level idea to a more concrete product.
Implementation	Implementation of commercial solutions, configuring these to align with the business requirements and implement custom solutions based on clear goals and requirements. The latter can be a next step in a Innovation process after it is concluded the innovation is viable to be implemented.

Key Takeaways

Software development or implementation is usually about applying known solutions for common business problems, within the capabilities of your team. Tweaking commercial software should have preference here.
Success comes from being predictable and dependable to your sponsors, the business.
Definitely do innovation, find your differentiator where it is adding to either your bottom line, (end)customer satisfaction or efficiency.
Remember that risk and reward go hand in hand so make it clear to the team what you are expecting. Don't measure stability and reliability on a brand new innovative product. Measure stability and reliability on a honed down, stabilized (supportable) solution.
Support the teams on the agreed success factors and celebrate when they are doing what they should be doing.

References and further reading

An Approach for Software-Intensive Business Innovation Based on Experimentation in Non-software-Intensive Companies - PMC

Several companies whose businesses are not centered on technology might fail to innovate and get advantages over their competitors. For them, meaningful innovations are not necessarily related to the usage of new technologies but the optimization of…

NCBI home page✉Corresponding author.

‼️

Acknowledgment:
AI has been used for spell checking and reviewing this article's content. All content, including ideas, suggestions and learnings are my own.