Why Most IT Projects Fail. And How Agile Principles Help

This content is syndicated from by Kelly Waters. To view the original post in full, click here.

why most IT projects fail, and how agile principles helpI recently wrote a blog post explaining why most IT projects fail to meet expectations.

As a follow-up, here I take a quick look at the common reasons for project failure, and how I think agile software development methods and agile principles mitigate these risks and issues.

I have a strong view that agile methods help significantly with a lot of these areas of project risk, although I'm sure not all. This is an enourmous subject, so I can't really do it justice and keep this blog post a reasonable length. So you'll just have to take or leave my comments on face value :-)

Here's how I think agile principles help, in my experience:

Common cause of failure How agile helps
Project Initiation & Planning Issues
Unclear or unconvincing business case Agile principles don't help directly with the busines case - in fact it can be hard to make a business case for agile projects due to their agile (i.e. unpredictable) nature. However the principles of incremental delivery and frequent deliver of products can help to get an initial solution out, test the proposition with the market and get real customer feedback to inform the priorities for further development.
Insufficient or non-existent approval process Agile business cases and new business propositions benefit from the same diligence and challenge before investing large amounts of money, just as in any other approach to development.
Poor definition of project scope and objectives Agile projects also benefit from clear definition of scope and objectives, even though details are allowed to emerge throughout the development.
Insufficient time or money given to project If only agile could solve this!
Lack of business ownership and accountability Active user involvement, or involvement from a key user representative in the business, creates an environment that fosters close collaboration and cooperation.
Insufficient and/or over-optimistic planning This is an interesting one. I (like many agile enthusiasts) believe that it's practically impossible to plan every detail of many software development projects up front, hence expectations are better managed by active involvement in the project, frequent delivery of product on an incremental basis.
Poor estimating Agile methods provide some important principles to help with accuracy of estimating: estimating should be done by the whole team as a collaborative process; tasks should be broken down into micro pieces (ideally less than 1 day) so progress is measurable on a daily basis; 'velocity' is calculated on the number of estimated hours delivered only when a feature is 100% complete, providing a gauge for how much development can be safely included in future iterations. In non agile methods, this approach can also be adopted and is known as 'earned value analysis'. So even if you're terrible at estimating, this approach can be self correcting as long as you're consistently terrible :-)
Long or unrealistic timescales; forcing project end dates despite best estimates Agile projects encourage short and regular iterations, developing the software and delivering working product in small bitesize pieces.
Lack of thoroughness and diligence in the project startup phases Rather than diligent and thorough planning, agile principles propose to deliver small increments of working product and get continuous feedback from active user involvement throughout the development cycle.
Technical & Requirements Issues
Lack of user involvement (resulting in expectation issues) Active user involvement and continuous feedback is one of the most important principles of an agile approach.
Product owner unclear or consistently not available One of the reasons product owners are unclear in traditional projects is because they are asked for far more detail than they can handle, too early in a project and when they cannot visualise the solution. Instead, agile requirements are kept lightweight and visual, and delivered just in time for a feature to be developed. Availability must be forthcoming for agile principles to work, as it's essential for constant collaboration.
Scope creep; lack of adequate change control Agile projects may stick to the broad scope of the project, but requirements are allowed to emerge and evolve. However the project must include non-essential requirements at the outset, in order for emerging requirements to be traded with original scope.
Poor or no requirements definition; incomplete or changing requirements Agile projects expect requirements to be incomplete and changing. That's the nature of software. Instead of resisting this, agile projects provide for it by allowing requirements are allowed to emerge and evolve. Requirements being produced on a feature-by-feature basis, just in time to be developed, helps with definition because it breaks this intensive task into small pieces instead of being a mammoth effort up front.
Wrong or inappropriate technology choices Agile projects can surface inappropriate technology choices early, as they encourage frequent delivery of product on an incremental basis. Testing is integrated throughout the development cycle, testing each feature as it's developed. Doing so can help to ensure inappropriate technology choices are identified early, before too much of the software has been developed.
Unfamiliar or changing technologies; lack of required technical skills Agile methods don't help directly with this issue, although can help to surface such issues early, and make them visible.
Integration problems during implementation Agile projects are delivered in short iterations, with testing integrated throughout the development. This requires continuous integration of the code and frequent builds, removing the need for a lengthy or problematic integration phase at the end of the project.
Poor or insufficient testing before go-live Testing is integrated throughout the development.
Lack of QA for key deliverables Working software is the key measure of progress, as the software is developed and delivered in regular iterations. This helps to ensure that the adequacy of any other deliverables is highlighted early and made visible.
Long and unpredictable bug fixing phase at end of project Testing is integrated throughout the development.
Stakeholder Management & Team Issues
Insufficient attention to stakeholders and their needs; failure to manage expectations Active user involvement ensures two way feedback throughout the development.
Lack of senior management/executive support; project sponsors not 100% committed to the objectives; lack understanding of the project and not actively involved All projects need this, agile or otherwise.
Inadequate visibility of project status Agile projects provide clear visibility of measurable progress on a daily basis.
Denial adopted in preference to hard truths Humans, eh? Who needs 'em!
People not dedicated to project; trying to balance too many different priorities I don't think this ideal is specific only to agile methods, but agile principles propose small, multi-displined teams dedicated to the development of the product.
Project team members lack experience and do not have the required skills Agile principles may help to surface such issues early, as they may well be evident in early iterations of the software. Frequent delivery of iterations and continuous testing can help to mitigate this risk when it might otherwise go unnoticed until much later in the project.
Team lacks authority or decision making ability Agile teams must be empowered.
Poor collaboration, communication and teamwork Close cooperation and collaboration between all stakeholders is essential.
Project Management Issues
No project management best practices Can be an issue when applying agile methods, unless using an agile management practice such as Scrum.
Weak ongoing management; inadequately trained or inexperienced project managers Agile methods and principles are just management tools. A fool with a tool is still a fool!
Inadequate tracking and reporting; not reviewing progress regularly or diligently enough Agile practices have daily status reporting built into the process, providing clear visibility and measurable progress on a very regular basis.
Ineffective time and cost management Daily visibility of measurable progress.
Lack of leadership and/or communication skills Sadly, adopting agile principles doesn't make inspirational leaders. However agile principles do encourage a particular form of servant leadership, empowering the team to take responsibility and make timely decisions.

Of course most of the things I've cited as agile mitigations can be applied in any project methodology, including waterfall. It's just that they are more the norm and explicitly emphasised in agile principles and practices, which is something I really like.

Most of all, agile principles help enormously with visibility, collaboration and engagement. This can transform the levels of trust and teamwork between the product team and its key stakeholders. The consequence of this is enhanced satisfaction, due to a much greater understanding of the commitment and expertise of the team, and the issues and risks they face. If failure, by definition, is a problem of not meeting expectations, you're half way there when you adopt an agile approach.

However, all this, in the end, ultimately still relies heavily on the skills of the team. Just because agile methods state these principles and have some inherent risk management built within the process, it doesn't mean the team necessarily has all the skills and experience it needs to execute the princples consistently.


See also:
Most IT Projects fail. Will yours?
DIY Guide for Fixing a Failing Project - by Mike Cohn
10 Key Principles of Agile Software Development
10 Good Reasons to do Agile Software Development

Leave a Reply

Your email address will not be published. Required fields are marked *

9 + 13 =

There are 101 ways to do anything.
To find the best way, sometimes you need expert help

What People Say

“Kelly was engaged as a Program Director on a complex business and technology transformation program for Suncorp Commercial Insurance. Kelly drew on his key capabilities and depth of experience to bring together disparate parties in a harmonised way, ensuring the initiate and concept phases of the program were understood and well formulated. Excellent outcome in a very short time frame. ”


“Kelly and I worked together on a very large project trying to secure a new Insurer client. Kelly had fantastic commercial awareness as well as his technical expertise. Without him I would never had secured this client so I owe a lot to him. He is also a really great guy!”


“I worked with Kelly whilst at Thoughtworks and found him to be a most inspiring individual, his common-sense approach coupled with a deep understanding of Agile and business makes him an invaluable asset to any organisation. I can't recommend Kelly enough.”


“Kelly is an Agile heavy-weight. He came in to assess my multi-million $ Agile development program which wasn’t delivering the right throughput. He interviewed most of the team and made some key recommendations that, when implemented, showed immediate results. I couldn’t ask for more than that except he’s a really nice guy as well.”


“Kelly revolutionised the way our digital department operated. A true advocate of agile principles, he quickly improved internal communication within our teams and our internal clients by aligning our business and creating a much enhanced sense of transparency in the decisions the business was making. Kelly also introduced a higher sense of empowerment to the development teams...”


“Kelly’s a leading program director with the ability to take charge from day one and keep strong momentum at both a program and project level driving prioritisation, resourcing and budgeting agendas. Kelly operates with an easy-going style and possesses a strong facilitation skill set. From my 5 months experience working with Kelly, I would recommend Kelly to program manage large scale, complex, cross company change programs both from a business and IT perspective.”


“I worked with Kelly on many projects at IPC and I was always impressed with his approach to all of them, always ensuring the most commercially viable route was taken. He is great at managing relationships and it was always a pleasure working with him.”


“Kelly was a great colleague to work with - highly competent, trustworthy and generally a nice bloke.”


“Kelly was a brilliant CTO and a great support to me in the time we worked together. I owe Kelly a great deal in terms of direction and how to get things done under sometimes difficult circumstances. Thanks Kelly.”


“Kelly came to the department and has really made a huge impact on how the department communicates, collaborates and generally gets things done. We were already developing in an agile way, but Kelly has brought us even more into alignment with agile and scrum best practices, being eager to share information and willing to work with us to change our processes rather than dictate how things must be done. He is highly knowledgable about agile development (as his active blog proves) but his blog won't show what a friendly and knowledgeable guy he is. I highly recommend Kelly to anyone looking for a CTO or a seminar on agile/scrum practices - you won't be disappointed!”


“Kelly is an extremely talented and visionary leader. As such he manages to inspire all around him to achieve their best. He is passionate about agile and has a wealth of experience to bring to bear in this area. If you're 'lucky' he might even tell you all about his agile blog. Above all this, Kelly is great fun to work with. He is always relaxed and never gets stressed - and trust me, he had plenty of opportunity here! If you get the chance to work with Kelly, don't pass it up.”



To explore how we can help you, please get in touch