This article is a guest blog post by Justin Leto
Warning: This article could save your Agile project and your career.
Despite the now long standing and growing debate over what Agile is and whether it’s effective, the only question that matters to you and the business is whether you are effective. If project success is what the business is using to measure your performance who cares about convention? I don’t. Neither should you.
Yet my experience suggests that a lack of confidence in one’s ability to solve problems often causes project leaders to turn their brains off and cling to dogma, hoping it will provide cover in the event of failure. Spoiler alert: it doesn’t.
Management is encouraged to enable this behavior by showing “management support” for Agile. On the contrary, management should demand success and support learning and change when there’s failure. What I see instead are practitioners doing the same thing over and over again regardless of the outcome—in deference to ideology. Management should simply charge the operators with the responsibility of bringing about the most success possible with the resources available, regardless of the methodology employed. Otherwise you provide the grounds for excuses and failure.
Some tenets of Agile are relevant to this discussion. For example:
The primary goal of Agile is working software.
While paying lip service to doing “just enough” documentation, the inference here is that face to face verbal communication and writing code is emphasized over writing requirements. Frequent deployments and receiving customer feedback is favored over a more structured requirements effort.
This premise—and the notion that frequent deployments are an appropriate substitute for requirements verification—is flawed. It should be:
The primary goal of Agile is working software that meets the business requirements.
I would argue that business acceptance has to be a requirement. Otherwise, where’s the accountability? Code that wasn’t designed to do what the business needs creates several costs (4 of which I list here):
1) The cost of developing a solution to the wrong spec.
2) The cost of fixing a solution to the right spec.
3) The opportunity cost of what you could have developed while engaged in 2.
4) The cost of lower motivation and reduced morale from throwing work over the wall and losing the confidence of the business and its customers.
Number 4 is particularly disruptive because it threatens future productivity and is more difficult to resolve.
Agile proponents would argue that meeting the business requirements is assumed, but the process offered is not credible for ensuring this for certain kinds of projects.
My goal for this article is to not just highlight where the agile methodology fails, but to explain exactly why it fails.
Complexity
Here’s a simple fact that should be self-evident:
As the complexity of the requirements increase, the resources needed to understand and communicate the requirements will also increase.
Critics of Agile often point to the type of project or the size of the team as indicating appropriateness. But I believe these criticisms mask “complexity” as the true cause. If the requirements are simple enough and could be easily understood and communicated, little would have to be written and a small team requiring informal communication could deliver.
Complexity in the requirements quickly causes an Agile project to go off the rails to a hybrid Agile/Scrum/Waterfall method affectionately dubbed “Water-Scrum” or “Scrum-Fall.”
Here’s why.
Of the additional resources required to address complex requirements, the most impactful is time. It simply takes longer for architects and developers to elicit, refine, verify, communicate and absorb complex requirements. Business users, customers or subject matter experts have other priorities that do not allow for them to commit large blocks of time on the development team’s schedule.
Result: Lack of stakeholder availability starts to push the schedule and now the team is idling. Sound familiar?
If Agile suggests that requirements should be gathered and delivered just in time for development, understanding how long this process takes is important. It requires planning. Understanding that a stakeholder can only contribute 15-25% of their time to the requirements process often means lengthening the runway beyond a sprint. It could also mean rearranging the order of features scheduled for development based on a combination of rated complexity and business value rather than business value alone.
Several important corollaries can be derived from this observation.
Corollary 1: Parallel development paths
Agile suggests a serial development path, but parallel paths may be needed to maximize efficiency. This might mean developers devote 15-25% of their time gathering requirements for the next feature to be implemented 2 or 3 iterations from the current one.
Corollary 2: Written requirements
Remembering what someone said 3 weeks ago in order to start development at the beginning of the sprint is not realistic. Requirements gathered over a long period of time and alongside other development paths will inevitably require that artifacts be created and managed. Yes, requirements need to be written down. This allows for both accountability and asynchronous communication, which adds value. The formula to calculate how many communication paths there are in a team is N(N-1)/2 where N is the size of the team. N does not have to grow very large to prove the value of asynchronous (written) communication.
Corollary 3: Scalability requires Specialization of Resources and a Division of Labor
I’ve heard people discuss what it takes to implement Agile. They claim that the best developers with a full vertical stack of competencies who are self-organizing and doing everything is what is needed. First of all, who are these people? Do they exist? Are there enough to scale your team? Better question: Are they available for your project right now? Spoiler alert: Probably not.
The problem with this thinking is that often these resources are either not available or they cost too much. What value does Agile bring if it requires the very best and brightest to implement it? Shouldn’t the job of a manager be to inspire and lead an ordinary team to perform exceptionally? Has moneyball taught us nothing?
Having someone do everything does not eventually make them great at everything. More than likely it makes them mediocre at everything including what they were originally great at. Requiring frequent context switching is not only inefficient, it degrades the quality of the developers’ core work product: the code.
The most important question here is: who will take pride in and be celebrated and rewarded for producing high quality requirements at the most useful level of abstraction? The coder? No. The business users? No.
When scaling up projects and teams, we need to address the time needed for complex requirements. We need a way to free the business users from the requirements process. To do this someone will need to be charged with that responsibility and commit him or herself to develop a specialized knowledge of the features, processes and rules of the products and the business. This person would ideally be 100% available to communicate, refine and verify requirements with the developers. Traditionally this role has been filled by product managers or business analysts.
Volatility
Market or customer demands may make requirements highly volatile.
On the flip side, a good question to ask is: are requirements changing because developers didn’t fully understand or ask the right questions about the business, the product or the customers? Is volatility indicative of an inadequate requirements effort?
Other features may have very low volatility. They may be core proprietary algorithms or well-defined formulas that do not fundamentally change often. Document what doesn’t change often.
This decouples documentation from the purpose of supporting initial development to instead capturing for the business the product features and business processes needed to educate new employees. It is a critical function of knowledge management and knowledge transfer. Systems need to be maintained and migrated over time making documentation highly valuable in the long run. By Agile’s own standard, do documentation that adds value. Failing to do so adds technical debt.
Severability
Like in contract law, I use the term severability to describe the interdependence between requirements comprising a feature or feature set. Complexity alone does not indicate low severability. There could simply be a high number of requirements or a depth to the requirements that makes a feature complex, but each requirement may not be dependent on others.
Low severability will affect the scrum process because building and deploying the full feature will require more time to complete than a typical sprint may allow. Often doing the up front work on a larger body of requirements is needed to present a clear and complete view of the feature to the developers at iteration start. Adjusting the length of the sprint to match the demands of the requirements is often the right approach.
Summary
Knowing how to evaluate project requirements for complexity, volatility and severability allows practitioners to more effectively manage schedules, resources, and risk. Understanding and communicating not just where Agile fails but why it fails in certain scenarios will aid in getting team buy-in for an alternative course of action.
Justin J. Leto has over 10 years of experience as a Senior Software Engineer. He holds a B.S. in Computer Engineering and an M.B.A. from The Pennsylvania State University. He is also a licensed Professional Engineer and a certified Project Management Professional. He can be reached at jleto [at] letoconsulting [dot] net.