Too often, I have seen postings by project team members out there in the corporate world who just got placed into an Agile SDLC (Software Development Life Cycle) from a previous Waterfall SDLC, who are confused and bewildered at the lack of process and the apparent chaos which has been bestowed upon them. There are numerous examples of an Agile SDLC being adopted incorrectly and unfortunately, as a result, there are now many software professionals who continue to be skeptical of Agile and frameworks like Scrum.
Former Waterfall team members are used to producing very thorough and specific blueprints of how to build every single aspect of a software application, with very little latitude given. Most everything is done for the designers and developers in advance, and with little thought and only needing comprehension of the blueprints constructed, they implement.
Some team members excel in a Waterfall environment, while other “thinkers” feel constrained by it. While both may be open to the changes offered by Agile, when not properly implemented, a new Agile SDLC can leave them perplexed and confused. An improperly implemented SDLC can cause sprints to fail, projects to overrun, team turnover to skyrocket, and jaded team members who now won't consider future Agile projects due to this negative experience. But lets not kid ourselves, even when doing everything right in Agile, sprints can still fail and projects can overrun. But one of the key differences is, you usually know about it well in advance so you can setup measures to get back on track.
More often than not, many people that adopt a framework like Scrum, go through the motions of what the framework specifies like time boxing, planning sessions and retrospectives, but lacks the complete implementation which is actually needed which a ScrumMaster training class just does not provide. Being a transformative leader in an Agile organization means you are looking at much more than the work that the development team is doing. (See my blog post on all the things a ScrumMaster should be doing BESIDES operating the framework)
The testimonials and case studies of companies who've successfully transitioned to an Agile SDLC can create high expectations in organizations about to make the transition themselves. However, many organizations fail to realize that this success did not come all at once, and that each organization went through a series of implementation phases (Exploring, Piloting, Expanding, Matured) to produce a truly lean and efficient SDLC that produces great value.
Researching the fundamentals of the Agile SDLC, learning processes and forming a strategy to implement those processes into the organization at all levels of the business' operation. This adoption phase can be the most important part of the organization’s adoption. Specifically, an organization creates in-depth business process models to communicate to the entire organization how it will operate under the new SDLC. The explorers will work to ensure that the developed process are in accordance with Agile SDLC “best practices” and will safeguard them from being compromised by “old” SDLC philosophies and business practices. As part of the exploration process, explorers will answer some critical questions like;
- What management tools will be used?
- How will requirements be managed?
- How will project work and budget estimates be calculated?
- What will the Sprint duration be, and why was it chosen? “Why” is important because the choice will have a dramatic effect on the productivity curve.
- How will the product backlog be managed?
- How will the release plans be drafted?
- Who will be the Product Owner?
- How will the Product Owner interact with key business decision makers and stakeholders?
- How will the daily standup be handled?
- How will Sprint demos be performed?
- What mechanisms will be used to ensure the SDLC is test-driven and what will the quality process flow be like?
These and many more critical questions need to be answered before any further adoption should be attempted. Diving straight into the Agile SDLC adoption without any process planning for anything other than a micro organization can be adoption suicide and cause the executives in the company to abandon it all together. The more people that are involved the more robust the adoption, coaching and training plans need to be. For a couple of reasons I even recommend applying an Agile approach to the Exploration phase itself. Doing this will allow those involved to learn, practice, and internalize Agile principles which they can pass on to others and leverage in later work. Knowledge is shared, and unless the organization is bringing in Agile consultants to train and coach, the exploration team needs to be put through the ringer of trial and error as soon as possible. The idea here is to fail early so you can succeed early.
Every new Agile organization starts out as “Exploring” the Agile SDLC to learn what the possibilities and rewards are of implementing the SDLC. Commonly though, an organization will focus so much on the possibilities, success stories and the definition or manifesto1 of Agile, that they fail to really grasp what Agile is and how to implement it with a team that has never experienced it properly. Possible root-causes of this failure, for example, can be:
- pressure on the explorer(s) for a rush towards implementation, not being given adequate time to understand the SDLC and how to implement it;
- pressures on the explorer(s) to mix old Waterfall production processes in with the new Agile elements or the organizations insistence that a critical Agile SDLC process be excluded in favor of internal corporate processes which might be political or obstructive in nature;
- a misunderstanding of the SDLC in its entirety because of reference material used to obtain the knowledge;
- and even an outright lack of interest in putting together a documented business process of how the Agile SDLC will work in the organization which adheres to the Agile SDLC standards.
Most often, organizations find it very difficult to shed their old processes and attempt to mingle what they know, with what they don’t know in an attempt to make better sense of it. As part of the exploration process, it is essential that the Explorers protect the Agile SDLC from being degraded by “old” processes and practices from the deprecated SDLC. As the adoption moves forward into the piloting phase, it will be up to the Project Manager or ScrumMaster as well as team leaders to continually protect the SDLC from this.
In a large organization conducting the first project with a small team is critical to rolling out Agile effectively. Attempting to roll out a broad Agile adoption without having significant coaching mechanisms available is very risky. It takes no less than 3 to 4 Sprints for a Waterfall team to start getting the hang of it. Adopting Agile into an organization with a small team at first, who can later act as Agile coaches and become team leaders to future Agile workers is preferred. Of course organization-wide adoption from the beginning is possible, but only with a very aggressive coaching and facilitation program staffed with experts who know what they are doing. To help facilitate this, companies may hire independent consultants who are seasoned Agile professionals and can coach the entire organization based on their years of experience. In any case, Agile adoption by an organization is no different than attempting to adopt any other piece of software, technology, mission or objective. Even with expert training and coaching, there is a learning curve for everyone involved and some “try and fail as you go” should be expected -- but the likelihood of success is far greater with good training, coaching and a proper SDLC implementation. Notwithstanding, with a properly setup Agile SDLC, you can better respond to any failures very quickly without it affecting the overall project’s success. This is because the SDLC maintains constant introspection.
It is common for people to read the Agile manifesto and related artifacts and incorrectly perceive that:
- budgets must be open ended and you cannot with any degree of accuracy predict how much money, time or effort will be needed to produce an Agile project;
- business people and developers must work together daily throughout the project;
- detailed requirements are not needed, or at least not until the developers start;
- business process models, use cases, test cases, data models and etc are abolished and left to the engineer to figure out proactively (while this can be done on elite or pole position teams, it is rare with drone teams);
- business people write User Stories or developers pull requirements from users;
- business people are the decision makers;
You just can’t throw together a team from a Waterfall environment, say you’re going to do the project with Agile, and then expect anything good to come of it. Similarly, doing a standup each day and mandating requirements be specified at a Sprints' conception by the developers and that those same developers figure everything out won’t be efficient either.
Agile is not a shortcut whereby you can disregard planning, nor is it an “easy button”. It is however an amazing way to streamline the SDLC into something which is lean and a super “business and user value” production machine. Agile, when implemented properly, takes everything we know about the typical requirements of an SDLC and gets rid of the fat, streamlines communication and focuses on what generates value.
The great thing about Agile, is that there are hundreds of ways to implement it. Unfortunately, some of those ways are damaging when misunderstood. That said, a successful implementation requires processes and roles to be drafted in advance, with a clear understanding of how they comply with the Agile SDLC guidelines.
The Piloting phase has sub phases. As an organization adopts Agile, they commonly roll it out from a single amateur team to a single intermediate team, followed by multiple amateur and intermediate teams under one program. An organization should not consider expanding its Agile adoption until all the initial teams have adopted Agile and are performing sprints with success.
The expanding phase is when an organization is migrating from a single program having one or more intermediate teams (pilot adoption) to multiple programs having multiple intermediate teams on each. Prior to this phase, the organization should have proven its current adoption under a single program using quantifiable measurements derived from the business and SDLC process. From there, an organization can take its template of success and expand it across all of its programs, with each program going through the adoption phases. However this time, new teams under the programs would benefit from having a proven, organization-specific blueprint for success.
The reality is that every organization is different. For this reason, every organization requires a unique Agile adoption plan. What made one organization successful might not work for another organization. Similarly, the plan used for one program might not produce the same results for another program, even within the same Agile organization. While they might be very similar, teams may decide that something else works better and produces greater value in their unique program requirements. An organization should always be open to slight variations of business and SDLC processes from program to program. The focus of Agile is being lean, and to create as much business and user value as possible each sprint. This is why organizations should use the retrospective to constantly update its SDLC process. The team should always be the arbiter of what is most efficient, and what produces the greatest value for the business and users. While the explorers established the business and SDLC framework for which the programs would be governed by in the piloting phase, the teams conducting the actual sprints will need to make whatever adjustments are necessary to make things lean, produce great value, and meet the SDLC success criteria.
As such, an expanding Agile adoption will frequently adapt its SDLC from program to program based on success and failures, until such time that the entire organization is running multiple programs which have seamless SDLC productions which meet all the SDLC success criteria.
Some example success factors for an Agile organization (Cornacchia, 2009) would be that each team is cross-functional, self-managed and directed, produces nothing but value, focuses output based on the people who add value, measures value as a derivative of demand, optimized across the entire organization and programs.
A mature Agile adoption is really an organization which has become an Agile SDLC center of excellence, meeting and exceeding all of its SDLC success criteria and are experts in eliminating SDLC waste in multiple teams, and multiple programs. Small organizations could consider themselves mature, so as long as they never have the need to operate multiple programs; however, even a small organization must have experience running multiple teams under a single program before they could be considered mature.
A mature or “expert” Agile organization will have removed all waste from their SDLC. The seven key factors of waste (Poppendieck, 2002) which are most prevalent in an SDLC are;
- Extra Processing Steps
Managing Requirements with Agile
From what I have experienced, one of the most common mistakes a novice makes while implementing an Agile SDLC is thinking that defining requirements and making thorough plans are wasteful, obsolete and not needed. Some new practitioners might read the manifesto as well as other pro-Agile literature and come to the conclusion that project planning in the form of detailed requirements, use cases, wire frames and etc would be considered a poor implementation of the Agile SDLC is used. This simply is not true.
Agile at its core is all about lean processes which produce great value in frequent increments. How an organization manages its requirements is still very important to the SDLC. No matter what SDLC an organization uses, if the requirements are not properly managed and communicated – it will undoubtedly lead to project failure. As many as 71 percent of software projects that fail do so because of poor requirements management, making it the single biggest reason for project failure (Lindquist, November 15th 2005). Additionally, 9,236 IT projects surveyed in 2004 found that the top three causes of project failure were lack of user input, incomplete requirements or changing requirements (The Standish Group International, Inc., 2004). Every Agile project should have a plan based on value which properly captures scope and requirements so budget and resources can be properly calculated to complete the release plan with success.
In Waterfall, as you probably know, projects live or die by the plan, and are approached as if they were buildings to be constructed, or perhaps even paint by number painting. We don’t need to argue the benefits of Agile over Waterfall, but Waterfall’s concept would be great, if only building software was like building a house; which it isn’t. To me, developing software is more like producing a movie. As with a movie, software development needs an executive producer, a script, a director and a cast. And just like a movie, the production is constantly developing based on the script, but not always following the script line by line. Talent, free thinking and creativity are essential to producing something amazing.
I have read countless blog posts over the years whereby authors cite that a Business Analyst has no place in an Agile SDLC. The Business Analyst is still a very important team member in the program, just as they were under Waterfall. As software development is more like producing a movie, a Business Analyst is the script writer and the production team is the cast who act out the script using their talent, free thinking and creativity to make the final product. However there is a distinct difference between what a Business Analyst does under the Agile SDLC versus the Waterfall SDLC. Specifically, A Business Analyst in an Agile SDLC will only produce requirements, models, test cases and prototype simulations which are demanded by the business and team members. And they only produce what is minimally required for the team to understand the vision. However, under Waterfall, the Business Analyst would produce an entire arsenal of models and related materials in an effort to blueprint the entire system from top to bottom, regardless if it was demanded or not. A Business Analyst in a Waterfall SDLC specifies a massive number of software features, and as you might already know , a Waterfall SDLC, produces unused features resulting in waste. So under Agile, we reverse this trend and only produce what is demanded by the people who will be performing the development.
Remember that value is determined by demand in an Agile organization. Just like the program teams strive towards producing only business and user value in their software applications, the same value proposition is made for the Business Analyst. The Business Analyst in an Agile SDLC should be considered to be producing for the program teams, and as such, should only be producing value for those teams. While the Business Analyst would still produce a complete set of business and user requirements to drive development, the Analyst would not produce for each requirement a use case, test case, data model, business model, wire frame, etc unless the team specifically demanded it for a particular requirement while grooming the backlog.
Most people in the Agile community are familiar with tools like VersionOne or Rally. Having a collaborative platform whereby everyone can elicit and consume project requirements is essential. Even on small programs, teams can get lost in traceability. As teams move from Sprint to Sprint, often dependencies can be overlooked and requirements misunderstood as they might apply to other requirements. Even with Scrum and XP, using these requirements management applications can be extremely beneficial before the project team begins their work. But hey! You can still use the sticky notes and note cards too! In many regards, some organizations get lost in the tool. And as a matter of fact, I recommend that any team or organization just getting started with Agile NOT use a tool. Use a Scrum board to start with. Doing so will help team members grasp the concepts much faster, and also help to prevent traditional Project Managers from using it like they would MS Project.
But whatever your tool is to manage requirements and team boards, it’s essential that it is done well. Software makes it easier and successful, but software can also make your SDLC really fat and process heavy. Remember the value proposition when creating requirements and you can maintain a lean Agile SDLC.
The fact is, your new Agile adoption will have impediments. Waterfall teams will struggle with the change for the first few sprints, but with each new one you can expect dramatic improvements from the last. The important thing is that you not rush, and you implement Agile properly as a lean SDLC that produces value each Sprint and adapts as needed using lessons learned from the previous Sprint. There is no de facto standard and each and every organization requires a specific Agile adoption that works best for them and delivers the leanest and most valuable SDLC. Your adoption should focus on value, even when it comes to the production and management of requirements. And always remember that value is based on demand.
Cornacchia, C. (2009, March). Scaling Agile with Lean. Webinar . rallydev.com.
Lindquist, C. (November 15th 2005). Fixing the Requirements Mess. CIO Magazine .
Poppendieck, M. (2002). Principles of Lean Thinking. Eden Prairie, MN: Poppendieck.LLC.
The Standish Group International, Inc. (2004). CHAOS Report. Boston, Massachusetts.