This can promote a "best of breed" approach for project management, and risk mitigation....
Topics covered in this article:
Our Background
RJB has successfully applied SDLC processes for many years, by recognizing the strengths and weaknesses of different approaches and tailoring them where necessary. SDLC processes are not ideological dogma, but rather they are guidelines. When the concept of risk management and mitigation is taken into account by a competent practice leader, those guidelines can be successfully applied to real projects. RJB has successfully applied Agile processes on many projects using the Onshore-Offshore model. In this article, we examine the process adaptations required to achieve success using this paradigm.
This article is a brief overview of our experience and capability. We hope you will find it useful. For more information, or for advice on your software project, don’t hesitate to get in touch.
What is Agile?
Agile focuses on short delivery schedules called sprints, where Features are evaluated, prioritized, and reprioritized as necessary, sometimes as often as daily. Focus is placed on tangible deliverables e.g. Features that can be demonstrated to a user. Properly applied, this can result in an improvement in quality of the final deliverable, and in some cases result in some Features not being built at all in that they were recognized as not necessary during the course of the project.
This flexibility and the iterative nature of Agile can come at a cost – rework of deliverables after review, for instance, can result in better quality at the expense of more effort. It’s not uncommon to budget as much as 25% extra effort in an iterative project to account for such reviews and resulting rework. It’s not that someone just plain “got it wrong” – it’s just that they had an idea during the project for an improvement that was deemed worth incorporating into the next release.
There is a time and a place for everything. True Agile, with its often very short delivery schedules, might not be appropriate where infrastructure is concerned. As much as possible, infrastructure should be planned out in advance and implemented prior to the sprint in which it will be used. It’s definitely not a good idea to develop infrastructure in parallel with the client sprints.
Conversely, not all infrastructure needs to be in place before the first sprint starts – it can be implemented incrementally as well. Since infrastructure is often something that runs in the background, it will likely need a sprint to demonstrate its effectiveness. Use of already established frameworks, whether open source or commercial, can reduce the risk of bringing an untested framework to a sprint. Where new frameworks are involved, expect some growing pains when it is used for the first time.
Similarly, client and delivery team composition and distribution need to be factored into an Agile approach. Distributed teams, for example those that use the Onshore-Offsource model, need to take those factors into account. The question is: How do we account for these real life scenarios while taking advantage of the benefits that an Agile development process offers?
Combined Agile and Waterfall SDLC
Much has been made of the pros and cons of various SDLC approaches such as Agile, and Waterfall. Agile skeptics (and there are many) point to the lack of project governance and the increasingly common challenge of distributed software development across many continents and time zones as inhibiting factors when adopting Agile. Waterfall skeptics (and there are many) claim that the approach is too rigid and unresponsive to change, resulting in poorer quality software and potentially elevating project risks due to a perceived lack of prototyping inherent in the methodology. There are other criticisms of each approach but we do not intend to provide an exhaustive list here, as they have been documented in great detail elsewhere. Instead we will focus on whether a marriage can be or ever should be made between these two disparate approaches.
Combining Rather Than Merging
Some have counseled that Agile and Waterfall can be merged into a Hybrid Agile approach that addresses the critics on both sides; however, that argument too has met with a great deal of skepticism. Chief amongst these arguments is that both sides feel they have lost the advantages of their respective models. We would argue on the other hand that the way we combine Agile and Waterfall is inherent to the nature of the project and the organization. It’s true that in a way Agile and Waterfall are like oil and water – they don’t mix. But what if we don’t mix them? What if we use these methodologies side-by-side, for tasks that they each excel at, in a complimentary manner rather than an opposing manner. In other words, for tasks that require:
Significant pre-planning of budget and schedule, and a global project level viewpoint, we consider a Waterfall approach
Iterations of feature/functionality that evolve towards a finished high-quality product, we consider an Agile approach
The notion of combining Agile and Waterfall rather than merging is not simply a matter of word-smithing. A merged discipline would result in a single methodology, which as stated previously, would please no one. However, using each methodology for the portions of the project that they are eminently suited for can be a match made in heaven.
An Example
Let’s envision the ground-up development of a new system for a large corporation.
Start with Waterfall
We start the project with a Waterfall approach, ideally suited towards pre-planning, budgeting, and scheduling. It may also involve co-ordinating schedules with multiple departments in the company, so that they are aware of what is happening, even though they might not be direct participants in the project. This phase of the project only lasts until we have created enough artifacts to:
- Articulate the overall architecture, high level requirements, business model, schedule, and budget
- Produce a manufactured prototype that demonstrates that the overall concept holds together. The term “manufactured” simply means that the prototype is not a throw away – it will in fact become the starting code base for the first sprint. Selecting the functionality for the prototype is important – it should not be too big, but it should use the full stack and exemplify the decisions made to articulate the vision in the previous point
These artifacts reduce overall risk for the project. The outcome from the articulation step provides a vision that the entire project team can adhere to, throughout the Agile Sprints that follow. The outcome from the prototyping step provides example working code, using the full stack as conceived by the Architect, including the standards and conventions to be used throughout the project for consistency. At the end of this phase, the team can credibly revisit their original estimate of schedule and budget for the entire project, reducing risk in those areas as well.
Switch to Agile
With a global project vision and a proven architecture, we can unleash the power of Agile feature delivery. Imagine teams distributed around the globe working on sprints that adhere to a common well defined architecture and platform (there are some other factors that affect distributed teams that we defer discussion on for now).
Agile sprints proceed in all their glory towards a fully featured product on a well defined architecture. No need to delve into the details of sprints because that has been well documented elsewhere, and …oh yes, we are intentionally not changing the Agile methodology.
Switch back to Waterfall
The Waterfall approach will kick in again towards the end of the project as we approach final User Acceptance Testing and ultimate deployment to production. The reason is again because these activities will likely involve significant pre-planning and co-ordination of activities within the company as a whole, which are tasks better suited to Waterfall. The figure below shows this approach graphically:
So it’s clear that we don’t need to merge Agile and Waterfall into a Hybrid model. There is no need to compromise our principals in either discipline. However, there are a few cautions:
Agile sprints are flexible in that the feature/functionality being developed can change over the course of time. These changes can affect project schedule and budget. We should anticipate this and build in a buffer to the project schedule and budget to allow for rework and changes. We have seen buffers of 15-25% built into the project schedule for refinement of feature/functionality. The exact value of this buffer is project dependent, but this is a typical range. The benefit gained is a higher quality outcome which has been refined based upon feedback.
Sometimes part of the project team may have to jump back into Waterfall during the course of the project. The occurrence of this is most likely due to a significant change to the project scope e.g. the project sponsor has decided they will need a document management component and this was not factored into the original proposal. This can entail a Change Management Request, and perhaps a follow-on manufactured prototype for the new component. Once the prototype has been successfully completed, further feature/function work for the documentation component will be done using Agile.
To reiterate the point, the above discussion doesn’t mean that every project is governed by a Waterfall => Agile => Waterfall approach. The statement being made is that we can consciously make decisions about when to switch gears in order to obtain a more optimal result. There are numerous factors that can feed into this decision process, which is why we say that the specifics of the chosen process must take into account the nature of the project and the organization. In order to choose rightly which methodology to employ at a given point in the project, one must understand both the benefits of a particular methodology, and understand what we are trying to achieve at that point in the project. Ideology wars are to no one’s benefit, except perhaps to someone who is trying to sell something.
Decision Factors
So what are some of the factors that can play into these decisions? They often revolve around scope, schedule, cost, and quality. These are key concerns for any project, so it should come as no surprise that an evaluation of these concerns would point to a particular “best of breed” approach to address same. A proper evaluation of these criteria in the context of a specific project is beyond the scope of this article, but here are a few points to consider:
Scope – Is a fixed scope something that is desired for the project? Sometimes it is. Think of a government project that is tendered out via RFP – typically the scope is fixed, and the government wouldn’t have it any other way because that’s the way their organization operates. In this scenario, introducing innovation might jeopardize the project. For example, the government body might not be able to invest the resource necessary to participate in an Agile process. On the other hand, a startup company that is developing a new product may need to experiment a bit to find the best performer for their prospective customers. In this case it might be better to allow some scope creep to occur if it turns out a better product. These two scenarios produce vastly different visions of the scope of their respective projects.
Schedule – How flexible is the schedule? Sometimes you have a drop dead date, such as the date for implementation of the online sign up for the Affordable Care Act (ACA). On that date you MUST have the system up and running, and it MUST allow customers to sign up. Those things MUST be in place even if some lower priority Use Cases are not completed. In this case, schedule trumps everything else. However, in the case of a manned rocket launch we wouldn’t launch unless everything else was in place – in this case schedule, while important, takes a back seat to scope, quality, and risk.
Cost – Are cost overruns allowed? Is it better to sacrifice some scope in order to deliver something within a fixed budget? In some cases that might be true. A lot depends on the means by which the money is allocated to the project. An angel investor might be more flexible than a corporate or government budgetary process. Fixed cost is a bit like fixed schedule – it’s better to be aware of that from the start and adjust scope and schedule where needed to remain within budget.
Quality – Everyone wants high quality software. You’ll seldom have someone tell you they want low quality software. The question is what they will spend to get it. Sometimes it is important to be first into a particular market even if there are still a few glitches in the product. That can mean turning out what is a slightly less than a stellar quality product in order to capture early market share. Agile, with it’s frequent feedback points, is perfectly suited for balancing quality against scope, schedule, and cost.
Just a word about risk management. Here we are recommending approaches to adopting methodologies that reduce risk for projects, by choosing and adapting methodologies to the nature of the project and the organization. There are many forms of risk management. It’s important to ensure that your project adopts an appropriate risk management model. See risk management for more information on this subject.
The techniques described in this article have been used successfully on many projects, some of which were budgeted in the millions of dollars. For more information, or for advice on your software project, don’t hesitate to get in touch.