Moving from a traditional waterfall methodology to an agile one

Question

My development team is in the process of 'going agile' and lot of people are looking at each other blankly, not really too sure what it means. All management are saying is 'start working agile' without suggesting any particular methodology so they're waiting to see if we can come up with something ourselves. Has anyone been in this situation before and how did you handle it? Any do's and don'ts? I'm one of the senior members of the team so there's an expectation to show some initiative. I know that what works for one won't necessarily work for another so I'm more interested in the approach not so much the end result because I think the result will come naturally with the right approach.

Answer 1

I have experienced such situation in a company attempting to make the agile move.

The agile way is fairly simple and straight forward and actually removed (or should have) a lot process heaviness. But it was difficult and to my knowledge they are still struggling with it.

The biggest hurdle was that people lacked understanding of what going agile means. I think the change is similar when going from a procedural language to object oriented programming. Writing classes and doing inheritance does not mean you are doing OOP, in much the same way doing daily stand-up meeting and shortening the iterations does not mean you are agile.

Agile is about getting feedback on your work as soon as possible such that this feed back can be integrated back in the product AS IT IS BEING CREATED.

Waterfall is about attempting to get everything down before starting and not look back until it is done. Only a finished product warrants review and starts a new iteration.

I guess you could say in some ways the difference between agile and waterfall is akin to the difference between guided missiles and ballistic missiles. One fires right away and continuously adjust the trajectory in-flight while the other aims very carefully before firing a dead weight.

What ended up being done as "agile" was that they split the waterfall process in smaller iterations. But they still worked all before it started. Though this enabled some parallelism the work was still divided in the basic waterfall view, first get requirements, then design, then code, then test and ship, each part by a different team each team working independently. They applied the agile in apparence but their hearts and minds remained firmly in the waterfall way. When told it was not agile enough they just dropped the design part and created a mess !

It may take some time for your team to truly grok what agile is all about. Not all believe in the agile way and it is very hard to get a non believer to abide to it. Perhaps some external fresh set of eyes and ears can get your team off the ground faster and more efficiently.

---

EDIT

I should point out that the process in place at the company, modelled after waterfall, was broken before they decided to move to a more Agile way. They thought that this new thing called agile would solve their problems but they never really analysed what their problem was to begin with. Thus they went from bad to worst.

Also, judging from the comment threads I guess my over simplification of both description deserves some clearing up.

In their own ideals both process would attempt to seek feedback from customers, ideally from the actual users of the system being created. For Waterfall this means that each successive stage should have an open dialogue with the stage before to understand and perhaps fix problems before it reaches them. However the cost of finding a problem increases dramatically as the product nears completion especially so if the problem comes from a few stages back. This implies rigorous checking akin to what a carpenter mean by saying "measure 5 times, cut only once". Waterfall is not inherently broken and has worked very well and continues so in cases where the product, once delivered, is not meant to be changed easily. Getting feedback from customer from a large document describing the engineer's understanding of their need is not easy, yet many company will ask customer to commit to a set of highly technical documents they are often not equipped to understand.

Agile embraces the dynamic nature of human nature and how easily Software can follow. The goal of Agile is to translate the requirements in a form the customer will more easily relate to and as such gain more reliable feedback, I believe this was the essence of the Agile Manifesto. It was not meant as a proletarian revolution of software engineers over the establishment, though it is sometimes, unfortunately so, wielded as a hammer meant to destroy all that do not agree. Most agile techniques will hinge on the fact that software should be easily modifiable given a competent team to do so, that the most difficult part is to make sure the project is on track to satisfy an actual existing need as opposed to the perception of this need by an analyst. This requires a deep change in philosophy and does not integrate well with the present corporate structure. Projects must gain financing, for this financing projects must be evaluated very precisely before it ever gets started. This structure tends to favour a waterfall mindset, where once a contract signed it is often worded in a way where the team is bound to the approved documents and must follow a pre-set plan rather than pursue the need that originated the project. A software company will truly embrace Agile when all it's structure will embrace the agile mindset starting with how the work is billed to customers, how the projects are framed legally and contractually etc. At this point my experience is skim and going further would be purely speculative for me but it is certainly something I would be very interested in learning more about.

Answer 2

1. (if you haven't done so yet) Start by browsing through some well known Agile processes such as Scrum and XP to get an idea of their philosophy, ways of approach, tools and methods.

2. Discuss these with team members and management, and decide which practices are (not) suitable for you.
   E.g. it might be that management is opposing pair programming (for real or imaginary reasons), or that end users can't physically participate in the teamwork and planning sessions; there is no point endlessly arguing over it, it may be better to just leave it out from the initial process.
   (You seem to have management support which is indeed an absolutely necessary, Very Good Thing - but you must ensure this support stays with you even in the face of difficulties. For this you should ensure that management actually understands what Agile roughly means, and what (not) to expect from it.)

3. Start doing it, with a simple initial set of rules which is acceptable to everyone. No problem if the edges are rough, just define an initial process and start practicing it as soon as possible. Since you need time to get used to any new process and way of thinking, expect an initial period of confusion, chaos and subsequently lower productivity; you must survive these weeks and months in order for your efforts to start bearing fruits.
   One thing you should absolutely include in the process is retrospectives: regular meetings at the end of each iteration/sprint to discuss

   • what went well (to be continued),
   • what went badly (to be stopped or replaced by something better),
   • what could be done better and how.

4. This should result in some action points to successively fine-tune and improve your process.

The result* may be a "strange" mix of ideas and practices from different methodologies, but it will be your own personal process, tailored for your own team and project.

_{*In fact there is no "end result", just as there is no end result in the process of evolution. Process improvement never ends. Circumstances change, projects change, team members change, and a good Agile process is kept adapted to all these changes.}

Answer 3

A few things we did that helped us with our transition.

1. Agile coach: The one thing that made the biggest positive for our transition (which so far is pretty successful) is getting someone in from outside to help us. We were lucky enough that a smart and respected fellow that used to work for us had gone on to work at a few different agile shops, seeing some successes and some failures. We snared him for a while to help us pitch, plan, and ultimately execute, a switch to agile. Coming in, he had a good grasp of our business and the things that make us special/different/difficult/whatever, and a good grasp on agile. He took a fair amount of time to talk to people, consider end-state models, consider a transition plan, and make sure we understood the costs and benefits.

2. Transition team - Once we knew this was for us, we formed a small team under the guidance of the agile coach, with experienced people from most of the functional areas of our department. This team works as an agile team, with a backlog that models the things we need to plan or execute for the transition. Membership has been reasonably static; team members have their "day jobs" as well as their transition team duties. Collectively, the team know pretty much everyone, understand what makes people tick, know the processes we use in every corner of the company, and what's going to break as we start changing things.

3. Executive support - I'm amazed at what a difference this makes. For us, the desire to go agile came from the top, based on the perceived benefits both to the organisation and to the individuals in the organisation. When we interview folks, it sounds like most places looking to go agile have a grassroots agile movement that tries to get going in spite of a lack of management support, or lip service. The way you commit to work changes dramatically; without support from the top, I can't imagine how a transition would be successful. You're getting pushed in this direction from management; I assume they want more than "just be agile please", they want something in particular out of it.

The transition itself is a tricky beast. All of those practices we value under agile (stories, teams do estimation, set iterations, access to stakeholders, focus on testing and continuous integration, etc) are hugely important, but properly managing the transition to using those is difficult. It's a project in itself. We're a mid-sized place with a bunch of average-sized scrum teams, so your actual mileage may vary, but I'm sure the concepts would still apply.

Answer 4

Agile is all about short iterations and react to changes and have runnable version all the time. The rest are more or less details you could read up and do one after the other. I would suggest to read the following:

• http://www.extremeprogramming.org/
• http://en.wikipedia.org/wiki/Extreme_Programming
• http://gettingreal.37signals.com/toc.php

Answer 5

Don't drop the current methodology all of a sudden. Instead adapt it to be more agile one step at a time. The current methodology can be the outer framework that gives structure to a more agile inner one.

1. Find ways in which to simplify administrative and operational documentation. Drop some documents from the process.
2. Shorten the release-to-deployment cycles. Change projects of more than six months of schedule into several versions released each three months, and revise the plan and features for the next version before starting to work on it.
3. Use stories, scenarios, or use cases as the feature units, have developers estimate them, and base the project plans on those estimations.
4. Plan to deliver working use cases in iterations of two to four weeks.
5. Measure planned versus working features (ideal versus actual plus the velocity) and modify the general plan and the following iterations accordingly.
6. Invest in the automation of functional tests, and make executing those tests part of the daily routine.
7. Introduce unit tests and a personal code integration process that uses them. Don't let developers break the code in shared repositories.
8. Tell management that frequent and expedite access to those who represent the client or the users is indispensable when going agile. Answers to questions are required when the features are being built, not after they've been released.
9. If you can't or won't do pair programming, at least have developers work on different parts of the system on different iterations. That will make the code be reviewed by everyone, and will increase the knowledge everyone has about system components and technologies, thus reducing the risks of component/module ownership.
10. Start with a pilot, low-risk project.

I've consulted in introducing agility to software teams and companies, and what has worked is to gradually introduce agile ways at the core of what they already do. The keywords are simplify and iterate (something that also applies to the process of introducing agile).