Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Throughout my career I had worked at companies that had a collection of different environments for different purposes. We always had more or less our desktop environment, a test environment, a QA environment, a staging environment and a production environment. This went for both servers/applications and any data sources we were using.

When I started at my current company I found that 90% of the apps were either developed on a desktop environment against production data sources or developed directly on the production server depending on the platform. I wasn't fazed because I was hired in part to make changes to improve the way the development team functioned, which was clear from my interview process. We slowly started to turn the philosophy and pretty soon, most of the apps could be run in either a desktop, test or production environment. Not too long after that staging came around as well.

Now most of our developers see the benefit of this methodology and defend it vigilantly. However, we have a number of legacy apps that never got migrated. We also have a number of legacy programmers who think of this as a waste of time. Unfortunately, we got lip service but never full buy-in from management. We got what we thought was a commitment to invest substantially in this about a year ago, but nothing materialized despite the considerable planning that we put into it. Now we are finding that we need more and more environments. We need help from the server/network administration teams for setup and we need participation from the business stakeholders to support the release cycle. We are at a place now where a project can function what reasonable developers would consider "normally" only if you have the right people on the project and the time to set up the proper environments.

I'd love to present a complete argument, but management really has no time and interest in hearing me out until there is a critical issue. I can't really articulate the benefits simply as it always just seemed second nature to me. I was wondering if there are any good, simple, irrefutable reasons for the separation of environments that would get managers lacking development experience to support this idea. Are there any good resources/literature on the topic?

share|improve this question
1  
Great question, I am interested to hear what others have to say. I don't have a good answer for you because managers want hard numbers, and all of benefits of having multiple environments are hard to measure in hard numbers. –  maple_shaft Nov 28 '11 at 20:36
4  
How has there not yet been a critical issue? If applications are being developed in production environments, it should be common (and is common in normal dev enviroments) for basic mistakes to disable features, cause error conditions, and even crash the entire application. Is the application so non-mission-critical that these problems are not critical failures? –  JGWeissman Nov 28 '11 at 20:52
1  
It's not a case of it not resulting in critical issues. It is a case of them not understanding how it is the cause of the critical issues. I suppose I didn't word it well enough. –  smp7d Nov 28 '11 at 20:56
1  
Wish I had a fortune to start a bounty! –  Kris Dec 1 '11 at 11:53
2  
For anyone who cares... it has been two years since I asked this question and we have a clear separation of environments now. It happened because of repetition. We continually said we needed it and we lost some employees who were against it and won over others. Slowly the tide turned. I wish there were a formula to get it, but I guess the culture just had to naturally adopt it. –  smp7d Aug 23 '13 at 18:21
show 1 more comment

11 Answers

up vote 66 down vote accepted

The answer: Money

I don't care what the actual reason is. Money MUST be at the root of all of your reasoning, especially when dealing with management.

If we both sat in a room for 2 hours, we could come up with dozens of reasons why it is better to have multiple environments.

Here's the problem: If the reasons are not based on money, then none of them matter.

Programmers are not hired to be smart. They're not hired to be creative. They're not even hired to program (although they tend to do it). They're hired to increase revenue -- either by earning money or saving money. If you're not doing either one of those, you'd better get your resume together.

When looking at it from that standpoint, the answer is simple:

Having only one environment increases our downtime and results in lost revenue. Multiple environments allows us to protect our profits by giving our users a front-end that is just as reliable and dependable as our company.

Repeat it every day.


EDIT 2: There are some great comments below that add some real value to this answer, so I'll mention them:

  • Karl Bielefeldt had a great point when he mentioned that Cost/Benefit analysis is an important factor. An economist might refer to it as the opportunity cost of pursuing multiple environments. While it may be surprising to hear, there are scenarios where multiple environments may not be the answer! If the website of your company is a very minor addition, then unexpected downtime may actually be the more cost effective way of doing business. This doesn't sound like the position you are in, but it is worth mentioning.

  • BlairHippo had a good point in that you should feel free to make it seem like a catastrophe (and if you lose your data, it is!). Liability is a great tool for persuading managers, but still for the same reason--lawsuits are expensive. Avoiding them saves money.


EDIT: As an addendum, I found this article to be quite good. It doesn't directly answer your question, but enables you to recognize how programmers are viewed to management, which in turn, leads to this answer. Good read.

share|improve this answer
6  
+1 For money being the only language management understands. Great quote by the way. It is succint and perfect. –  maple_shaft Nov 28 '11 at 20:57
5  
Great answer. Just wanted to add that the benefit must exceed the cost. After a certain threshold adding more test environments will cost more than it saves. –  Karl Bielefeldt Nov 28 '11 at 21:02
4  
+1 for the "Don't call yourself a programmer" article –  nwahmaet Nov 28 '11 at 21:38
2  
Great answer. I'd also add: feel free to catastrophize a bit. As long as you're releasing under-tested code on production data, there's always the possibility of accidentally nuking said data. Money may be the language spoken by all managers, but Liability is at least a popular dialect. –  BlairHippo Nov 28 '11 at 21:43
    
There are many right answers to this question, but this one is the best of the bunch. –  smp7d Dec 6 '11 at 19:40
show 1 more comment

Single Point of Failure

By not having a development or staging environment you have a Single Point of Failure for those legacy applications. Management will hear you if you describe the legacy applications in those terms.

You need to be able to pitch your message in sound bytes that makes sense to them. Take the "Programmer Speak" out of the discussion and replace it with "Manager Speak". Also pretend you have a 30 second elevator ride to get your point accross.

I had a situation where my boss was an Infantry Marine. I kept telling him I needed software tools and computer training for my Marines in order to be more productive. He didn't get it. I finally went into his office one day and told him things were f'd up.

I said something to the effect... "If I was fighting a war I'd be using sticks, rocks and tree branches. What I need are grenades, bazookas and machine guns." He got the message.

share|improve this answer
    
Haha, Thanks for the good answer. I agree that being direct and aggressive is the solution to get what you want. I've never had a marine as manager but looking forward to using bazookas and machine guns in an argument. –  Filip Nov 29 '11 at 5:33
add comment

It sounds like you have all the "right" arguments already in place. Instead, you are experiencing a "red-herring", if you will. Or, "chasing the carrot"

management really has no time and interest in hearing me out until there is a critical issue

That's what I consider to be the real problem. In my experience, if a company has sub-par development practices as poor as you describe. It's not simply a matter of "we didn't know any better". Rather, its a compilation of technical debt caused by an upper management team that doesn't know (care?) about the problems it presents.

In such cases, a good pep-talk is not going to suddenly swing things your direction. Maybe a severe trauma (product failure visible to client and directly tied to poor practices), but I'm sure sane savvy techies before you have tried the talk thing.

My suggestion is to either suck it up and take things for what they are or look for a new position.

share|improve this answer
add comment

Is it actually critical?

I can understand the desire to use separate environments. The non-obvious question is:

Is it actually critical to migrate a legacy system?

I think most technically minded people tend to focus on the academic question of which way is better which is fine in academia. In business though best doesn't always win out. I'm not saying this to be negative, or start a flame war. I am stating the obvious, or what should be obvious to those of us who have been in the software business for a few years.

All business decisions are typically made based on the perceived cost/benefit. So the question the business is probably asking is:

Is the cost of the additional system and development investment in a legacy application worth the benefit versus putting that same investment into another project/product?

I have and still do regular cost benefit analysis to make determinations not just in migrating/rewriting software, but in day-to-day decisions a lead typically engages in. I've passed on rewriting/migrating old software because it had limited life and therefore value.

Separating Environments

The business reasons to separate environments.

  • Less risk in releases, and bug fixes. Prove it with numbers. How many times has the product failed and cost customer revenue because of a bad release/bug.
  • Less risk in development. Accidentally blowing away the dev db is different from accidentally blowing away the production db
  • The ability to clearly separate roles and access ie. better security. limiting the number of fingers in the production pie is a good thing
  • The ability to separate environments, and the practices and procedure that go along with this style of development allow for future build out into Cloud Systems.
  • The separation of environment should force efficiencies in replicating environments which may be useful in scheduled as well as dynamic scaling.
share|improve this answer
    
+1 For pointing out that it's important to look at cost. –  sleske Dec 1 '11 at 11:17
    
Love your business reasons to separate environments. Especially the first 3. Best answer. Thanks. –  John Assymptoth Sep 27 '13 at 14:21
add comment

How many groups of people plan to be working on the app at a time? Usuaslly I've seen one environment for each group of people. This is the Developers (they get a DEV environment and a DEV Integration environment - some would say not 100% necessary, I'd say it varies by project), two testing environments (one group of testers doing very detailed testing, the other for high-level QA testers - usually they are actual business users, not trained testers). There is also usually an isolated Performance testing environment (so you can test huge volumes of data, simulate huge volumes of users, etc... g).

Why all these environments? So different groups can test different features without stepping on each others' toes. If developers and testers work in the same environment, it's a nightmare: a tester can open a defect on a feature that is actively being changed every minute by a developer. If there are two levels of testing, they can focus on different activity and not worry about messing up each other's data. Having an isolated performance environment allows you to run tests which may hang the machine, but if it's isolated, no other testers will be impacted.

When too many people try to do too many different things in the same environment, you end up with a lot of wasted time as one group waits for another group's test to finish so they can run theirs. And that wastes time, and wasted time can lead to wasted money, which Stargazer712 pointed out could make the strongest arugment.

Another reason (not as common) is data: if your application has sensitive personal data or credit card data, you usually cannot put that into test environments, and there is usually masking requirements for everything except the QA and Production environments.

share|improve this answer
    
Can anyone explain the downvote? –  FrustratedWithFormsDesigner Nov 28 '11 at 20:49
    
+1 to counter the unjust downvote –  maple_shaft Nov 28 '11 at 20:58
    
@maple_shaft: LOL! I'd have rather had an explanation, so I could fine-tune my answer. –  FrustratedWithFormsDesigner Nov 28 '11 at 21:00
1  
What downvote? I don't see a downvote... –  Yannis Rizos Nov 28 '11 at 22:17
    
@YannisRizos: There was one... but it was never explained. –  FrustratedWithFormsDesigner Nov 28 '11 at 22:22
add comment

Here's one: testability.

Having a testing environment gives you the freedom to perform tests on a database that would be inadvisable to perform in a production environment.

share|improve this answer
add comment

You want to change how your organization develops its software? Forget worrying about "reasons" for "doing things differently". Humans don't change behavior because of rational arguments. They change because of psychological influences on their habits.

So, where am I going with this?

While occasionally you can successfully change an organization's behavior through argumentation, there are other tactics that work better. These include:

  • Grassroots support: Find ONE other "legacy" developer who is willing to give you a shot. Partner with him and change how things work. Don't announce the change. Just make the change. If anyone ever asks you about it, just say "Oh yeah, that's how we do it now."

  • Take over responsibility. Volunteer to handle deployments for the legacy folks. Act like you love it. They may be happy to relinquish that responsibility. Then run it how you want to.

  • Blame the right people for their mistakes. The next time a legacy app bug is introduced into production because of your stone age deployment mechanism, point it out. Do it subtly... Not in an email. Next time you're in a meeting with a manager, just casually mention the example of a reason that the deployment was problematic. "Yeah, remember how we were scrambling last Friday because of the Foo bug Bob checked into production? Yeah, that was a lot of wasted effort!"

  • Make it easy to do it the better way. Look at the iphone, for example. There is ONE button on it. (Well, two). It's VERY easy to turn on. Make deployment-to-multiple-environment crazy stupid easy. Make it so easy all the managers can do it!

share|improve this answer
    
Humans don't change behavior because of rational arguments. They change because of psychological influences on their habits. How depressingly true that is. Whether it comes to software or the "free market", the belief that people make rational decisions in their best interest is a fallacy. –  maple_shaft Nov 28 '11 at 23:22
add comment

Its more of an issue when you start dealing with interconnected or legacy systems, systems the business depends on to be working and accurate. Its important because there needs to be segregation between the stages, its the reason why you dont DEV on PROD because it has the potential to cause millions of dollars worth of damage in lost time.

We always do DEV -> QA -> PROD (occasionally those steps are broken up into smaller pieces) with identical hardware behind them. Current production data is always pushed from PROD to QA to DEV.

DEV: is intended to be the development sandbox, where things are tried, iterated and beaten on any data in this environment should never be trusted and is regularly trashed by developers simply finding ways to solve a problem.

QA: Once your developers are satisfied with unit testing, its time for the test group to get there eyes on it. They run test cases, performance testing and find bugs. Those bugs/enchancements get fed back to DEV and the cycle continues until everyone is happy.

PROD: Once you get to this stage you should be sure the code works in conjunction with current data and your QA group/business users are happy with the implementation. If you did everything correctly you should simply be able to update the code and be done with it.

In the same way you would never release an untested product to customers you should never release untested code to a production environment.

If the company is not willing to invest the time to do it properly they will pay back the cost in emergency maintenance and errors 10 fold.

As a small example: We had one company who decided to make a change to a report in production on his own. No one knew that it changed until we came in to address a variety of issues a year or two down the line.

When we pointed out the irregularity in the report the CFO's face went white, turns out they were losing ~$250,000 a quarter because of someone making a quick change.

Happens more often then you think, if you cant afford to do it properly then dont do it.

share|improve this answer
    
Nice example. Of course, accountability is an important reason for separating DEV and PROD. That way you can have extremely strict controls on PROD, while giving DEV the freedom it needs. –  sleske Dec 1 '11 at 11:19
add comment

You seem to have invested a great deal of effort in bringing about a cultural change in your workplace. This is a great achievement as change is difficult at the best of times, yet cultural change is not simply about changing people's minds, but about changing habits, breaking prejudices, and ultimately about opening potentially closed minds to greater possibilities. So the question to ask yourself at this point is "What did I miss?". The easy answer is that you may not have fully engaged with management.

Getting buy in from management is easy, but even harder is getting acceptance. Regardless of the arguments about money etc, the reality is that you need to be able to influence management's view of priority. Your manager will have a budget, and will want to show that budget has been applied sensibly and in keeping with company values and priorities. Some of those priorities will be fiscal, but others will be about serving other needs. In some cases, this may mean greasing the palms of other managers in order to get that promotion your boss has always wanted. In most cases though, it will likely be about finding ways to gain more business, or improve relationships with partners and customers. If you can't make your case in these terms, you will only be able to go so far before you find yourself at an impasse.

My suggestion is to try to make a case about productivity and how this affects budget, as others have suggested, but also to make the case in terms of your company's priorities and how your productivity might directly impact on the company's relationships with other companies.

share|improve this answer
    
"changing habits, breaking prejudices, and ultimately about opening potentially closed minds to greater possibilities" - in retrospect this was the key and I cannot point to any single reason as to why it eventually happened –  smp7d Aug 23 '13 at 18:25
add comment

Management has a big part behind the success of the Software Companies and Software Products which required to generate these environments. Let take an example of your project. If your software is developed on a large scale then if you do not manage your project requirements, process control, Test Builds etc then this is chances of failure. so that Project Management exist.

I am somewhat agree with @Stargazer712 that your statement point to the Money matters, But Check the following statement that i have gotten from Marc Hamilton's book on Software Development: Building Reliable Systems (Prentice Hall PTR, 1999, ISBN 0-13-081246-3). After looking all of these factors; my Opinion about your question is that Single Environment is not give you savings, it will make a long term process to complete the project/software. Distributed Environments will save Time and revenue as i learned and seen in my experience that what happened with the start-up software companies from which i have start my carrier.

There are lots of articles demonstrate that what matter for success, Check this one Organizing for Successful Software Development

Each individual in an organization has certain skills, and these skills are typically measured against formal or informal performance metrics leading to rewards (compensation) as incentives for future performance. The people in an organization establish its culture—those behavior patterns and values that are generally recognized as being adopted.

A great set of software developers will struggle and ultimately fail to meet their goals if they have to spend all their time fighting against an inappropriate organizational structure.

Many a software startup begins life with no more than a couple of developers working out of a garage. Not much organizational structure is required at this point in a company's history, but organizational structure still exists. For instance, in 1977, when Bill Gates and Paul Allen formed their partnership and officially named it Microsoft, the company had minimal organizational structure. Fewer than a dozen employees worked at Microsoft's first office in Albuquerque, New Mexico, and everyone knew who was in charge. No complicated organizational charts were needed to figure out everyone's reporting structure. At the same time, all employees knew their role in the company and what they were trying to accomplish. This was because any organizational structure that was needed could be informally communicated between each of the employees.

share|improve this answer
add comment

When it comes to making a change, you will be lucky to have someone who will just listen to your professional opinion and implement suggested changes.

On my experience, each time I had to make a major change, I had to justify it in terms of savings the business will make. For example, introducing ReSharper into the development pipeline was pretty easy, as I was able to say something on the lines off:

ReSharper costs around £50 per developer. Average developer cost per year is £40k. ReSharper should increase developers productivity by at least 20% given its used to its full potential. Say developer spends 75% of their time actually writing code in IDE. 75% of 40k is £30k. £30k is now the cost of developer's productivity. An additional percent of productivity (1%) per year costs £300. To get additional 20% productivity, the business would have to spend £6k.

If you were to put this into perspective to the business, you can say that you can hire somebody else and get additional 20% productivity for £6k, or you can get the same result by spending £50 on a ReSharper license. Once the figures are in front of the business, the decision will be easy to make.

Now in regards to your questions with having multiple environments, all you have to do is find a way of calculating how much it costs the business every year to now have these environments.

You can ask your fellow developers to keep track of hours spent each week on configuring applications for development, deployment etc. For example, ten hours of senior developer's time might cost business £500. That's 10 hours that can be spent on development, or something much more important. You gather the figures for some period of time and provide business with a yearly cost.

I personally hate this kind of politics, but it's common and we have to live with it.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.