Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Today I got yelled at for developing an application on a production server. Quote, "developing on a production server is not acceptable - ever!"

Here is the situation.

  1. I set up a development instance: http://example.com:3000
  2. The production instance is: http://example.com
  3. I complete all my development work on http://example.com:3000 and when the client is pleased with the changes, I move them over to http://example.com.

The application I am working with is an old Ruby on Rails application, and I have to say that initially, I did try to set up an development environment locally, but I could never get it running. After trying for a while, I gave up and decided to develop on the production server.

Again, am I an idiot? Probably so, but I've been doing web development for a couple of years now, and I have never encountered a situation like this. Who is right and why?

share|improve this question
42  
The only valid retort would be "having a production server that can't be reproduced in development is not acceptable - ever." –  Blrfl Jan 11 '13 at 3:03
    
haha, thanks for the encouragement. –  luk3thomas Jan 11 '13 at 3:06
46  
To me, the real problem here is that you have a production system where you have no idea what's running or how it is configured. What would you do if your production machine crashed, losing all its data? If you can't set up a proper development environment now, what will you do when that's your only option? –  Greg Hewgill Jan 11 '13 at 4:15
25  
Greg is right, if you don't enough about the application to reproduce the environment on a development machine, then not only should you not develop and test on the production machine, you shouldn't even be allowed access to deploy to that machine. You were clearly in the wrong, but I would have yelled at your boss for giving you access in the first place before you fully understood what you were doing. –  maple_shaft Jan 11 '13 at 10:26
2  
If you can't setup local development environment, you should not develop at all - ever. –  Jas Jan 16 '13 at 13:42
show 2 more comments

9 Answers

up vote 56 down vote accepted

I used to develop on the production server. It can work fine, but it is inadvisable for at least two reasons:

  1. Development code can cause infinite loops, memory leaks, or other problems that lock up the CPU, eat up all the memory, or otherwise affect the server in a way that will impact your production code.
  2. If you need to make changes to components of the server environment as part of your development work, like the version of Ruby or MySQL or whatever, you'll be in a bind.
share|improve this answer
1  
Yes, that is a good point. The more I think about it you guys are totally correct. –  luk3thomas Jan 11 '13 at 4:30
6  
There's a third issue: security. What if someone portscans the production server and discovers your (development) application -- and as a result compromises the production application? Again, while this isn't a likely scenario that's pretty much what everyone says after a server or system has been compromised. –  Marco Jan 15 '13 at 20:58
    
The infamous infinite loop problem... –  Mansuro Jan 19 '13 at 17:26
2  
@Marco Actually that's very likely if the server is publicly accessible. Development servers typically have very bad security (because conveniences like debuggers and stack traces are liabilities when it comes to security). If an attacker can escalate privileges by exploiting the dev server, he/she can completely own the production app. –  mehaase Feb 24 '13 at 20:34
add comment

This is really a protocol issue. Generally, this is not something that you would want to be doing. You want to leave production machines alone. They may contain sensitive data, and you don't want to compromise the performance or stability of production sites with non-production ready code.

That said, there are times where this is commonly done. If you are in a position where you are pumping out low traffic brouchureware or simple CMS sites, this will likely be less of an issue. But if you are working on anything with sensitive data or "business critical" processes, you really shouldn't be risking having development code on the same machine.

share|improve this answer
    
Ok, thanks. Can development code make a machine unstable? I am working on an old rails app. It seems to me (naive person) that development code for http://example.com:3000 would not affect http://example.com. –  luk3thomas Jan 11 '13 at 3:16
2  
@luk3thomas: well, sure, it shouldn't. What if there's a bug in the webserver or Rails framework, though, that crashes the server? What if, as Jacob Terry suggested in his answer, an infinite loop or memory leak in your dev code sucks up all the resources on the production server and compromises the performance of the live site? –  Carson63000 Jan 11 '13 at 20:41
1  
Sometimes it's a requirement. Such as shops that license expensive software and don't have the budget for a second copy just for dev work. –  Brian Knoblauch Jan 16 '13 at 12:52
add comment

I did try to set up an development environment locally, but I could never get it running. After trying for a while, I gave up and decided to develop on the production server.

I DO support the statements to AVOID development on a production server. You may only be justified to do under the GUN, if it is a typo correction in config file and insisted by your manager.

WHY NOT? - Because, it is very risky and pron to become a habit later on that would catch you up badly. Because, fatal production errors/failures may cost you to be fired from your work.

Let me re-iterate it again, even-though if you requested to do typo correction on production server, first do it on Staging. or in another words test your changes, test it, and again test it before placing in production.

This is something that happens often in places where culture of "do it quick and dirty" seems to be a norm.

Edit: Developing on production server, as a separate environment is also NOT acceptable. Any issues caused in your work may simply bring down the production server, and affect the production application performance. As example, i do remember a case when there was a security hole, when my previous co-worker tried to use production server WinServer 2003 for development purposes.

share|improve this answer
1  
Very valid points. Thanks for educating me. –  luk3thomas Jan 11 '13 at 4:33
    
This is a good summary of pitfalls with developing in a production environment, but the question was about developing in a separate environment on production hardware. –  Carson63000 Jan 11 '13 at 20:39
    
Thanks, i have added an edit that addresses concerns of doing it. –  Yusubov Jan 11 '13 at 20:55
add comment

Another important reason not to develop directly in production is that a development instance will usually produce and show verbose errors and stack traces. You never want to expose that to the user.

Yes, you can log them instead of showing them to the client, but that makes debugging that much less amusing than it already is.

Added Addressing your side-problem of having trouble with your development instance: I've had great success deploying a VirtualBox-based VM which duplicates our production environment (exclusive of hardware, of course) with an Ubuntu Server.

share|improve this answer
3  
noted, thanks for the advice w/ VirtualBox –  luk3thomas Jan 11 '13 at 4:28
1  
@luk3thomas It's also free! There are tonnes of tutorials online for VirtualBox + Ubuntu (probably one of the most common virtualization combinations). –  msanford Jan 11 '13 at 5:08
add comment

I always try and ask other developers what the procedures are for the particular company. In general yes, you should always:

  1. Build locally.
  2. Push it to some type of box that mimics production as much as possible to see if it plays nice there.
  3. Possibly push it to a QA or certification instance to pass to the client/QA team to review the changes.
  4. Push to production.

You can use Capistrano recipes paired with GitHub to handle all of this stuff for you. If you have to do this by hand every time, it can get old fast.

share|improve this answer
    
rails 2.3.11, I'll probably end up doing something like that –  luk3thomas Jan 11 '13 at 13:59
add comment

As others have stated, coding on the PROD environment exposes your users to your bugs. Even if you've started a different instance, you've still got shared hardware resources and can still access production files and databases. And as some of the comments point out, if your Dev instance gets hacked (for example, because you forget to wipe it and someone then discovers a massive security exploit in Rails), you've now got a publicly accessible machine with your app acting as a gateway in. Which would be... unfortunate.

Different businesses have different responses to this, but it can be generally broken down like this:

  • Did a screw-up occur?
  • How long would it take to revert a change (I primarily work in C++, so rolling back a binary can take significantly longer than in Ruby, especially when you've "lost" the old binary and have to recompile)
  • What the effect of the change (rough guide: screwing up stored data is so much worse than not storing or displaying data, which in turn is worse than not showing the page at all)
  • If you screwed up then walked out the door, would anyone know what you'd done?
  • Was there another deployment option that would have prevented/minimized/detected the screw-up before impact?

This gives you the final calculation:

  • How much would this completely preventable screw-up have cost the business?

This is now how much less your entire management structure is worth to the guy making budget decisions. Hence shouty.

If you're working on the company's internal "About Us" page and typo your own name to be L. "God-like" Thomas, embarrassing nickname problem; if you're working on the business-critical purchasing app, and it ends up accidentally plain-text debugging out credit card details to the homepage... lawsuit problem. Between those extremes lie everything from mischarging, crippling productivity, and all the other things that can drive away customers.

The reason not to allow it even for the "About Us" page is because coding directly in production is addictive. You start by only doing it for the minors, but over time, it's so much quicker not to have to get the DEV env up to scratch.

Beyond that, the size of the business can have a big effect. In a two-man team, when something goes phut, you lean over your shoulder and go "Oi, jackass, put it back". In a 300-person company, you have to start worrying about whether it was incompetence or malice, managers can be held responsible for stuff they had no control over, etc.

At the end of the day, if you follow procedure and screw up, they check what's wrong with the procedure. If you skirt procedure and screw-up, it's now your responsibility alone, even if the blame gets spread out a bit. Whether you want to roll the dice on that is up to you.

share|improve this answer
    
That is a good perpspective, thanks for sharing –  luk3thomas Jan 11 '13 at 14:00
    
This is a good summary of pitfalls with developing in a production environment, but the question was about developing in a separate environment on production hardware. –  Carson63000 Jan 11 '13 at 20:39
    
@Carson63000 Agreed, and Jacob's answer is definitely the better one on that side. I've altered mine slightly. –  deworde Jan 12 '13 at 18:37
add comment

I'm quite astonished no one mentioned the most important reason yet, why it's absolutely forbidden to develop on production servers:

Don't mess with production data, which can happen oh so easily!

A tiny error in one place leads to gigantic troubles in other calculations and then, on the next day, all the data is garbage and the customer is pissed. This is much, much worse than a bug in the UI or a little crash here and there.

For most applications, the value lies in the data and not in the routines.

share|improve this answer
1  
You learn this pretty quickly after you've screwed up production data. I'm guessing he doesn't have a DB. –  LachlanB Jan 12 '13 at 12:41
add comment

Rules that use "always" or "never" are usually ill defined. There will be edge cases where breaking a best practice will be justified. Much better advice will be "Don't touch the production servers unless you have very good reasons"

In my career I have found only two reasons to change code on production servers:

  1. Bugs or behavior that happen only there and are not reproducible on the development environment. These are rare but can be very annoying and hard to find.

  2. Directly fixing critical bug that you just cannot afford to wait to pass trough the normal deployment process if it takes more than a few minutes. After this has been cleared with management. If you are lucky you should only have few of those for your entire career.

Both are best left to senior developers that know the systems intimately.

share|improve this answer
    
If you have bugs that only occur on the production environment, your development environment is not close enough to the production environment. You should AT THE VERY LEAST have a staging environment with the exact same configuration as the production environment, down to the exact OS settings, processing hardware and installed software. –  Nate Kerkhofs Nov 28 '13 at 10:45
add comment

Another problem with developing on prod is that, sometimes, these things get missed in source control and a future release might undo your quick-fix change.

If you are in a publicly traded company in the US, you should not even have access to prod for regulatory reasons. In general, in no company should a developer have prod access (for the resons stated in all the answers as well as possible legal reasons), so your manager is also in the wrong for allowing you the rights to that server.

share|improve this answer
add comment

protected by maple_shaft Jan 11 '13 at 10:23

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.