Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

This has happened to most of us...

You come to work one day. Everything seems normal - the sun is shining, birds are chirping, but you notice a couple of weird things on your way to work that remind you of the déjà vu cat in the Matrix.

You get into the office and there are a lot of phones ringing - but it could just be that they are doing a new sales promotion. You settle in, when you notice a dark cloud hovering over you.

It takes you a couple of moments, but you recognize the cloud is your boss. Usually he checks on you every morning with his "Soooo Peeeeter, how about those TCP/IP reports?" routine, but today he forgot everything about common manners and rudely invaded your personal space. No "Good Morning", just some drooling, grunts and curses. He reminds you a bit of a neanderthal who is trying to get away from a cyber -toothed tiger, fear and panic all compressed in a tight ball. You try to decipher the new language that he created since yesterday and you start understanding that something bad happened overnight - the production system went down.

Now, your system is usually used by clients during regular working hours from 9-5, but for whatever reason you didn't get any alerts on your beeper (for people under 30 - a beeper was like a mobile phone that could only ring and tell you who beeped you). You'll need to remember to charge it next time.

So it is now 8:45am, and the system MUST be up at 9am. Every 10 seconds, your boss lets out yet another curse which communicates to you that another customer is having problems getting into the system. Also, several account managers are now hovering over your boss trying to make him understand how clients are REALLY REALLY suffering.

Everyone is depending on you to get the system up ASAP and at the same time is hindering your progress by constantly distracting you.

How do you keep cool in a situation like this?

share|improve this question

closed as off topic by Yannis Rizos Mar 25 '12 at 0:25

Questions on Programmers Stack Exchange are expected to relate to software development within the scope defined by the community. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about reopening questions here.If this question can be reworded to fit the rules in the help center, please edit the question.

33  
Step One: Compose a 300 word post on programmers.stackexchange. –  kubi Feb 25 '11 at 11:02
7  
Not saying it is happening right now. Wait let me check... –  Mag20 Feb 25 '11 at 11:03
5  
+1 for "cyber tooth tiger" :-) –  Péter Török Feb 25 '11 at 11:19
1  
Is this a problem unique to developers? If something you are responsible for isn't working you need to be able to deal with the pressure regardless of what that "thing" is. –  ChrisF Feb 25 '11 at 12:32
3  
This reads like the start of an entry on TheDailyWTF! –  Grant Palin Feb 25 '11 at 17:14

11 Answers 11

up vote 40 down vote accepted

In the situation, ask your boss to help you by keeping all the other folks away from you (which gives him something to do somewhere else).

When you get it up and running again, ask your boss for a meeting to evaluate and establish procedures for avoiding this happening again.

share|improve this answer
1  
+1 for sane and sensible advice –  Gary Rowe Feb 25 '11 at 14:35
1  
+1. Disaster recovery drills are good ways to gauge reactions and reponse times. Shame I don't see enough of it done. –  Desolate Planet Feb 25 '11 at 14:47
    
@DP yah, but we can't do that because it would mean the people and equipment aren't available for actual emergencies while the drill is going on (yes, I've heard that argument more than once). Of course if there were enough people, and equipment, you could train one team on one set while the other is on duty... –  jwenting Apr 26 '11 at 7:47
    
@jwenting sounds like saving on the fire alarm. –  user1249 Mar 15 '12 at 10:42

The first thing to do is to remove the distractions as politely as possible. Nobody can work with someone wittering in your ear about how bad it is for your customers. This is of course easier said than done if your boss is a maniac, but if that is the case, you might want to consider finding another job anyway.

Then make a quick appraisal of the real loss the error is causing and how (if at all) it can be mitigated quickly. With a bit of practice, you can also do a quick check on log files, which you'll need to form a plan of action.

If the problem is complex, concentrate on the most severe part of it. Think two or three steps ahead before you leap into action. Also, make sure you know how to back out of any plan before you act.

And the most important thing: Don't panic!

share|improve this answer

Situations like this are common in industrial control systems. The production line goes down in the middle of the night, the company is typically losing hundreds, or even thousands, of dollars per minute, and they're looking at you to fix the problem. You handle it thus:

  1. Explain to them what you know
  2. Explain what you don't know (but need to know to solve the problem)
  3. Explain how you're going to find out what you don't know
  4. Give them an estimate of how long that's going to take (use a range)
  5. Ignore everything around you while you focus on following through on your plan
share|improve this answer

First thing is to have repeatedly practiced disaster recovery (without people standing over your shoulder) so you know exactly what steps you need to take to diagnose and fix the problem without having to resort to questions on SO to find out what to do. Once you feel confident in your recovery skills, the pressure and stress is much much lower.

Next is to get the people out of your hair while you work. Your boss wants something he can go to his boss with. Give them some information on what you intend to do and how long it may take to do and then regular progress reports especially if you find something that means it will take considerably longer than you told them. Yes progress reports take time awy from fixing it, but hovering bosses and users take even more time away. Me, I go for the progress reports each time. Once they are confident you will keep them up-to-date, they will trust you to do your job more and leave you alone more.

If users are going to be blocked for some time, then send an email to them if this is an option or put up a notice on the website, saying the site is down for maintenance and when they should be able to try back. (This is possibly one task you can give your boss to find someone to do to keep him out of your hair as well.) People are less cranky about not being able to log in when they know someone is working on the problem. When things are fixed, if you sent out an email, email the same group to tell them it is fixed. Can't tell you how many times I've seen people forget this and users still think they can't log in when they can. The goal isn't just to get things up but to get people working with the system again.

Breathe deeply (deep breaths are calming) and plunge into the problem. It's good to have the things you need to do written down somewhere because in an emergency sometimes your brain synaspes don't pull up information as quickly as normal. You don't want to look like an idiot muttering: "I know we have a log, where the hell is it?"

If you are in a job where you support production systems, it is best to be the kind of person who reacts well in an emergency in general. I'm not sure you can learn this really. If someone riding a horse in front of you fell off (a not-so-random example taken from my life) and was lying bleeding on the ground, are you the kind of person who stands there with his mouth hung open or are the one who calls the ambulance, puts the pressure bandage on the bleeding and directs someone to catch the horse? If you are the first type of person, perhaps this isn't the right line of work for you.

share|improve this answer

Tell them this is a good reason why you need a backup server, and by that I mean a second server that runs the same as the primary one that can be switched to immediately if the first one goes down.

share|improve this answer
    
I've seen a backup server switched on, and it had the same problem as the primary server. It doubled the hardware cost, added to the configuration cost, and was a total waste of expense. If you're doing high-availability work, then sure, but you have to right-size your hardware to the problem. –  Scott Whitlock Feb 25 '11 at 12:56
    
one (extreme) example of the backup system being affected by the same bug as the primary system is Ariane 5 Flight 501 –  Andre Holzner Feb 25 '11 at 16:46

It's bad enough when you're surrounded from all sides by people who are angry with you for a problem you created, though it's twice as bad when it's a problem you didn't create. It's happened to me more than once that the client just configured it badly, meaning the fault is in communicating with the client (whether the fault is the client for not listening or the marketer for not explaining well, you'll never know).

How do you explain that they screwed up? Never an easy task, especially when your boss is breathing down your neck because he doesn't know any better than to assume the customer is always right.

So how do you keep cool in a situation like this? Politely remind your boss that the sooner you get to work, the sooner this problem will get fixed.

share|improve this answer

By seeing this event as an opportunity to show how valuable I am (to the business) by getting the production system back running as quickly as possible (if not before 9am ;-)).

Obviously, hoping I didn't break it in the first place ;-)

share|improve this answer
  • sh_t happens
  • there must be a solution to a problem
  • if someone in the world knows the solution, I can be one of them
  • if there is no solution, panic does not help
  • again, sh_t happens
share|improve this answer

Well definitely ask your boss that you will get back to him when you get the issue resolved; though in these sort of situations the management usually gets other people involved to get it resolved ASAP and then dea, with the "concerned" person later... That is the norm with any firm regardless of the industry; as for business the Customer is usually the King!!

share|improve this answer

Situations like that just motivate me more to have a thorough documentation of everything, and a thorough plan for dealing with any kind of situation.

Even if we can not predict every possible problem, but we can work our behind's off, in being more prepared, and organized, and documented.

share|improve this answer
1  
I've never solved a production problem (i.e. system down) by using documentation. –  Marcie Feb 25 '11 at 17:14
1  
No, but if you need to look something up, like specs, table definitions, server settings, it pays to have it documented. –  crosenblum Feb 25 '11 at 17:20

I spent 8 years doing maintenance on B52G bombers on a 5 minute alert for World War Three. That puts everything in perspective for me.

A down production system is important, but it is not going to kill millions or billions of people.

Find out what is wrong, find the cause, fix it. Establish clear communications with those that matter and keep them informed. Tell your boss what you are doing and when you will be able to update him can prevent a continuing steam of "is it fixed yet" messages and conversations.

do a through post-mortem and figure out how to prevent and limit the effects of such incidents in the future.

If you are on call, having a dead battery on a cellphone or beeper is extremely unprofessional. This is made up general scenario, but if this happened to a person working for me, there would be a serious discussion and if it was repeated, they would no longer be working for me. Yes I am a hardass.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.