Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

This isn't a programming question per se - more about application management. I work for what started out as quite a small company, which has grown rather quickly and become very succesful in a relatively small period of time. Nearly all trade happens online, and our IT team is very small relative to the amount of money that gets pumped through the site (it's me and two others).

There are occasionally "blips" in the framework outside of hours, which inevitably results in either myself or one of my colleagues receiving a phone call. Our call centre aren't particularly great at analyzing errors etc.. and 99% of the time the solution is actually a data entry / user training issue. 1% of the time it's a real bug.

Our company is now insisting that we three developers figure out an on-call rota where we can be contacted with assurances that we won't be out of town etc. unpaid whether we are called or not, in case something goes wrong out of hours. Not all of our skills overlap, so I may not be able to quickly diagnose and fix a problem that one of the others could (e.g. database timeouts would need the DBA, email issues would need our infrastructure guy, website issues would need me).

None of us are particularly happy about this, so I was wondering if anyone else has had a similar experience and how you dealt with the situation?

EDIT: Thanks all for the suggestions - I've marked one as the right answer purely on the basis as it made me feel a bit better, but all suggestions are pretty much bang on what we've been discussing internally. We do feel like it's a little unfair we now have to give up our weekends so everyone else can get rich. We are attempting to push towards a more mature development process to minimize bugs too, but that would require the rest of the company to change too and they really don't want anything to change outside of IT!

share|improve this question

migrated from stackoverflow.com Jul 21 '11 at 11:32

This question came from our site for professional and enthusiast programmers.

    
"they really don't want anything to change outside of IT!" Why should they? If they are making money then nothing is broken in their eyes and if it ain't broke don't fix it. Your company probably views IT as a cost center and not as as means to save departments money, increase efficiency and grow revenue. You would think escaping this situation and putting them in trouble will teach them your true value but this almost always has the opposite effect in such an organization who will blame the incumbents for not being prepared for such an event, causing things to become WORSE (cont...) –  maple_shaft Jul 26 '11 at 11:22
    
... for the people you leave behind. Like I said earlier, escape and screw your friends, or take power and control to improve your immediate situation, things like bolstering front-line support and adopting better internal processes to deal with the unexpected, also automating as many manual tasks as possible. –  maple_shaft Jul 26 '11 at 11:24

7 Answers 7

up vote 1 down vote accepted

I am disappointed at the number of answers here that suggest that the OP should get paid for his after hours support when he clearly states in the question that management will not not only leave it up to them to determine how after hours support will be handled, but also that they will NOT reimburse them extra in anyway.

I am going to tell you from experience that when a small company rises quickly like this that management "sees green" and looks for ways to maximize the windfall profits, or they still haven't adjusted IT budgets to a much higher revenue and operate on a small company poverty budget.

It will get worse before it gets better and you might want to explore other opportunities.

In the meantime to make your horribly unfair experience a little more bearable, try to find out where front-end support is lacking and bolster their knowledge base as much as you can. Try directly training them how to handle common issues, write them helpful tools to do their job better, and improve training and help documentation. This will be the most effective way to improve your immediate situation.

share|improve this answer
2  
Strongly disagree. Mgmt in such a fast-paced organisation will trade off money for stability. Losing even one of their three experienced IT people is a large loss. They of course won't say to initially, but they'll probably buckle under pressure. Understanding this gave me the biggest raise I ever got in my career. –  MSalters Jul 26 '11 at 9:40
    
They would be logical to do such and I am glad that worked out for you but my experience differs greatly. If in this case they valued the stability that the team brings why would they essentially impose a policy on them that many job seekers would view as a deal breaker if they were seeking employment with said company? Perhaps if they hired a handful more people to help reduce the total on-call time for the veterans and expected more cross training then I would agree with your argument of stability vs. money, but then this doesn't sound like the case. –  maple_shaft Jul 26 '11 at 11:15
1  
I'm not arguing that mgmt will magically see the light, it will require some serious negotiation. But business is business, and getting paid for your job is business too. Don't take an initial "no" as an answer, that's what I'm arguing. That's just their cheap opening offer. –  MSalters Jul 26 '11 at 12:44
1  
@MSalters I've never worked anywhere where management trade off money for stability :) –  Mikey Hogarth Aug 1 '11 at 10:52

What does your contract say? Does it state working hours? If so, then this is an alteration of contract, which cannot be enforced unless you agree to it (at least here in the UK it is).

Many companies have paragraphs in there stating that you may need to work occasionally out of hours. This is not occasionally.

Having said this, I had to solve a similar situation at my previous company, but as the manager. We eventually did two things:

  1. We agreed that being on standby was not going to be paid, but any call-out would be compensated with a fixed amount for the first 2 hours, and then at an agreed hourly rate thereafter. We also agreed that the compensation would not be paid to the person taking the call, but to the person having to go on site and fix the issue (which was rarely the same person).

  2. I then instituted a fairly rigorous review system for every single callout that happened, and applied the motto of "fixing it twice". The callout would fix the immediate problem and the review yielded actions that fixed the underlying root cause.

Within less than 3 months we basically had no more call outs for anything other than third party faults or hardware failures.

IANAL!

share|improve this answer

It's not nice but the simplest approach would be for all of you to request a paid on-call structure.

My last job had on-call and they tried to reduce the per call/hour payment - everyone in the rota said they would not participate if that was to happen.

share|improve this answer
    
tried it, not on the table unfortunately :( –  Mikey Hogarth Jul 26 '11 at 8:50

I think this is pretty common, there are a few things that should be mentioned to your company that they should do to compensate you for the extra burden:

  1. If you are on call - they should provide you with a phone or pay for part of your bill
  2. Every effort should be made to allow you to fix problems remotely, and the company should pay for all / part of your internet bill
  3. You should be allowed some flex time for this. I've either been able to come in late after working on something the previous evening or my current situation is that I just get an extra 1/2 day of vacation a month.

I'd even ask for mileage reimbursement if you have to drive in.

I've also heard of places that pay a couple bucks an hour the the dev on call, not even minimum wage but people mind it less when they are paid. Good luck

share|improve this answer

Don't let them pass the buck.

Make management come up with the rotation systems. Simply Ignore there request to self manage, untill they get involved. Each of you should blame the other, till they are forced to intervene.

Once they do they will have to: 1) Give you more responsibility, clearly defining who has to do what. 2) THEN at some point they will HAVE to compensate you for it. Most likely it will be negotiated out when they are coming up with the after hours support schedule.

(They may also just hire someone else, or perhaps one of you team members will volunteer to be a sucker in hope of it paying off later)

share|improve this answer

Okay. A couple of comments.

Is there anything - and I mean - anything at all - about support for unexpected events in your contracts of employment? If there is, you may have to negotiate quite carefully.

Typically, out of hours coverage is pretty much an insurance - you pay for it in the hope that you never have to claim against it. Out of hours coverage should either be included as a component in your salary, or you should be able to claim specifically for it. Different companies apply different methods of payments, be it on a flat fee, a flatfee with call out charge or a call out charge only plus mileage, and coverage of comms costs. It is not really fair that it be done "free".

In my view, if you can get over that hurdle, there's no reason why you shouldn't organise the roster yourselves - it will allow you more flexibility in terms of swaps.

I would also strongly consider that you track every out of hours item and highlight that incidents are arising because your frontline is not adequately trained. If that was dealt with then the cost to the business of out of hours disruptions would fall.

share|improve this answer

Providing OOH support is very common, and if your company has started being successful its something they're going to need. Don't try and fight it; its inevitable that the company will need to protect its business interests from IT problems.

In my experience (in the UK) companies always pay for OOH support. Generally this is a flat daily/weekly rate for being on call plus payments for actually being called (sometimes its per hour, sometimes its a scale, say £x for a problem that took < 1h, £y for a problem < 3h and £z for >3h).

Its best if you can work the rota out between you, and have some flexibility to cover for each other when you have personal/social events. Plan the rota 6-8 weeks in advance and publish it to anyone who might need to call you (or at least give it to your manager).

You should also agree on a service level that you will provide. This should include;

  • The hours/days when OOH support is provided. This is usually everything outside of office hours, but it needs to be written down.
  • The response time for providing support. This can be something like; you will be logged on and investigating the problem within 20 minutes of being called. Avoid quicker response times than 10-15 mins as when you get called at 3am you will need to wake yourself up and get a coffee. Working on live servers while still half asleep is not a good idea.
  • Dispensation if you spend a lot of time working OOH, for eg if you are up all night doing support, then your manager shouldn't expect you in at 9am for the day job. You should be allowed to catch up on a little sleep and maybe get in at lunchtime.

You need to negotiate the service level and pay with your manager. Get together with the other IT guys and agree on what you want, then go to your manager and negotiate. Its important to negotiate together, not as individuals.

The company should provide any equipment required, such as a business mobile phone and laptop. That way the call centre can just ring the number without having to know the rota. It also means you don't have to give your personal phone number out, which can lead to being called when you are not on the rota. The laptop should be capable of remotely logging onto the servers to perform problem analysis and resolution, including checking log files, accessing databases and application servers. You might need to use a VPN for this. Having to drive into the office to provide support is not realistic.

General advice

  • When you are on rota, make sure you are available. Don't drink, don't drive long distances, don't go anywhere where you will be out of mobile phone reception for long.
  • Think carefully before taking action. This is very important if you've just been woken in the middle of the night. A colleague of mine once forgot the precidate on a SQL delete statement and turned a 5 min support call into a 6 hour restore from backup.
  • Be flexible with your colleagues. If one of them needs to you to cover for them in an emergency then help them out; you never know when you may need them to return the favour.
  • Document all callouts and make a league table. Every month or so take a look at the table and fix the top problem. Even if it isn't a software bug, do something to get around the need to be called (such as providing a crib list for the call centre to resolve the problem themselves).
  • Once you come off rota, give the phone and laptop to the next guy on the rota. Also do a little hand-over such as telling him what calls you got during the week and what he might expect during his week. Prep him with solutions to any expected problems so he doesn't have to figure it out himself at 3am with the call centre screaming at him.
share|improve this answer
1  
Being on-call without any sort of compensation is a non-starter. –  Robert Harvey Jul 21 '11 at 15:20

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.