Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I'm leading a largish development team (~35 developers). We are doing primarily Web Development work on a number of sites.

Historically the knowledge on the teams has been pretty siloed. If you worked on Site A you will know how to troubleshoot it, but you would not be a lot of help on Site B.

We also have a few cross-cutting concerns, i.e. common components used between sites which require specialized knowledge to troubleshoot.

With all this in mind, I'm trying to understand the best way to setup an on-call team.

This would be a team of programmers who would be available to deal with out-of-hours emergency issues occasionally (say one call every 2 weeks). They may be required to deploy emergency fixes.

Part of me is saying we can't have a big on-call team with shallow knowledge, instead we need a smaller team with deep knowledge who can expect to be on-call more often and remunerated as such.

Does anyone have any suggestions based on experience on how to setup this team?

Thanks in advance.

share|improve this question
3  
The need for out-of-hours emergency patches is usually a sign of extreme organizational problems. Care to expand a bit on why you need an on call team anyway? –  Yannis Rizos Nov 2 '12 at 11:05
    
What YannisRizos says is right. This seems like more of a problem for system administrators not software developers. If software developers need to write emergency code patches in the middle of the night and apply them to production immediately then you have some pretty serious problems. –  maple_shaft Nov 2 '12 at 11:07
    
@YannisRizos, Companies that deploy frequently and have software that is used 24/7 internationally often have these needs. For these types of companies, I would claim this is good organizational planning and not "extreme organizational problems". –  cdkMoose Nov 2 '12 at 16:31
1  
I can't speak for OP's environment, but my team deploys new functionality on as frequent as bi-weekly basis. We can't expect sysadmins to be up to date on all of our changes (and every other team like us in the company). We don't get called often and we seldom actually patch production code immediately, but when that is needed, it is critical and I don't want just anyone dealing with a production issue at 3 AM. It is well worth identifying the right people in the first place. –  cdkMoose Nov 2 '12 at 16:33
    
These devs may be handling level 2 calls that don't always require programming to solve or caused the system to crash. –  JeffO Nov 2 '12 at 16:45
add comment

1 Answer 1

Where I work, the management first asked for volunteers, and then divided the on-call times based on the amount of available people. In practice, there's three on-call periods for each person during a two month period. They run in a weekly schedule of:

Person 1: Monday 08:00 until Friday 08:00 (64 hours of on-call)
Person 2: Friday 16:00 until Monday 08:00 (64 hours of on-call)

During office hours (08:00 - 16:00) there is no on-call, of course, as calls will go directly to the on-duty phone.

During cases where the on-call person is unable to solve the issue, he can call the technical lead and the overtime hours will start running for the Technical Lead, and the on-call person continues his on-call, though usually working together with the Technical Lead until the issue is fixed.

Update: Oh and the on-call person only handles issues related to his area of the system. If the issue goes beyond that, there is a different on-call person in another team. For example the Unix team has a separate on-call person, and the Database team has one, etc.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.