Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

We're in the business of automated trading and our team consists of two bigger groups, I call them data producers and data consumers.

The producers' primary task is to maintain a chain of smaller tools that push some real-time data through an indicator system and out comes an order. All the data that was needed or produced is logged into files, one file per tool per run.

The data consumers on the other hand, used to backtests and captured in their backoffice world, want to fragments of the data produced in the different runs, polished to their needs, more specifically one big post-processed chunk of data per day.

Now the problem that has split our team into two well-distinguishable sides is that the data producers consider it their responsibility to provide comprehensive data without any loss of information and want the consumers to cherry-pick whatever they need in a pre-processing step.

The consumers on the other hand want to see live trading as a black-box, to them it shouldn't be different to the backtest which means the data producers in their eyes lack a crucial post-processing step without which they can't start their task.

Now clearly there has to be some glue between the two teams, my question is whose task is it? Or is there to be a third group in the middle that provides the glue? What does the theory say about this (can we apply the producer/consumer pattern to `real life')?

And just to make the problem a real one: The data producers consider it ugly to boil down the data into consumable chunks, mainly because the consumers' side keeps changing their requirements. The consumers on the other hand are not skilled enough to do the proposed cherry-picking.

share|improve this question
up vote 4 down vote accepted

I think it's appropriate to have someone who has a view into what each team is doing, and can identify potential problems, opportunities, redundant work and so on and make recommendations to improve things. It'd be that person's responsibility, from the technical side, to determine whether what the different groups are working on can fit well together and will solve the business problems. It would not (necessarily) be that person's responsibility to decide whether the problems being solved are the right ones, though. Call that person a technical officer, an architect, a principal engineer, or whatever.

I don't think you'd want a third "interface group", which would essentially have the responsibility to show whether the two other groups were doing compatible work, but would probably lack the authority to make changes to what each group was up to. Because now you have three problems: the two official interfaces that will spring up between the first-second and second-third groups and the unofficial one that arises to work around the bureaucracy inherent in communicating between the first-third groups.

share|improve this answer
Well yeah, I definitely don't want a third group. I was after some: Look, the theory says ... and that approach facilitates ... and everyone's happy. But apparently, typical design pattern don't work in real life :( – hroptatyr Oct 24 '11 at 10:59
@hroptatyr Yup, there is no theory, but Graham's second paragraph pretty much reflects real life scenario (at least the one I was involved in). Group "in the middle" will either have no power to make something happen or will wrap around "producers" group, sometimes fixing their bugs and bloating their own codebase (which will happen for sure, because group "in the middle" will be exposed to direct presure from "consumers"). – Jacek Prucia Oct 24 '11 at 11:57
@Jacek I know, too right. I guess I have to compromise on this one, my argument was: If this was a software project and the producers were a library, then consumers either adapt their thinking to the library's or they use a different better-suited library, doesn't work like this in RL sigh – hroptatyr Oct 24 '11 at 12:31
@hroptatyr a better argument would be non-technical. Just pick up two departments from your company that have the same producer-consumer relationship and propose department between them. Such absurd scenarios seem to be more comprehensible. – Jacek Prucia Oct 24 '11 at 12:43
@Jacek that is a VERY good idea, I will do that. – hroptatyr Oct 24 '11 at 12:49

You're asking a very specific question based on a very specific project and set up but it actually points to a generic answer that applies to all projects:

It's whoever it's agreed to be by the people involved.

There is no one right team structure, even for a specific situation. It will depend on the people involved, their skills, the time they can commit, the business and so on and so on.

Which means that the only approach you can take is to make sure that when you set up a project responsibilities are agreed within the team and that, as in this case, if you miss something, it gets picked up and assigned as soon as it's identified.

The first thing to try would be to get the two teams to talk it through and agree something. Ideally they would and they'd just take it to the PM and say "this is what we think" and he or she would say "great".

If that doesn't happen (for instance because they don't agree), then you need to take it to the PM and / or whoever determined that you will have these two groups. That decision was presumably taken based on some criteria or other and that will usually be a good starting point.

But there's nothing authoritative that's going to tell you how this situation should work, certainly not a group of people on the internet who understand none of the subtleties of the situation and have nothing at stake...

share|improve this answer
well my first problem is to make the PM aware of the difficulty, showing up with a possible solution is quite optional, I was looking for some formal, maybe even non-technical thing that makes it easy for everyone to agree to disagree and then we'll be at a point to start a proper discussion. At the moment it looks like the consumers clearly see us producers as culprits. – hroptatyr Oct 24 '11 at 12:52
@hroptatyr - For me I'd start with you having an informal discussion with the consumers, understanding their position and seeing what you do agree on. I suspect it's going to require someone independent or dispassionate to sort the whole thing out but the two of you may be able to reduce the size of the area under disagreement. – Jon Hopkins Oct 24 '11 at 15:39
Yes, good thinking, so far we positively identified what can be done without effort because the data is either there or it's trivial to convert the data to what the consumers need. The problem is a lot smaller now but it's not negligible, but I'll see what further discussions might bring. – hroptatyr Oct 24 '11 at 15:49

The way I see it you split a larger project into two chunks. The only sensible thing to split projects into are modules. Modules have and interface and an implementation. A module is good, if the interface is clear and leaks no information about the implementation. The way it seems now, the "producer team" did not create a module, only an unusable set of tools.

OTOH you can't expect their module to provide data in a structure perfectly suitable for every scenario. The needs of your clients may change and that actually shouldn't be affecting their work.

Depending on how much work needs to be done to transform a well designed producer module's output to a suitable consumer module input you then might need to create another module, who's interface depends on your needs and who's implementation depends on the interface of the producer module (and its own interface of course). If the producer interface is any good and the consumer requirements are the least bit reasonable, implementing such a module is trivial and is something you can demand of the producer team, assuming this really has an organizational benefit.

share|improve this answer
Ok, thanks for the input, I will definitely throw it into the discussion. Let's hope the producer team (which I'm part of) is up for it :) – hroptatyr Oct 24 '11 at 11:41

This doesn't sound much like a programming problem. Rather it sounds more like a teamwork/business/management problem.

If the two groups already had a mindset that they were in the same team, working together to make the business succeed, this problem may have already solved itself. At the least, they could bring any time/personnel constraints to the appropriate higher-ups. Then, an appropriate decision could be made, with all 3 groups working with the same goal - here's a problem that needs solving so our business can succeed.

It sounds like you're dealing with a situation where neither team wants to do the work. Generally that's a bad sign for the business' health.

share|improve this answer
+1 for the social aspect perspective – sleske Oct 26 '11 at 14:59

Are the data consumers the users the data producers are producing tools for? In this case, what they need is ther requireement, not what you would rather do.

share|improve this answer
no, the producers produce as much data as necessary to drive the live trading, it's only now that the consumers wish to gain more insight and want to use their tools that proved suitable when dealing with backtest data. – hroptatyr Oct 24 '11 at 15:47

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.