Team member missing? Glue between data producers and data consumers

Question

We're in the business of automated trading and our team consists of two bigger groups, I call them data producers and data consumers.

The producers' primary task is to maintain a chain of smaller tools that push some real-time data through an indicator system and out comes an order. All the data that was needed or produced is logged into files, one file per tool per run.

The data consumers on the other hand, used to backtests and captured in their backoffice world, want to fragments of the data produced in the different runs, polished to their needs, more specifically one big post-processed chunk of data per day.

Now the problem that has split our team into two well-distinguishable sides is that the data producers consider it their responsibility to provide comprehensive data without any loss of information and want the consumers to cherry-pick whatever they need in a pre-processing step.

The consumers on the other hand want to see live trading as a black-box, to them it shouldn't be different to the backtest which means the data producers in their eyes lack a crucial post-processing step without which they can't start their task.

Now clearly there has to be some glue between the two teams, my question is whose task is it? Or is there to be a third group in the middle that provides the glue? What does the theory say about this (can we apply the producer/consumer pattern to `real life')?

And just to make the problem a real one: The data producers consider it ugly to boil down the data into consumable chunks, mainly because the consumers' side keeps changing their requirements. The consumers on the other hand are not skilled enough to do the proposed cherry-picking.

Answer 1

I think it's appropriate to have someone who has a view into what each team is doing, and can identify potential problems, opportunities, redundant work and so on and make recommendations to improve things. It'd be that person's responsibility, from the technical side, to determine whether what the different groups are working on can fit well together and will solve the business problems. It would not (necessarily) be that person's responsibility to decide whether the problems being solved are the right ones, though. Call that person a technical officer, an architect, a principal engineer, or whatever.

I don't think you'd want a third "interface group", which would essentially have the responsibility to show whether the two other groups were doing compatible work, but would probably lack the authority to make changes to what each group was up to. Because now you have three problems: the two official interfaces that will spring up between the first-second and second-third groups and the unofficial one that arises to work around the bureaucracy inherent in communicating between the first-third groups.

Answer 2

You're asking a very specific question based on a very specific project and set up but it actually points to a generic answer that applies to all projects:

It's whoever it's agreed to be by the people involved.

There is no one right team structure, even for a specific situation. It will depend on the people involved, their skills, the time they can commit, the business and so on and so on.

Which means that the only approach you can take is to make sure that when you set up a project responsibilities are agreed within the team and that, as in this case, if you miss something, it gets picked up and assigned as soon as it's identified.

The first thing to try would be to get the two teams to talk it through and agree something. Ideally they would and they'd just take it to the PM and say "this is what we think" and he or she would say "great".

If that doesn't happen (for instance because they don't agree), then you need to take it to the PM and / or whoever determined that you will have these two groups. That decision was presumably taken based on some criteria or other and that will usually be a good starting point.

But there's nothing authoritative that's going to tell you how this situation should work, certainly not a group of people on the internet who understand none of the subtleties of the situation and have nothing at stake...

Answer 3

The way I see it you split a larger project into two chunks. The only sensible thing to split projects into are modules. Modules have and interface and an implementation. A module is good, if the interface is clear and leaks no information about the implementation. The way it seems now, the "producer team" did not create a module, only an unusable set of tools.

OTOH you can't expect their module to provide data in a structure perfectly suitable for every scenario. The needs of your clients may change and that actually shouldn't be affecting their work.

Depending on how much work needs to be done to transform a well designed producer module's output to a suitable consumer module input you then might need to create another module, who's interface depends on your needs and who's implementation depends on the interface of the producer module (and its own interface of course). If the producer interface is any good and the consumer requirements are the least bit reasonable, implementing such a module is trivial and is something you can demand of the producer team, assuming this really has an organizational benefit.

Answer 4

This doesn't sound much like a programming problem. Rather it sounds more like a teamwork/business/management problem.

If the two groups already had a mindset that they were in the same team, working together to make the business succeed, this problem may have already solved itself. At the least, they could bring any time/personnel constraints to the appropriate higher-ups. Then, an appropriate decision could be made, with all 3 groups working with the same goal - here's a problem that needs solving so our business can succeed.

It sounds like you're dealing with a situation where neither team wants to do the work. Generally that's a bad sign for the business' health.

Answer 5

Are the data consumers the users the data producers are producing tools for? In this case, what they need is ther requireement, not what you would rather do.