We're in the business of automated trading and our team consists of two bigger groups, I call them data producers and data consumers.
The producers' primary task is to maintain a chain of smaller tools that push some real-time data through an indicator system and out comes an order. All the data that was needed or produced is logged into files, one file per tool per run.
The data consumers on the other hand, used to backtests and captured in their backoffice world, want to fragments of the data produced in the different runs, polished to their needs, more specifically one big post-processed chunk of data per day.
Now the problem that has split our team into two well-distinguishable sides is that the data producers consider it their responsibility to provide comprehensive data without any loss of information and want the consumers to cherry-pick whatever they need in a pre-processing step.
The consumers on the other hand want to see live trading as a black-box, to them it shouldn't be different to the backtest which means the data producers in their eyes lack a crucial post-processing step without which they can't start their task.
Now clearly there has to be some glue between the two teams, my question is whose task is it? Or is there to be a third group in the middle that provides the glue? What does the theory say about this (can we apply the producer/consumer pattern to `real life')?
And just to make the problem a real one: The data producers consider it ugly to boil down the data into consumable chunks, mainly because the consumers' side keeps changing their requirements. The consumers on the other hand are not skilled enough to do the proposed cherry-picking.