In preparation for a new release of software our company maintains, I've been working on what I believe to be a really good approach to solving our scalability issues. I have every intention of putting together a proof of concept to validate the design on paper will actually do what I want. When I briefed it to the team, the boss had a counter proposal, inspired in part by the way I described the problem areas. The boss also accepted my proposal to do two proofs of concept to evaluate the alternatives.
So, what's the best way to work through the proof of concept shoot out? We have both objective and subjective criteria that we are using to evaluate the solutions. I'd like to make sure we are comparing apples to apples with these fairly different approaches.
- We have requirements for throughput and size. In short, we know we need to process a certain number of objects per second and maintain that rate for an hour.
- We need to evaluate scalability (both by adding more cores and by increasing the number of objects)
- We need to evaluate ease of development (subjective)
- We need to evaluate how easy it is to understand the algorithm (subjective)
I have my theory about which way things will lean, but I don't want that to influence my results. Any input on how to maintain objectivity in this process, and things I might need to consider would be greatly appreciated.