Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

In preparation for a new release of software our company maintains, I've been working on what I believe to be a really good approach to solving our scalability issues. I have every intention of putting together a proof of concept to validate the design on paper will actually do what I want. When I briefed it to the team, the boss had a counter proposal, inspired in part by the way I described the problem areas. The boss also accepted my proposal to do two proofs of concept to evaluate the alternatives.

So, what's the best way to work through the proof of concept shoot out? We have both objective and subjective criteria that we are using to evaluate the solutions. I'd like to make sure we are comparing apples to apples with these fairly different approaches.

  • We have requirements for throughput and size. In short, we know we need to process a certain number of objects per second and maintain that rate for an hour.
  • We need to evaluate scalability (both by adding more cores and by increasing the number of objects)
  • We need to evaluate ease of development (subjective)
  • We need to evaluate how easy it is to understand the algorithm (subjective)

I have my theory about which way things will lean, but I don't want that to influence my results. Any input on how to maintain objectivity in this process, and things I might need to consider would be greatly appreciated.

share|improve this question
up vote 9 down vote accepted

Generally what we do is...

  • set a deadline like 1 month
  • use your requirements that you listed and code it up.
  • have some moderator who isn't writing either program control the requirements and doesn't see them think of a hidden requirement that should be doable in a short amount of time.

  • When the rest of the team looks over the programs to evaluate them THEY have to add in the requirement to each system so they get a feel for both of them. After the team dig into the code and add in the missing requirement they should have a reasonable feel for the codebases and pick which one they would rather develop off of.

    • then take the best remaining peices of the 'loser' and incorperate them into the winning architecture.
share|improve this answer
i like this idea – Justin Ohms Jun 30 '11 at 1:58

For the subjective items come up with a numerical rating of some time and try to obtain unbiased feedback. Example: for "Understanding the algorithm" have a programmer who did not write either look at both and rank each against each other.

You can also factor in objective measures on the code like the codes "Complexity", there are some tools to measure that based on number of control statements etc.

Take your ranking in each category and sum them up to a "Total Score" for each approach.

share|improve this answer

how to maintain objectivity in this process

You have only one objective criteria. The throughput.

Everything is subjective. You can't be "objective". All you can do is be "fair". World of difference.

The final decision is always political. As long as all the available information was provided; you've done all you can do.

Don't stress out over trying to make the perfect ("objective") point. What you see as right or best may simply be overturned by some ridiculous excuse like "the team doesn't have the required skills for your proposed solution".

Just build the demos. Run them. Be prepared for random decision-making. The very best you can hope for is informed and fair. You can't get to "objective" very easily.

share|improve this answer
It's not about winning and losing. It's about picking the best option. The team here is wonderfully apolitical, a rare find indeed. They also only care about picking the best choice. BTW: scalability is also measureable. – Berin Loritsch Jan 25 '11 at 13:47
@Berin Loritsch: The "best" option cannot easily be foreseen. Scalability is entirely based on assumptions about the future. Since we can't foresee the future, we don't have very objective criteria. We only have a bunch of disputable assumptions. "Best" is hard to do; perfection is unachievable. Relax. Pick something and go forward. A year from now, you'll realize a few of the mistakes you made. – S.Lott Jan 25 '11 at 14:08

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.