Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

At our company, we frequently use third-party libraries and frameworks in our products. Often, we are faced with the task of evaluating one or more libraries that full fill similar requirements (in a specific case we had to choose between some encryption libraries). We often do a simple prototype to try to figure out, if the library somehow works for us. However, often the prototypes get really complex and when we are at a point where we figure out, that the library is not suited for our purposes, lots of time was spend, that we could have used to write the desired functionality ourselves.

My question would be, do you have or practice any guidelines when it comes to evaluating third-party libraries? More specifically, do you have something like a maximum time that will be spend on prototypes and do you have general requirements for libraries you use (license issues, support, documentation, ...)?

share|improve this question
up vote 4 down vote accepted

Speaking only for myself (and having done dozens of these):

If it's closed-source, it's not worth my company's time.

If the open-source license would cause even slightly raised eyebrows in Legal, it's not worth my company's time. LGPL is fine, Apache is fine, MIT is fine, GPL (any version) is not.

If the documentation doesn't include at minimum full interface docs, a quicky tutorial, and a FAQ, it's not worth my company's time.

If the library requires me to build it before I can use it, it's not worth my company's time.

If the library doesn't appear to have a google-detectable user community, it's not worth my company's time.

If the library vendor won't publish a roadmap or open up their issue-tracking system (even read-only), those are bad sign, but not immediate downchecks unless all the other candidates do better.

That cuts out the deadwood. Once we're there, it's time to design the bake-off. A useful bake-off shall have a fixed time-scale, a fixed set of evaluation criteria, and a fixed feature set, all of which should be written down. All of the features in the feature set should be tied to an evaluation criterion (ideally exactly one of the evaluation criteria). For longer bake-offs (particularly those with >2 candidates), there should be a checkpoint. At the checkpoint, all of the candidates should be examined to see if any one of them completely is completely dominated on all evaluation criteria. Any candidate so dominated (or even almost completely dominated) should be dropped mid-way through the bakeoff.

This all sounds fussy, but the alternative appears to be endless evaluation criteria that go off into the weeds until the original business justification for the technology is obsolete or dropped through sheer exhaustion.

share|improve this answer
Closed source - not worth your company's time. Seriously? Practically every robust commercial library I've ever used has been closed source (NAG, IMSL, RAD, ...). – Rook Aug 18 '11 at 23:17
+1 on the whole, although the 'requires me to build...' argument doesn't mean much when building is painless - in other words, when the usual ./configure, make, sudo make install works out-of-the-box and doesn't take more than five minutes, I don't see the problem. – tdammers Aug 19 '11 at 6:31
This will help to build our own third-party libary checklist. However, I also do not agree with the closed source objection (at some areas you do not even have a choice - for example if you program in a printdata or mainframe enviroment) – Benjamin Rogge Aug 19 '11 at 14:29
Fair enough. For many years I've been working in B2B startups, where closed source libraries are simply not considered. Additionally, I've been in the Java ecosystem, where pre-build library archives are (almost) standard practice. In different spaces, your mileage may vary. – Dave Griffith Aug 19 '11 at 15:45

The single most important guideline I always try to follow when choosing stuff like that is to have a documented checklist

  • for third party library my list may look like this: maven; maven - release version; code examples; open source; understandable javadoc; compact API; reference to specification used; reference to public spec used (like RFC, JSR etc); releases changelog; bug tracking; active; positive feedback (most recent version of the list documented and explained here: csv api for java:)

do you have something like a maximum time that will be spend on prototypes

In cases like you describe - yes. More generally, whenever I sense anything resembling a deadline, I immediately add it to my checklist. ...; tentative date to decide whether to accept or reject a prototype

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.