Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Modular programming and reusable software routines have been around since the early 1960's, if not earlier. They exist in every programming language. Conceptually, a software library is a list of programs, each with its own interface (entry points, exit points, function signatures) and state (if applicable). Libraries can be high quality because modules are focused on solving narrow problems, have well-defined interfaces, and the cost can be amortized across all future software programs that will ever use them. Libraries are purely additive: adding new modules does not introduce bugs or limitations in existing modules.

So why don't software libraries solve all our problems? Why can't we write software as merely a composition of high-quality software modules?

Can we fix these problems with software libraries and unleash the full potential of this incredibly powerful mechanism for writing high-quality programs faster? Or are these problems intrinsic to libraries and can never be solved?

Note: Several comments said that I have too many questions. Please treat the above questions as the real questions and everything that follows as points for discussion.

It is true that software libraries are widely used. Some programming languages such as C, Java and Python have enormous software libraries. But there are numerous problems.

  1. Some well-known languages have less than ideal library support (e.g. C++, Lisp). To some extent this is mitigated by piggybacking on a virtual machine platform (e.g. JVM, CLR). A corollary question, should all future software be written for a virtual machine platform to increase library support? This is problematic for scripts that don't want to incur the cost of launching a virtual machine every time.
  2. There is a lot of "reinventing the wheel". Have you ever written a linked-list module in C? Yes, of course you have. I don't enjoy writing linked lists in C, but what is the alternative?
  3. Can a given library (e.g. libfoo-0.1.2) be trusted as as a basis to write your important software? Is the library tested, documented, and does it implement the features you need? How can you tell?
  4. Learning a library's API can be as time consuming as learning a whole new programming language.
  5. If a bug is discovered in a library, what is the proper procedure for fixing the bug in your software? How should the bugfix be distributed to all library users?
  6. How should libraries be built and distributed? (For example, autotools obviously got it wrong.) Semantic Versioning is good, but how can this be verified and enforced?
  7. How should library dependencies be handled? Can we download and install automatically? Will the versions and licenses be compatible?
  8. Is the license for a given library compatible with your software? How can you tell?
  9. What should you do if the library is abandoned before starting your software? What about after starting your software?
share|improve this question

closed as not constructive by Mark Trapp Feb 22 '12 at 19:28

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.If this question can be reworded to fit the rules in the help center, please edit the question.

    
" Why can't we write software as merely a composition of high-quality software modules?" Do you use Python? That's actually a complaint about Python programming. 99% of Python is just reassembly of libraries. Your question isn't really as general as you think. You might want to specify the specific language(s) around which you have this question. –  S.Lott Feb 22 '12 at 13:59
1  
C++ certainly does NOT have poor library support. Your question is not clear. You have far to many questions. –  Ramhound Feb 22 '12 at 15:19
1  
Hi Nathan, Stack Exchange is not a discussion forum: it's not appropriate to add points for discussion because they're off-topic here. If you'd like to have such a conversation with other programmers, feel free to use chat. If you have a specific problem you'd like others to help you solve, feel free to ask about that, instead. –  user8 Feb 22 '12 at 19:30
    
I would vote to close as 'not a real question' - but I see it's already closed... –  Vector Aug 22 '13 at 9:08
add comment

7 Answers

Libraries are great and certainly solve a lot of problems very quickly. However they never solve every problem and even gluing together many libraries isn't enough to solve many specific use cases.

Programmers are not typists. We've had that discussion before. Despite all the libraries, frameworks and out of the box solutions, there is always something your framework, library or SaaS product is not going to do for you. You're going to need to store your data in a different way, you're going to need to read your data faster, you're going to need to validate input in a way your libraries don't.

In any programming project of significant size, there is going to be some problem you need to solve that's so specific to your use case, so unusual, or so complex/simple that no library covers it properly. You're going to need to write your own code, your own objects or maybe even your own libraries or frameworks.

There's also lots of business reasons libraries aren't or can't be used. Maybe we can't use this library because of its license. Maybe we don't trust the source, or we'd have to modify it, and we don't trust an altered library. Maybe management believes all code needs to be purely owned, created and maintained by the company. But for various reasons, good, bad and spurious, many IT shops don't or can't use as many libraries as possible.

Some of these problems simply can't be fixed; some management types (or coder types!) are simply never going to trust code they don't own. The quality of libraries will never trump dogma.

Other problems are more solvable (quality of libraries, licensing) but the fact of the matter is that the legal and practical implications are often large and complex. Until all libraries are public domain and are 100% stable while containing all possible features (with perfect performance!) there's always going to be a reason to write your own code rather than use a library.

This isn't to say that libraries don't result in better programs; libraries solve a lot of problems and keep you from endlessly solving and debugging the same problems or working on hacked solutions. But the practical barriers mean they'll never be a panacea.

share|improve this answer
    
I agree that every software project has some subset of the code that needs to be written new. But couldn't that new code be put into a library and then all future software would benefit? –  Nathan Farrington Feb 22 '12 at 15:14
2  
@Nathan You could, but when you attempt to librarize (don't look that up) something, you might be spending time on something tangential to your task. Of course, once you hit a second (or third) situation where you do need that behavior, then you should probably take the time to break out that functionality and genericize it for consumption. –  birryree Feb 22 '12 at 15:23
    
@birryree, if we could lower the cost of implementing libraries then it would not be so bad. Also, if code is important, shouldn't it be in a library rather than locked into a single application? –  Nathan Farrington Feb 22 '12 at 15:32
1  
@Nathan Yes, but it tends to start in an application and migrate into a library. More awkward is when the code deserves to be a library but is so dependent on the features of being an application (e.g., it assumes that it can pop up a dialog box to ask the user a question any time it wants) that splitting it out is really difficult. That happens rather often, alas. –  Donal Fellows Feb 22 '12 at 15:39
2  
@Nathan Farrington: You seem to assume that it's a fairly trivial operation to take application code and turn it into a generally useful library, and it really isn't. You have to design a generally useful API, document it, and test it as if it were new code. Brooks, in "The Mythical Man-Month", estimated that coming up with a good library would be about three times the work as doing the same in an application. –  David Thornley Feb 22 '12 at 16:04
show 2 more comments

Libraries were created by someone to fit some purpose they had. It might sound like a problem you want to use the library to solve, but it might not fit your exact usage. When considering a library you do have to evaluate it.

As a developer, one of your responsibilities is to think about how any library may fit into your software, and whether or not you can make your software and libraries work well together. Certainly as you gain more experience that becomes easier, but when you're a blank slate on libraries, there is a higher initial ramp-up time.

That said, I did find your questions interesting so I took my shot at answering them.

Some well-known languages have very poor library support (e.g. C++, Lisp). To some extent this is mitigated by piggybacking on a virtual machine platform (e.g. JVM, CLR). A corollary question, should all future software be written for a virtual machine platform to increase library support? This is problematic for scripts that don't want to incur the cost of launching a virtual machine every time.

Define "Very poor." C++ has boost, qt, Windows MFC and Win32 API, poco, WxWidgets, and those are just the general use frameworks and libraries similar to what you'd get with, say, the Java API. There are C++ libraries for many things, without having to factor in a CLR or JVM. Plus you can take advantage of the many C libraries out there (glib, curl, Apache Portable Runtime) and wrap them in C++ if you want to feel more C++-ey.

Lisp, though. There are many varieties of Lisp. You have modern Lisps like Clojure (on the CLR and JVM) and they can take advantage of their respective platforms. Common Lisp as a standard has many implementations, and I know there are companies out there building software in Common Lisp. I don't know enough about the kind of library support it has though.

Also, whether or not you should use a VM-based language. That's really up to you and what you want to use and what your company/employer/team are fine with. I will use what I think is a suitable tool for the job, as well as what I'm productive in for a given task.

I use Python and Ruby heavily because it's easy for other people to understand and I can quickly do a task in them.

There is a lot of "reinventing the wheel". Have you ever written a linked-list module in C? Yes, of course you have. I don't enjoy writing linked lists in C, but what is the alternative?

I have, but only as an academic exercise. There are several widely used libraries that provide data structures like linked lists for C, namely Apache Portable Runtime and glib.

The general alternative is to not leap before looking - you'll find reinvention in all languages, and much of that may be due to lack of knowledge about existing solutions.

Also, languages and libraries evolve over time. You're not going to find a library that fully suits your needs at some point in time, and in certain fields, algorithmic invention might be very commonplace.

Can a given library (e.g. libfoo-0.1.2) be trusted as as a basis to write your important software? Is the library tested, documented, and does it implement the features you need? How can you tell?

I base it on usage, community, maintenance activity, and if I must, I will check out its bug/task queue and see if it moves along smoothly. If it's commercially supported, I like to see if they actively update it. I also go by recommendations from developers I trust, as well as from communities like StackOverflow, where certain libraries are just mentioned significantly more than others.

There are libraries which I will trust without question, like boost and qt, and I will trust things from Google to perform correctly because I know Google eats their own food and uses their own code internally to process the bajillions of gigabytes of data they get every day.

For other things, I will evaluate it, read the code, and tailor it or scrap it if it's unsuitable.

Learning a library's API can be as time consuming as learning a whole new programming language.

Languages are easy to pick up, especially once you have experience in other languages and a similar paradigm. Ultimately they boil down to you expressing your logic in text, and being sure the compiler/interpreter knows what you mean.

This could be a factor of poor documentation, or you're using a framework that is significantly scoped (like the Java API). But the secret is, you don't necessarily have to learn 100% of a library to be effective using it. I consider myself a proficient Java developer, but I have not touched parts of the Java API because I have never had a need to.

Single purpose libraries are easier to learn in that regard - they have more limited functionality and hopefully documentation is simpler for it.

If a bug is discovered in a library, what is the proper procedure for fixing the bug in your software? How should the bugfix be distributed to all library users?

I have a lot of experience with this (making boost work on Solaris, making boost work on AIX, making tons of other things work on Solaris, like graphics libraries). Fixing it is the easy part.

If this is proprietary code, then my company would do something like patch releases if it was minor issues. Basically a bunch of fixes rolled into one, and then we notified customers via email that we had a new patch release.

For critical or per-customer issues, the fixes were put into a quick patch and we would either distribute just patched files to users, or release a quick patched release.

For open source projects - file a ticket with the project's bug tracker, and if you fixed it, submit your fix recommendation and a patch.

How should libraries be built and distributed? (For example, autotools obviously got it wrong.) Semantic Versioning is good, but how can this be verified and enforced?

autotools is a bear, but modern make systems are nicer, like cmake. There are more like Apache Maven, and you can use these systems to build and package distributables. I also like buildr, which is a Ruby alternative to Maven (compatible with maven).

How should library dependencies be handled? Can we download and install automatically? Will the versions and licenses be compatible?

Using cmake or maven can help you take care of dependencies, and maven can pull in remote dependencies or from a local server. cmake can do the same, but you do have to tell both these build systems what your dependencies are. Versioning and licensing is mostly your job, but if you were on a system like a Linux with a package manager, this could be mitigated.

Is the license for a given library compatible with your software? How can you tell?

You need due diligence here, because licensing is legal, not a software issue. You can see what the licenses are for various things in your system, and there are sites like FSF's License List and Comments which give FSF's interpretation of various licenses' compatibility with the GPL. Also, consult a lawyer.

What should you do if the library is abandoned before starting your software? What about after starting your software?

If it's abandoned before starting your software - evaluate to see if it's still useful to you, and check it's bug list. You could contact the maintainer and offer to take on the project.

If it's abandoned after - again, you can decide if it's worth it to continue, hopefully it's not so deeply embedded into your code that you couldn't switch it out with some effort. Of course, there are softwares where that is a problem and they get pegged to some old version. You can, though, also take the project under your wing and tailor it to your needs. Contact the maintainer.

share|improve this answer
    
Thanks for these points. I think by listing common problems with libraries, I may have obscured my real question. Can we fix these problems and how? For example, how can we automate away each of these problems. Computers are fast and people are slow. Can we automate away licensing, packaging, installing, dependencies, bugfixes, discovering libraries that do what we need, writing new libraries, etc. How can we do this? Given enough time, each of us could make the perfect libfoo, but if we can lower the cost of making the perfect libfoo, the we will all have better libraries. –  Nathan Farrington Feb 22 '12 at 15:25
    
Each of those points can be automated to a degree - package managers/software updaters attempt to get rid of the headache of dependencies, and put the onus on just a small set of people who do the packaging and verification. Library discovery - yeah this one is a lot of information that floats around on the internet - I still discover new libraries all the time, I don't know a good way for library/framework discovery other than reading and hearsay. Licensing - maybe we can build an auto-license compatibility checker. Fixes and writing libraries - well, not everything can be free, work-wise. –  birryree Feb 22 '12 at 15:30
1  
@NathanFarrington: "I may have obscured my real question." I would agree. Please update the question to be your real question. Don't clarify your question in the comments to an answer. Actually rewrite the question to be your real question. –  S.Lott Feb 22 '12 at 16:58
add comment
  1. Depends on the language. JRuby is performance-competitive with Ruby, but you would not want to run C on the JVM.

  2. I'm not an expert in C, but if you google 'linked-list library C' you will find several results.

  3. I trust libraries based on the backing (is it a library Google uses or an obscure one), how active is it (likely to be supported), and is it open-source (can I fix bugs/extend it myself).

  4. Learning library APIs does take a lot of time. Fortunately, you only need to learn what you need to solve a particular problem, and many of the paradigms are the same across libraries (data structure names, file IO, etc).

  5. Fix the bug and submit the fix. Hope it is included in the next release, then everyone will get it eventually. In the meantime people googling the same problem will find your fix.

  6. Not sure what you are asking on this one.

  7. Many possible answers here, Apache Maven is interesting. I don't think there is one 'correct' way to go about it.

  8. Google the license. If your software will be commercial or closed-source it may limit your options.

  9. Before starting the software you probably can defer deciding which libraries to use, but try to choose ones with high levels of activity. After starting software, hopefully you wrapped a risky library in a layer such that you can replace the implementation, otherwise you will have to start supporting it yourself.

Honestly, none of these are the reason software libraries don't solve all our problems. The reason is that problems are always changing and evolving. From moving to dynamic websites, mobile devices, or solving obscure problems in a domain, if there was a library to do it then companies wouldn't need to hire more than one programmer.

share|improve this answer
add comment

First of all, there's a lot more code reuse than most people give credit for, just not for every single application. Have you tried making a website using a modern CMS? "Merely a composition of high-quality software modules" is a perfect description.

There is an inherent coupling between the general applicability of an interface and its complexity. Let's say you wanted to make the container to end all containers. The interface couldn't be as simple as an array's because some people need O(1) insertions into the middle, some need fast lookup, some need persistent storage, some need space efficiency, etc. and they need a way to specify and operate upon those needs.

When you try to do too much in a library, your interface ends up with a complexity comparable to the programming language itself, so you may as well make narrowly-applicable interfaces and use the programming language as the glue, which is precisely what we have done.

share|improve this answer
    
You are right, I did not give enough credit to how pervasive software libraries are in modern software development. I'm looking for the limits of libraries. What is keeping them from being better than they are? For example, is it possible to write an application and extract a library from it automatically? How much can be automated? –  Nathan Farrington Feb 22 '12 at 17:39
add comment

The reason that software libraries can't solve all our problems is because software libraries just solve many (not all) of the problems of software developers. They don't solve the problems of software users!

Suppose your organization decides the world needs a really good 3D visual interactive aerospace design application? You, as the software developer, may well start with an existing 3D graphics library. For this exacting application you're almost certain to find that there is no library that does everything you need, you'll have to extend what already exists. And certainly there is no 'design a Boeing 747 or Airbus 380' library! It will take years of work to develop this application.

Suppose you are a cabinet maker, a customer comes in with a hand drawn sketch of a cabinet she wants created. You have a shelve full of woodworking books and a shop full of good woodworking tools.

The tools and books help, but you still have to make the cabinet!

share|improve this answer
add comment

So why don't software libraries solve all our problems? Why can't we write software as merely a composition of high-quality software modules?

People are trying to make this easier every day but there biggest hurdle I can think of is : Every 3rd party software library is still disjoint with how you store data.

How could a 3rd party PDF library know I want to create a PDF from my books table, author table and the sum of the prices for that author?

ORMs help tackle this but it's short of the sentient components you're hoping for.

Also, try to come up with a more concise question. Discussion is difficult with so many questions.

share|improve this answer
    
Sorry, I agree that is a lot of questions, but hopefully they are all related. If we ignore the numbered list of problems then the concise questions are "why don't software libraries solve all of our problems?" and "can we fix the existing problems to allow software libraries to realize their potential?" It sounds like you are saying that the primary problem with libraries is their different and arbitrary interfaces? –  Nathan Farrington Feb 22 '12 at 14:58
    
The primary reason they can't all be plugged into one another to make something meaningful, yes. I wouldn't say this is THE primary problem with 3rd party libraries though (you've listed quite a few yourself in the original question that are more concerning) –  brian Feb 22 '12 at 15:19
    
so you are saying the primary problem is a technical one where the libraries simply do not interoperate well, vs. a business problem like others have suggested? So in principle we could solve this technical problem of interoperability? Would the JVM and CLR be the way to go since they simplify library and programming language interoperability? –  Nathan Farrington Feb 22 '12 at 15:37
add comment

A difficulty that I don't see described clearly in other answers is how to find and exploit a relevant library efficiently. Even if all available libraries were searchable at http://code.google.com/hosting/ (and of course they are not) it would sometimes take longer for a programmer to complete that search process than to improvise an acceptable solution using the libraries that are already at hand.

Mathematicians have a similar problem to find math theorems that have already been proved. No one has yet perfected "mathematical knowledge management" software for that purpose. Aspects of the problem may be beyond the possibility of automating. The conceptual extreme case is Jorge Luis Borges's "Library of Babel" in which everything, as well as every possible variation on everything, is in the collection. Then the only unambiguous way to describe an item from the collection is to recreate it.

Edit: And where I just mentioned the possibility of having to choose between many variations on software libraries, a highly relevant example is different trade-offs between space (data storage) and time (CPU usage).

Edit 2: OK, my Library of Babel observation isn't mathematically correct. We can give names to the items in that catalog that interest us. But what I understand of Chaitin-Kolmogorov complexity leads me to believe that any such naming system leaves plenty of (perhaps unwanted) information resources whose most concise name is just as verbose as their complete content.

share|improve this answer
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.