My feeling is no.
What I suspect you'd find if you did this is that instead of having individual teams producing libraries that no-one outside that team used, you'd have a specialised team producing libraries that no-one outside the team used (and doing so at considerable additional cost).
There are assorted problems with the sort of team you describe, but for me the main on is that it doesn't address the issue you actually have.
The problem you have is not who produces the libraries (by the sounds of things you already have many solutions to these problems so how is one more going to help?), it's that the teams aren't talking and interacting.
There are good reasons why teams don't reuse each others code (for instance that the problems while superficially similar are subtly different, or that the project timing just doesn't allow for the additional dependency of developing something together), but you need to look at how you can get them to interact when it is possible.
- rotate teams between projects
- hold inter team lunches and discussion groups
- post project reviews going over how problems were solved (attended by the other teams)
- set up an area of the wiki outlining code which might be reusable (and who to talk to about it)
- think about incentivising good re-use - seriously actually pay people extra for doing it. If re-using a component saves 5 days and $2000 in costs, why not give $200 of what is now extra profit to the team for a night out at the end of the project (when you've validated the saving was genuine)
A libraries team would be, I suspect, overhead with no benefit.
In terms of it being a common project that developers work on for fun - no company should rely on programmers working on things in their own time. That's just unpaid overtime and is, in any case, not dependable as there will likely be large periods where no-one wants to work on things.
If you're saying it would be people working in company time between projects then maybe it can work but I still don't think it's the real problem. You still need to work out how you're going to get people to use the libraries. As I said, you already have solutions to these problems which are being developed on each project, your issue is why aren't they being shared.