I tend to work most with web applications and even though I'm trying to be general, my answer might not apply to your area of programming.
I am also going to use "framework" synonymical with "library".
Before implementing a framework, one must consider a few things, here are a few general examples.
#1. Will the framework save time and effort?
The answer to this question is almost always yes. Frameworks tend to be built to solve specific problems, and solve them very well. For instance, frameworks such as EntityFramework can save you entirely from writing SQL-code. Which can be fantastic if your programming team is not fluent in SQL.
Frameworks are built to either, a) add a programmer-friendly interface to otherwise complex components or b) add abstraction to already well known (or established) components.
The latter (or even the former in some cases) can actually get in the way of development. This applies especially when you or your programming team is going to implement a new framework, in which they have never worked before.
This could potentially slow down your development process, which could potentially be costly.
#2 The scale of your application
It is said that "anything worth doing is worth overdoing", but usually that's not the case. There's probably no good reason to implement a super-sized framework if the point of your application is to print "potato".
On the other hand if a framework does a lot of dirty work on the inside (i.e. database frameworks), then it might be viable to implement it, even if you only are "partially" using the framework. A good anecdote would be to not try to build your own ADO.NET or MongoDB-driver, just because you don't need to utilize the entire library.
Sometimes frameworks come open-source (and with 'do-whatever-you-want'-licenses). This opens up a new possibility where a programming team might only opt for parts of a framework.
This ultimately ties back to question #1 and #3.
Sometimes implementing a framework can directly impact the end user.
Even in other types applications, end-users might be negatively impacted by your application-dependencies. Frameworks at least always take up some disk space, and if you are developing a mobile app (or even a desktop app) this might be needed to be taken into consideration.
Server-side frameworks (even more web-specific) will most likely not affect your end-users, but it will affect your infrastructure. Some frameworks have dependencies themselves which might require you to restart your web server, either just the service or the server entirely.
Some frameworks might also be very resource-heavy.
This of course ties back to point #1 and #2.
It's all just a big "circle of considerations", and there is no real scientific method for deciding whether you should implement a framework or not.
Corbin March summarized it very well:
The groups I've worked with all do the same thing - make a guess at
costs and benefits, choose the most productive route, and hope they're
right. It's not terribly scientific - one part intuition, three parts
experience, one part susceptibility to marketing, one part cunning,
and five parts rank opinion.
It's also important not to be elitist. Frameworks are tools which are meant to be used. I know people of both extremes; on the one side you have the guy making life very hard for himself, on the other side you have the guy who builds slow, bloated applications.
All frameworks have use-cases, it's just a matter of implementing them for the right purposes.