I'd really like to focus on why some new programming languages are adopted in the mainstream, and others remain relatively niche. I'd like to know about things like specific use cases, backwards compatibility, or some new features, simple or complex implementation difficulty. Specific examples would be appreciated, but let's not get caught up on the exact definition of "mainstream" or "niche" here.
If anyone really knew, they'd be very rich people.
That said, here's my guess:
BASIC is an awful little language that came with (all?) PCs when they first came out. The Apple II (the default computer for many high schools), and the ATARI 400/800 (the first super cheap home computer) both came with BASIC.
If you had a UNIX machine (either AT&T system V or Berkeley's) you had C. If you wanted to program the machine, that's what you had to use.
EDIT: With the advent and ubiquity of the 'net, this requirement transforms into:
Ease of Transition
Java is a good example here. If you knew C programming, Java was not a very far jump ahead, and yet it gave many of the features that C lacked and C++ failed to provide cleanly.
C++ for obvious reasons was an easy step forward from C, and it was easy to sell to management as an "improved" C. C++ had the added benefit of being backward compatible with much of the existing legacy C code base.
Perl was an amalgamation of C, awk, sed, and other Unix utilities all in one bundle. Prior to its appearance, most system administration was done through shell scripts gluing everything together in an unsatisfactory way. Bringing everything under one process with the data structures and control of a C-like language was a godsend.
Fills a Need
C took off because it allowed you to produce close-to-assembly efficiencies without getting bogged down in the machine-specific, hard to maintain world of assembly.
FORTRAN took off because it allowed for easy translation of mathematical ideas into code without having to get lost in the details of the machine.
Likewise for LISP and symbolic manipulation.
Python grew out of the need for a "better" Perl. (I'm biased here, so I won't say more.)
PHP was essentially the BASIC for the web -- it was installed by default on many web servers, and it was easy to hack together something useful quickly.
Advocacy, User-base, Contributed Code
Let's face it, Haskell would not be anywhere as popular as it is for a not-in-production language if it weren't for the tireless advocacy of its developers and user base.
Many languages have a cult-of-personality behind the language's creator(s), and we all know who they are.
FORTRAN has huge sets of established and vetted mathematics code; same for Java and the web/systems-integration/MVC-systems; same for Perl and CPAN; same for TEX and document management; etc.
The It Factor
For some reason some languages just seem to have the right amount of new, with enough of a nod to the old, with a way that makes it seem easy or needed. That is, it makes its own case. And who knows just how this happens?
Anyway, that's my best guess for why some make it. As for why some don't... well, if they don't meet the above criteria, that's probably why they failed.
Languages becomes popular because they have an advantage over existing languages in an area that is needed.
I'll be a cynic: money and coming with that, marketing.
It's no coincidence that C# is supported by Microsoft, Java by Oracle and Objective C by Apple. Only Google's Go hasn't really lifted of so far.
Of course money is not the only reason but having deep pockets sure helps to place your language in the market.
On one hand it is marketing, more precisely presentations, blogs etc. It is important to have features that mainstream programmers can relate to and see benefits over what they know (Java vs. C++ - garbage collector vs. memory management). Last, but not least is to have low entry barrier - examples, good documentations, seamless install, good community and support, vibrant development.
Almost all the languages had non-trivial, real-world problem solving programs written using them very early in their life.
Unix was written in C, so were the tools on Unix, when C was very young and evolving.
Anaconda (RedHat's installer program) was written in Python when Python was young and did not have the popularity of today.
These are what I can recall off handed. This list could touch each of the languages that has survived its formative years.
Then, large scale adoption in the universities can help a language's longevity. Java is very popular at universities as a teaching language. To some extent, Lisp and dialects of Lisp enjoy this status too.