Rather than trying to find what OOP is ill-suited to, perhaps it would be more productive to look at some alternatives like functional programming, generic programming and aspect-oriented programming, and considering what they do particularly well and areas where they provide advantages over OOP.
Compared to generic programming, OOP requires that you define a relationship between types, even if there is none. In reality, "debt" and "rainbow" don't have much in common -- but if we're going to use them in Java (for example), they're going to have some common traits that most normal people would never associate with either one (e.g., hashCode, wait, and notify).
In theory the classes at the bottom of the hierarchy are concrete and those at the top abstract. In reality, as you get close to the top of the hierarchy, classes rarely represent a meaningful, coherent abstraction -- rather, they represent the the union of the requirements of a miscellaneous set of lower-level classes that somebody decided needed to be able to apply to all the other classes (e.g., the aforementioned hashCode, so you can store any object in a hash-based collection). I don't mean to pick on Java in particular here either -- .NET is similar in this regard, and Smalltalk is (if anything) considerably worse -- when you get to the base of its hierarchy, you have a "knot" that almost nobody even attempts to understand; a pretzel is hardly twisted all compared to what your mind has to do to fool itself into believing that mess represents anything even similar to a coherent abstraction of anything!
Generic programming does away with specifying a common base class to represent the set of operations required by a particular collection or algorithm or whatever. Instead, you define a set of requirements, and work with anything that meets those requirements. This, of course, can have consequences of its own -- e.g., mistakes with C++ templates are notorious for producing error messages that contain reams of meaningless garbage.
Aspect-oriented programming deals with a somewhat similar problem, but in a considerably different way. AOP is generally concerned (at least primarily) with cross-cutting concerns. A classic example is security. In a typical program, you end up with security-oriented code scattered throughout the program, "tangled up" with implementations of everything else. This means when/if you want to modify the security aspects of the program, you typically end up having to look at nearly everything else. AOP attempts to address that so you can move all the security code into its own, independent module, and specify points in the main stream of the code where the security code will be injected (so to speak) to do its job.
Functional programming can address a couple of completely different areas in which OOP is (or can be) weak. One is the ability to formally prove correctness -- lack of side effects makes formal proofs (and reasoning in general) easier. Note, that this is frequently misstated as "possible" rather than "easier" -- but this is incorrect. Formal proofs are possible in the presence of side-effects, but they are substantially more difficult.
Functional programming also limits communications between different parts of the code (for the most part, the only incoming communication is a function's parameters, and its own outgoing communication is the return value). This can be extremely useful for things like distributed systems. It's relatively easy to detect exactly where communications will be needed, and what parts of the system can run in parallel.
