Precisely because highly performing software components are generally orders of magnitude more complex than other software components (all other things being equal).
Even then it is not as clear cut, if performance metrics are a critically important requirement then it is imperative that the design have complexity to accomodate such requirements. The danger is a developer who wastes a sprint on a relatively simple feature trying to squeeze a few extra milliseconds out of his component.
Regardless, complexity of design has a direct correlation with the ability of a developer to quickly learn and become familiar with such a design, and further modifications to functionality in a complex component can result in bugs that might not be caught by unit tests. Complex designs have many more facets and possible test cases to consider making the goal of 100% unit test coverage even more of a pipe dream.
With that being said it should be noted that a poorly performing software component could perform poorly just because it was foolishly written and unnecessarily complex based on the ignorance of the original author, (making 8 database calls to build a single entity when just one would do, completely unnecessary code that results in a single code path regardless, etc...) These cases are more a matter of improving code quality and performance increases happening as a consequence of the refactor and NOT the intended consequence necessarily.
Assuming a well designed component however, it will always be less complex than a similarly well designed component tuned for performance (all other things being equal).