have I just been lucky enough to not to have to worry too much about it, or am I a bad programmer?
Do you care about your requirements? If performance isn't a requirement then don't worry about it. Spending any significant time on it is a disservice to your employer.
To an extent performance is always a requirement. If you can hit it without thinking about it, you are justified in not thinking about it.
Personally, I'm most often driven by performance when my tests take to long to pass. I'm too impatient to wait 5 minutes while a set of tests pass. But that's usually solved by fiddling with the tests.
My question is why is it that a large number of programmers care so much? Is it really an issue for most developers,
There are large numbers of programmers who are justified in how much they care. There are large numbers who aren't. Let's talk about those who aren't.
One of the first things programmers learn in school, after how to make things actually work, is big O notation. Many of them learn the lesson properly and thus properly focus on things dramatically impacted by n. Others don't get the math and only take away the lesson that once it works in needs to be fast. Worse, some of these students never learn anything else about what's important to do with your code besides make it work and make it work fast. The missed lessons: make it readable, design it well, don't play around in it for no reason.
Knuth was right: premature optimization is the root of all evil. But once it works what is the next step? Fast right? NO! The next step is readable. Readable is the first, next, middle, and last step. Many of the people I find doing unneeded performance optimizations are throwing readability under the bus.
Some even get a perverse thrill from how unreadable their code is. They've had to suffer looking at hard to understand code created by others so now it's their turn at payback.
I know this because I used to do this. I once refactored a perfectly readable 5 line if structure down to an indecipherable one line boolean expression and proudly sent it to my professor expecting to impress since I could create something so compact and intimidating. I didn't get the praise I was hoping for.
If code stays readable making it fast later is easy. That's why Knuth emphasizes "premature" not "unneeded". Because sure, faster is better. But better is only better depending on what you sacrifice for it. So wait until you know what performance you really need before you make sacrifices for it. Sacrifice readability reluctantly because once it's gone, it's hard to get back.
Beyond readability is the whole world of software design. What this site is about. Some have no clue what to do as far as design. So since they can't impress with design they make an indecipherable mess so people can't tell they have no clue. Since no one ever fixes their code it must be good code right?
For some, performance is the catch all excuse to do whatever they want. Programmers have a lot of power and autonomy. Trust has been put in them. Don't abuse the trust.