I guess programming is sometimes viewed as a second-rate role because "sometimes" (in fact, typically) programmers are not the most important people in a randomly-selected company, and also are not the ones who take the highest-level decisions. I wouldn't say myself that project managers or hedge fund managers are inherently superior to programmers. But if someone is of the opinion that they are, then clearly programmers will be viewed as secondary. And in a certain context they objectively might be superior, in the sense that if you need someone with particular skills then someone who has them is superior to someone who doesn't (or at least, whose role doesn't say they do). So in that context again programmers are inferior.
Leaving aside when programmers are viewed as inferior, be aware that the word "just" means "only" (this and nothing else) as well as "merely" (no better than). Saying "are you just a programmer?" does not necessarily imply that programmers are inferior to something else, and likely wasn't intended to mean that. It questions whether or not you are something else in addition to being a programmer. So, while I think there's a perfectly good answer to your question, I don't think your question is fully justified by your example :-)
I would say that pretty much always, if you want to move from role X to role Y you could reasonably be asked "are you just an X?", as a (rather blunt) challenge to prove that you're fit for Y. So a hedge fund manager who talks about writing triple-A game titles might well be asked "aren't you a hedge fund manager?" and vice-versa.
In the company where I work, a small programming department supports a large group of analysts. Programming is secondary to what the company really does, which is not to produce software. As such, in the context of a discussion about analysis, I am "just" a programmer. I'm not completely unfamiliar with the analysis, but I'm naturally not as familiar with it as the analysts are, and I could not simply step in and start doing their jobs. If that made me feel "second-rate" then I'd have a problem. What's really happening though is just a consequence of specialization. A hammer and a screwdriver are equally useful tools, unless you have a nail to knock in.
Finally, consider that the career path of "someone who makes software" might leave the word "programmer" behind as you get more senior. Words like "developer" or "engineer" or "software architect" all suggest that literally programming, that is to say typing code into a computer, is not all that's required to make software. In that context, again, someone who is "just a programmer" is by implication not a senior software engineer, and so might be considered directly junior to one. That senior software engineer would be "more than just a programmer" as far as that company's hierarchy is concerned.