Java isn't fast enough for most game development. It's far slower than using C++/Assembly, which is the standard. It's the same reason more game development isn't done using C# or VB. Game developers need and plan every last clock cycle that they can get their hands on for things like physics calculations, AI logic, and environment interactions.
For simpler games, Java could be used quite effectively. If you want to create a Tetris clone or Bejeweled, or something else of that level of detail, then Java would work fine. But Java can't possibly create games like Halo, Medal of Honor, Command & Conquer, and so forth and make it playable. At least as it exists nowadays.
And the reasons you list in your question are valid as well. Except, I think, for the lack of game developers with Java expertise. Many games on phones and other portable devices are written in Java (including most Android games), and some of the games are quite excellent. So I think there is a decent, and growing, base of game developers with Java knowledge.
The thought is changing on the ability to use these higher level languages for some of the more advanced games. For instance, one of my favorite games, Auran's Train Simulator, is written with large portions in C#, and it works quite well. So the base is growing and will continue to evolve.