I'm in a similar situation and here's what I've found:
- Any job will limit your availability/energy for side projects and extra-curricular learning.
- No job can stop you from doing what you love.
There are also things that you can and will get from working on an enterprise product.
- Teamwork is usually a big deal on enterprise product teams
- Tools are often good
- Your soft skills become important
- Processes are often well laid out
Some of these things will help as a developer or are directly translatable to a game development position.
I think the key is that if you really want to get into the game dev field you have to keep at it. Practice in your own time to build experience in game industry tools/technologies and whenever possible, apply to positions in the game industry that you would like to do and are capable.
In my experience, the only difficult is that I enjoy my job and the people. I have good tools, good coworkers, time and resources to learn, decent pay and get to write code. That satisfies much of my desires. At a previous job, where I hated pretty much everything about the place, I coded at home all the time because the itch burned like a raging fire. Now, I get to scratch it almost enough to placate the urges, and that's sometimes sad because I'm only learning things that are related to my work - not my outside interests.