My company has a suite of mature (roots go back 15+ years), computationally intensive applications (simulations) which we use to do consulting work. I work on the simulation/consulting side of the house, not the software development side. I recently learned about a particular feature of the architecture which concerns me: our large (200-500MB, sometimes more) set of input data are first completely loaded into memory from the database. Then, the simulation engine access the input data from memory as it goes.
The struck me as "pre-mature optimization" and/or an old-fashioned idea, as I believe there are database servers out there that can be quite fast. The architecture causes another problem, which is our simulation database can only be so big (a little over a GB) before a sim cannot run at all. The chief architect is a scientist, not a software developer, and somewhat hard-headed at that, so I am seeking comments on this question from the community.
Is our architecture hurting us? Are there any papers/work/test on the subject-- tradeoffs between database reading as-you-go and a database copy in memory? Any comments or insight would be appreciated. I would like to get schooled up before I take the issue up with my company.