Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a company intranet that utilizes a similar set of data across many different pages and many different methods.

Speaking in generalities, is it closer to "best practice" to retrieve a large dataset and filter it for each specific page or method or retrieve many smaller specific sets of data to use? I cannot really test both approaches, as I would like to pick a design approach from the beginning and use it throughout. This is for a redesign. Thanks.

share|improve this question

The ideal is to define a single, unified, structure for your data. Then to put the data into a database with appropriate indexes so that each specific page or method can efficiently retrieve exactly that subset of records that it needs.

There are lots of reasons why people do not follow this ideal in practice. But it is what you should do if you can.

share|improve this answer
Agreed. There are a million variables, but this is the default approach. SQL databases are good at this kind of thing and you shouldn't try outsmart it unless it's a proven bottleneck. – jiggy May 14 '13 at 2:44

I think this requires more context to answer with a yes or no, you'll probably get more/better answers if you provide more detail.

With that said, I tend to go down the route of retrieving only the data that's required, when its required. Since you're talking about an intranet it seems reasonable to assume that there will be no problems with fetching data (as you're db server will be on the same network as your intranet server), which means there's not much reason to fetch it all at once.

Keeping unneeded data out of the way saves resources (memory on the system requesting it, disk IO on the system serving it) and if you pull all the data at the start of a session, it may well be out of date in five minutes time when The user is still browsing.

In most cases, pulling it all at once is the lazy way. You would probably know if there was some reason which made it more efficient.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.