Short Version: What is the term/concept which describes the efficient design of programming the data layer for efficient data access and what are some sources/books/articles/website which discuss this concept?
Long Version: So I had a recent conversation with the senior developer at my small company about how and why we access data the way we do. He told me that it was the single most important aspect of the program and explained how it is the largest bottleneck in efficiency if done incorrectly. He also said he had never learned about it from the courses he took and he never read about it in any books, despite the fact that he thinks it's the first thing he chooses to teach others.
I should explain that this is my first year at a job programming, that I have a degree in Electrical Engineering, but not in CS, so I wouldn't know if it was covered in school. Also, the language I work in is Java, but this question was meant to ask about any language's access to data in a general sense.
So what am I talking about? I'll explain through an example. We work with a lot of data, so the data layer is a bit complicated. It uses a combination of singleton pattern and factory design pattern to achieve different access.
For example, an Object, let's say Customer would be most often instantiated as follows:
Customer customer = Customer.getDefaultInstance(CustomerAccessType.LIST)
Customer customer = Customer.getDefaultInstance(CustomerAccessType.ONE_RECORD)
Customer customer = Customer.getDefaultInstance(CustomerAccessType.LIST_BY_ACCOUNT)
As you may have already guessed, each access type accesses the database in a different manner. The LIST access type most of the time has very few columns, so even though it's grabbing many rows of data, in some cases up to tens of thousands of records (maybe not with customers, but maybe something like orders from a website you can imagine that becoming very large), it might only contain the primary keys on the table and some other very specific information. But when you edit specific data you access using type "ONE_RECORD" and with only a single row you have all the columns of the table (and most probably joins to a lot of other tables).
This is more apparent than ever when you're looking at a screen that is painting information in front of tellers and the information is changing constantly. The information shown may be from a table that has as previously mentioned, tens of thousands of records currently being worked on. A LIST type object will grab only the primary keys from the table, then only the records that require painting are accessed, one row at a time, and the information is painted to the screen.
So I have two questions as a result of all this:
What is the term/concept which describes the efficient design of programming the data layer for efficient data access? Most large programs that have to deal with a lot of data access must have to deal with these same issues. Surely others have coined a term for it?
What are some good resources to learn more about what else is out there on this subject? Again, surely others have dealt with these same issues/bottlenecks we are dealing with. Surely other brilliant minds have tackled the problem and written extensively on it, no?
What are other ways that perhaps your personal experience has shown you in dealing with this problem, even if you've never read it anywhere else?
Your help and knowledge is much appreciated,