Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I'm looking to make some performance enhancements to our site, but I'm not sure exactly where to begin. We have some custom object caching, but I think that we can do better.

Our Business

We aggregate news stories on a news type of web site. We get approximately 500-1000 new stories per week. We have index pages that show various lists of the items and details pages that show the individual stories.

Our Current Use case: Getting an Individual Story

  1. User makes a request
  2. The Data Access Layer(DAL) checks to see if the item is in cache and if item is fresh (15 minutes).
  3. If the item is not in cache or is not fresh, retrieve the item from SQL Server, save to cache and return to user.

Problems with this approach

  • The pull nature of caching means that users have to pay the waiting cost every time that the cache is refreshed. Once a story is published, it changes infrequently and I think that we should replace the pull model with something better.

My initial thoughts

  • My initial thought is that stories should ALL be stored locally in some type of dictionary. (Cache or is there another, better way?). If the story is not found, then make a trip to the database, update the local dictionary and send the item back.
  • Since there may be occasional updates to stories, this should be an entirely process from the user.
  • I watched a video by Brent Ozar, How StackOverflow Scales SQL Server, in which Brent states "the fastest database query is the one that you don't make".

Where do I start?

At this point, I don't know exactly what the solution is. Is it caching? Is there a better way of using local storage? Do I use a Dictionary, OrderedDictionary, List ? It seems daunting and I'm just looking for some good starting points to learn more about how to do this type of optimization.

share|improve this question
    
"We get approximately 500-1000 new stories per week.": looks like you can keep everything entirely in cache even on a machine with not too much RAM. Am I wrong? If yes, can you explain more? –  MainMa Jan 26 '12 at 16:31
    
Certainly, keeping as much in memory is desirable. A week's worth should be a breeze. How much to put into memory beyond that is a question. Also, I'm looking for a starting place for how to do that in code. –  John Jan 26 '12 at 16:44
    
"How much to put into memory beyond that is a question.": again, why not putting everything in memory, permanently? If data doesn't change very often and changes do not depend on users' interaction, you can cache everything forever at application startup (i.e. when web server starts), and update it through a scheduled task. –  MainMa Jan 26 '12 at 16:59
    
1000 stories / week x 52 weeks x 10 years of data = 520,000 stories. I don't know how much memory that would eat up. –  John Jan 26 '12 at 17:14
add comment

1 Answer

If you were to choose a caching system:

Did you check the following article available at MSDN: http://msdn.microsoft.com/en-us/library/ee817645.aspx ?

It is outdated ("retired" as they say) but still a good read, particularly the second chapter for you since you did not yet made up your mind on the technique/technology you want to use. The following techniques are mentioned:

  • Using the ASP.NET Cache
  • Using Remoting Singleton Caching
  • Using Memory-Mapped Files
  • Using Microsoft SQL Server 2000 or MSDE for Caching
  • Using Static Variables for Caching
  • Using ASP.NET Session State
  • Using ASP.NET Client Side Caching and State
  • Using Internet Explorer Caching

Also, I've never used it, but I heard a lot of good stuff about Memcached, there is an interesting answer to the Memcached with Windows and .NET question that explains how it might work along a .NET environment.

However, I don't know how long is a story, how many bits it would take, but 520000 in ten years does not seem that much. Maybe Memcached is overkill in that case, I'm not sure...

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.