Tell me more ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.
up vote 7 down vote favorite
2

Notice how view counts of youtube videos are always laggy? For example, a video has like 1000 comments and still has 500 hits, and will have 10000 hits hours after.

Youtube isn't alone in this. Most message boards are implemented that way and view counts are updated like every 10 minutes or so.

Does anyone know the reason behind this?

Thanks.

share|improve this question

4 Answers

up vote 17 down vote accepted

Recording views is very simple, simply add a row to a table that represent the "view" action. This is fast because no locking is required in the database, you're just adding a row onto the end of a heap.

Aggregating that into the total number of views requires something like doing SELECT COUNT(*) FROM ... which means you have to lock the table while the calculation is progressing. Alternatively, UPDATE ... SET num_views = num_views + 1 also requires that you lock that particular row every time someone views it.

So from a scalability point of view, it's much more efficient to add a row each time someone views the video and then do the SELECT COUNT(*) FROM ... every ten minutes or so.

Note I don't actually know the architecture of YouTube, or whether they even use a relational database to store their data, but whatever they do use, the principle is likely the same: inserting data is cheap, aggregating values is (relatively) expensive.

share|improve this answer
3  
Doesn't it use BigTable with the rest of Google? – TheLQ Apr 8 '11 at 17:48
@Dean Harding Thanks, but doesn't it mean the table would have billions, if not trillions, of records for a web site even with moderate traffic, much less youtube? With such massive records, I suspect that SELECT COUNT(*) would have a performance impact on the DB even if it runs only every 10 minutes. This would also require more disk space for the database and backup. I'm not saying locking the table on every page hit is any better, but I just find it hard to understand how big web sites would handle such huge data. – Tom Tucker Apr 8 '11 at 17:52
It's not the first time I hear this. What really puzzles me, is that incrementing a counter in a threadsafe manner is harder or more expensive, than appending to a list. If you can solve the latter, the former should be really easy. – back2dos Apr 8 '11 at 17:55
1  
@Tom Tucker: yes, but we're talking about Google here, remember :-) One way that I've solved this problem on a smaller scale is that once I've finished the aggregation, I would truncate the table that the aggregated data was calculated from. So you never get more than an hour (or whatever you update interval is) of "raw" data. – Dean Harding Apr 8 '11 at 18:12
3  
Also keep in mind that the data in your "actions" table can be used for more than just calculating the "number of views". You can also use it to implement IP blocks (i.e. "no more than 1 comment every 10 seconds from the same IP" etc). You could also generate graphs showing the number of views over time, and other kinds of things that a simple num_views = num_views + 1 doesn't allow. – Dean Harding Apr 8 '11 at 18:14
show 3 more comments
up vote 7 down vote

Most likely the value has been cached somewhere along the way so you are seeing stale data. Because it is not critical for this data to be accurate the developers have decided to favour performance over getting up to date data. You really wouldn't want to go to the database and do a row count for every hit on the site just to update this figure so they don't, they just cache it for a while.

share|improve this answer
up vote 4 down vote

In order for large sites to scale, they have to perform caching at several stages. That can be page caching, sub-page caching, and/or record caching. You might have a combination of all of them in effect. For example, if the youtube page is cached until a new comment is added, you'll see some lag until someone posts a comment.

There are several ways of measuring page views:

  • Store it in the database as a record: easy to insert, however it is a major maintenance overhead for records that are only providing a count.
  • Store it in the database as a record and roll up the counts periodically: easy to insert, batch processing to gather the stats you want, and cleans up after itself.
  • Update a count column in the database: expensive to update (assuming row locking), no maintenance overhead, negative performance when dealing with multiple people requesting the same page at the same time.
  • Process the access log file when it rolls over: no extra data in the database, all processing is done in batches off-line, and the summary stats you want are updated when it is time.

Out of the items above, all except one option suggests that the updates will be done in batches. The number of views is not really a time critical attribute, so this is OK. However, keeping people waiting to view a video on YouTube because the backend database can't keep up is a time critical action. That means that updating a column in the database isn't going to work for a site as large as YouTube. I personally wouldn't be surprised if they opted for the final option. The web servers will be recording a whole host of information for every visit including what IP you are using, how you were referred to the page, etc. It only makes sense to process those in batches and summarize the results as necessary.

share|improve this answer
Never thought of the last solution - very clever! That alone is worth +1. – Tom Tucker Apr 8 '11 at 18:29
1  
We used that approach to handle the rolling "most popular" page lists for the day/week/month. We rolled the counts up to a simple properties file for days, weeks, and months. The current day would get reprocessed every hour, and the remaining summary files were treated like the grandfather/father/son backup tapes. Essentially we needed no more than 8 summary files (weekly summaries, and a summary file for each day of the current week). – Berin Loritsch Apr 8 '11 at 18:47
That's kind of similar to how RRDTool works, although RRDTool is much more complex than your solution with its elegant simplicity. – Jörg W Mittag Apr 8 '11 at 20:02
up vote 0 down vote

This could be due to a number of reasons. It all boils down to the algorithms used by each respective website. Unless someone here is actually a YouTube developer I doubt your going to get an exact answer here.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.