Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I'm building an API and want to count hits for each user. It's a HTTP API implemented in Python. I could keep the count in a database (using PostGreSQL) but it'll be a very busy API, so I don't want the overhead in the DB. I'm looking at Redis or just writing plain text logs. But is there a better approach?

It's a map tile service, so bursts of hundreds of hits per sec. I want to report usage per month to the user—not keep total count of hits.

share|improve this question
3  
Read the count from the DB at start. Maintain the count in memory, flushing to the DB periodically and at shutdown. –  Steven Burnap Mar 18 '13 at 16:24
1  
I'm a developer on a system that keeps counts in the database for a similar purpose and I highly advise against this. As your table grows locking will become and issue and it'll have a serious impact on your API's performance. I'd stick the requests in Redis or some other non-relational data store and run a background job that calculates new counts periodically. Are you counting hits for statistical purposes or does each client get a set number of requests? –  binarycleric Mar 18 '13 at 16:41
    
Yes, I want to impose a monthly limit. But also for statistical purposes. –  mhoegh Mar 18 '13 at 16:53
    
I already running ElasticSearch on my server, so maybe I can use that instead of Redis? –  mhoegh Mar 18 '13 at 16:56
3  
Might just be me, but if this API and infrastructure is so fragile that a single small write will break the camel's back then you might want to think about re-engineering. Also, look at message busses as that is what you need here. Sometimes they even use hip and cool technologies like redis. –  Wyatt Barnett Mar 18 '13 at 18:23

1 Answer 1

up vote 1 down vote accepted

If you are writing logs anyway, I would advice using a background job to periodically extract the counts from the logs and update the counts in your DB. Consider using a logrotate postrotate script to extract data in order to avoid problems due to rotating logs.
In that scenario any DB is ok, so stick with whatever DB you are already using.

If, however, you do not write logs then I would go with @Shinary's proposal (see comments) to write the hits to redis (or similar) and periodically update from there. As you need this functionality in the context of a web-app, I would advice implementing it as a middleware (and depending on (a) the store you have chosen and (b) the severity of single hits being lost - using asynchronous writes/sends to avoid slowing down the main app.)

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.