Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Question --

I currently have a large monolithic Java application which I am working to break up into services components. One of the components I want to separate out is the service which handles and presents users and profiles.

Currently the core service uses a MySQL backend and is written in Java, and is only hosted on one set of servers (web, app, db). My primary goal is to get the user/profile service to be n+ scalable across multiple web, and app servers and trying to figure out a way to do it across the database layer.

What are some of the best practices to achieve high scalability with this environment? (We currently have a few million users, but expect to grow past that).

Thanks!

share|improve this question
1  
Authentication and user profiles are not typically the most resource-intensive activity; is there a particular reason you've singled this one out, and why you can't just (for example) cache them? –  Aaronaught May 12 '11 at 2:46

1 Answer 1

Why are you seeking to break out service components?

Unless the application is having trouble fitting in memory, you really don't want to do that. Seriously. Communication via RPCs (particularly if you are using XML for serialization) adds a ton of overhead over simple method calls, which costs a lot of performance. Furthermore it adds a bunch of complexity that makes debugging problems much more difficult. I've seen the internals of websites that did this without good reason, and it is amazing how many scalability problems were created with no gain. (I've also seen the internals of a well-known website that did this for very good reason with very good results, and I'm aware of how much constant discipline they need to keep it from devolving into a complete disaster. Very few organizations can manage that.)

Of course you should make sure that your code is modular with well-defined APIs, such that it would be simple to swap out a local or remote implementation of the API. That's just good software design. But don't actually do it until you need to because your application is having trouble fitting in memory..

But what you really need to do instead is make sure that you are horizontally scalable. Meaning that you can run multiple web servers at once. And multiple app servers at once. But it is OK for each to be vertically integrated.

As for the database, there are several common strategies for scaling that when you hit the limits. Use any combination of these that make sense for your application:

  1. Add caching layers (eg memcached) to avoid having to go to the database all of the time.
  2. Add read-only slaves of your database that you can direct some read-only queries to.
  3. Get a bigger database. (You would be amazed at how much you can scale the database. Last I heard, all of PayPal runs off of one huge Oracle database.)
  4. Shard your data. Sharding data means that you can have multiple master databases, because each only has a chunk of your data.
  5. Investigate NoSQL solutions that relax database consistency guarantees in return for easy horizontal scalability. See http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis for a quick comparison between several of those options.
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.