I'm currently setting up a similar system (on a professional level) and this is the design I've chosen:
- Two Nginx loadbalancers (both active, both failover for the other, balanced with DNS round robin)
- Two MySQL Databases in master master replication mode
- Two Tomcat instances as a tomcat cluster
- Two Memcached instances for both caching and session state sharing for the Tomcat cluster
This will achieve a redundant, high availability, scalable solution.
The loadbalancers (on decent hardware) will easily loadbalance a saturated 1gbit line each. This is also a great place for SSL offloading.
You can save your session info in memcached. In case a tomcat instance fails, another tomcat instance can retrieve relevant session information and the clients won't notice a thing. Don't forget to combine this with sticky sessions too. (To keep network traffic down)
Tomcat clustering also has an option to share session information among the cluster in real time, without using memcached. Although I think performance wise, using Memcached will be better.
If you need more power in any of these applications:
- Nginx: Add more loadbalancers, although I don't think this will be the bottleneck very soon.
- Tomcat: you can easily increase the size of the Tomcat cluster or add more clusters
- Mysql: Add some read-only slaves or increase the cluster size (depending on your application, but since you wrote a REST-based application, this should not be a problem)
- Memcached: Add more nodes, Memcached scales pretty well I believe.
I don't know how your application is build and what the big resources hogs are, but if you see a high database load (during your loadtests!), adding a cache between the application and database could certainly improve performance a lot. But don't forget that not everything is cachable, if your queries are always different, caching won't help (much)
My advice would be to download VMware Workbench (or similair virtualization software) and try to create a simple setup. No loadbalancing or clustering, just the basics and work from there. One by one add more features (balancing, caching, clustering, etc.) and make sure to do some research on each topic, so you'll know you made the right pick.
If you keep running the same performance tests during this process, you can see for yourself if using X is better than using Y in your setup, or what impact caching will have, etc.
In the end, a setup like this really depends on the requirements of your application and its clients, everything can be done in various ways, each with it's own strengths and weaknesses.
Any more questions?