I'm trying to design a web app with ability to scale but can't wrap my heads around few concepts. I want to design it right but im not a experienced programmer, i have more of a system engineering background.
Basic architecture look like this:
Web Server -> File processing server -> NoSQL DB -> Search server
So the main scenario is follows:
- User uploads a file via site
- File is send for processing to a server(python script)
- Results of processing is send to NoSQL DB
- Results processed by search server and returned to user
We can scale web frontends via load balancing. Something like nginx+apache.
Database scaling is taking care by Cassandra or MongoDB. Search scaling is taking care by elasticsearch or sphinx clustering.
Now I want to be able to add multiple file processing servers in case file uploaded is too big. So I need to somehow split file into chunks and process it simultaneously on multiple nodes plus if node goes down while working it shouldn't affect anything and data must be saved. So I need something else which will be allocating tasks to my file processing servers, balancing load and control execution of tasks.
How to design custom applications for that kind of things? Should I use message queuing?