Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I'm trying to design a web app with ability to scale but can't wrap my heads around few concepts. I want to design it right but im not a experienced programmer, i have more of a system engineering background.

Basic architecture look like this:

Web Server -> File processing server -> NoSQL DB -> Search server

So the main scenario is follows:

  • User uploads a file via site
  • File is send for processing to a server(python script)
  • Results of processing is send to NoSQL DB
  • Results processed by search server and returned to user

We can scale web frontends via load balancing. Something like nginx+apache.
Database scaling is taking care by Cassandra or MongoDB. Search scaling is taking care by elasticsearch or sphinx clustering.

Now I want to be able to add multiple file processing servers in case file uploaded is too big. So I need to somehow split file into chunks and process it simultaneously on multiple nodes plus if node goes down while working it shouldn't affect anything and data must be saved. So I need something else which will be allocating tasks to my file processing servers, balancing load and control execution of tasks.

How to design custom applications for that kind of things? Should I use message queuing?

share|improve this question
2  
This type of problem is often solved with MapReduce programming, using tools such as Hadoop. –  dan1111 Mar 27 at 17:24
    
I know about hadoop, but i've heard it too slow for those kind of things. Anyway i want to start small and without hadoop initially so i more interested in design of custom applications –  Coolface Mar 28 at 4:04

2 Answers 2

You will have to use ajax (or some async callback or a handler) to hit the server only when you have the whole file, that way you won't need to split the file into chunks. Just tell the user that his file are going to be processed and send him a message when its finished.

From there, you can have async worker method (on the server) per uploaded file to send it to the python script for processing.

share|improve this answer

I'm not sure I quite follow what you're doing (like what does "file is sent for 'processing'" mean -- processing?). Maybe that's not all that important for the context.

I guess my first instinct would be to ask why your "processing" can't happen on your app server (as part of the upload process). Then, whatever scaling solution you use for your app server, your "processing" scales with it.

However, I also get the hint that perhaps you're trying to design a massively scalable solution, with independently tunable subsystems, so you probably aren't wanting to link your app server to your "processing" event. In that case, there are a few different ways to go.

I've used message queuing (as you suggested) for this type of problem before. Then, you just fire up additional listeners as your load increases. You have to be careful about what triggering mechanism you use though, and you have to deal with transaction processing appropriately (syncpoints / rollback and commits), otherwise queuing can become a mess to debug. It seems trivial, but in my experience, it's not. When everything's working, it's great, but you really have to test the heck out of the cloudy-day scenarios to make sure you don't get dropped messages or messages processed more than once. "Guaranteed delivery" isn't all it's cracked up to be.

I also recently solved a problem like this using a shared memory data grid (Hazelcast). The API actually implements a queue for you, so you don't have all the overhead of setting up a queue manager as you would with the solution above. In my case I was needing to run a transformation process against hundreds of billions of rows of data from a table, and by writing a multithreaded app that spun up table readers at different points in the table, I threw the records into the data grid, and I wrote a separate process that I could instantiate as many times as necessary to transform the data -- so it scaled as far as I needed it to. I suppose it's just a variation on the solution above. I took a process that took about 3 days to run in the original approach, to about 15 minutes, but it was spread across a 50 node cluster for those 15 minutes.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.