Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I have a 10K LOC project written in Django with quite a deal of Celery (RabbitMQ) for asynchronicity and background jobs where needed, and have come to the conclusion that parts of the system would benefit from being rewritten in something other than Django for better concurrency. Reasons include:

  • Signals handling and mutable objects. Especially when one signal triggers another, handling them in Django using the ORM can be surprising when instances change or disappear. I want to use some messaging approach where the data passed along doesn't change in a handler (Clojure's copy-on-write approach seems nice, if I got it right).
  • Parts of the system are not web-based, and need better support for performing tasks concurrently. For example, the system reads NFC tags, and when one is read an LED goes on for a few seconds (Celery task), a sound is played (other Celery task), and the database is queried (other task). This is implemented as a Django management command, but Django and its ORM being synchronous by nature and sharing memory is limiting (we are thinking of adding more NFC readers, and I don't think that the Django+Celery approach will work any longer, I'd like to see better message-passing capabilities).

What are the pros and cons of using something like Twisted or Tornado compared with going for a language such as Erlang or Clojure? I am interested in practical benefits and detriments.

How did you come to the conclusion that some parts of the system would fare better in another language? Are you suffering performance problems? How severe are those problems? If it can be faster, is it essential that it is faster?

Example 1: Django at work outside an HTTP request:

  1. An NFC tag is read.
  2. The database (and possibly LDAP) is queried, and we want to do something when data becomes available (red or green light, play a sound). This blocks using the Django ORM, but as long as there are Celery workers available it doesn't matter. May be a problem with more stations.

Example 2: “message-passing” using Django signals:

  1. A post_delete event is handled, other objects may be altered or deleted because of this.
  2. At the end, notifications should be sent to users. Here, it would be nice if arguments passed to the notification handler were copies of deleted or to-be-deleted objects and guaranteed not to change in the handler. (It could be done manually simply by not passing objects managed by the ORM to handlers, of course.)
share|improve this question

migrated from stackoverflow.com Aug 20 '11 at 14:58

This question came from our site for professional and enthusiast programmers.

    
I think better answers will happen if you explain more about why you've come to the conclusion –  Winston Ewert Aug 20 '11 at 15:07
5  
Before anyone says that language choice questions are off-topic, I'll chime in saying I think this one is fine since it's a practical problem with specific requirements. I hope it draws some detailed comparisons. –  Anna Lear Aug 20 '11 at 15:09
    
Twisted is the opposite of concurrent! It is a single threaded server that is event driven, it won't get you anywhere if you need true concurrency. –  Jarrod Roberson Aug 22 '11 at 1:45

4 Answers 4

Opening Thoughts

How did you come to the conclusion that some parts of the system would fare better in another language? Are you suffering performance problems? How severe are those problems? If it can be faster, is it essential that it is faster?

Single-Thread Asynchrony

There are several questions and other web resources that already deal with the differences, pros, and cons of single-thread asynchrony vs. multi-thread concurrency. It’s interesting to read about how Node.js's single-thread asynchronous model performs when I/O is the major bottleneck, and there are many many requests being serviced at once.

Twisted, Tornado, and other asynchronous models make excellent use of a single-thread. Since a lot of web programming has lots of I/O (network, database, etc.), the time spent waiting on remote calls adds up significantly. That is time that could be spent doing other things—like kicking off other database calls, rendering pages, and generating data. The utilisation of that single-thread is extremely high.

One of the greatest benefits of single-thread asynchrony is that it uses much less memory. In multi-thread execution, each thread requires a certain amount of reserved memory. As the number of threads increases, so does the amount of memory required just for the threads to exist. Since memory is finite, it means there are bounds on the number of threads that can be created at any one time.


Example

In the case of a web server, pretend each request is given its very own thread. Say 1MB of memory is required for each thread, and the web server has 2GB of RAM. This web server would be capable of processing (approximately) 2000 requests at any point in time before there just isn't enough memory to process any more.

If your load is significantly higher than this, requests are going to take a very long time (when waiting for older requests to complete), or you’re going to have to throw more servers into the cluster to expand the number of concurrent requests possible.


Multi-thread Concurrency

Multi-thread concurrency instead relies on executing several tasks at the same time. That means that if a thread is blocked waiting on a database call to return, other requests can be processed at the same time. Thread utilisation is lower, but the number of threads executing is much larger.

Multi-thread code is also much harder to reason about. There are issues with locking, synchronisation, and other fun concurrency problems. Single-thread asynchrony doesn't suffer from the same problems.

Multi-thread code is much more performant for CPU intensive tasks, however. If there exists no opportunities for a thread to “yield”—such as a network call that would normally block—a single-thread model just isn't going to have any concurrency whatsoever.

Both May Coexist

There's of course overlap between the two; they are not mutually exclusive. For instance, multi-thread code can be written in a non-blocking way, to better utilise each thread.


The Bottom Line

There are many other issues to consider, but I like to think about the two like this:

  • If your program is I/O bound, then single-thread asynchrony is probably going to work quite well.
  • If your program is CPU bound, then a multi-thread system will probably be best.

In your particular case, you need to determine what kind of asynchronous work is being completed, and how often those tasks arise.

  • Do they occur on every request? If so, memory is probably going to become an issue as the number of requests rise.
  • Are these tasks ordered? If so, you’re going to have to consider synchronisation if using multiple threads.
  • Are these tasks CPU intensive? If so, is a single-thread able to keep up with the load?

There's no simple answer. You must consider what your use cases are, and design accordingly. Sometimes an asynchronous single-thread model is better. Other times, using a number of threads to achieve massive parallel processing is required.

Other Considerations

There are other issues you need to consider also, rather than just the concurrency model you choose. Do you know Erlang or Clojure? Do you think you'd be capable of writing safe multi-thread code in one of these languages such that you improve the performance of your application? Is it going to take a long time to get up to speed in one of these languages, and will the language you learn benefit you in the future?

How about the difficulties associated with communication between these two systems? Will it be overly complex maintaining two separate systems in parallel? How will the Erlang system receive tasks from Django? How will Erlang communicate those results back to Django? Is performance significant enough a problem that the added complexity is worth it?


Final Thoughts

I've always found Django to be quick enough, and it is used by some very heavily trafficked sites. There are several performance optimisations you can make to increase the number of concurrent requests and response time. Admittedly, I haven’t done anything with Celery thus far, so the usual performance optimisations are probably not going to solve any issues you may be having with these asynchronous tasks.

Of course, there’s always the suggestion of throwing more hardware at the problem. Is the cost of provisioning a new server cheaper than the development and maintenance cost of an entirely new subsystem?

I've asked far too many questions at this point, but that was my intention. The answer is not going to be easy without analysis and further detail. Being able to analyse the problems comes down to knowing the questions to ask, though…so hopefully I've helped on that front.

My gut feeling says that a rewrite in another language is unnecessary. The complexity and cost will probably be too great.


Edit

Response to Follow-Up

Your follow-up presents some very interesting use cases.


1. Django working outside HTTP requests

Your first example involved reading NFC tags, then querying the database. I don’t think that writing this part in another language will be that useful to you, simply because querying the database or an LDAP server is going to be bound by network I/O (and potentially database performance). On the other hand, the number of concurrent requests will be bound by the server itself, since each management command will be run as its own process. There will be setup and teardown time that impacts performance, since you aren't sending messages to an already running process. You will, however, be able to send multiple requests at the same time, since each will be an isolated process.

For this case, I see two avenues you can investigate:

  1. Ensure that your database is capable of handling multiple queries at once with connection pooling. (Oracle, for example, requires that you configure Django accordingly 'OPTIONS': {'threaded':True}.) There may be similar configuration options at the database level or Django level that you can tweak for your own database. No matter what language you write your database queries in, you will have to wait for this data to return before you can light up the LEDs. The performance of the querying code can make a difference though, and the Django ORM isn't lightning fast (but, usually fast enough).
  2. Minimise setup/teardown time. Have a constantly running process, and send messages to it. (Correct me if I’m wrong, but this is what your original question is actually focusing on.) Whether this process is written in Python/Django or another language/framework is covered above. I don’t like the idea of using management commands so frequently. Is it possible to have a small piece of code running constantly, that pushes messages from the NFC readers onto the message queue, which Celery then reads and forwards to Django? The setup and teardown of a small program, even if it’s written in Python (but not Django!), should be better than starting and stopping a Django program (with all of its subsystems).

I’m unsure what web server you’re using for Django. mod_wsgi for Apache lets you configure the number of processes and threads within the processes that service requests. Be sure to tweak the relevant configuration of your web server to optimise the number of serviceable requests.


2. “Message-passing” with Django signals

Your second use case is also fairly interesting; I’m not sure if I have the answers for that. If you’re deleting model instances, and wish to operate on them later, it might be possible to serialize them JSON.dumps and then deserialize JSON.loads. It’ll be impossible to fully recreate the object graph later on (querying related models), since related fields are lazy loaded from the database, and that link will no longer exist.

The other option would be to somehow mark an object for deletion, and only delete it at the end of the request/response cycle (after all signals have been serviced). It might require a custom signal to implement this, rather than relying on post_delete.

share|improve this answer
6  
+1 for the length and good writing of this answer –  eyquem Aug 20 '11 at 16:14
1  
lots of FUD and doubts about locking and other things that are not issues with Erlang, none of the traditional shared state issues you list are considerations with a language and runtime specifically designed to not share state. Erlang can handle tens of thousands of discreet processes in very little ram, memory pressure is not an issue either. –  Jarrod Roberson Aug 23 '11 at 1:28
    
@Jarrod, I personally don't know Erlang so I'll accept what you say in that regard. Otherwise, nearly everything else I mentioned is relevant. Cost, complexity and whether the current tools are being properly utilized or not. –  Josh Smeaton Aug 23 '11 at 7:38
    
Node.js is not single threaded. –  Robert Harvey Mar 28 '13 at 15:54
    
This is the kind of epic answer I really love to read ^^. +1, good job! –  Laurent Bourgault-Roy Mar 28 '13 at 16:46

I did some very sophisticated highly scalable development for a major US ISP. We did some serious tranasaction numbers using a Twisted server, and it was a nightmare of complexity to get Python / Twisted to scale on anything that was CPU bound. I/O bound is not an issue, but CPU bound was impossible. We could put together systems quickly, but getting them to scale to millions of concurrent users was a nightmare of configuration and complexity.

I wrote up a blog post about it, Python/Twisted VS Erlang/OTP.

Spoiler, Erlang won.

share|improve this answer

Practical problems with Twisted (which I love and have used for about five years):

  1. The documentation leaves something to be desired, and the model is quite complex to learn anyway. I find it hard to get other Python programmers to work on Twisted code.
  2. I ended up using blocking file I/O and database access for lack of good blocking APIs. This can really hurt performance.
  3. There does not seem to be a huge community and healthy community using Twisted; for instance Node.js has a lot more active development especially for web back-end programming.
  4. It is still Python, and at least CPython isn't the fastest thing around.

I have done a bit of work using Node.js with CoffeeScript and if concurrent performance is your concern then that may be worth the leap.

Have you considered running multiple instances of Django with some arrangement to spread clients between instances?

share|improve this answer
1  
Python documentation in general leaves something to be desired :/ (Not saying it's that bad, but for a language that popular one would expect for it to be a lot better). –  Rook Aug 20 '11 at 19:32
2  
I find Python documentation and in particular Django documentation to be some of the best docs for any language hands down. Many third-party libraries leave something to be desired though. –  Josh Smeaton Aug 22 '11 at 0:04

I'll suggest the following before you consider switching to another language.

  1. Use LTTng for recording system events like page faults, context switches, and system call waits.
  2. Convert whereever it's taking too much time to use the C library, and use any design pattern you like (multi-threading, signal event based, call back async, or Unix traditional select) which is good for I/O in there.

I would not use threading in Python once the application has priority in performance. I would take the above option, which can solve many problems like software reuse, connectivity with Django, performance, ease of development, etc.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.