Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

So, I've finally gotten myself to a point where I'm comfortable enough with Python (using Pyramid as my framework of choice) to undertake a rather large personal project. As it's a personal project, I have the luxury of taking my time having absolutely no deadlines other than self-imposed ones.

I love learning new frameworks, languages, etc. so if given a reason, I don't mind pushing back on development for a month or so while learning a new language and framework (takes longer when you're doing it on your own time ;)).

I recently learned about CPython's GIL (Global Interpreter Lock), which raised my eyebrow a bit. If I understand it correctly, this means that if I have a Queue in my web app and have threads that complete jobs in the queue, then the code is locked until the thread for that request is complete, meaning that any subsequent request is locked while the previous thread is executing.

Question:

Has anyone in "real world" applications found this to be a problem? Is it worthwhile to learn a language that supports real concurrency out of the box, such as Erlang? I'm most interested in any benchmarks that anyone has done in real world apps and whether or not anyone has seen any real noticeable issues with the GIL.

share|improve this question

2 Answers 2

up vote 4 down vote accepted

I have run up against the GIL in server side programming in almost every instance where I need something to scale to millions of concurrent users on multiple core machines.

Python is great for command line tools and things that don't need true concurrency to extract every last bit of performance from a given piece of hardware.

But for things that really need to squeeze everything out of something like a Sun T2000, you don't want to write anything in Python, it will be a operational maintenance nightmare running 32 separate processes and trying to management them all.

I abandoned Twisted in favor of Erlang a few years ago, Python just doesn't cut it in the large scale concurrency space. The transparent distributed nature of Erlang means it scales horizontally as well as vertically.

share|improve this answer

The lock doesn't prevent you switching between threads. It prevents multiple threads from running at the same time. This only has an effect for multiple cores. Python's threads do not take advantage of multiple cores. Really, that is the only difference.

To reiterate, the code is not locked while the request is executed. The requests will happen side-by-side as the computer switches between threads. This will function exactly the same as any other language running a single-core machine. Only if you have multiple cores will you see a difference.

This is probably not a concern for a web application. Your web application will spend most of its time waiting for the network connection. The amount of time spent doing CPU heavy work is so small you won't notice. The bottleneck will still be at the network so the fact that you couldn't take advantage of multiple cores was not really a big deal.

If you really do need to make use of multiple cores python has a multiprocessing module. This allows you to control seperate processes which will allow you to execute using multiple cores. It works somewhat differently then a threading module, but I find it easier to work with.

My own experience is that I created a little library that uses python, network communication, and multiple processes to distribute python work over a cluster of computers. It works great. I'm able to easily distribute work (as long as it can be split up easily) across a collection of computers. So basically, nothing in python prevents working with this sort of thing.

share|improve this answer
2  
Actually performance actually gets significantly worse running multiple threads on a multicore machine, because of the way the GIL is implemented. Check out this video for details. It's kinda scary once you see what's really going on under the hood. –  Mason Wheeler Sep 21 '11 at 0:16
    
@MasonWheeler, yes you are right. I forget about that because I use eventlet which doesn't use real threads and doesn't hit that problem. (It still can't use multiple cores) –  Winston Ewert Sep 21 '11 at 0:47

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.