Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

The global interpreter lock (GIL) seems to be often cited as a major reason why threading and the like is a touch tricky in Python - which raises the question "Why was that done in the first place?"

Being Not A Programmer, I've got no clue why that might be - what was the logic behind putting in the GIL?

share|improve this question
4  
The Wikipedia article states that "the GIL can be a significant barrier to parallelism—a price paid for having the dynamism of the language", and goes on to say that "Reasons for employing such a lock include: increased speed of single-threaded programs (no necessity to acquire or release locks on all data structures separately), and easy integration of C libraries that usually are not thread-safe." –  Robert Harvey Feb 13 '13 at 1:17
    
@RobertHarvey, Dynamism has nothing to do with it. The problem is mutation. –  dan_waterworth Jun 13 '13 at 12:00

3 Answers 3

up vote 29 down vote accepted

There are several implementations of Python, for example, CPython, IronPython, RPython, etc.

Some of them have a GIL, some don't. For example, CPython has the GIL:

From http://en.wikipedia.org/wiki/Global_Interpreter_Lock

Applications written in programming languages with a GIL can be designed to use separate processes to achieve full parallelism, as each process has its own interpreter and in turn has its own GIL.

Benefits of the GIL

  • Increased speed of single-threaded programs.
  • Easy integration of C libraries that usually are not thread-safe.

Why Python (CPython and others) uses the GIL

In CPython, the global interpreter lock, or GIL, is a mutex that prevents multiple native threads from executing Python bytecodes at once. This lock is necessary mainly because CPython's memory management is not thread-safe.

The GIL is controversial because it prevents multithreaded CPython programs from taking full advantage of multiprocessor systems in certain situations. Note that potentially blocking or long-running operations, such as I/O, image processing, and NumPy number crunching, happen outside the GIL. Therefore it is only in multithreaded programs that spend a lot of time inside the GIL, interpreting CPython bytecode, that the GIL becomes a bottleneck.

Python has a GIL as opposed to fine-grained locking for several reasons:

  • It is faster in the single-threaded case.

  • It is faster in the multi-threaded case for i/o bound programs.

  • It is faster in the multi-threaded case for cpu-bound programs that do their compute-intensive work in C libraries.

  • It makes C extensions easier to write: there will be no switch of Python threads except where you allow it to happen (i.e. between the Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros).

  • It makes wrapping C libraries easier. You don't have to worry about thread-safety. If the library is not thread-safe, you simply keep the GIL locked while you call it.

The GIL can be released by C extensions. Python's standard library releases the GIL around each blocking i/o call. Thus the GIL has no consequence for performance of i/o bound servers. You can thus create networking servers in Python using processes (fork), threads or asynchronous i/o, and the GIL will not get in your way.

Numerical libraries in C or Fortran can similarly be called with the GIL released. While your C extension is waiting for an FFT to complete, the interpreter will be executing other Python threads. A GIL is thus easier and faster than fine-grained locking in this case as well. This constitutes the bulk of numerical work. The NumPy extension releases the GIL whenever possible.

Threads are usually a bad way to write most server programs. If the load is low, forking is easier. If the load is high, asynchronous i/o and event-driven programming (e.g. using Python's Twisted framework) is better. The only excuse for using threads is the lack of os.fork on Windows.

The GIL is a problem if, and only if, you are doing CPU-intensive work in pure Python. Here you can get cleaner design using processes and message-passing (e.g. mpi4py). There is also a 'processing' module in Python cheese shop, that gives processes the same interface as threads (i.e. replace threading.Thread with processing.Process).

Threads can be used to maintain responsiveness of a GUI regardless of the GIL. If the GIL impairs your performance (cf. the discussion above), you can let your thread spawn a process and wait for it to finish.

share|improve this answer
7  
Sounds like sour grapes to me. Python can't do threads properly, so you make up reasons why threads are unnecessary or even bad. "If the load is low, forking is easier", seriously? And the GIL is "faster" for all those cases only if you insist on using referencce-counting GC. –  Michael Borgwardt Feb 13 '13 at 9:38
1  
s/RPython/PyPy/g. @MichaelBorgwardt Giving reasons pro GIL is kind of the point of the question, isn't it? Though I would agree that some of the contents of this answer (namely discussion of alternatives) is beside the point. And for better or for worse, refcounting is now almost impossible to get rid of -- it is deeply ingrained in the entire API and code base; it's almost impossible to get rid of it without rewriting half the code and breaking all external code. –  delnan Feb 13 '13 at 13:35
2  
Don't forget the multiprocessing library - standard since 2.6. It's worker pools are a super-slick abstraction for some simple types of parallelism. –  Sean McSomething Feb 13 '13 at 18:52

First off: Python doesn't have a GIL. Python is a programming language. A programming language is a set of abstract mathematical rules and restrictions. There is nothing in the Python Language Specification which says that there must be a GIL.

There are many different implementations of Python. Some have a GIL, some don't.

One simple explanation for having a GIL is that writing concurrent code is hard. By placing a giant lock around your code, you force it to always run serially. Problem solved!

In CPython, in particular, one important goal is to make it easy to extend the interpreter with plugins written in C. Again, writing concurrent code is hard, so by guaranteeing that there will be no concurrency, it makes it easier to write extensions for the interpreter. Plus, many of those extensions are just thin wrappers around existing libraries which may not have been written with concurrency in mind.

share|improve this answer

What is the purpose of a GIL?

The CAPI documentation has this to say on the subject:

The Python interpreter is not fully thread-safe. In order to support multi-threaded Python programs, there’s a global lock, called the global interpreter lock or GIL, that must be held by the current thread before it can safely access Python objects. Without the lock, even the simplest operations could cause problems in a multi-threaded program: for example, when two threads simultaneously increment the reference count of the same object, the reference count could end up being incremented only once instead of twice.

In other words, the GIL is prevents corruption of state. Python programs should never produce a segmentation fault, because only memory safe operations are permitted. The GIL extends this assurance to multi-threaded programs.

What are the alternatives?

If the purpose of the GIL is to protect state from corruption, then one obvious alternative is lock at a much finer grain; perhaps at a per object level. The problem with this is that although it has been demonstrated to increase the performance of multi-threaded programs, it has more overhead and single-threaded programs suffer as a result.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.