Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I read that some constructs of Python are more efficient because they are compiled in C.

https://wiki.python.org/moin/PythonSpeed/PerformanceTips

Some of the examples used were map() and filter(). I was wondering how Python is able to do this? It's generally interpreted, so how does some of the code get compiled while another is interpreted - and in a different language? Why not just compile the whole thing?

share|improve this question
    
When you write map(something...) in Python, nothing gets compiled to C. It calls code that was already written in C. Imagine that there was a piece of code in the Python interpreter like this: if(!strcmp(functionName, "map")) {/* mapping code goes here */} else if(!strcmp(functionName, "filter")) {/* filter code goes here */} else if ... else {error("Unknown function!");} –  immibis Aug 24 at 2:57
    
You may benefit from reading this: programmers.stackexchange.com/questions/24558/… –  Pharap Aug 24 at 7:15

4 Answers 4

up vote 4 down vote accepted

I read that some constructs of Python are more efficient because they are compiled in C.

That's not true.

First off, there is nothing in the specification of the Python language that requires that certain functions have to be implemented in a certain language. A Python implementor can choose to implement any function and language construct anyway he wants. For example, in Jython, those functions are implemented in Java, not in C. In IronPython, they are implemented in C#. In PyPy, they might be implemented in RPython or just in Python. In Pynie, they are probably implemented in Python.

Seondly, there is nothing in the specification of the C language that says that it must be compiled. There are interpreters for C.

Thirdly, just because it is in C doesn't mean it's fast. There are C compilers out there which produce really terrible code, and there are pretty fast Python implementations.

And fourth, even if the function implemented in C is blazingly fast, that doesn't necessarily translate into faster execution speed of the overall program. Optimizations typically don't work across languages. For example, the optimizer can't inline the call to map into your program, because your program is written in Python but map is written in C. But inlining is pretty much the foundation of all optimizations, because inlining (and loop unrolling) give nice long straight paths of code without branches or calls, which is what optimizers love.

It's generally interpreted,

Actually, all currently existing Python implementations always compile Python code, they never interpret it.

Why not just compile the whole thing?

That's a good question! Writing everything in the same language has many advantages, some of which I outlined above. (Another one is that it's easier to find collaborators who know one language than two.)

If everything is written in the same language, then performance improvements to that language get multiplied throughout the whole system. If everything is written in C, then making the Python 10 times faster isn't going to speed up your program much, because most of the code isn't Python. Sure, the code you wrote is running 10 times faster, but that code mostly consists of calls to C functions which have the same speed as before.

But if everything is written in Python, then making Python faster will have a ripple effect: the primitive types get faster, the datastructures built on top of those primitive types get faster, the algorithms using those datastructures get faster, the modules using those algorithms get faster and so on.

share|improve this answer
    
This gave me a lot of insight about how languages can work, thank you (and all the other responders as well). I knew programming languages were complicated, but there is so much information about them and so many idiosyncrasies to each one. –  Howcan Aug 24 at 0:58
5  
I think "all currently existing Python implementations always compile Python code, they never interpret it" is overstating the case. There is no implementation of the full Python language that compiles entire programs ahead-of-time to machine code, which is what one normally means when one says X is a compiled language. The most commonly used implementations compile ahead-of-time to some sort of byte code, which is then executed by an interpreter. I haven't seen a language that interpreted the source code directly since the days of C64 BASIC. –  Zack Aug 24 at 1:22
    
@Zack Heck, even C64 BASIC didn't interpret the source code directly. –  fluffy Aug 24 at 5:24
    
@Zack: "I haven't seen a language that interpreted the source code directly" – Languages don't interpret source code, interpreters do. A language is just a set of abstract mathematical rules. A language isn't compiled or interpreted. It just is. Compilation and interpretation or traits of the compiler or interpreter (duh!), not the language. Every language can be interpreted. Every language can be compiled. If English were a typed language, the term "compiled language" would be a type error. That term just doesn't make any sense whatsoever. –  Jörg W Mittag Aug 24 at 6:45
1  
@Zack: "Compile" doesn't mean "translate to machine code", it means "translate to a different language" (or actually the same one). –  Jörg W Mittag Aug 24 at 6:48

The interpreter itself is written in C (or at least the reference implementation CPython is). That means every language feature is implemented with a tiny C function. All the interpreter does is read the sourcecode to find out which of these tiny functions to call.

This means it is not a problem to also delegate more complex functions to C implementations.

These functions are not compiled with your program. They already exist in compiled form in the Python interpreter which just calls them when it encounters them in your sourcecode.

share|improve this answer
1  
"That means every language feature is implemented with a tiny C function." Not quite true, there are some stuff in the stdlib (which I consider to be a language feature) that's written in Python and not C. –  Terry Chia Aug 24 at 3:14
    
@TerryChia But what does that Python code in the stdlib boil down to in the end? The most basic language elements which are then implemented in C. When you dig down, you will always end up in a C implementation. –  Philipp Aug 24 at 10:35

Saying that they are "compiled in C" is a bit confusing statement. Let me rephrase it: CPython (the most widespread implementation of the Python interpreter, to which you are referring) is a C program, and those are C functions built in the interpreter, not Python code that the interpreter magically chooses to compile to C.

When you feed a .py file to Python, it parses it and converts each function into bytecode, i.e. a compact binary form of the operations that the function does . This is fed to the Python virtual machine, a piece of the interpreter which reads the bytecode and does what it tells it to do.

When you call functions written in Python nothing special happens: the VM will just jump to interpret their bytecode; but when you call a function that is built in inside the interpreter, the interpreter calls the corresponding C function that it has built in (this happens all the time, for example most operations on primitive types are actually performed by code compiled directly inside the interpreter).

Function written in C in CPython are generally faster, both because they are written in a language that is tailored for high performance, compiled to native code (no need for an interpreter, runs directly on the CPU) and has simpler, less dynamic semantics (for example, in Python each access to a member is a complex hashtable lookup, in C it can boil down to a single assembly instruction).

It's generally interpreted, so how does some of the code get compiled while another is interpreted - and in a different language? Why not just compile the whole thing?

You seem to think that the it's the interpreter to decide what code is compiled and what is interpreted; that's a wrong premise. The compiled parts are code that it's written directly in C by the CPython developers, and is compiled inside the interpreter when the interpreter is built (other code written in C or other compiled languages can be called by Python if it's compiled in Python extensions or via ctypes). All the CPython interpreter can do when you run it is just to interpret Python and call ready-made C functions.

share|improve this answer

It's not "compiled in C" per se: map and filter are built-in functions from the standard library, which is provided by the language implementation (in this case, CPython). Since CPython is written in C, it can easily implement any of its primitive functions in C if needed.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.