Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I just read through the documentation on the Codecs module, but I guess my knowledge/experience of comp sci doesn't run deep enough yet for me to comprehend it.

It's for dealing with encoding/decoding, especially Unicode, and while to many of you that's a complete and perfect explanation, to me that's really vague. I don't really understand what it means at all. Text is text is text in a .txt file? Or so this amateur thought.

Can anyone explain?

share|improve this question
    
I would suggest starting by reading the Wikipedia article on Unicode. It is an essential foundation to understand the concepts related to character encodings. If you transcend an anglophone's view of text, it suddenly becomes a rather more complicated affair than text is text is text. –  Pedro Romano Oct 12 '12 at 10:23
    
I have read that as well, so I have some understanding. But I do not see what advantage OutFile = codecs.open('example.txt', 'a', 'utf-8') affords over OutFile = open('example.txt'). FYI the code I am trying to understand is a webcrawler writing to a text file, if that context is important. –  Aerovistae Oct 12 '12 at 10:28
7  
@Aerovistae I agree with Pedro Romano that some background is needed - these two are slightly more lighthearted and high-level explanations: Absolute minimum every developer should know about Unicode and There ain't no such thing as plain text. –  Daniel B Oct 12 '12 at 11:25
    
@Aerovistae I read the first sentence and was immediately about to go dig up that article, beat me to it. Anyone who thinks reading blogs is not a good use of an engineer's time is off their nut :) –  Jimmy Hoffa Oct 12 '12 at 15:20

2 Answers 2

up vote 4 down vote accepted

Having understood why Unicode is necessary (recommended reading: link, link :thanks @DanielB for the excellent links) and that character encodings are what computers use to represent real world characters, it becomes clear that when Python is reading or writing bytes representing text from a stream (which can be a file, a pipe, a socket, ...), it needs to know which character encoding is being used so those bytes are meaningful as human readable text.

Python uses ascii as the default encoding. The ascii encoding is inadequate for almost all languages (including English: there's no £ in ascii!): so you need to specify an alternate encoding when writing to streams if you intended to use any characters not part of ascii.

When reading from streams, you need to know which encoding was used to write to the stream and use the same encoding to read from it, otherwise the decoded result will be wrong. Try writing a string with Cyrillic characters with the ISO-8859-5 codec and reading it back with the UTF-8 codec: you'll see they don't match, because different byte sequences mean different characters in the two encodings.

So to answer your specific question,

out_file = open('example.txt', 'w')

is actually opening the file for writing using the ascii codec implicitly. If you want to specify another codec, you need to either use the encoding parameter of the open function in Python 3.x:

out_file = open('example.txt', 'a', 'utf-8')

or if you are still using Python 2.x (the latest is Python 2.7.3 at the time of this writing), you need to use the functions from the codecs module:

out_file = codecs.open('example.txt', 'a', 'utf-8')

since open in Python 2.x doesn't allow you to specify an encoding (you can use it to read the byte stream into a byte string in memory though, and then decode that string).

share|improve this answer

Have a look at Dive Into Python 3 by Mark Pilgrim, Chapter 4. Strings. It is for Python 3 -- the chapter explains what is the difference between the sequence of bytes and abstract strings (text).

The codecs documentation is about the same things. But the explanation in the chapter is more understandable for newcomers.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.