Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I got this error in my program which grab data from different website and write them to a file:

'charmap' codec can't encode characters in position 151618-151624: character maps to <undefined>

I am not familiar with all the encoding decoding thing and I have been okay with what python 2 did. Although the python officially said that they made the change in order to make things better, it seems to get worse.

I have no idea how to fix these errors. However I am a pro-active person so I would really like to know what is causing the problem and how to solve it. I have check the official site but the words are hard to understand.

Could I have a simple elaboration on that? Another page is also acceptable.

EDIT: I've check this page, the Unicode HOWTO in Python 2.7. My understanding is that we must translate the unicode string into binary format while we're writing it to file and it require an encoding. Obviously 'utf-8' is the best one, but why it didn't force the python interpreter to use 'utf-8'? Instead, it use some strange codec such as 'charmap' and 'cp950' and 'latin-1'.

share|improve this question
add comment

1 Answer

See the Python wiki on the subject. You are trying to encode a unicode string (the default string type in Python 3) with an encoding that doesn't support some of the characters in your string.

>>> '\u0411'.encode("iso-8859-15")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/encodings/iso8859_15.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_table)
UnicodeEncodeError: 'charmap' codec can't encode character '\u0411' in position 0: character maps to <undefined>
>>> '\u0411\u0411\u0411\u0411\u0411'.encode("iso-8859-15")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/encodings/iso8859_15.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_table)
UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-4: character maps to <undefined>

As you can see, some encodings use a charactermap internally, and detect encoding errors for multiple characters at once.

You'll have to narrow it down to the exact code points (characters in position 151618-151624) and the encoding used.

share|improve this answer
    
@MartjinPieters Why python 3 doesn't force the default encoding to 'utf-8' but let the system choose the encoding? –  lamwaiman1988 Jul 26 '12 at 16:13
2  
Py2 had ASCII as the default encoding for both source code and unicode conversions. On Py3 the UTF-8 default only applies to source code, and the default encoding for file*names* is set by the "LANG" environment parameter. See docs.python.org/py3k/howto/unicode.html –  Martijn Pieters Jul 26 '12 at 16:20
    
Not on Windows, where filenames are encoded in UTF-16 and can contain characters not supported by the locale's "ANSI code page". Makes it pretty much impossible to use the C standard library if you care about supporting non-ASCII characters. –  dan04 Aug 26 '12 at 5:50
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.