Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I thought Unicode was designed to get around the whole issue of having lots of different encoding due to a small address space (8 bits) in most of the prior attempts (ASCII, etc.).

Why then are there so many Unicode encodings? Even multiple versions of the (essentially) same one, like UTF-8, UTF-16, etc.

share|improve this question
10  
UTF-8 is not the same as UTF-16. The list will grow as soon as we encounter other solar systems with earth-like planets. –  Joset May 20 '11 at 5:31
    
@Joset: We already have Klingon. We have most earth languages on the BMP with slight spillage into plains 1,2. If current theries are correct and there are only 42 sentient species in the galaxy that reach a point where they can use space travel (thus allow first contact) we should be able to squeeze all the characters in all languages into UNICODE (assuming we can expand from 21 to 22 bits to allow 64 plains). That even leaves 10 bits of buffer space if we want to include the primitive species that have not achieved space flight. –  Loki Astari May 20 '11 at 6:18
7  
@Kevin Hsu: UTF-7,8,16LE,16BE,32LE,32BE. So, at least 6 real encodings exist. UTF-9 and UTF-18 are April Fools. –  MSalters May 20 '11 at 9:49
8  
The good thing about standards is that there's so many of them –  konrad May 20 '11 at 12:13
2  
@Dean: If you plug in the best guess values into the THE DRAKE EQUATION you can get 42 (of course with minor modifications you can move this value) but it coincides so nicely with Douglas Adams –  Loki Astari May 20 '11 at 14:03

9 Answers 9

up vote 26 down vote accepted

Because people don't want to spend 21 bits on each character. On all modern systems, this would essentially mean using three bytes per character, which is three times more than what people were used to, so they were unwilling to adopt Unicode at all. Compromises had to be found: e.g. UTF-8 is great for English text because legacy ASCII files need not be converted at all, but it is less useful for European languages, and of little use for Asian languages.

So basically, yes, we could have defined a single universal encoding as well as a single universal character chart, but the market wouldn't have accepted it.

share|improve this answer
7  
+1 Great answer. To be truly honest it's the only one that really answers this question. All other answers are (more or less) about how bytes are laid out in all different unicode encodings. –  Jacek Prucia May 20 '11 at 12:02
    
Historically it's a simple matter of disagreement. However, I don't see much use for anything but UTF-8 today, while there are theoretical scenarios where UTF-16 would consume less space, it's not by a big margin, and they are rare. The most prominent place where you would want to save space is for websites, but they are full of HTML codes which are by far shortest using UTF-8. You could for instance use Shift JIS to make a Japanese website smaller than the UTF-8 equivalent, but that only works because it's a charset specifically for Japanese. –  eBusiness May 20 '11 at 14:46
2  
Not really true either. As the compressed formats are really only used for transport and storage. Within an application it is more usually to use UCS-2 or UCS-4 as these are fixed width but these take up 2 or 4 bytes per character. So applications are willing to give up the space for the ease of use. –  Loki Astari May 22 '11 at 18:45

Unicode is a 21 bit character encoding the uniquely describes "CodePoints" each code points being represented by a a glyph (a graphical representation).

  • 16 bits used to identify a code point in a plane (most code points are on plane 0).
  • 5 bits to identify the plane.

The encodings supported are:

  • UTF-8 (to encode each point using 8 bit values)
  • UTF-16 (to encode each point using 16 bit values)
  • UTF-32 (to encode each point using 32 bit values)

But no matter what the encoding when you decode they all map back to a specific codepoint that has the same meaning (which is why it is cool).

UTF-8

This is a variable sized format. Where each codepoint is represented by 1 to 4 bytes.

UTF-16

This is a variable sized format. The code points on the "Basic Multilingual plane" (BMP or Plane 0) can be represented by 1 single 16 bit value. Code points on other planes are represented by a surrogate pair (2 16 bit values).

UTF-32

This is a fixed size format. All code points are represented by a single 32 bit value.

share|improve this answer
2  
I like this answer, too. Was writing one similar, but this one is clear. I'd also add that UTF-8 is also useful in that ASCII strings are automatically UTF-8. –  Kevin Hsu May 20 '11 at 7:38
3  
Please, it's the Basic Multilingual Plane, not a plain. –  JSBձոգչ May 20 '11 at 12:31
3  
This is a good answer, but I think that it still begs the question, "Why?", although this answer implicitly touches on that. To elaborate: UTF-32 is a more direct (some would say easier) approach of encoding Unicode characters, but it also wastes a lot of space, as each and every character takes up 4 bytes. UTF-8 is much more compact and backwards-compatible with ASCII, but it's not regular: a character can take anywhere from 1 to 4 bytes to encode, which makes it harder to work with. UTF-16 is a sort of hybrid approach between the two, mostly with the pros and cons of each. –  mipadi May 20 '11 at 21:39
4  
There's a tradeoff between memory usage (where UTF-8 is best, since the most common characters are single-byte) and processing speed (where UTF-32 is best, because all characters are the same size, allowing for certain optimizations and giving perfect 32-bit alignment in memory). As a result, network protocols and file formats commonly use UTF-8 (to save bandwidth / storage space), while script interpreters and language runtimes may prefer UTF-16 or UTF-32. –  tdammers May 21 '11 at 0:06
1  
@Marcel: A "CodePoint" is a "CodePoint" not a character (as a character may be constructed from multiple "CodePoints"). Don't get the two terms confused. But you are correct "CodePoints" do not refer to glyphs. A Glyph is just a graphical representation of a code point. A subtle but important difference. –  Loki Astari Feb 15 '13 at 17:04

I think it's useful to separate the 2 ideas:

  1. Unicode - mapping of characters from all over the world to code points.
  2. Encoding - mapping of code points to bit patterns (UTF-8, UTF-16, etc).

UTF-8, UTF-16, and other encodings has each own advantages and disadvantages. Better consult Wikipedia about it.

share|improve this answer
    
@jfs: Why have Unicode at all though if there's still going to be a dozen or more different encoding which are all different on the wire anyway? What use does having a global mapping have in and of itself? –  Matthew Scharley May 20 '11 at 5:54
9  
@Matthew Scharley: You are looking at it wrong. UNICODE maps all characters from all languages (including Klingon) to a UNIQUE ID (codepoint). The encodings are merely a way of compressing the codepoints onto disc or a stream across a network. UTF stands for "UNICODE Transport format". You should always think of a UNICODE codepoint as a 21 bit value. The advantage over other formats is that all characters are uniquely identified and do not overlap (Unlike Latin-1, Latin-2 etc). –  Loki Astari May 20 '11 at 6:09
    
@Matthew Scharley Why have a global mapping? Actually everyone had his own mapping at the past (remember code pages?). I think a silly example will clear things out. Imagine the idea of love. How will you represent it to someone? Give flowers? Say "I love you"? Everyone has his own way of expressing it. Love (which is an abstract idea) is like the code points. Expressing it is like the encodings. :) –  jfs May 20 '11 at 7:12
4  
Unicode is the global alphabet. UTF-x is the way it's transported by computers, since it's difficult to shove paper through wires. –  Mel May 20 '11 at 11:50
1  
@Martin, Klingon actually didn't make it. Nor did the Tengwar or Cirith, used for writing Tolkein's elven tongues. –  TRiG May 23 '11 at 11:53

UTF-7, UTF-8, UTF-16 and UTF-32 are simply algorithmic transformation formats of the same coding (codepoints) of characters. They are encodings of one system of codification of characters.

They are also algorithmically easier to navigate forward and backward than most previous schemes for dealing with character sets larger than 256 characters.

This is very different than the generally country- and sometimes vendor-specific codification of glyphs. In Japanese alone, there were a ton of variations of JIS alone, not to mention EUC-JP and the codepage-oriented transformation of JIS that DOS/Windows machines used called Shift-JIS. (To some extent, there were algorithmic transformations of these, but they were not particularly simple and there were vendor-specific differences in characters that were available. Multiply this by a couple hundred countries and the gradual evolution of more sophisticated font systems (post greenscreen era), and you had a real nightmare.

Why would you need these transformation forms of Unicode? Because a lot of legacy systems assumed sequences of ASCII-range 7 bit characters, so you needed a 7-bit clean solution safely passing data uncorrupted through those systems, so then you needed UTF-7. Then there were more modern systems that could deal with 8-bit character sets, but nulls generally had special meanings to them, so UTF-16 didn't work for them. 2 bytes could encode the entire basic multilingual plane of Unicode in its first incarnation, so UCS-2 seemed like a reasonable approach for systems that were going to be "Unicode aware from the ground up" (like Windows NT and the Java VM); then the extensions beyond that necessitated additional characters, which resulted in the algorithmic transformation of the 21 bits worth of encodings that were reserved by the Unicode standard, and surrogate pairs were born; that necessitated UTF-16. If you had some application where consistency of character width was more important than efficiency of storage, UTF-32 (once called UCS-4) was an option.

UTF-16 is the only thing that's remotely complex to deal with, and that's easily mitigated by the small range of characters that are affected by this transformation and the fact that the lead 16-bit sequences are neatly in a totally distinct range from the trailing 16-bit sequences. It's also worlds easier than trying to move forward and backward in many early East Asian encodings, where you either needed a state machine (JIS and EUC) to deal with the escape sequences, or potentially move back several characters until you found something that was guaranteed to only be a lead byte (Shift-JIS). UTF-16 had some advantages on systems that could chug through 16-bit sequences efficiently, as well.

Unless you had to live through the dozens (hundreds, really) of different encodings out there, or had to build systems that supported multiple languages in different encodings sometimes even in the same document (like WorldScript in the older MacOs versions), you might think of the unicode transformation formats as unnecessary complexity. But it's a dramatic reduction in complexity over the earlier alternatives, and each format solves a real technical constraint. They are also really efficiently convertible between each other, requiring no complex lookup tables.

share|improve this answer
1  
The various JIS and EUC state machines are really nasty, and doubly so if you're working with transforming between them. Unicode hugely simplifies that. The only major problem with Unicode is that you've got to stop thinking of bytes as characters, you ASCII-using small-character-setted chauvinist you! –  Donal Fellows May 20 '11 at 12:20

Unicode was not designed to get around the whole issue of having lots of different encodings.

Unicode was designed to get around the whole issue of one number representing many different things depending on the code page in use. Numbers 0 - 127 represent the same characters in any Ansi code page. This is what is also known as the ASCII chart or character set. In Ansi code pages, which allow for 256 characters, numbers 128 - 255 represent different characters in different code pages.

For example

  • Number $57 represents a capital W in all code pages, but
  • Number $EC represents the inifinity symbol in code page 437 (US), but a "LATIN SMALL LETTER N WITH CEDILLA" in code page 775 (Baltic)
  • The Cent Sign is number $9B in code page 437, but number 96 in code page 775

What Unicode did, was turn this all upside down. In Unicode there is no "reuse". Each number represents a single unique character. Number $00A2 in Unicode is the cent sign and the cent sign appears nowhere else in the Unicode definition.

Why then are there so many Unicode encodings? Even multiple versions of the (essentially) same one, like UTF-8, UTF-16, etc.

There are no multiple versions of the same encoding. There are multiple encodings of the same Unicode character definition map and these have been "invented" to administer to storage requirements for different usages of the various lingual planes that exist in Unicode.

Unicode defines (or has the space to define) 4.294.967.295 unique characters. If you want to map these to disk/memory storage without doing any algorithmic conversions, you need 4 bytes per character. If you need to store texts with characters from all lingual planes, then UTF-32 (which is basically a straight 1 character - 4 byte storage encoding of the unicode definition) is probably what you need.

But hardly any texts use characters from all lingual planes. And then using 4 bytes per character seems a big waste. Especially when you take into account that most languages on earth are defined within what is known as the Basic Multi-lingual Plane (BMP): the first 65536 numbers of the Unicode definition.

And thats where UTF-16 came in. If you only use characters from the BMP, UTF-16 will store that very efficiently using only two bytes per character. It will only use more bytes for characters outside of the BMP. The distinction between UTF-16LE (Little Endian) and UTF-16BE (Big Endian) really only has something to do with how numbers are represented within computer memory (byte pattern A0 meaning hex $A0 or meaning $0A).

If your text use even fewer different characters, like most texts in Western European languages, you will want to restrict the storage requirements for your texts even more. Hence UTF-8, which uses a single byte to store the characters present in the ASCII chart (the first 128 numbers) and a selection from the Ansi characters (the second 128 numbers of the various code pages). It will only use more bytes for characters outside of this "most used characters" set.

So to recap:

  • Unicode is a mapping of the characters in all of the languages on earth (and some Klingon to boot) and then some (mathematical, musical, etc..) to a unique number.
  • Encodings are algorithms defined to store texts using the numbers of this unique character map as space efficiently as possible given the "average usage" of characters within texts.
share|improve this answer
2  
"Numbers 0 - 127 represent the same characters in any code page." - well, unless you're talking EBCDIC, in which case $57 is not a W –  MSalters May 20 '11 at 9:51
    
@MSalters: you are absolutely right. EBCDIC is different (and there are others EBCDIC). I guess that my mainframe days are so long behind me that I didn't remember, or I have repressed these memories too hard and too long... :-) –  Marjan Venema May 20 '11 at 11:49
    
"Numbers 0 - 127 represent the same characters in any code page." There are actually encodings, such as BinarySignWriting, which are not supersets of ASCII. BinarySignWriting, in fact, does not include any ASCII characters at all. –  TRiG May 20 '11 at 12:08
    
@TRiG: That's why I edited my statement to be specifically about Ansi code pages. Must have done that before you refreshed... –  Marjan Venema May 20 '11 at 12:13
    
Yes. There was an extra comment and a post update made while I was writing my comment. Still, BinarySignWriting is interesting. –  TRiG May 20 '11 at 12:18

The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)

http://www.joelonsoftware.com/articles/Unicode.html

share|improve this answer

Unicode defines the map between numbers and characters. However, when you send a number to a receiver, you still need to define how to represent that number. That's what UTF is for. It defines how to represent a number in a byte stream.

share|improve this answer

You know that a zip-file can compress a file to be much smaller (especially text) and then uncompress it to an identical copy of the original file.

The zipping algorithm actually have several different algorithms with different characteristics to choose from: stored (no compression), Shrunk, Reduced (methods 1-4), Imploded, Tokenizing, Deflated, Deflate64, BZIP2, LZMA (EFS), WavPack, PPMd, where it theoretically could try all of them and choose the best result but usually just go with Deflated.

UTF works much the same way. There are several encoding algorithms each with different characteristics, but you usually just pick UTF-8 because it is widely supported as opposed to the other UTF-variants, which in turn is because it is bitwise compatible to 7-bit ASCII making it easy to use on most modern computer platforms which usually use an 8-bit extension of ASCII.

share|improve this answer
    
ørn: The difference with a zip file is that there's a header that tells you what compression is in effect. With text files, we still need to guess don't we? –  Matthew Scharley May 21 '11 at 1:19
    
There is a special sequence that tells exactly that. Due to backward compatibility with ASCII it is optional. –  user1249 May 21 '11 at 7:38

The rationale behind UTF-32 is simple: It's the most straightforward representation of Unicode code points. So why isn't everything in UTF-32? Two main reasons:

One is size. UTF-32 requires 4 bytes for every character. For text that uses only characters in the Basic Multilingual Place, this is twice as much space as UTF-16. For English text, it's 4 times as much space as US-ASCII.

The bigger reason is backwards compatibility. Each Unicode encoding other than the "unencoded" UTF-32 was designed for backwards compatibility with a prior standard.

  • UTF-8: Backwards compatibility with US-ASCII.
  • UTF-16: Backwards compatibility with UCS-2 (16-bit Unicode before it was expanded beyond the BMP).
  • UTF-7: Backwards compatibility with non-8-bit-clean mail servers.
  • GB18030: Backwards compatibility with the GB2312 and GBK encodings for Chinese.
  • UTF-EBCDIC: Backwards compatibility with the Basic Latin subset of EBCDIC.

I thought Unicode was designed to get around the whole issue of having lots of different encoding

It was, and it did. It's much easier to convert between UTF-8, -16, and -32 than deal with the old system of hundreds of different character encodings for different languages and different OSes.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.