Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Hi all I was wondering what limitations will we have if Unicode had decided to assign one and only one codepoint to every user-perceived character?

Currently, Unicode has code-points that correspond to combining characters. Such characters combine with a previous code-point (or sequence thereof) to present to the user what appears to be a single character.

From what I can see, the current standard is full of complexities. Even if I try to avoid any kind of complexities by using an encoding like UTF-32, this problem still persists. It's not an encoding issue at all.

For example, in Unicode when a grapheme cluster is represented internally by a character sequence consisting of base character + accent, then when the user clicks the > button to skip to the next user-perceived character, we had to skip from the start of the base character to the end of the last character of the cluster.

Why does it need to be so hard? Why can't we assign a single code point to every user perceived character such going to the user-perceived character is simply a matter of reading the next code point?

The unicode website seems to acknowledge this complexity, (although I could not understand what exactly is so rare about having to figure out the number of user-perceived character counts in a string):

In those relatively rare circumstances where programmers need to supply end users with user-perceived character counts, the counts should correspond to the number of segments delimited by grapheme clusters. Grapheme clusters may also be used in searching and matching

Diacritics are also the reason why things don't work as expected. For example, if I throw 2 ピ characters (japanese katakana PI using the unicode representation U+30d2 U+309a) into a String builder and reverse it, I would naturally expect the output to be 2 ピ characters (i.e. ピピ), but it gives an invalid output of ゚ピヒ !

If Unicode had assigned individual code points for each user-perceived character and scrapped the idea of grapheme clusters, this wouldn't have happened at all.

What I would like to know is, what exactly is wrong with representing every user-perceived character as one unicode codepoint? Is it likely that doing so would would exceed U+10FFFF possible code points (if it does exceed U+10FFFF code points I see no reason why they couldn't have set the limit to 2^32 in the first place), Even when there is so much spare space to include the whole family of Elf Languages ?

Unicode states:

If you wanted, for example, to give each “instance of a character on paper throughout history” its own code, you might need trillions or quadrillions of such codes;

But seriously that doesn't make sense at all. Where is the source for making such a claim? A trillion codes is quite an over-statement.

share|improve this question
    
Welcome to the real world. Who knows the exact reasons why... perhaps it is true that some developers/designers over-complicate things(subconsciously even...?) just to give themselves more work. Small example - why does Windows use the "\" while Unix does "/" in directories? Don't get me started on SQL variants... I dunno, anyway you seem to be quite expert here already, I hope someone slightly more expert comes around! –  Adel Dec 8 '11 at 23:38
3  
@Adel On why Windows uses backslash: blogs.msdn.com/b/larryosterman/archive/2005/06/24/432386.aspx –  Yannis Rizos Dec 8 '11 at 23:45
add comment

3 Answers

If a script has M letters and N diacritics, then a precomposed representation requires up to M*N code points, while a decomposed representation requires only M+N. Combining characters prevent Unicode from "running out of code points".

share|improve this answer
    
I don't think running out of code points was the problem. Only the BMP is full. The SMP is packing up but not full. Another 3/4 are very sparsely populated. Which leaves about 10 planes empty (each plane can have 65K code points). –  Loki Astari Dec 9 '11 at 1:14
1  
@Loki: Remember that Unicode originally planned to only have one "plane". –  dan04 Dec 9 '11 at 2:43
    
In fact the standard could have taken 32 bits for each code point, in such a way that they will never "run out of code points" (2^32 possible points) –  Pacerier Dec 9 '11 at 7:39
    
@Pacerier You really are missing the point. You cannot possibly have one code point per extended grapheme cluster, because the number of the latter easily exceeds 32 bits, let alone 21 of them. It just isn’t possible. You need to learn to iterating by grapheme clusters just as you need to learn how to iterate by words. Both are higher-level concepts of any open-ended problem space that is subject to infinite variation. –  tchrist May 13 '12 at 14:24
    
@tchrist Iterating graphemes is starkly different than iterating words. In a language, there's no fixed number of possible word combinations, but there's certainly a fixed number of possible characters. As an example, Unicode allows us to merge e with with with ˆ with ˝. But the resulting character is unused in any language. It is unlikely that there are 2^32 (4 billion) standalone characters, because there's no such thing as language characters being subject to infinite variation. –  Pacerier May 13 '12 at 15:41
show 6 more comments

it's because back in the olden days a unit of data was 8 bits, the widespread 32-bit architecture is younger than the old ASCII standard on which the UTF variants are based (for backwards compatibility)

besides programmers on those old (sizeof(int)== 16 bits) architectures never liked using more resources than needed (memory in particular remember that UTF32 takes 4 times as much memory than UTF8 for text only in the latin alphabet) as back in the day you only got a few hundred kilobytes at your disposal at best

that and you don't need any other characters than those in ASCII if you are only programming for the english speaking (or any language with no accents)

one big lesson you can here learn is: anything that grows towards changing requirements is rarely future proof against the next change in requirements and generally only good enough for the current requirements and if someone thinks of a solution that exceeds the requirements it will likely get shot down for various reasons (most common will be performance, memory use and compatibility)

also allow me to point you towards this xkcd comic that might shed some light on the situation

share|improve this answer
    
Yes but at times some standards are so horrible to live with it. Unicode has way way too many flaws. No one gets it right. Not even Microsoft nor Google. We should disband it for a simpler standard that actually has a chance of getting things right. –  Pacerier Dec 8 '11 at 23:49
add comment

The origins of Unicode date back to 1987–1988, when Joseph D. Becker published the first Unicode draft proposal. A 16-bit fixed width encoding (UCS-2) was an indispensable part of his design. He was very proud of the distinction he made between characters and glyphs, just to squeeze everything to this small range. Unfortunately his naïve assumptions were proven wrong, or at least unrealistic, when in 1996 Unicode code-space was increased beyond the 16-bit limit and UCS-2 was extended to UTF-16. Currently Unicode has almost twice as much allocated codepoints than the code-space of Becker's original design.

[...] if it does exceed U+10FFFF code points I see no reason why they couldn't have set the limit to 2^32 in the first place [...]

It is currently limited by U+10FFFF because it's the maximum range possible to be encoded with UTF-16. UTF-16 is an ugly hack applied late in the day to extend the code-space. It's not extensible. UTF-8 on the other hand is. UTF-8 was originally allowed to encode 231 codepoints and was later limited to just match the same range as UTF-16. In fact you can extend the scheme used by UTF-8 indefinitely, even to encode 2128 codepoints and preserve all the nice properties of UTF-8 at the same time (lexicographically comparable, unambiguous substring matching with UTF-8 unaware code, etc...) except that you won't be able to determine the length of the codepoint from the first byte.

From what I can see, the current standard is full of complexities. Even if I try to avoid any kind of complexities by using an encoding like UTF-32, this problem still persists. It's not an encoding issue at all.

Exactly! I want to point out that variable length encodings are not the problem. Indeed, finding the next code-point in a UTF-8 string is simple, just scan for the next byte of the form 0xxxxxxx or 11xxxxxx. So we could easily increase the available codespace without increasing the average memory consumption. The real problem is that to find the next grapheme you must have a Unicode database available (in some form), which is definitely complex and costly for many simple applications.

share|improve this answer
    
So in fact if we scraped the idea of UTF-16 and extend codepoint to 2^32, we could actually have enough space to incorporate all natural languages and a few art language ? Btw what's the solution to that real problem? Or what solutions can we use to attempt to solve that problem? –  Pacerier Dec 31 '11 at 20:27
    
@Pacerier: one grapheme cluster may have arbitrary number of combining character, so no fixed-width codepoint per grapheme may do. What you may do though is 1) use a variable length encoding that encodes arbitrary length codepoints so that each represents a grapheme cluster or 2) leave the multiple codepoints per grapheme cluster used today but change the assignment of codepoints so you can determine the properties of codepoints by some simple rules (e.g. test some bit to determine if it's a base character). –  ybungalobill Dec 31 '11 at 20:40
    
Also note that Unicode had more constraints. It took into consideration making conversion from existing encodings to Unicode simpler. –  ybungalobill Dec 31 '11 at 20:45
    
The problem I have with unicode is that they are trying to do too many things. End up, they do things half-assedly. For example, unicode actually wants other people to tailor and invent algorithms for determining user-perceived-characters when that exactly is what they should have been doing. I've certainly not heard of anyone who actually did any tailoring for Unicode (unless its Apple/Google/MS etc). This is crazy, if all they did was KISSS and assigned one code point to every user-perceived character (which now I believe is possible having over a million –  Pacerier Feb 1 '12 at 14:41
    
code points stackoverflow.com/q/9047318/632951), now we wouldn't be talking about allographs, diagraphs, graphemes, ligatures, grapheme clusters, glyphs etc etc. Yes the encoding issue would still persist, but compared to this Unicode mess of having code points all over the place and used for multiple purposes, encoding issues are nonexistent. Seriously Unicode Is Flawed, we –  Pacerier Feb 1 '12 at 14:41
show 5 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.