Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I'm kind of horrified that people actually suggest using UTF-8 inside a std::string and are OK with UTF-16 std::wstring.

Do you actually do this? Do you think that it is OK?

share|improve this question
1  
Perhaps you could share your thought process, before this gets closed as flamebait? –  Robert Harvey Oct 29 '10 at 15:22
add comment

3 Answers

up vote 6 down vote accepted

My convention is:

  • std::string = UTF-8. For general use.
  • Utf16String = UTF-16. Used mainly for Windows system calls.
    • std::wstring in Windows
    • std::basic_string<uint16_t> in Linux
  • Utf32String = UTF-32.
    • std::basic_string<uint32_t> in Windows
    • std::wstring in Linux.
  • std::vector<unsigned char> = non-string binary data

You just have to get used to thinking of strings as being an array of code points instead of an array of characters. With this view, all string methods work correctly except for find_first_of/find_last_of/find_first_not_of/find_last_not_of.

share|improve this answer
    
I don't disagree about thinking of strings as arrays of code points, but in many respects that's not practical in a number of applications (think, for example, a password verification routine that must impose restrictions like "at least 8 characters", or a "display name" verification routine that must impose restrictions like "at most 256 characters"). What we need is to bite the proverbial bullet and agree on one damn encoding, instead of having 739 different ones, each with subtle and highly esoteric differences. –  Nik Bougalis Nov 17 '12 at 4:40
1  
@NikBougalis A display routine would then better not count combining characters, unless at the beginning of the string. Yes, we really should agree on one encoding. (My favorite one would be UTF-8, but I'd also happily accept UCS-4, a.k.a. UTF-32. UTF-16 mixes the worst of both and is not really an option.) But that would not solve issues like “what does the user perceive as a single character.” –  Christopher Creutzig Dec 30 '13 at 16:36
1  
@ChristopherCreutzig Absolutely. Diacritics, combining marks, non-space space characters and so on. It's unclear what “single character” means when dealing with these kinds of encodings. –  Nik Bougalis Dec 30 '13 at 19:51
add comment

Well, let's say that many programmers still don't know that UTF-16 is a multibyte encoding (they still think that 16-bit wide char is enough to represent all the Unicode characters, but actually they are stuck with the old UCS-2).

However, there's no real drawback in using a wstring for storing UCS-16 text, but you should know that its length doesn't necessarily represent the number of text symbols that will be printed.

share|improve this answer
    
Well, if you are ready to throw pretty much all methods out of the window, shouldn't you use vector instead of base_string? Having a string implies, that methods actually work. –  Let_Me_Be Oct 29 '10 at 17:27
add comment

Even if you store UCS-4 in a string, you still can't guarantee a 1:1 relationship between an item stored in the string and (at least a typical user will view as) a single character when it's displayed or printed (if the string happens to contain any combining character sequences).

Now, it's true that with UCS-4 a single item in the string maps to a single code-point even if that's not necessarily a single character.You can avoid multiple string elements mapping to a single code point. You can't (entirely) avoid multiple code points mapping to a single "character" (and canonicalization to even get close is non-trivial). The problem is that "code point" is an almost entirely artificial construct. Getting excited about/condemning one while ignoring the other doesn't (IMO) make a whole lot of sense.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.