4

I'm kind of horrified that people actually suggest using UTF-8 inside a std::string and are OK with UTF-16 std::wstring.

Do you actually do this? Do you think that it is OK?

1
  • 2
    Perhaps you could share your thought process, before this gets closed as flamebait? Oct 29, 2010 at 15:22

3 Answers 3

8

My convention is:

  • std::string = UTF-8. For general use.
  • Utf16String = UTF-16. Used mainly for Windows system calls.
    • std::wstring in Windows
    • std::basic_string<uint16_t> in Linux
  • Utf32String = UTF-32.
    • std::basic_string<uint32_t> in Windows
    • std::wstring in Linux.
  • std::vector<unsigned char> = non-string binary data

You just have to get used to thinking of strings as being an array of code points instead of an array of characters. With this view, all string methods work correctly except for find_first_of/find_last_of/find_first_not_of/find_last_not_of.

3
  • I don't disagree about thinking of strings as arrays of code points, but in many respects that's not practical in a number of applications (think, for example, a password verification routine that must impose restrictions like "at least 8 characters", or a "display name" verification routine that must impose restrictions like "at most 256 characters"). What we need is to bite the proverbial bullet and agree on one damn encoding, instead of having 739 different ones, each with subtle and highly esoteric differences. Nov 17, 2012 at 4:40
  • 1
    @NikBougalis A display routine would then better not count combining characters, unless at the beginning of the string. Yes, we really should agree on one encoding. (My favorite one would be UTF-8, but I'd also happily accept UCS-4, a.k.a. UTF-32. UTF-16 mixes the worst of both and is not really an option.) But that would not solve issues like “what does the user perceive as a single character.” Dec 30, 2013 at 16:36
  • 1
    @ChristopherCreutzig Absolutely. Diacritics, combining marks, non-space space characters and so on. It's unclear what “single character” means when dealing with these kinds of encodings. Dec 30, 2013 at 19:51
2

Well, let's say that many programmers still don't know that UTF-16 is a multibyte encoding (they still think that 16-bit wide char is enough to represent all the Unicode characters, but actually they are stuck with the old UCS-2).

However, there's no real drawback in using a wstring for storing UCS-16 text, but you should know that its length doesn't necessarily represent the number of text symbols that will be printed.

1
  • Well, if you are ready to throw pretty much all methods out of the window, shouldn't you use vector instead of base_string? Having a string implies, that methods actually work. Oct 29, 2010 at 17:27
2

Even if you store UCS-4 in a string, you still can't guarantee a 1:1 relationship between an item stored in the string and (at least a typical user will view as) a single character when it's displayed or printed (if the string happens to contain any combining character sequences).

Now, it's true that with UCS-4 a single item in the string maps to a single code-point even if that's not necessarily a single character.You can avoid multiple string elements mapping to a single code point. You can't (entirely) avoid multiple code points mapping to a single "character" (and canonicalization to even get close is non-trivial). The problem is that "code point" is an almost entirely artificial construct. Getting excited about/condemning one while ignoring the other doesn't (IMO) make a whole lot of sense.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.