Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I'm working on an cross platform C++ project, which doesn't consider unicode, and need change to support unicode.

There is following two choices, and I need to decide which one to choose.

  • Using UTF-8 (std::string) which will make it easy to support posix system.
  • Using UTF-32 (std::wstring) which will make it easy to call windows API.

So for item #1 UTF8, the benefit is code change will not too many. But the concern is some basic rule will broken for UTF8, for example,

  • string.size() will not equal the character length.
  • search an '/' in path will be hard to implement (I'm not 100% sure).

So any more experience? And which one I should choose?

share|improve this question
3  
wstring is a basic_string<wchar_t>. wchar_t is implementation-defined in size and definitely not enough bits for UTF-32 on Windows. –  Lars Viklund Apr 17 at 13:05

1 Answer 1

up vote 12 down vote accepted

Use UTF-8. string.size() won't equal the amount of code points, but that is mostly a useless metric anyway. In almost all cases, you should either worry about the number of user-perceived characters/glyphs (and for that, UTF-32 fails just as badly), or about the number of bytes of storage used (for this, UTF-32 is offers no advantage and uses more bytes to boot).

Searching for an ASCII character, such as /, will actually be easier than with other encodings, because you can simply use any byte/ASCII based search routine (even old C strstr if you have 0 terminators). UTF-8 is designed such that all ASCII characters use the same byte representation in UTF-8, and no non-ASCII character shares any byte with any ASCII character.

The Windows API uses UTF-16, and UTF-16 doesn't offer string.size() == code_point_count either. It also shares all downsides of UTF-32, more or less. Also, making the application handle Unicode probably won't be as simple as making all strings UTF-8; good Unicode support can require some tricky logic changes like normalizing text, handling silly code points well (this can become a security issue for some applications), making string manipulations such as slicing and iteration work with glyphs or code points instead of bytes, etc.

There are more reasons to use UTF-8 (and reasons not to use UTF-{16,32}) than I can reasonably describe here. Please refer to the UTF-8 manifesto if you need more convincing.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.