Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

At work I come across a lot of Japanese text files in Shift-JIS and other encodings. It causes many mojibake (unreadable character) problems for all computer users. Unicode was intended to solve this sort of problem by defining a single character set for all languages, and the UTF-8 serialization is recommended for use on the Internet. So why doesn't everybody switch from Japanese-specific encodings to UTF-8? What issues with or disadvantages of UTF-8 are holding people back?

EDIT: The W3C lists some known problems with Unicode, could this be a reason too?

share|improve this question
    
Actually more and more popular sites are in UTF-8, one example is ニコニコ動画 and はてな –  Ken Li Jun 8 '11 at 8:35
4  
Why doesn't everybody switch from ISO-8851-1 to UTF-8 ? –  ysdx Jun 8 '11 at 11:35

4 Answers 4

In one word: legacy.

Shift-JIS and other encodings were used before Unicode became available/popular, since it was the only way to encode Japanese at all. Companies have invested in infrastructure that only supported Shift-JIS. Even if that infrastructure now supports Unicode, they are still stuck with Shift-JIS for various reasons ranging from it-works-so-don't-touch-it over encoding-what? to migrating-all-existing-documents-is-too-costly.

There are many western companies that are still using ASCII or latin-1 for the same reasons, only nobody notices since it's never causing a problem.

share|improve this answer
2  
Japanese software industry... slower than dirt at utilizing new software/standards. –  Mark Hosang Jun 8 '11 at 8:11
1  
@Mark Truer words were ne'er spoken! (I'm working in/with Japanese IT... -_-;;) –  deceze Jun 8 '11 at 8:21
    
True, but Western companies have the excuse that our legacy software is full of hard-coded assumptions that 1 byte = 1 character, which makes the transition to UTF-8 harder than for Asians who have long had to write MBCS-clean code. –  dan04 Nov 14 '11 at 7:36

deceze's answer has a very strong element of truth to it, but there is another reason why Shift-JIS and others are still in use: UTF-8 is horrifically inefficient for some languages, mostly in the CJK set. Shift-JIS is, IIRC, a two-byte wide encoding whereas UTF-8 is typically 3-byte and occasionally even 4-byte in its encodings with CJK and others.

share|improve this answer
2  
While that is true, there's always the alternative of UTF-16, which may be as efficient as Shift-JIS. I'd also argue that the headache of dealing with different encodings far outweighs the slight increase in size in this day and age. To put it another way, I have never heard the argument of efficiency for Shift-JIS by anybody still using it. ;-) –  deceze Jun 8 '11 at 9:32
3  
I've heard the efficiency issue used as an excuse for sloth and inertia, though. –  JUST MY correct OPINION Jun 8 '11 at 9:42
    
Hehe, same result then. :o) –  deceze Jun 8 '11 at 9:45
1  
@JUST My correct OPINION: Try "View Source" or the equivalent. Assuming all the actual text is in Japanese, there's likely to be a lot of keywords and the like that were derived from English, and are represented in ASCII. –  David Thornley Jun 8 '11 at 15:48
1  
This sounds to me like a reason to do so we find afterwards. I am pretty sure efficiency has close to absolutely nothing to do with the status quo. To me it's just inertia and legacy. Actually I also think it has to do with the fact that most code produced by Japanese programmers is for other Japanese people, so they don't even feel the need to use something like Unicode. –  Julien Guertault Jun 9 '11 at 1:15

Count string size/memory usage amongst the primary reasons.

In UTF-8, east-asian languages frequently need 3 or more bytes for their characters. On average they need 50% more memory than when using UTF-16 -- the latter of which already is less efficient than native encoding.

The other main reason would be legacy, as point out by deceze.

share|improve this answer

Legacy and storage size, as others said, but there is one more thing: Katakana characters.

It takes only one byte to represent Katakana characters in Shift-JIS, so Japanese text including Katakana takes less than 2 bytes per character (1.5 for a 50/50 mix), making Shift-JIS somewhat more efficient than UTF-16 (2 bytes/char), and much more efficient than UTF-8 (3 bytes/char).

Cheap storage should have made this a much smaller problem, but apparently not.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.