Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

On my filesystem (Windows 7) I have some text files (These are SQL script files, if that matters).

When opened with Notepad++, in the "Encoding" menu some of them are reported to have an encoding of "UCS-2 Little Endian" and some of "UTF-8 without BOM".

What is the difference here? They all seem to be perfectly valid scripts. How could I tell what encodings the file have without Notepad++?

share|improve this question
There is a pretty simple way using Firefox. Open your file using Firefox, then View > Character Encoding. Detailed here. –  Catherine Gasnier Apr 15 at 16:26

2 Answers 2

up vote 8 down vote accepted

Files generally indicate their encoding with a file header. There are many examples here. However, even reading the header you can never be sure what encoding a file is really using.

For example, a file with the first three bytes 0xEF,0xBB,0xBF is probably a UTF-8 encoded file. However, it might be an ISO-8859-1 file which happens to start with the characters . Or it might be a different file type entirely.

Notepad++ does its best to guess what encoding a file is using, and most of the time it gets it right. Sometimes it does get it wrong though - that's why that 'Encoding' menu is there, so you can override its best guess.

For the two encodings you mention:

  • The "UCS-2 Little Endian" files are UTF-16 files (based on what I understand from the info here) so probably start with 0xFF,0xFE as the first 2 bytes. From what I can tell, Notepad++ describes them as "UCS-2" since it doesn't support certain facets of UTF-16.
  • The "UTF-8 without BOM" files don't have any header bytes. That's what the "without BOM" bit means.
share|improve this answer
Why would a file that starts with a BOM be auto-detected as "UTF-8 without BOM"? –  Michael Borgwardt Feb 15 '13 at 10:36
And if a file started with 0xFF,0xFE it should be auto-detected as UTF-16, not UCS-2. UCS-2 is probably guessed because it contains mainly ASCII characters and thus every other byte is null. –  Michael Borgwardt Feb 15 '13 at 10:39
@MichaelBorgwardt You are definitely right on the the UTF-2. The UCS-2/UTF-16 is a bit less clear. Will update my answer. –  Baqueta Feb 15 '13 at 10:48
Gah, meant to say "UTF-8" not "UTF-2" in my previous comment. –  Baqueta Feb 15 '13 at 11:01

You cannot. If you could do that, there would not be so many web sites or text files with “random gibberish” out there. That's why the encoding is usually sent along with the payload as meta data.

In case it's not, all you can do is a “smart guess” but the result is often ambiguous since the same byte sequence might be valid in several encodings.

share|improve this answer
OK, then, does the Windows OS store that information (meta data) actually somewhere? In the registry probably? –  Marcel Feb 15 '13 at 10:18
You're wrong. That is codepages- not quite the same. There are algorithms to guess at the Unicode encoding. –  DeadMG Feb 15 '13 at 10:24
@Marcel: No. That's why "text files" are so problematic for anything except pure ASCII. –  Michael Borgwardt Feb 15 '13 at 10:37

protected by gnat Apr 15 at 16:49

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.