29,429 reputation
249115
bio website coderscentral.blogspot.com
location Colorado Springs, CO
age 50
visits member for 4 years, 7 months
seen 4 hours ago

Started programming on a Control Data mainframe in FORTRAN IV, back when that was still a new thing. Was apparently quite masochistic, because I kept programming anyway. For that matter, I still do...

Gold C++ badge #20
Gold C badge #12
Legendary badge #35


Mar
15
awarded  Scholar
Mar
15
accepted What mistakes in managing software products must be avoided to keep people from hating the vendor?
Mar
12
revised Advice for an ambitious student on building your own kernel
added 9 characters in body
Mar
8
revised Is Bubble Sort the slowest sorting algorithm?
added 704 characters in body
Mar
4
answered In ifs inside for loops, prefer checking for true, or for false and continue?
Mar
3
revised Bad sign if nobody can comprehend one's code?
added 514 characters in body
Mar
3
revised Bad sign if nobody can comprehend one's code?
added 223 characters in body
Mar
3
revised Bad sign if nobody can comprehend one's code?
added 800 characters in body
Mar
2
comment How can frond-end developer support different encoding (UTF-8 , UTF-16,..)other than ASCII?
@Deduplicator: I doubt there would be a strict 1:1 mapping from code points to tokens, but if you were going to use Unicode at all, it seems to me there are enough sufficiently obvious direct mappings like these that (as I've said all along) UTF-32 would make (by far) the most sense.
Mar
2
comment How can frond-end developer support different encoding (UTF-8 , UTF-16,..)other than ASCII?
@Deduplicator: I guess I'm a bit lost here. Even just checking equality between one code point and some known code point becomes non-trivial if you deal directly with UTF-8. Let's say we decide to support things like U+221a for square root, and U+2265 for "greater than or equal to". With UTF-32, this is something like switch (input_codepoint) { case 0x2265: ...; case 0x221a ...;} With UTF-8, this simplicity is lost, and just figuring out what we've read becomes a non-trivial exercise.
Mar
2
comment How can frond-end developer support different encoding (UTF-8 , UTF-16,..)other than ASCII?
@Deduplicator: Again, it seems to me that precisely the opposite is true. Essentially the only rational way to deal with UTF-8 is to start from UTF-32, convert to UTF-8, then on the other end convert back to UTF-32. In this case, there's simply no meaningful motivation to convert the data to UTF-8 for storage in that buffer. For a file or network transfer, you can make a minimal case based on reduced storage/bandwidth, but (IMO) even there it's pretty minimal (and if it matters, better served by a compression algorithm).
Mar
2
comment How can frond-end developer support different encoding (UTF-8 , UTF-16,..)other than ASCII?
It sounds to me an awful lot like you're starting from the conclusion that UTF-8 is the right answer, and you're just groping for any excuse you can come up with to justify that conclusion.
Mar
2
comment How can frond-end developer support different encoding (UTF-8 , UTF-16,..)other than ASCII?
@delnan: As you undoubtedly already know, extracting the Nth code point in constant time (without pre-indexing) is an obvious one. Consider it from the other direction though. If you turned every text file on your hard drive to UTF-32, would that consume a whole dollar's worth of extra space? Fifty cents? Even for files using only US-ASCII, does the savings from using UTF-8 justify any complexity at all? Short answer: no! Not even close.
Mar
2
comment How can frond-end developer support different encoding (UTF-8 , UTF-16,..)other than ASCII?
Yes ftp and quite a few things that manipulate files on Windows are basically broken. As for leaking file system details: no. All I'm saying is that the language spec shouldn't address how the transcoder learns the encoding at all. As far as the language spec cares, all input as just UCS-4. The transcoder is just an external tool that turns raw input (in whatever format, detected in whatever way) into a form acceptable to the lexer.
Mar
2
comment How can frond-end developer support different encoding (UTF-8 , UTF-16,..)other than ASCII?
@delnan: I have read it. It's a giant mistake. And no, being able extract a code point quickly does not mean you're mistaking that for a grapheme cluster. That's simply a fantasy that's been made up to support a position that can't be supported by real facts.
Mar
2
comment How can frond-end developer support different encoding (UTF-8 , UTF-16,..)other than ASCII?
It should go into a (more or less) dedicated place provided by the file system. For example, on NTFS it could be written into an alternate file stream. Unfortunately, most Unix-like file systems fail to provide a good way to do this.
Mar
2
comment How can frond-end developer support different encoding (UTF-8 , UTF-16,..)other than ASCII?
@amon: Quite the contrary--for anybody who knows what they're doing, UTF-32 is the default option for all cases where you're actually manipulating code points. The space expansion is almost entirely irrelevant--we're talking about something that takes input, transcodes it to UTF-32, then passes it to the lexer, so (at most) you're talking about quadrupling the size of buffer used by (for example) an anonymous pipe. It's been a long time since using 16 kilobytes instead of 4 kilobytes was a significant problem (and avoiding UTF-8 will save more code than that anyway).
Mar
2
comment How can frond-end developer support different encoding (UTF-8 , UTF-16,..)other than ASCII?
Trying to encode the encoding into the file is used by several different things (e.g., also used by XML) but it's a terrible idea. The problem is simple: you have to know the encoding to read the comment to find out the encoding. For something to really work, the encoding for the file should be in some metadata separate from the content of the file itself. The current scheme only works at all by restricting the range of inputs and putting the comment early enough that we assume we can treat everything as ASCII (or similar) until we get to the comment saying to use something else.
Mar
2
comment How can frond-end developer support different encoding (UTF-8 , UTF-16,..)other than ASCII?
@delnan: Please not UTF-8. UTF-8 was only ever intended for storage in files (and such). It's a mediocre choice for files, and a terrible choice for internal use. If you're going with Unicode, use UTF-32/UCS-4 internally. It's much cleaner and simpler to deal with. But the encoding of string literals can be (and usually should be) almost entirely independent of the encoding of the rest of the source code.
Mar
2
comment How can frond-end developer support different encoding (UTF-8 , UTF-16,..)other than ASCII?
@CodeMan: In that case, handling the encoding is normally handled entirely in the lexer. Once lexing is done, it's all tokens and you no longer care how they were originally encoded.