Encodings

It’s actually pretty easy to identify text files encoded in UTF-8 because of the way the non-ASCII characters have a particular format to the binary representation which is quite distinctive and quite unlike Latin 1. You have to use heuristics, but it’s pretty reliable.

Also, well-formed Unicode text files should start with a Unicode BOM.

Also, well-formed Unicode text files should start with a Unicode BOM.

Well, I’d agree if you said it the other way round: a file starting with a Unicode BOM is much more likely to be a Unicode file, but for UTF-8 a BOM is not required. The Wikipedia article gives various reasons in favour or against use of a BOM with UTF-8.

Clive wrote

double acute

Yes, it’ll always be ‘hungarumlaut’ in my head because that’s its Postscript name. ;-)

Keyboard diagrams…

Oh very nice.

a file starting with a Unicode BOM is much more likely to be a Unicode file, but for UTF-8 a BOM is not required

Indeed. BOMs are a screaming hint to ancient systems that can’t be bothered to recognise what they’ve got. They are much more useful for UTF-16 than UTF-8.

it’ll always be ‘hungarumlaut’ in my head because that’s its Postscript name

Oh, I know that. I wasn’t blaming you!!

Let’s get a text editor that allows you to specify the encoding on input and output before worrying about characters in filenames!

Sorry for resurrecting an old thread…but I’ve been out of the loop somewhat.

I did start work on an editor that could do as you ask (although I was only going to have it save/load between UTF-8 and the Latin1 in RISC OS). Haven’t got very far at all with it but its on a list of things I would like to do with RISC OS at some point!

If anyone is interested progress will likely be on here when progress happens:

xsltpro.co.uk/content/thorn