Subject: Re: Draft proposal: Clarify guidance for use of a BOM as a UTF-8 encoding signature
From: Tom Honermann (tom_at_[hidden])
Date: 2020-10-13 16:06:28
On 10/13/20 4:42 PM, Shawn Steele wrote:
> My assertion is that if the application cannot change to UTF-8 due to
> legacy considerations, that the subtleties of whether to use a BOM or
> not also cannot be prescribed.Â If the application could follow best
> practices, it would use UTF-8.Â Since it cannot use UTF-8, therefore
> it can't follow any prescribed behavior.Â Therefore anything beyond
> "Use Unicode!" is merely suggestions.Â Terminology like "require"
> implies a false sense of rigor that these applications can't follow in
This is why the prescription remains abstract:
* If possible, use something other than a BOM.
* As a last resort, use a BOM.
I am effectively proposing that as a best practice.
> Eg:Â Presume I have a text editor that has been used in some context
> for some time. Â If I'm told "use UTF-8", that's cool, I could try to
> do that, but if I cannot, then I'm in an exceptional path.Â Unicode
> could suggest that I consider behavior for BOMs (such as ignoring them
> if present), however I'm already stuck in my legacy behavior, so
> there's a limit to what my application can do.
This scenario fits the advice above.Â The "use something other than a
BOM" could mean adding a command line option, adding a menu option,
remembering what encoding was used for that file last time, performing a
heuristic analysis (that may or may not include the presence of a BOM in
its calculation), prompting the user, etc...
> However, if Unicode says "if you see a BOM, then you must use UTF-8",
> then users of my legacy application that is difficult to change, may
> have expectations of the application that don't match reality.Â They
> could even enter bugs like "The app isn't recognizing data being
> tagged with BOMs."Â Or "your system isn't compliant, so we can't
> license it."Â If the app could properly handle UTF-8, we'd have been
> captured in the first requirements and wouldn't even be having this
> part of the conversation.Â Since they can't handle UTF-8, trying to
> enforce it through the BOM isn't going to add much.
No part of this proposal states "if you see a BOM, then you must use
UTF-8".Â It only suggests guidelines; requirements are imposed by
protocols as deemed appropriate by the protocol designers.
> IMO it's better that everyone involved understand that this legacy app
> that can't handle UTF-8 by default isn't necessarily going to behave
> per any set expectations and likely has legacy behaviors that users
> may need to deal with.
> Granted, the difference between "requiring," and "suggesting" or
> "recommending", may be subtle, however those subtleties can sometimes
> cause unnecessary pain.
> I don't mind mandating UTF-8 without BOM if possible.Â I don't really
> mind mandating that BOMs be ignored if "without BOM" isn't reasonable
> to mandate.
> After that though, it's trying to create a higher order protocol for
> codepage detection.Â BOM isn't a great way to identify UTF-8 data.Â
> (It's probably more effective to decode it as UTF-8.Â If it decodes
> properly, then it's likely UTF-8.Â With a certainty of about as many
> "nines" as you have bytes of input.Â Linguistically appropriate
> strings that fail that test are rare.)
We are agreed on these points.
> *From:* Tom Honermann <tom_at_[hidden]>
> *Sent:* Tuesday, October 13, 2020 1:04 PM
> *To:* Shawn Steele <Shawn.Steele_at_[hidden]>; Alisdair Meredith
> *Cc:* sg16_at_[hidden]; Unicode Mail List <unicode_at_[hidden]>
> *Subject:* Re: [SG16] Draft proposal: Clarify guidance for use of a
> BOM as a UTF-8 encoding signature
> On 10/12/20 4:54 PM, Shawn Steele wrote:
> I'm having trouble with the attempt to be this prescriptive.
> These make sense:Â "Use Unicode!"
> * If possible, mandate use of UTF-8 without a BOM; diagnose the
> presence of a BOM in consumed text as an error, and produce
> text without a BOM.
> * Alternatively, swallow the BOM if present.
> After that the situation is clearly hopeless. Â Applications should
> Use Unicode, eg: UTF-8, and clearly there are cases happening
> where that isn't happening.Â Trying to prescribe that negotiation
> should therefore happen, or that BOMs should be interpreted or
> whatever is fairly meaningless at that point. Â Given that the
> higher-order guidance of "Use Unicode" has already been ignored,
> at this point it's garbage-in, garbage-out.Â Clearly the
> app/whatever is ignoring the "use unicode" guidance for some
> legacy reason. If they could adapt, it should be to use UTF-8.Â
> Â It **might** be helpful to say something about a BOM likely
> indicating UTF-8 text in otherwise unspecified data, but
> prescriptive stuff is pointless, it's legacy stuff that behaves in
> a legacy fashion for a reason and saying they should have done it
> differently 20 years ago isn't going to help ð
> There are applications that, for legacy reasons, are unable to change
> their default encoding to UTF-8, but that also need to handle UTF-8
> text.Â It is not clear to me that such situations are hopeless or that
> they cannot be improved.
> The prescription offered follows what you suggest.Â The first three
> cases are are all of the "use Unicode!" variety.Â The distinction
> between the third and the fourth is to relegate use of a BOM as an
> encoding signature to the last resort option.Â The intent is to make
> it clear, with stronger motivation than is currently present in the
> Unicode standard, that use of a BOM in UTF-8 is not a best practice today.
> *From:* Unicode <unicode-bounces_at_[hidden]>
> <mailto:unicode-bounces_at_[hidden]> *On Behalf Of *Tom Honermann
> via Unicode
> *Sent:* Monday, October 12, 2020 7:03 AM
> *To:* Alisdair Meredith <alisdairm_at_[hidden]> <mailto:alisdairm_at_[hidden]>
> *Cc:* sg16_at_[hidden] <mailto:sg16_at_[hidden]>;
> Unicode List <unicode_at_[hidden]> <mailto:unicode_at_[hidden]>
> *Subject:* Re: [SG16] Draft proposal: Clarify guidance for use of
> a BOM as a UTF-8 encoding signature
> Great, here is the change I'm making to address this:
> Protocol designers:
> * If possible, mandate use of UTF-8 without a BOM; diagnose
> the presence of a BOM in consumed text as an error, and
> produce text without a BOM.
> * Otherwise, if possible, mandate use of UTF-8 with or
> without a BOM; accept and discard a BOM in consumed text,
> and produce text without a BOM.
> * Otherwise, if possible, use UTF-8 as the default encoding
> with use of other encodings negotiated using information
> other than a BOM; accept and discard a BOM in consumed
> text, and produce text without a BOM.
> * Otherwise, require the presence of a BOM to differentiate
> UTF-8 encoded text in both consumed and produced
> text*unless the absence of a BOM would result in the text
> being interpreted as an ASCII-based encoding and the UTF-8
> text contains no non-ASCII characters (the exception is
> intended to avoid the addition of a BOM to ASCII text thus
> rendering such text as non-ASCII)*. This approach should
> be reserved for scenarios in which UTF-8 cannot be adopted
> as a default due to backward compatibility concerns.
> On 10/12/20 8:40 AM, Alisdair Meredith wrote:
> That addresses my main concern. Â Essentially, best practice
> (for UTF-8) would be no BOM unless the document contains code
> points that require multiple code units to express.
> On Oct 11, 2020, at 23:22, Tom Honermann
> <tom_at_[hidden] <mailto:tom_at_[hidden]>> wrote:
> On 10/10/20 7:58 PM, Alisdair Meredith via SG16 wrote:
> One concern I have, that might lead into rationale for
> the current discouragement,
> is that I would hate to see a best practice that
> pushes a BOM into ASCII files.
> One of the nice properties of UTF-8 is that a valid
> ASCII file (still very common) is
> also a valid UTF-8 file. Â Changing best practice would
> encourage updating those
> files to be no longer ASCII.
> Thanks, Alisdair.Â I think that concern is implicitly
> addressed by the suggested resolutions, but perhaps that
> can be made more clear.Â One possibility would be to
> modify the "protocol designer" guidelines to address the
> case where a protocol's default encoding is ASCII based
> and to specify that a BOM is only required for UTF-8 text
> that contains non-ASCII characters.Â Would that be helpful?
> On Oct 10, 2020, at 14:54, Tom Honermann via SG16
> <mailto:sg16_at_[hidden]>> wrote:
> Attached is a draft proposal for the Unicode
> standard that intends to clarify the current
> recommendation regarding use of a BOM in UTF-8
> text.Â This is follow up to discussion on the
> Unicode mailing list
> back in June.
> Feedback is welcome.Â I plan to submit
> this to the UTC in a week or so pending review
> SG16 mailing list
> SG16_at_[hidden] <mailto:SG16_at_[hidden]>
SG16 list run by email@example.com