Date: Wed, 14 Oct 2020 01:16:14 -0400
On 10/13/20 5:29 PM, Shawn Steele wrote:
>
> > The "use something other than a BOM" could mean adding a command
> line option, adding a menu option, remembering what encoding was used
> for that file last time, performing a heuristic analysis (that may or
> may not include the presence of a BOM in its calculation), prompting
> the user, etc...
>
> That’s the catch. “Adding… adding… remembering… performing.” If the
> code was doing the best practices/right thing, it’d be using UTF-8.
> It isn’t, and it’s sort of a given that it’s legacy behavior.
> Therefore “adding”, etc. means that changes have to happen to the
> applications &/or processes. Which aren’t necessarily going to be
> deployed promptly, if at all. This isn’t a problem that a standard or
> best practices can solve.
>
> Everyone already knows the best practice: “Use UTF-8”. Any
> resources/effort is going to be getting toward that best practice, not
> edge cases of legacy behaviors that are offshoots of something that
> isn’t the desired end state of “use UTF-8”.
>
My goal is exactly to ease migration to that end state. We can't
reasonably synchronize a migration of all C++ projects to UTF-8. To get
to that end state, we'll have to enable C++ projects to independently
transition to UTF-8. Such independent transition will be eased by
having a portable means to indicate that a source file is UTF-8 encoded
in such a way that a C++ compiler can process it correctly when it is
#included from a differently encoded source file. This would suffice
for a project to migrate to UTF-8 while being usable (e.g., having its
header files #included) by another UTF-8 encoded project, a Windows-1252
encoded project, or an EBCDIC encoded project. Those other projects can
then migrate to UTF-8 on their own schedule.
Use of a BOM would be one way to get to that desired end state but, as
you mentioned, a BOM isn't a great way to identify UTF-8 data. The
Unicode standard already admits this with the quoted "not recommended"
text, but it lacks the rationale to defend that recommendation or to
explain when it may be appropriate to disregard that recommendation. My
goal with this paper is to fill that hole. If you don't care for how
I've proposed it to be filled, that is certainly ok and alternative
suggestions are welcome.
> I sympathize with the problem, since I encounter variations of it
> every day, I just don’t think any tweaking of this text will have any
> practical impact with moving the needle.
>
That is entirely possible.
Tom.
> -Shawn
>
> *From:* Tom Honermann <tom_at_[hidden]>
> *Sent:* Tuesday, October 13, 2020 2:06 PM
> *To:* Shawn Steele <Shawn.Steele_at_[hidden]>; Alisdair Meredith
> <alisdairm_at_[hidden]>
> *Cc:* sg16_at_[hidden]; Unicode Mail List <unicode_at_[hidden]>
> *Subject:* Re: [SG16] Draft proposal: Clarify guidance for use of a
> BOM as a UTF-8 encoding signature
>
> On 10/13/20 4:42 PM, Shawn Steele wrote:
>
> My assertion is that if the application cannot change to UTF-8 due
> to legacy considerations, that the subtleties of whether to use a
> BOM or not also cannot be prescribed. If the application could
> follow best practices, it would use UTF-8. Since it cannot use
> UTF-8, therefore it can’t follow any prescribed behavior.
> Therefore anything beyond “Use Unicode!” is merely suggestions.
> Terminology like “require” implies a false sense of rigor that
> these applications can’t follow in practice.
>
> This is why the prescription remains abstract:
>
> * If possible, use something other than a BOM.
> * As a last resort, use a BOM.
>
> I am effectively proposing that as a best practice.
>
> Eg: Presume I have a text editor that has been used in some
> context for some time. If I’m told “use UTF-8”, that’s cool, I
> could try to do that, but if I cannot, then I’m in an exceptional
> path. Unicode could suggest that I consider behavior for BOMs
> (such as ignoring them if present), however I’m already stuck in
> my legacy behavior, so there’s a limit to what my application can do.
>
> This scenario fits the advice above. The "use something other than a
> BOM" could mean adding a command line option, adding a menu option,
> remembering what encoding was used for that file last time, performing
> a heuristic analysis (that may or may not include the presence of a
> BOM in its calculation), prompting the user, etc...
>
> However, if Unicode says “if you see a BOM, then you must use
> UTF-8”, then users of my legacy application that is difficult to
> change, may have expectations of the application that don’t match
> reality. They could even enter bugs like “The app isn’t
> recognizing data being tagged with BOMs.” Or “your system isn’t
> compliant, so we can’t license it.” If the app could properly
> handle UTF-8, we’d have been captured in the first requirements
> and wouldn’t even be having this part of the conversation. Since
> they can’t handle UTF-8, trying to enforce it through the BOM
> isn’t going to add much.
>
> No part of this proposal states "if you see a BOM, then you must use
> UTF-8". It only suggests guidelines; requirements are imposed by
> protocols as deemed appropriate by the protocol designers.
>
> IMO it’s better that everyone involved understand that this legacy
> app that can’t handle UTF-8 by default isn’t necessarily going to
> behave per any set expectations and likely has legacy behaviors
> that users may need to deal with.
>
> Granted, the difference between “requiring,” and “suggesting” or
> “recommending”, may be subtle, however those subtleties can
> sometimes cause unnecessary pain.
>
> I don’t mind mandating UTF-8 without BOM if possible. I don’t
> really mind mandating that BOMs be ignored if “without BOM” isn’t
> reasonable to mandate.
>
> After that though, it’s trying to create a higher order protocol
> for codepage detection. BOM isn’t a great way to identify UTF-8
> data. (It’s probably more effective to decode it as UTF-8. If it
> decodes properly, then it’s likely UTF-8. With a certainty of
> about as many “nines” as you have bytes of input. Linguistically
> appropriate strings that fail that test are rare.)
>
> We are agreed on these points.
>
> Tom.
>
> -Shawn
>
> *From:* Tom Honermann <tom_at_[hidden]> <mailto:tom_at_[hidden]>
> *Sent:* Tuesday, October 13, 2020 1:04 PM
> *To:* Shawn Steele <Shawn.Steele_at_[hidden]>
> <mailto:Shawn.Steele_at_[hidden]>; Alisdair Meredith
> <alisdairm_at_[hidden]> <mailto:alisdairm_at_[hidden]>
> *Cc:* sg16_at_[hidden] <mailto:sg16_at_[hidden]>;
> Unicode Mail List <unicode_at_[hidden]> <mailto:unicode_at_[hidden]>
> *Subject:* Re: [SG16] Draft proposal: Clarify guidance for use of
> a BOM as a UTF-8 encoding signature
>
> On 10/12/20 4:54 PM, Shawn Steele wrote:
>
> I’m having trouble with the attempt to be this prescriptive.
>
> These make sense: “Use Unicode!”
>
> * If possible, mandate use of UTF-8 without a BOM; diagnose
> the presence of a BOM in consumed text as an error, and
> produce text without a BOM.
> * Alternatively, swallow the BOM if present.
>
> After that the situation is clearly hopeless. Applications
> should Use Unicode, eg: UTF-8, and clearly there are cases
> happening where that isn’t happening. Trying to prescribe
> that negotiation should therefore happen, or that BOMs should
> be interpreted or whatever is fairly meaningless at that
> point. Given that the higher-order guidance of “Use Unicode”
> has already been ignored, at this point it’s garbage-in,
> garbage-out. Clearly the app/whatever is ignoring the “use
> unicode” guidance for some legacy reason. If they could
> adapt, it should be to use UTF-8. It **might** be helpful to
> say something about a BOM likely indicating UTF-8 text in
> otherwise unspecified data, but prescriptive stuff is
> pointless, it’s legacy stuff that behaves in a legacy fashion
> for a reason and saying they should have done it differently
> 20 years ago isn’t going to help 😊
>
> There are applications that, for legacy reasons, are unable to
> change their default encoding to UTF-8, but that also need to
> handle UTF-8 text. It is not clear to me that such situations are
> hopeless or that they cannot be improved.
>
> The prescription offered follows what you suggest. The first
> three cases are are all of the "use Unicode!" variety. The
> distinction between the third and the fourth is to relegate use of
> a BOM as an encoding signature to the last resort option. The
> intent is to make it clear, with stronger motivation than is
> currently present in the Unicode standard, that use of a BOM in
> UTF-8 is not a best practice today.
>
> Tom.
>
> -Shawn
>
> *From:* Unicode <unicode-bounces_at_[hidden]>
> <mailto:unicode-bounces_at_[hidden]> *On Behalf Of *Tom
> Honermann via Unicode
> *Sent:* Monday, October 12, 2020 7:03 AM
> *To:* Alisdair Meredith <alisdairm_at_[hidden]>
> <mailto:alisdairm_at_[hidden]>
> *Cc:* sg16_at_[hidden] <mailto:sg16_at_[hidden]>;
> Unicode List <unicode_at_[hidden]> <mailto:unicode_at_[hidden]>
> *Subject:* Re: [SG16] Draft proposal: Clarify guidance for use
> of a BOM as a UTF-8 encoding signature
>
> Great, here is the change I'm making to address this:
>
> Protocol designers:
>
> * If possible, mandate use of UTF-8 without a BOM;
> diagnose the presence of a BOM in consumed text as an
> error, and produce text without a BOM.
> * Otherwise, if possible, mandate use of UTF-8 with or
> without a BOM; accept and discard a BOM in consumed
> text, and produce text without a BOM.
> * Otherwise, if possible, use UTF-8 as the default
> encoding with use of other encodings negotiated using
> information other than a BOM; accept and discard a BOM
> in consumed text, and produce text without a BOM.
> * Otherwise, require the presence of a BOM to
> differentiate UTF-8 encoded text in both consumed and
> produced text*unless the absence of a BOM would result
> in the text being interpreted as an ASCII-based
> encoding and the UTF-8 text contains no non-ASCII
> characters (the exception is intended to avoid the
> addition of a BOM to ASCII text thus rendering such
> text as non-ASCII)*. This approach should be reserved
> for scenarios in which UTF-8 cannot be adopted as a
> default due to backward compatibility concerns.
>
> Tom.
>
> On 10/12/20 8:40 AM, Alisdair Meredith wrote:
>
> That addresses my main concern. Essentially, best
> practice (for UTF-8) would be no BOM unless the document
> contains code points that require multiple code units to
> express.
>
> AlisdairM
>
>
>
>
>
> On Oct 11, 2020, at 23:22, Tom Honermann
> <tom_at_[hidden] <mailto:tom_at_[hidden]>> wrote:
>
> On 10/10/20 7:58 PM, Alisdair Meredith via SG16 wrote:
>
> One concern I have, that might lead into rationale
> for the current discouragement,
>
> is that I would hate to see a best practice that
> pushes a BOM into ASCII files.
>
> One of the nice properties of UTF-8 is that a
> valid ASCII file (still very common) is
>
> also a valid UTF-8 file. Changing best practice
> would encourage updating those
>
> files to be no longer ASCII.
>
> Thanks, Alisdair. I think that concern is implicitly
> addressed by the suggested resolutions, but perhaps
> that can be made more clear. One possibility would be
> to modify the "protocol designer" guidelines to
> address the case where a protocol's default encoding
> is ASCII based and to specify that a BOM is only
> required for UTF-8 text that contains non-ASCII
> characters. Would that be helpful?
>
> Tom.
>
> AlisdairM
>
>
>
>
>
> On Oct 10, 2020, at 14:54, Tom Honermann via
> SG16 <sg16_at_[hidden]
> <mailto:sg16_at_[hidden]>> wrote:
>
> Attached is a draft proposal for the Unicode
> standard that intends to clarify the current
> recommendation regarding use of a BOM in UTF-8
> text. This is follow up to discussion on the
> Unicode mailing list
> <https://corp.unicode.org/pipermail/unicode/2020-June/008713.html>
> back in June.
>
> Feedback is welcome. I plan to submit
> <https://www.unicode.org/pending/docsubmit.html>
> this to the UTC in a week or so pending review
> feedback.
>
> Tom.
>
> <Unicode-BOM-guidance.pdf>--
> SG16 mailing list
> SG16_at_[hidden]
> <mailto:SG16_at_[hidden]>
> https://lists.isocpp.org/mailman/listinfo.cgi/sg16
>
>
>
>
>
>
> > The "use something other than a BOM" could mean adding a command
> line option, adding a menu option, remembering what encoding was used
> for that file last time, performing a heuristic analysis (that may or
> may not include the presence of a BOM in its calculation), prompting
> the user, etc...
>
> That’s the catch. “Adding… adding… remembering… performing.” If the
> code was doing the best practices/right thing, it’d be using UTF-8.
> It isn’t, and it’s sort of a given that it’s legacy behavior.
> Therefore “adding”, etc. means that changes have to happen to the
> applications &/or processes. Which aren’t necessarily going to be
> deployed promptly, if at all. This isn’t a problem that a standard or
> best practices can solve.
>
> Everyone already knows the best practice: “Use UTF-8”. Any
> resources/effort is going to be getting toward that best practice, not
> edge cases of legacy behaviors that are offshoots of something that
> isn’t the desired end state of “use UTF-8”.
>
My goal is exactly to ease migration to that end state. We can't
reasonably synchronize a migration of all C++ projects to UTF-8. To get
to that end state, we'll have to enable C++ projects to independently
transition to UTF-8. Such independent transition will be eased by
having a portable means to indicate that a source file is UTF-8 encoded
in such a way that a C++ compiler can process it correctly when it is
#included from a differently encoded source file. This would suffice
for a project to migrate to UTF-8 while being usable (e.g., having its
header files #included) by another UTF-8 encoded project, a Windows-1252
encoded project, or an EBCDIC encoded project. Those other projects can
then migrate to UTF-8 on their own schedule.
Use of a BOM would be one way to get to that desired end state but, as
you mentioned, a BOM isn't a great way to identify UTF-8 data. The
Unicode standard already admits this with the quoted "not recommended"
text, but it lacks the rationale to defend that recommendation or to
explain when it may be appropriate to disregard that recommendation. My
goal with this paper is to fill that hole. If you don't care for how
I've proposed it to be filled, that is certainly ok and alternative
suggestions are welcome.
> I sympathize with the problem, since I encounter variations of it
> every day, I just don’t think any tweaking of this text will have any
> practical impact with moving the needle.
>
That is entirely possible.
Tom.
> -Shawn
>
> *From:* Tom Honermann <tom_at_[hidden]>
> *Sent:* Tuesday, October 13, 2020 2:06 PM
> *To:* Shawn Steele <Shawn.Steele_at_[hidden]>; Alisdair Meredith
> <alisdairm_at_[hidden]>
> *Cc:* sg16_at_[hidden]; Unicode Mail List <unicode_at_[hidden]>
> *Subject:* Re: [SG16] Draft proposal: Clarify guidance for use of a
> BOM as a UTF-8 encoding signature
>
> On 10/13/20 4:42 PM, Shawn Steele wrote:
>
> My assertion is that if the application cannot change to UTF-8 due
> to legacy considerations, that the subtleties of whether to use a
> BOM or not also cannot be prescribed. If the application could
> follow best practices, it would use UTF-8. Since it cannot use
> UTF-8, therefore it can’t follow any prescribed behavior.
> Therefore anything beyond “Use Unicode!” is merely suggestions.
> Terminology like “require” implies a false sense of rigor that
> these applications can’t follow in practice.
>
> This is why the prescription remains abstract:
>
> * If possible, use something other than a BOM.
> * As a last resort, use a BOM.
>
> I am effectively proposing that as a best practice.
>
> Eg: Presume I have a text editor that has been used in some
> context for some time. If I’m told “use UTF-8”, that’s cool, I
> could try to do that, but if I cannot, then I’m in an exceptional
> path. Unicode could suggest that I consider behavior for BOMs
> (such as ignoring them if present), however I’m already stuck in
> my legacy behavior, so there’s a limit to what my application can do.
>
> This scenario fits the advice above. The "use something other than a
> BOM" could mean adding a command line option, adding a menu option,
> remembering what encoding was used for that file last time, performing
> a heuristic analysis (that may or may not include the presence of a
> BOM in its calculation), prompting the user, etc...
>
> However, if Unicode says “if you see a BOM, then you must use
> UTF-8”, then users of my legacy application that is difficult to
> change, may have expectations of the application that don’t match
> reality. They could even enter bugs like “The app isn’t
> recognizing data being tagged with BOMs.” Or “your system isn’t
> compliant, so we can’t license it.” If the app could properly
> handle UTF-8, we’d have been captured in the first requirements
> and wouldn’t even be having this part of the conversation. Since
> they can’t handle UTF-8, trying to enforce it through the BOM
> isn’t going to add much.
>
> No part of this proposal states "if you see a BOM, then you must use
> UTF-8". It only suggests guidelines; requirements are imposed by
> protocols as deemed appropriate by the protocol designers.
>
> IMO it’s better that everyone involved understand that this legacy
> app that can’t handle UTF-8 by default isn’t necessarily going to
> behave per any set expectations and likely has legacy behaviors
> that users may need to deal with.
>
> Granted, the difference between “requiring,” and “suggesting” or
> “recommending”, may be subtle, however those subtleties can
> sometimes cause unnecessary pain.
>
> I don’t mind mandating UTF-8 without BOM if possible. I don’t
> really mind mandating that BOMs be ignored if “without BOM” isn’t
> reasonable to mandate.
>
> After that though, it’s trying to create a higher order protocol
> for codepage detection. BOM isn’t a great way to identify UTF-8
> data. (It’s probably more effective to decode it as UTF-8. If it
> decodes properly, then it’s likely UTF-8. With a certainty of
> about as many “nines” as you have bytes of input. Linguistically
> appropriate strings that fail that test are rare.)
>
> We are agreed on these points.
>
> Tom.
>
> -Shawn
>
> *From:* Tom Honermann <tom_at_[hidden]> <mailto:tom_at_[hidden]>
> *Sent:* Tuesday, October 13, 2020 1:04 PM
> *To:* Shawn Steele <Shawn.Steele_at_[hidden]>
> <mailto:Shawn.Steele_at_[hidden]>; Alisdair Meredith
> <alisdairm_at_[hidden]> <mailto:alisdairm_at_[hidden]>
> *Cc:* sg16_at_[hidden] <mailto:sg16_at_[hidden]>;
> Unicode Mail List <unicode_at_[hidden]> <mailto:unicode_at_[hidden]>
> *Subject:* Re: [SG16] Draft proposal: Clarify guidance for use of
> a BOM as a UTF-8 encoding signature
>
> On 10/12/20 4:54 PM, Shawn Steele wrote:
>
> I’m having trouble with the attempt to be this prescriptive.
>
> These make sense: “Use Unicode!”
>
> * If possible, mandate use of UTF-8 without a BOM; diagnose
> the presence of a BOM in consumed text as an error, and
> produce text without a BOM.
> * Alternatively, swallow the BOM if present.
>
> After that the situation is clearly hopeless. Applications
> should Use Unicode, eg: UTF-8, and clearly there are cases
> happening where that isn’t happening. Trying to prescribe
> that negotiation should therefore happen, or that BOMs should
> be interpreted or whatever is fairly meaningless at that
> point. Given that the higher-order guidance of “Use Unicode”
> has already been ignored, at this point it’s garbage-in,
> garbage-out. Clearly the app/whatever is ignoring the “use
> unicode” guidance for some legacy reason. If they could
> adapt, it should be to use UTF-8. It **might** be helpful to
> say something about a BOM likely indicating UTF-8 text in
> otherwise unspecified data, but prescriptive stuff is
> pointless, it’s legacy stuff that behaves in a legacy fashion
> for a reason and saying they should have done it differently
> 20 years ago isn’t going to help 😊
>
> There are applications that, for legacy reasons, are unable to
> change their default encoding to UTF-8, but that also need to
> handle UTF-8 text. It is not clear to me that such situations are
> hopeless or that they cannot be improved.
>
> The prescription offered follows what you suggest. The first
> three cases are are all of the "use Unicode!" variety. The
> distinction between the third and the fourth is to relegate use of
> a BOM as an encoding signature to the last resort option. The
> intent is to make it clear, with stronger motivation than is
> currently present in the Unicode standard, that use of a BOM in
> UTF-8 is not a best practice today.
>
> Tom.
>
> -Shawn
>
> *From:* Unicode <unicode-bounces_at_[hidden]>
> <mailto:unicode-bounces_at_[hidden]> *On Behalf Of *Tom
> Honermann via Unicode
> *Sent:* Monday, October 12, 2020 7:03 AM
> *To:* Alisdair Meredith <alisdairm_at_[hidden]>
> <mailto:alisdairm_at_[hidden]>
> *Cc:* sg16_at_[hidden] <mailto:sg16_at_[hidden]>;
> Unicode List <unicode_at_[hidden]> <mailto:unicode_at_[hidden]>
> *Subject:* Re: [SG16] Draft proposal: Clarify guidance for use
> of a BOM as a UTF-8 encoding signature
>
> Great, here is the change I'm making to address this:
>
> Protocol designers:
>
> * If possible, mandate use of UTF-8 without a BOM;
> diagnose the presence of a BOM in consumed text as an
> error, and produce text without a BOM.
> * Otherwise, if possible, mandate use of UTF-8 with or
> without a BOM; accept and discard a BOM in consumed
> text, and produce text without a BOM.
> * Otherwise, if possible, use UTF-8 as the default
> encoding with use of other encodings negotiated using
> information other than a BOM; accept and discard a BOM
> in consumed text, and produce text without a BOM.
> * Otherwise, require the presence of a BOM to
> differentiate UTF-8 encoded text in both consumed and
> produced text*unless the absence of a BOM would result
> in the text being interpreted as an ASCII-based
> encoding and the UTF-8 text contains no non-ASCII
> characters (the exception is intended to avoid the
> addition of a BOM to ASCII text thus rendering such
> text as non-ASCII)*. This approach should be reserved
> for scenarios in which UTF-8 cannot be adopted as a
> default due to backward compatibility concerns.
>
> Tom.
>
> On 10/12/20 8:40 AM, Alisdair Meredith wrote:
>
> That addresses my main concern. Essentially, best
> practice (for UTF-8) would be no BOM unless the document
> contains code points that require multiple code units to
> express.
>
> AlisdairM
>
>
>
>
>
> On Oct 11, 2020, at 23:22, Tom Honermann
> <tom_at_[hidden] <mailto:tom_at_[hidden]>> wrote:
>
> On 10/10/20 7:58 PM, Alisdair Meredith via SG16 wrote:
>
> One concern I have, that might lead into rationale
> for the current discouragement,
>
> is that I would hate to see a best practice that
> pushes a BOM into ASCII files.
>
> One of the nice properties of UTF-8 is that a
> valid ASCII file (still very common) is
>
> also a valid UTF-8 file. Changing best practice
> would encourage updating those
>
> files to be no longer ASCII.
>
> Thanks, Alisdair. I think that concern is implicitly
> addressed by the suggested resolutions, but perhaps
> that can be made more clear. One possibility would be
> to modify the "protocol designer" guidelines to
> address the case where a protocol's default encoding
> is ASCII based and to specify that a BOM is only
> required for UTF-8 text that contains non-ASCII
> characters. Would that be helpful?
>
> Tom.
>
> AlisdairM
>
>
>
>
>
> On Oct 10, 2020, at 14:54, Tom Honermann via
> SG16 <sg16_at_[hidden]
> <mailto:sg16_at_[hidden]>> wrote:
>
> Attached is a draft proposal for the Unicode
> standard that intends to clarify the current
> recommendation regarding use of a BOM in UTF-8
> text. This is follow up to discussion on the
> Unicode mailing list
> <https://corp.unicode.org/pipermail/unicode/2020-June/008713.html>
> back in June.
>
> Feedback is welcome. I plan to submit
> <https://www.unicode.org/pending/docsubmit.html>
> this to the UTC in a week or so pending review
> feedback.
>
> Tom.
>
> <Unicode-BOM-guidance.pdf>--
> SG16 mailing list
> SG16_at_[hidden]
> <mailto:SG16_at_[hidden]>
> https://lists.isocpp.org/mailman/listinfo.cgi/sg16
>
>
>
>
>
Received on 2020-10-14 00:16:19