On 10/13/20 5:19 PM, Shawn Steele wrote:
>
> IMO this document doesn’t solve your problem. The problem of
> encourage use of UTF-8 in C++ source code is a goal that most
> compilers/source code authors/etc are totally onboard with.
>
> The source is already in an indeterminate state. The desired end
> state is to have UTF-8 source code (without BOM), which is typically
> supported. The difficulty is therefore getting from point A to point
> B. As far as “Use Unicode” goes, there’s no issue, but trying to
> specify BOM as a protocol doesn’t really solve the problem,
> particularly in complex environments.
>
I think there is a misunderstanding. The intent of the paper is to
provide rationale for the existing discouragement for use of a BOM in
UTF-8 while acknowledging that, in some cases, it may remain useful. My
intent is to discourage use of a BOM for UTF-8 encoded source files -
thereby arguing against standardizing the behavior exhibited by
Microsoft Visual C++ today.
>
> If the compiler doesn’t handle BOM as expected, then you’ll get
> errors. This can be further complicated by preprocessors, #include,
> resources, etc. If “specifying BOM behavior in Unicode” could help
> solve the problem, then all of the tooling used by everyone would have
> to be updated to handle that (new) requirement. If you could get
> everyone on the same page, they’d all use UTF-8, so you wouldn’t need
> to update the tooling. If you don’t need to update the tooling, you
> wouldn’t need to update the best practices for BOMs.
>
This paper does not propose "specifying BOM behavior in Unicode". If
you feel that it does, please read it again and let me know what leads
you to believe that it does.

The tooling isn't the problem. The problem is the existing source code
that is not UTF-8 encoded or that is UTF-8 encoded with a BOM. The
deployment challenge is with those existing source files. Microsoft
Visual C++ is going to continue consuming source files using the Active
Code Page (ACP) and IBM compilers on EBCDIC platforms are going to
continue consuming source files using EBCDIC code pages. The goal is to
provide a mechanism where a UTF-8 encoded source file can #include a
source file in another encoding or vice versa. Any solution for that
will require tooling updates (and that is ok).

> Personally, I’d prefer if cases like this ignore BOMs (or use them to
> switch to UTF-8); eg: treat BOMs like whitespace. But this isn’t a
> problem solvable by any recommendation by Unicode.
>
When consuming text as UTF-8, I agree that ignoring a BOM is usually the
right thing to do and would be the right thing to do when consuming
source code.
>
> As you noted, many systems provide mechanisms for indicating that code
> is UTF-8 or compiling with UTF-8, regardless of BOM.
>
Yes, but there is no standard solution, not even a defacto one, for
consuming differently encoded source files in the same translation unit.
>
> A rather large codebase I’ve been working with has been working to
> remove encoding confusion, and it’s a big task 😁
>
Yes, yes it is.

Tom.

> -Shawn
>
> *From:* Unicode <unicode-bounces_at_[hidden]> *On Behalf Of *Tom
> Honermann via Unicode
> *Sent:* Tuesday, October 13, 2020 1:47 PM
> *To:* J Decker <d3ck0r_at_[hidden]>; Unicode List <unicode_at_[hidden]>
> *Cc:* sg16_at_[hidden]
> *Subject:* Re: [SG16] Draft proposal: Clarify guidance for use of a
> BOM as a UTF-8 encoding signature
>
> On 10/12/20 8:09 PM, J Decker via Unicode wrote:
>
> On Sun, Oct 11, 2020 at 8:24 PM Tom Honermann via Unicode
> <unicode_at_[hidden] <mailto:unicode_at_[hidden]>> wrote:
>
> On 10/10/20 7:58 PM, Alisdair Meredith via SG16 wrote:
>
> One concern I have, that might lead into rationale for the
> current discouragement,
>
> is that I would hate to see a best practice that pushes a
> BOM into ASCII files.
>
> One of the nice properties of UTF-8 is that a valid ASCII
> file (still very common) is
>
> also a valid UTF-8 file. Changing best practice would
> encourage updating those
>
> files to be no longer ASCII.
>
> Thanks, Alisdair. I think that concern is implicitly
> addressed by the suggested resolutions, but perhaps that can
> be made more clear. One possibility would be to modify the
> "protocol designer" guidelines to address the case where a
> protocol's default encoding is ASCII based and to specify that
> a BOM is only required for UTF-8 text that contains non-ASCII
> characters. Would that be helpful?
>
> 'and to specify that a BOM is only required for UTF-8 ' this
> should NEVER be 'required' or 'must', it shouldn't even be
> 'suggested'; fortunately BOM is just a ZWNBSP, so it's certainly a
> 'may' start with a such and such.
>
> These days the standard 'everything IS utf-8' works really well,
> except in firefox where the charset is required to be specified
> for JS scripts (but that's a bug in that)
>
> EBCDIC should be converted on the edge to internal ascii, since,
> thankfully, this is a niche application and everything thinks in
> ASCII or some derivative thereof.
>
> Byte Order Mark is irrelatvent to utf-8 since bytes are ordered in
> the correct order.
>
> I have run into several editors that have insisted on emitted BOM
> for UTF8 when initially promoted from ASCII, but subsequently
> deleting it doesn't bother anything.
>
> I mostly agree. Please note that the paper suggests use of a BOM only
> as a last resort. The goal is to further discourage its use with
> rationale.
>
> I am curious though, what was the actual problem you ran into that
> makes you even consider this modification?
>
> I'm working on improving support for portable C++ source code. Today,
> there is no character encoding that is supported by all C++
> implementations (not even ASCII). I'd like to make UTF-8 that
> commonly supported character encoding. For backward compatibility
> reasons, compilers cannot change their default source code character
> encoding to UTF-8.
>
> Most C++ applications are created from components that have different
> release schedules and that are maintained by different organizations.
> Synchronizing a conversion to UTF-8 across dependent projects isn't
> feasible, nor is converting all of the source files used by an
> application to UTF-8 as simple as just running them through 'iconv'.
> Migration to UTF-8 will therefore require an incremental approach for
> at least some applications, though many are likely to find success by
> simply invoking their compiler with the appropriate
> -everything-is-utf8 option since most source files are ASCII.
>
> Microsoft Visual C++ recognizes a UTF-8 BOM as an encoding signature
> and allows differently encoded source files to be used in the same
> translation unit. Support for differently encoded source files in the
> same translation unit is the feature that will be needed to enable
> incremental migration. Normative discouragement (with rationale) for
> use of a BOM by the Unicode standard would be helpful to explain why a
> solution other than a BOM (perhaps something like Python's encoding
> declaration
> <https://docs.python.org/3/reference/lexical_analysis.html#encoding-declarations>)
> should be standardized in favor of the existing practice demonstrated
> by Microsoft's solution.
>
> Tom.
>
> J
>
> Tom.
>
> AlisdairM
>
>
>
> On Oct 10, 2020, at 14:54, Tom Honermann via SG16
> <sg16_at_[hidden] <mailto:sg16_at_[hidden]>>
> wrote:
>
> Attached is a draft proposal for the Unicode standard
> that intends to clarify the current recommendation
> regarding use of a BOM in UTF-8 text. This is follow
> up to discussion on the Unicode mailing list
> <https://corp.unicode.org/pipermail/unicode/2020-June/008713.html>
> back in June.
>
> Feedback is welcome. I plan to submit
> <https://www.unicode.org/pending/docsubmit.html> this
> to the UTC in a week or so pending review feedback.
>
> Tom.
>
> <Unicode-BOM-guidance.pdf>--
> SG16 mailing list
> SG16_at_[hidden] <mailto:SG16_at_[hidden]>
> https://lists.isocpp.org/mailman/listinfo.cgi/sg16
>
>
>