Hi Corentin,

Thank you for the update! I would be satisfied with either wording.

I actually have a slight preference for wording option 2, because I like the fact that it avoids the “sequence of integers” and “associated encoding scheme” circumlocutions.

I remain hopeful that we will land P2295 for C++23.

Best wishes,

                        Peter

From: Corentin <corentin.jabot_at_[hidden]>
Sent: 09 June 2022 15:23
To: C++ Core Language Working Group <core_at_[hidden]>
Cc: Alisdair Meredith <alisdairm_at_[hidden]>; Peter Brett <pbrett_at_[hidden]>; SG16 <sg16_at_[hidden]>
Subject: P2295 Support for UTF-8 as a portable source file encoding

Hello folks,
We have not talked about P2295 for a while, but given that multiple people have signaled to me they are interested in seeing progress,
I would like to see whether we can find a majority consensus on wording.
We have 2 options to choose from, I have a very strong preference for option 1 which is a more direct description of reality ("a kind of source file" as suggested by option 2 is a bit too vacuous for my taste).

The last sentence of both wordings is extracted from P2348 - Whitespaces Wording Revamp, as this avoids having to retain a note about "end of line indicator" for the non utf-8 case, and a note saying there are no such "end of line indicator" for the ut-8 case. The term "end of line indicator" was never defined, and because the mapping is implementation defined, it is a given that implementations can introduce whatever characters they like.

I tweaked option 2 slightly from what was suggested by Mike/Huber to avoid repetition of the definition of a UTF-8 source file.

It is important to me that, in addition to achieving the design goals of P2295, the wording remains as clear as possible.

Let me know what you think.

Regards,

Corentin

Option 1

A source file is a sequence of integers with an associated encoding scheme that is determined in an implementation-defined manner.
An implementation shall support the UTF-8 encoding scheme, and may support an implementation-defined set of additional encoding schemes.
If encoding schemes other than UTF-8 are supported, an implementation shall provide a means by which the UTF-8 encoding scheme can be specified, independent of the content of that source file. [Note: In other words, recognizing the U+FEFF Byte Order Mark is not sufficient. --end note]

If the encoding scheme of a source file is determined to be UTF-8, then the source file shall be a well-formed UTF-8 code unit sequence. The source file is decoded to produce a sequence of UCS scalar values that constitutes the sequence of elements of the translation character set.

For any other encoding scheme supported by the implementation, source file characters are mapped, in an implementation-defined manner, to a sequence of translation character set elements.

Option 2:

An implementation shall support UTF-8 source files. It may also support an implementation-defined set of other kinds of source files, and, if so, it shall provide an implementation-defined means of designating a file as a UTF-8 source file, independent of the content of that source file. [Note: In other words, recognizing the U+FEFF Byte Order Mark is not sufficient. --end note].

If a source file is determined to be a UTF-8 source file, then it shall be a well-formed UTF-8 code unit sequence and its content is decoded to produce a sequence of UCS scalar values that constitutes the sequence of elements of the translation character set.

For any other kind of source file, characters are mapped, in an implementation-defined manner, to a sequence of translation character set elements.