On 6/30/20 10:49 AM, Corentin Jabot via Core wrote:

On Tue, Jun 30, 2020, 16:32 Tom Honermann <tom@honermann.net> wrote:
On 6/30/20 1:31 AM, Corentin Jabot wrote:

On Tue, 30 Jun 2020 at 06:49, Tom Honermann <tom@honermann.net> wrote:
On 6/30/20 12:15 AM, Corentin Jabot wrote:

On Tue, Jun 30, 2020, 05:52 Tom Honermann <tom@honermann.net> wrote:
On 6/28/20 2:03 AM, Corentin Jabot wrote:

On Sun, 28 Jun 2020 at 07:37, Corentin Jabot <corentinjabot@gmail.com> wrote:

On Sun, Jun 28, 2020, 06:50 Tom Honermann via SG16 <sg16@lists.isocpp.org> wrote:

A new draft revision of P2029 (Proposed resolution for core issues 411, 1656, and 2333; numeric and universal character escapes in character and string literals) is now available at https://rawgit.com/sg16-unicode/sg16/master/papers/d2029r2.html.  This addresses the CWG feedback provided during the March 23rd, 2020 core issues processing teleconference.

Wording review feedback prior to the next Core issues processing teleconference would be much appreciated!

I really like the overall direction, a few comments:
- Can we not make conditionally supported escape sequences part of the grammar?
This was requested by Core in the 2020-01-16 issues processing telecon.

What I would do:
    any member of the basic source character set other than u, U, x, and the members of octal-digit

And in 5.13, keep 
Escape sequences not listed in Table 9 are conditionally supported, with implementation-defined semantics
What problem would that solve?

Not having separated grammar for non standard features, simpler grammar.
I prefer the current approach in the paper, but I have no objection to doing what you suggest if the CWG expresses such a preference.

- Can we not add notes for stateful encodings? It doesn't add anything.
Stateful encodings were discussed in the 2020-03-23 issues processing telecon.  

Sure, it is still a level of detail that doesn't add anything. I would like to avoid people in 30 years wondering why that this sentences are here.
Stateful encodings are still a thing.  They may still be a thing in 30 years.
I am not saying they aren't and wouldn't be, I am saying that the current wording was enough for that to be implemented correctly while the new wording does not.
I'm not following.  What do you believe the new wording changes?  Discussion of stateful encodings is limited to non-normative notes.
-- Wide multi character literals were not a thing, let's not make them one now. same for  conditional character literals and conditional wide character literals.

Instead, please add text in (Z) to describe them?

-ordinary and wide characters literal consisting of a single basic-c-char, simple-escape-sequence, or universal-character-name that specifies a character that either lacks representation in the associated character encoding or that cannot be encoded as a single code unit
are conditionally supported and have an implementation-defined value
- A wide character literal consisting of multiple c-chars is conditionally-supported and has an implementation-defined value.
Giving these odd literals a name was suggested by Core.  I agree with their suggested direction; giving them a name makes it easier to discuss and define them. 

No, especially wide multi characters that are simply not a thing, let's not make them one. The reason multi character literals exists and have a name is because their type is different from character literals.
They are a thing in C (see WG14 N2176 (the final draft WP before C18), "Character constants", p11).  I believe their omission in C++ is just an oversight.  Compilers support them.  I think they are a thing and giving them a name is useful.
They don't have a name in C either
I don't see how giving them a name is in any way detrimental.
Should I send a mail to core? Because I really do not like that direction. (Especially as what you call wide multi character literal doesn't behave at all as multi character literals). We should also look at making them ill formed rather than giving them a name

Arguably, you have already sent that mail to Core :)

Haha indeed, nice :)

I don't know what behavioral difference you are concerned about.  The primary reason for differentiating them is to allow the multicharacter case to be ill-formed (conditionally-supported) and/or to have an encoding that differs from single c-char literals.

I think the standard should reflect existing practice.  These odd literals are supported in common compilers.  If you would like to make them ill-formed, you are certainly free to write a paper, but implementations are already free to make them ill-formed and I suspect the ones that don't would retain support for them as an extension anyway.

I am very concerned about giving names to anti features that didn't have a name for the past 30 years, especially those that are not used and were previously not a thing in C++ ( i guess we disagree on our reading on the C standard). I am not concerned about behavior changes
Describing them in a bullet point, rather than in this table keep the table readable and meaning full and leave us with the following features:

* ordinary/wide/utf character literal
* multi character literal

Which is then mostly symmetric with the table for strings.
The bullet point then describes these odds behaviors, which again is not a behavior change. We are just talking about naming and presentation. 

(multi character literal does need a name, both because it always had one, and because it has a different type - also it is also somewhat used)
Whether you consider these literal kinds anti-features is not relevant.  Again, I don't see how giving them names is a bad thing; it doesn't legitimize them any more than describing them in a bullet list does.  A name can acquire both positive and negative connotations.  The current direction in the paper reflects CWG guidance.  If you would like to participate in the next CWG review of this paper and argue your position, please feel free.  I don't intend to argue it further in this email thread.

Please change 
The sequence of characters denoted by each contiguous sequence of basic-s-chars, r-chars, simple-escape-sequences ([lex.ccon]), and universal-character-names ([lex.charset]) is encoded to a code unit sequence
Each basic-s-chars, r-chars, simple-escape-sequences ([lex.ccon]), and universal-character-names ([lex.charset]) is encoded to a code unit sequence
The intent is to make it clear that these sequences are encoded as a group.  This is necessary for stateful encodings with SI/SO characters since such characters don't necessarily contribute a code unit sequence on their own.  This was also requested during the 2020-03-23 issues processing telecon.

The effect is that I can encode things like e,U+0301 as a single code unit, which at the very least should not be allowed in a wording change.
Please read the wording again.  I don't think it states that.  If you still think it does, please elaborate in detail.

 You use the term character ( which in this context is synonym of abstract character) 
 The sequence of characters denoted by each contiguous sequence of basic-s-chars, r-chars, simple-escape-sequences ([lex.ccon]), and universal-character-names ([lex.charset]) is encoded to a code unit sequence using the string-literal's associated character encoding. If a character lacks [...]

Maybe : Each codepoint denoted by a single basic-s-chars, r-chars, simple-escape-sequences ([lex.ccon]), and universal-character-names is encoded to a code unit sequence using the string-literal's associated character encoding. If that codepoint lacks representation in the associated character encoding,

Note that codepoint isn't particularly meaningful in this context , could be "element", for example. The point is the sequence is not converted as a whole.
Changing that is design ( I don't have a terribly strong opinion either way, but it needs to be discussed outside of core, notably because it would allow implementation to handle combining characters differently).
Again, that wording reflects prior CWG guidance.  Since the (wide) execution encoding is implementation-defined, I don't agree that this reflects a design change, but I defer to our benevolent CWG chair to make that determination as he sees fit.

Please understand that none of my comments are addressed at you , but rather at the proposed wording, which I understand is following cwg guidance. I am disagreeing with that guidance

It's also a terrible reason as c-char and UCNs are Unicode characters at this point and cannot correspond to a statefull character as the source of the conversation. The thing they are converted to being an implementation defined sequencee of code unit, the possibility of a state shift is implied.

What are you referring to as a "terrible reason"?

> The intent is to make it clear that these sequences are encoded as a group.  This is necessary for stateful encodings with SI/SO characters since such characters don't necessarily contribute a code unit sequence on their own

 - These characters  appear as ucn and they should in fact contribute to a code unit sequence 
 - They are used as part of a stateful source encoding and would have not been conserved past phase 1.
In discussion elsewhere, we've discussed the example of a decomposed 'é' and noted that at least some compilers will encode the 'e' and combining acute separately, perhaps substituting a character for the combining acute if the execution character set doesn't support combining characters as distinct characters.  Unless someone has proven otherwise, I believe it would be conforming for an implementation to convert the decomposed code point sequence to a composed character that is representable in the execution encoding (e.g., ISO-8859-1; noting that such normalization is not desirable for Unicode encodings).  I see no reason to be more restrictive here.

We have established elsewhere that implementation can't currently do that in phased 5 and that they should not be allowed to (we didn't poll that , but either ways this paper changes the status quo)

I don't agree that has been established.  If you feel otherwise, please provide a reference.  I don't think this changes the status quo since both translation phases 1 and 5 are implementation-defined (and I believe it has been established that implementations are granted considerable freedoms here; e.g., trigraphs and implementations that map "private" to "public").


SI/SO characters exist in Unicode and can therefore be represented as UCNs.  In translation phase 5, an implementation can treat them as part of a shift sequence when converting to the execution encoding.

Again that is a design change
Per prior comments, I disagree and trust that the CWG chair will make an appropriate determination about this.

Sure. I'd be happy to participate to the review, let me know!

- please replace applicable character encoding by character encoding
That doesn't seem correct to me; the wording needs to indicate which character encoding.  Note that there are three occurrences of "applicable associated character encoding"; I'm not sure which use you were referring to.

Missed a word. Sorry. Meant associated character encoding. "Applicable associated" doesn't add anything. Maybe the "the literal associated encoding"

That says the same thing to me.  If CWG expresses a preference, I'll change it.

Yes it does, just trying to be consistent in terminology. associated literal encoding is consistent and what sg16 has been using (including you, maybe you came up with that term :p)

There are multiple "associated character encodings" to choose from; "applicable" indicates which one to choose.  Again, if the CWG requests a change, I'll change it.




Core mailing list
Subscription: https://lists.isocpp.org/mailman/listinfo.cgi/core
Link to this post: http://lists.isocpp.org/core/2020/06/9469.php