Apologies for the delay responding; I've been on vacation!

On 6/25/22 4:55 AM, Corentin wrote:


On Sat, Jun 25, 2022 at 12:04 AM Tom Honermann <tom@honermann.net> wrote:
On 6/24/22 3:10 AM, Corentin wrote:
Hello,
A couple of comments on P2572R0
Thank you, Corentin.


The intent is not clear to me here.
I don't think there are a precondition that unicode encoded strings are well-formed in format (and if there was that change is not necessary as there can be no codepoints that are not scalar value in a well-formed sequence), and if we want to enforce well-formedness i'd rather that it be stated that way. 

On the other hand, if we do not intend to ensure well-formedness, we should be mindful that
if we allow ill-formed sequences, then the standard practice is 1ef5114c3377bbf48ed14092587de9605fcda5d2to replace isolate surrogate by
� whose width is definitively 1 [1], and so the changes in 22.14.2.2 requires, in my mind further discussion and a broader solution. I would keep that a separate issue.
Unless you want to clearly state "the width of surrogates is undefined" or something like that.

Beside, I also don't think it would be useful to replace code point by scalar values in places where there is already a precondition or no possibility to have isolated surrogates, as scalar value, beside being a mouthful, is only applicable to unicode (unlike codepoint), and in places where well-formedness is desirable, "Precondition foo is a well-formed sequence in the bar encoding" would take care of scalar values.

This sounds like something that would be appropriate to discuss in LWG. This change is intended to be a drive-by correction and specifically not a design change. If LWG wishes to reject that change, that won't affect the rest of the paper.

I'm not aware of any wording that describes what happens if an ill-formed Unicode string is provided as the format string; that seems like UB by omission and implementations appear to agree. I agree that any change to perform character substitution should be addressed in a separate paper.

I'm not sure that talking about scalar value here makes it clear that passing non-well formed text is a precondition violation.
The change is not intended to add such a precondition.
As no such precondition seems to exist, and as we seem to agree that your proposed change does introduce one.

I don't think we agree on that. As I said, I think passing an ill-formed Unicode string is currently UB by omission. Regardless, [format.string.std]p11 states:

For a string in a Unicode encoding, implementations should estimate the width of a string as the sum of estimated widths of the first code points in its extended grapheme clusters. The extended grapheme clusters of a string are defined by UAX #29. ...

It isn't clear to me that UAX #29 is intended to define extended grapheme clusters for code point sequences that contain surrogate code points. UAX #29 does not use UCS scalar value terminology, but the document reads as though it is intended to treat all code points as "characters". The words "lone" and "surrogate" do not appear in it. It may be worth trying to clarify this with the UTC.

I would prefer to see explicitly something like "The width of a unicode string that is not well-formed is undefined" or something like that. it covers both lone surrogate and what happens if you cannot form a codepoint for the purposes of width estimation.

I think that would be fine, but I don't want to do that as part of this paper. I suggest filing a separate LWG issue for that.

At any rate, since this change seems to be contentious, I'll just remove it in the next revision.

Tom.



The wording in this context is Unicode specific; I agree we should not use UCS scalar value in a general sense.




A couple of comments on P2572R0
The 🤡 (U+1F921 CLOWN FACE) emoji has an estimated width of 2. The examples above that include that character illustrate the effect of the estimated width when that character is used as a fill character as opposed to when it is used as a formatting argument.

I had trouble understanding that, I would suggest including the information that 🤡 is of width 2 directly within the example or in a separate note directly below the example.

I'm not sure I'm following; why would relocating the note improve comprehension? Perhaps the note would read better as two paragraphs as below?

[ Note 3: If the width option is absent, then the field width is the estimated width of the formatted argument and the alignment option has no effect. If the estimated width of the formatted argument matches or exceeds the field width, then both the alignment and width options have no effect.

The width of any fill character is assumed to be 1. The 🤡 (U+1F921 CLOWN FACE) emoji has an estimated width of 2. The examples above that include that character illustrate the effect of the estimated width when that character is used as a fill character as opposed to when it is used as a formatting argument. — end note ]


Because there are two completely different statements here.

The width of any fill character is assumed to be 1.

This is a general statement - I could even argue that it could be normative rather than a note. It's important

It's followed by a description of the example. In such a way that makes it looks like we are still making general statements.
If you don't want to attach that part to the note you could maybe rewrite

The width of any fill character is assumed to be 1. The 🤡 (U+1F921 CLOWN FACE) emoji has an estimated width of 2. The examples above that include that character illustrate the effect of the estimated width when that character is used as a fill character as opposed to when it is used as a formatting argument. 

The width of any fill character is assumed to be 1. The examples above that include that character illustrate the effect of the estimated width when that character is used as a fill character as opposed to when it is used as a formatting argument. The 🤡 (U+1F921 CLOWN FACE) emoji has an estimated width of 2.  

At least that makes it clear that the statement about the emoji is related to the example.
It's a detail but i had to read that paragraph to understand why we were suddenly talking about a clown

But my preference would be

The align specifieroption applies to all argument types. The meaning of the various alignment options is as specified in Table 64
The width of any fill codepoint is 1.
[ Note 3: If the width option is absent, then the field width is the estimated width of the formatted argument and the alignment option has no effect. If the estimated width of the formatted argument matches or exceeds the field width, then both the alignment and width options have no effect.]

[ Example ]

[Note 4:  The examples above that include that character illustrate the effect of the estimated width when that character is used as a fill character as opposed to when it is used as a formatting argument. The 🤡 (U+1F921 CLOWN FACE) emoji has an estimated width of 2. ]


 



" The examples above that include that character illustrate the effect of the estimated width when that character is used as a fill character as opposed to when it is used as a formatting argument." is probably superfluous and could be omitted.
Hmm, I think that note is helpful for explaining the rather subtle reason that format("{:🤡^6}", "x") and format("{:*^6}", "🤡🤡🤡") produce the results that they do.

Fair enough! 

Do we need examples with extended grapheme clusters?
ie 🐻‍❄️ has 3 codepoints, width 1, but its use as fill-option is invalid

🐻‍❄️ is actually 4 code points (U+1F43B, U+200D, U+2744, U+FE0F) and has a width of 2 since its first code point is U+1F43B per [format.string.std]p11.

yes, sorry 4 codepoints, width 2.
 

There are such examples in the prose. I'm ambivalent with regard to whether we add such an example in the wording. If we do, I'm assuming something like the following would be what you want? (I substituted an EGC consisting of 2 code points with an estimated width of 1).

string s9 = format("{:^6}", "x");       // ill-formed; is U+0065, U+0301.
string s10 = format("{:*^6}", "
");    // value of s10 is "***"


Yes, exactly mirroring the clown.
It would illustrate both EGC for the string arguments being counted as one and show that they are ill-formed as a fill argument.
I would however strongly prefer an emoji sequence, not to bring normalization confusion in the mix.
 
Thanks,
Corentin



The rest looks great to me

Thank you.

Tom.


Thanks,
Corentin


[1] Unicode Standard 14.0 - 2.7 Unicode Strings
Whenever such strings are specified to be in a particular Unicode encoding form—even
one with the same code unit size—the string must not violate the requirements of that
encoding form. For example, isolated surrogates in a Unicode 16-bit string are not allowed
when that string is specified to be well-formed UTF-16. A number of techniques are available for dealing with an isolated surrogate, such as omitting it, converting it into U+FFFD
replacement character to produce well-formed UTF-16, or simply halting the processing of the string with an error. (See Section 3.9, Unicode Encoding Forms.)