On 12/4/21 3:26 AM, Corentin Jabot wrote:
>
>
> On Sat, Dec 4, 2021, 01:04 Tom Honermann <tom_at_[hidden]
> <mailto:tom_at_[hidden]>> wrote:
>
> On 12/3/21 4:47 PM, Corentin Jabot wrote:
>>
>>
>> On Fri, Dec 3, 2021, 22:03 Tom Honermann <tom_at_[hidden]
>> <mailto:tom_at_[hidden]>> wrote:
>>
>> On 12/1/21 2:28 PM, Corentin Jabot wrote:
>>>
>>>
>>> On Wed, Dec 1, 2021 at 8:13 PM Tom Honermann
>>> <tom_at_[hidden] <mailto:tom_at_[hidden]>> wrote:
>>>
>>> On 11/28/21 5:22 AM, Jens Maurer wrote:
>>>> On 28/11/2021 10.42, Corentin Jabot via SG16 wrote:
>>>>> On Sun, Nov 28, 2021, 01:31 Tom Honermann via SG16 <sg16_at_[hidden] <mailto:sg16_at_[hidden]> <mailto:sg16_at_[hidden]> <mailto:sg16_at_[hidden]>> wrote:
>>>>> 2. If the estimated width of the fill character is greater than 1, then alignment to the end of the available space might not be possible. The choice here is whether to under-fill or over-fill the available space. This possibility is avoided if fill characters are restricted to characters with an estimated width of exactly 1.
>>>>> std::format("{:🤡>4}", 123);
>>>>>
>>>>>
>>>>> Is there value in specifying it? Neither solutions are great nor terrible, i think saying unspecified would be fine, so would underfilling i guess.
>>>>>
>>>>> Hopefully, we are consistent and choose option 1 among those specified in the lwg issue
>>>>>
>>>>> For P2286R3<https://wg21.link/p2286r3> <https://wg21.link/p2286r3>, LEWG requested<https://lists.isocpp.org/sg16/2021/11/2845.php> <https://lists.isocpp.org/sg16/2021/11/2845.php> that SG16 consider the ramifications for support of user defined delimiters. We should also discuss the "?" specifier proposed to explicitly opt in to quoted and escaped formats for std::string, std::string_view, and arrays of char/wchar_t.
>>>>>
>>>>> Not sure the quoted thing is in our purview.
>>>>>
>>>>> For the delimiter, we should support codepoints, to be consistent with everything else. The issue is the we don't have experience with that afaik.
>>>> But the compile-time format string parser might not necessarily understand
>>>> the details of the literal encoding, so it's unclear how codepoints map to
>>>> code units. Or are you saying that the rest of std::format already requires
>>>> detailed understanding, anyway?
>>>
>>> I believe the compile-time format string parser is
>>> already required to understand such details. For
>>> example, if the literal encoding is Shift-JIS, then the
>>> parser would need to be able to differentiate byte
>>> values that appear as lead code units vs trailing code
>>> units (since, for example, a 0x5C code unit denotes the
>>> '\' character if it is a lead code unit, but that value
>>> may also appear as a trailing code unit for a double
>>> byte character).
>>>
>>> I think Jens is right. MSVC does handle Shift-JIS
>>> specifically but I'm not sure we can/should mandate
>>> something that work universally, the burden on
>>> implementation could be high)
>>
>> Are you suggesting that we should revisit the consensus for
>> the proposed resolution for LWG3576
>> <https://cplusplus.github.io/LWG/issue3576> from our
>> 2021-08-25 telecon
>> <https://github.com/sg16-unicode/sg16-meetings#august-25th-2021>?
>>
>>
>>
>> I am concerned about implementability
>> The current resolution calls for a compile time mechanism to read
>> a codepoint for arbitrary encoding.
>> Such mechanism currently doesn't exist.
>> For an implementation like GCC, the generic solution would be to
>> expose iconv facilities through builtins (the equivalent of mblen
>> or mbrtocX at least, i think, as Hubert pointed out)
>> This seems... A lot to ask in an issue resolution.
>> I don't remember if that was considered last time or if it
>> constitute new information in anyway but we might want to bring
>> that up again.
>
> I believe this requirement is already the status quo. Let me
> provide a better example than I did previously.
>
> std::format("<text>");
>
> If the literal encoding is not self-synchronizing then <text> may
> contain code units that correspond to the (single) code unit for
> '{' but that do not encode the '{' character. This can happen due
> to DBCS or shift-state encoding. An implementation needs to be
> able to recognize this case (for effected encodings) in order to
> avoid incorrectly interpreting the text as containing an
> introducer for a replacement field.
>
>
> I am well aware.
> I wonder if we understood that fully (compile time support and
> codepoint semantics were decision taken at about the same time
> independently of one another). I do not recall realizing that we were
> asking for full blown constexpr codepoint decode.
>
> I think I'd like to get input from implementers.
> If I understand this msvc PR, support for compile time non UTF-8 multi
> bytes encoding is limited
> https://github.com/microsoft/STL/pull/2221
> <https://github.com/microsoft/STL/pull/2221>
>
> I am not opposed to the direction to be clear, but I am reluctant to
> go further down this road without implementers support. We are asking
> a lot.

To be clear, your concern only has to do with the changes made via P2216
to require compile-time checking of format strings, yes? I don't see how
the question of custom delimiters makes the situation worse.

>
> For reasons, the work to add EBCDIC to clang has a home grown encoder,
> for example, as clang cares about environments where iconv is not present.
> This direction would likely, in addition to add constexpr builtins
> mandate that someone writes an EBCDIC -> utf decoder in clang or libc++.
>
> It makes me wonder if some of these features should be restricted to
> u8 formatting strings 😅

The PR suggests a way to reduce the burden; we could demote an invalid
format string from ill-formed to IFNDR so as to make this QOI rather
than a mandated diagnostic. Implementations would still have to diagnose
the bad format string at run-time in that case.

>
>
> It might turn out to be a non issue, but it's worth making sure we are
> all on the same page.
>
> Charlie, opinion?

I suggest starting a new message thread (with a suitable subject) that
copies relevant libstdc++ and libc++ maintainers as well as Charlie.

Tom.

>
> Tom.
>
>>
>> Tom.
>>
>>> I agree with Corentin that delimiters should be
>>> restricted to code points. That is consistent with the
>>> direction we have already advocated for fill characters
>>> in LWG3576 <https://cplusplus.github.io/LWG/issue3576>.
>>>
>>> Tom.
>>>
>>
>