On Tue, Apr 27, 2021 at 5:57 AM Tom Honermann <tom@honermann.net> wrote:
On 4/26/21 1:04 PM, Corentin Jabot via SG16 wrote:


On Mon, Apr 26, 2021 at 6:19 PM Tom Honermann via SG16 <sg16@lists.isocpp.org> wrote:
On 4/19/21 10:58 AM, Tom Honermann via SG16 wrote:

SG16 will hold a telecon on Wednesday, April 28th at 19:30 UTC (timezone conversion).

The agenda is:

LEWG discussed P2093R5 at their 2021-04-06 telecon and decided to refer the paper back to SG16 for further discussion.  LEWG meeting minutes are available here; please review them prior to the telecon.  LEWG reviewed the list of prior SG16 deferred questions posted to them here.  Of those, they established consensus on an answer for #2 (they agreed not to block std::print() on a proposal for underlying terminal facilities), but referred the rest back to us.  My interpretation of their actions is that LEWG would like a revision of the paper to address these concerns based on SG16 input (e.g., discuss design options and SG16 consensus or lack thereof).  We'll therefore focus on these questions at this telecon.

Hubert provided the following very interesting example usage.

std::print("{:%r}\n", std::chrono::system_clock::now().time_since_epoch());

At issue is the encoding used by locale sensitive chrono formatters.  Search [time.format] for "locale" to find example chrono format specifiers that are locale dependent.  The example above contains the %r specifier and is locale sensitive because AM/PM designations may be localized.  In a Chinese locale the desired translation of "PM" is "下午", but the locale will provide the translation in the locale encoding.  As specified in P2093R5, if the execution (literal) encoding is UTF-8, than std::print() will expect the translation to be provided in UTF-8, but if the locale is not UTF-8-based (e.g., Big5; perhaps Shift-JIS for the Japanese 午後 translation), then the result is mojibake. This is a good example of how locale conflates translation and character encoding.

Addressing the above will be our first order of business.  Please reserve some time to independently think about this problem (ignore responses to this message for a few days if you need to).  I am explicitly not listing possible approaches to address this concern in this message so as to avoid adding (further) bias in any specific direction.  I suspect the answers to the previously deferred SG16 questions will be easier to answer once this concern is resolved.

Now that we've all had some time to think about this issue, here are some possible directions we can pursue to resolve it.  These are presented in no particular order.

  • Specialize std::locale facets and related I/O manipulators like std::put_time() for char8_t.  This would allow std::print() to, when the literal encoding is UTF-8, opt-in to use of the UTF-8/char8_t facets and I/O manipulators.
  • When the literal encoding is UTF-8, stipulate that running the program in a non-UTF-8 based locale is non-conforming.  This would effectively require MSVC programmers to, when building code with the /utf-8 option, to also force selection of a UTF-8 code page via a manifest and require use of Windows 10 build 1903 or later.
  • When the literal encoding is UTF-8, specify that non-UTF-8 based locale dependent translations be implicitly transcoded (such transcoding should never result in errors except perhaps for memory allocation failures).
  • Drop the special case handling for the literal encoding being UTF-8 and specify that, when bypassing a stream to write directly to the console, that the output be implicitly transcoded from the current locale dependent encoding (whatever it is) to the console encoding (UTF-8). 

We have 2 things to explain to LEWG for print. And we do not need to operate change to the design, just to explain things to them in a terms they can understand (and they want to rely on our expertise which
implies consensus among ourselves)

1. It is always non-sense to interpret a string in encoding X when it is in fact not.
2. From there, if a string literal is in UTF-8, we HAVE to assume the execution encoding is also utf-8. Why rely on the literal encoding and not execution? it is resilient to call to setlocale and more efficient. Also, format strings are likely to be literals.
3. From there if that string is displayed on a terminal/console/screen/tty, it is text. So it has to be rendered correctly. On a specific system (windows) there is a way to enforce that. Because windows has a separate mechanism for unicode display and console handling that exists independently of the C++ execution encoding.
4. "we have to assume" in 2. implies a precondition. That is true REGARDLESS of utf-8 or not. in all cases the format string has to be interpreted as text, which assumes it is valid in the execution encoding. CF the Microsoft STL issue for braces in shift JS.
5. This means that converting to UTF-16 on windows for the purpose of console display is always valid (no ""transcosding"" error) within the contract of the function, and as such does not have to be specified. Preconditions violations are UB within the standard library and we should keep doing that. In practice the implementation (which is here the terminal, not the stl) will do character replacement the best it can, or render something horrible.

I agree with all of that, but I don't see how it relates to the problematic example above.  The issue with the example is that the "%r" field specifier may cause non-UTF-8 content supplied by the locale to be written.

I see two problems here.
One is that this should not be locale dependent by default - has that been discussed? It seems to run amok of fmt design.

The other is that, if print("xxx{}", foo) assumes that xxx is utf8, and the formated result is displayed onto a terminal, then the entire thing _has to_ be utf-8. note that this is because of
a precondition on the act if displaying on the terminal which has nothing to do with formatting it's a 2 step process format -> print on terminal both of which have different preconditions (formating puts a requirement on the format string, to parse it, print additionally puts preconditions that the resulting  thing will be utf8 such that individual arguments have to be to.
 


The locale in there is a red herring. Changing the execution encoding is always dicey - all  strings that were correctly interpreted correctly before the locale change are potentially no longer
correctly interpreted because their encoding no longer matches the new execution encoding.
The existence of a setlocale function doesn't imply that calling it leads to sensible results if the locale change also changes the encoding :)
The example doesn't assume a locale change, at least not beyond an initial std::setlocale(LC_ALL, "") during program startup.


> Specialize std::locale facets and related I/O manipulators like std::put_time() for char8_t.  This would allow std::print() to, when the literal encoding is UTF-8, opt-in to use of the UTF-8/char8_t facets and I/O manipulators.

This is a different issue, one Peter and I have discussed. we should not try to shove char into char8_t. Both char8_t and utf-8 char are valid use cases. Also, the whole point of fmt::print is to avoid all of that :)

I think this is strongly related, or we are misunderstanding each other.  I see the point of std::print() being to bypass the implicit (wrong) console transcoding.

fmt::print just dumps the bytes in the general case, similarly to printf, that is then interpreted incorrectly by the windows console. I don't see where there might be transcoding
in the program (I expect the console to do interesting things, but that's outside of C++).

C++ thinks a string is Utf-8
System (incorrectly) disagrees
System has a method that allows it to agree
Do we use that method?

I strongly agree that char8_t and UTF-8 char are valid use cases.


> When the literal encoding is UTF-8, stipulate that running the program in a non-UTF-8 based locale is non-conforming.  This would effectively require MSVC programmers to, when building code with the /utf-8 option, to also force selection of a UTF-8 code page via a manifest and require use of Windows 10 build 1903 or later.

If you program contains literals that are not correctly interpreted by the execution encoding, the behavior of your program cannot be correct <insert scary U word>. So they should probably do that but it seems out of scope.
The literalS encoding and the execution encoding should be consistent (each string literal should be correctly interpreted).

> When the literal encoding is UTF-8, specify that non-UTF-8 based locale dependent translations be implicitly transcoded
Sorry, can you detail what you mean? I do not understand, sorry
In the example above, the "%r" field specifier indicates that a locale dependent 12-hour clock time be formatted.  The AM/PM designator to be formatted is locale dependent.  If the locale is not UTF-8 based, then mojibake is produced (if the literal encoding is UTF-8).  This suggestion addresses the problem by implicitly transcoding the locale dependent AM/PM designator from the locale encoding to UTF-8 when formatting the output.

Think about cases in which that can happen
There is a non-utf8 locale and a utf8 string literal mixed together.
 

> Drop the special case handling for the literal encoding being UTF-8 and specify that, when bypassing a stream to write directly to the console, that the output be implicitly transcoded from the current locale dependent encoding (whatever it is) to the console encoding (UTF-8). 

Dropping the special case seems more difficult in terms of wording.
I think it is simpler actually; we would just have to say that the implicit transcoding is from the locale encoding to the console encoding.

It's really hard to know what the console encoding is (it is a very microsoft specific thing), and the windows console basically have a wide (utf16) and narrow encoding (not sure it works exactly like that but it's a good enough model)
Transcoding in the general case might be worse.
A wording that encourages vendors to... encourage utf8 content to not be misinterpreted as something else might help but good luck wording that!
Especially as it needs to handle file redirection, etc
 
If everything else fails, Microsoft could do the sensible thing as a matter of QOL.

Agreed.

Tom.


 

Please feel free to comment on these, or additional, approaches before our meeting on Wednesday.

I think it would benefit LEWG if a revision of the paper presented each of these possibilities, the consequences, and the rationale (and hopefully SG16 consensus) for the proposed direction.

Tom.

I do not intend to time limit discussion of P2093R5 as I believe this is an important matter to resolve.  If we are able to complete discussion of P2093R5, then we'll discuss P2348R0.

Tom.



--
SG16 mailing list
SG16@lists.isocpp.org
https://lists.isocpp.org/mailman/listinfo.cgi/sg16