On Mon, Mar 1, 2021 at 10:24 AM Corentin via SG16 <sg16_at_[hidden]>
wrote:

> Hey folks!
> Last meeting we talked about the relation between the literal & execution
> encoding.
>
> I think there is pressure to solve this issue (encoding names, std::print,
> other features).
> In P2297, I suggested that we say the execution character set is a
> superset of the literal character set, such that any character in the
> literal character set results in the same code unit sequence
> whether it is encoded in the literal encoding or execution encoding.
>
> Hubert was concerned this was too restrictive because some ebcdic &
> iso 646 have codepoints reserved for "national symbols".
> Even Shift-JIS is not 100% ascii compatible (Yen instead of backslash,
> overline instead of tilde)
>
> I've been thinking about that over the past few days, I think the solution
> is to not have requirements on the literal character set but rather on the
> literals themselves.
>
> If the execution encoding is UTF8, "ABC" is interpreted identically
> whether its encoding is ASCII, ISO 646-IT, or Shift-JS.
>
> However, "C:\\" would be interpreted as "C:\\", "C:ç" and "C:¥"
> respectively.
>
> So we need to only put requirements on the content of individual literals
> rather than on the entiere literal set (which, P1885 non whistanding, is
> not observable during execution anyhow)
>
>
> *A way to word that:*
>
> The execution encoding is the locale-specific encoding used to interpret
> character and NTMBS parameters in character functions, multibyte characters
> functions and other locale-specific functions.
>
I think we can start from something like this. I am guessing that the
parallel treatment for wide strings is intended?


>
> If character literals and string literals used as arguments to character
> functions and locale specific functions do not represent the same sequence
> of abstract characters whether they are interpreted with the literal
> encoding or the execution encoding the behavior is undefined.
>
I don't know how common it is in practice, but deliberately having mojibake
(as seen in the source) strings is currently a possible way to represent
strings that are meant to be interpreted at runtime using a specific
encoding (without immediately raising UB).

Presumably the scope of this wording is meant to encompass cases where the
contents of character/string literals made their way into the arguments via
assignment/memcpy/etc.? I think this quickly degenerates to
"string/character arguments to locale sensitive functions are taken as
being in the locale-specific encoding (and may be processed in a manner
that does not match their appearance in the source)".

Also, having '\x5c' included in the UB is presumably unintended. The user
intent is expressed by the numeric escape. Furthermore, consideration
should be given (and documented) about whether the UB should apply to
"unparsed" strings (like the argument given to printf for %s) for "locale
sensitive" functions. Again though, I think that we're really talking about
there being "natural consequences" of defined behaviour when "bad input" is
involved.


>
> I hope that this resolves Hubert concerns and that we can refine the
> general idea and put that in a paper :)
>
> Have a great week,
> Corentin
>
>
>
>
>
>
> --
> SG16 mailing list
> SG16_at_[hidden]
> https://lists.isocpp.org/mailman/listinfo.cgi/sg16
>