Mike, can you please confirm receipt of this issue request?

Tom.

On 7/2/20 5:22 PM, Tom Honermann via Core wrote:
>
> This issue was originally reported in
> https://lists.isocpp.org/core/2019/03/5770.php, but it looks like a
> core issue was never recorded. This request obsoletes the previous one.
>
> (Mike, could we please, pretty please with a cup of moonshine and a
> plate of chocolate truffles, get a new public facing publication of
> the active issues list? At present, we can't rely on
> https://wg21.link directing users to a paper with recently created
> issues or up to date status information)
>
> [lex.phases] <http://eel.is/c++draft/lex.phases> states that string
> literals are converted to the execution character set prior to being
> concatenated:
>
>> 5.Each basic source character set member in a /character-literal/ or
>> a /string-literal/, as well as each escape sequence and
>> /universal-character-name/ in a /character-literal/ or a non-raw
>> string literal, is converted to the corresponding member of the
>> execution character set ([lex.ccon]
>> <http://eel.is/c++draft/lex.ccon>, [lex.string]
>> <http://eel.is/c++draft/lex.string>); if there is no corresponding
>> member, it is converted to an implementation-defined member other
>> than the null (wide) character.
>> 6.Adjacent string literal tokens are concatenated.
> However, [lex.string]p11 <http://eel.is/c++draft/lex.string#11>
> specifies that encoding conversion is dependent on the presence of
> /encoding-prefix/es on the set of string literals that will be
> concatenated:
>
>> 11 In translation phase 6 ([lex.phases]
>> <http://eel.is/c++draft/lex.phases>), adjacent /string-literal/s are
>> concatenated. If both /string-literal/s have the same
>> /encoding-prefix/, the resulting concatenated /string-literal/ has
>> that /encoding-prefix/. If one /string-literal/ has no
>> /encoding-prefix/, it is treated as a /string-literal/ of the same
>> /encoding-prefix/ as the other operand. If a UTF-8 string literal
>> token is adjacent to a wide string literal token, the program is
>> ill-formed. Any other concatenations are conditionally-supported
>> with implementation-defined behavior. [/Note:/ This concatenation is
>> an interpretation, not a conversion. Because the interpretation
>> happens in translation phase 6 (after each character from a
>> string-literal has been translated into a value from the appropriate
>> character set), a /string-literal/'s initial rawness has no effect on
>> the interpretation or well-formedness of the concatenation. — /end
>> note/ ] Table 11 has some examples of valid concatenations.
>>
>> Table 11 <http://eel.is/c++draft/lex.string#tab:lex.string.concat>:
>> String literal concatenations
>> [tab:lex.string.concat]
>>
>>
>> Source Means
>> u"a" u"b"
>> u"a" "b"
>> "a" u"b"
>>
>>
>> u"ab"
>> u"ab"
>> u"ab"
>>
>>
>> Source Means
>> U"a" U"b"
>> U"a" "b"
>> "a" U"b"
>>
>>
>> U"ab"
>> U"ab"
>> U"ab"
>>
>>
>> Source Means
>> L"a" L"b"
>> L"a" "b"
>> "a" L"b"
>>
>>
>> L"ab"
>> L"ab"
>> L"ab"
>>
>>
>>
>> Characters in concatenated strings are kept distinct.
>>
>> [/Example:/
>>
>> "\xA" "B"
>>
>> contains the two characters '\xA' and 'B' after concatenation (and
>> not the single hexadecimal character '\xAB'). — /end example/ ]
>
> The intent expressed by the prose and the note seems to be that string
> literals with different /encoding-prefix/es be separately converted
> and then joined together, such that the resulting string literal
> potentially consists of code unit sequences corresponding to different
> character encodings. However, that conflicts with the intent expressed
> by the table that specifies that, for example, `u"a" "b"` means the
> same as `u"ab"`.
>
> There is implementation divergence. GCC and Clang implement the
> intent expressed in the table, but Visual C++ implements ... something
> else.
>
> The difference is illustrated at https://msvc.godbolt.org/z/Dcrgda
> using the code below:
>
>> const char8_t* u8_1 = "" u8"\u0102";
>> const char8_t* u8_2 = u8"" "\u0102";
>> const char8_t* u8_3 = u8"" u8"\u0102";
>>
>> const char16_t* u16_1 = "" u"\u0102";
>> const char16_t* u16_2 = u"" "\u0102";
>> const char16_t* u16_3 = u"" u"\u0102";
>>
>> const char32_t* u32_1 = "" U"\u0102";
>> const char32_t* u32_2 = U"" "\u0102";
>> const char32_t* u32_3 = U"" U"\u0102";
>>
>> const wchar_t* w_1 = "" L"\u0102";
>> const wchar_t* w_2 = L"" "\u0102";
>> const wchar_t* w_3 = L"" L"\u0102";
>
> GCC and Clang produce the same sequence of encoded code units for each
> combination of /encoding-prefix/ encoded according to the
> /encoding-prefix/. Visual C++ produces the same code unit sequence
> for the *_1 and *_3 variants, but not for the *_2 variants; the latter
> have their second string literal component first encoded to the
> execution character set (substituting a '?' character if the character
> is not representable in that character set), and then re-encodes that
> code unit sequence (interpreting it according to the current locale
> the compiler is operating in) using the encoding indicated by the
> /encoding-prefix/ of the other string literal component. The double
> encoding behavior of the Visual C++ compiler, and in particular the
> use of the wrong source encoding for the second conversion, are
> presumably unintended behaviors.
>
> In addition to the above, translation phase 5 states that conversion
> is unconditionally to the execution character set, but that is
> obviously incorrect for string literals that have an /encoding-prefix/
> present.
>
> Swapping translation phases 5 and 6 is not a viable resolution because
> that would interfere with processing of escape sequences. `"\33" "3"`
> is not intended to be the same as `"\333"` and separate processing of
> string literal components is necessary in some cases involving
> /hexadecimal-escape-sequence/. While the /octal-escape-sequence/ case
> can be worked around with `"\0333"`, the string intended by `"\xab"
> "c"` cannot be represented as a single string literal.
>
> Richard Smith suggested a possible resolution at
> https://groups.google.com/a/isocpp.org/forum/#!msg/std-discussion/qYf6treuLmY/dljWwyawCwAJ:
>
>> Swapping phase 5 and 6 is certainly wrong. See the example in
>> [lex.string]p12:
>>
>> "[Example:
>> "\xA" "B"
>> contains the two characters ’\xA’ and ’B’ after concatenation (and not
>> the single hexadecimal character
>> ’\xAB’). — end example]
>>
>> If you concatenate and then interpret escape sequences, you
>> misinterpret escape sequences.
>>
>> Instead, I think we should remove phases 5 and 6 entirely, parse one
>> or more string-literal tokens as a string literal expression, and only
>> perform the translation from the contents of the string literal tokens
>> into characters in the execution character set as part of specifying
>> the semantics of a string literal expression.
>
> Tom.
>
>
> _______________________________________________
> Core mailing list
> Core_at_[hidden]
> Subscription: https://lists.isocpp.org/mailman/listinfo.cgi/core
> Link to this post: http://lists.isocpp.org/core/2020/07/9537.php