On Wed, 10 Jun 2020 at 01:39, Hubert Tong <hubert.reinterpretcast_at_[hidden]>
wrote:

> On Tue, Jun 9, 2020 at 7:12 PM Steve Downey <sdowney_at_[hidden]> wrote:
>
>> While I understand what you are asking for, and I agree it doesn't seem
>> unreasonable, I also don't see how that it works with the machinery today?
>>
> I am not saying that the C++ wording today works for this by the letter
> (except for heavy-handed interpretations of phase 1). I consider it to be a
> bug that it doesn't.
>
>
>> All characters outside the basic source character set are mapped to
>> universal-character-names that are named by Unicode scalar values.
>> We'd need a mechanism to get back to the completely untranslated original
>> source.
>>
>
I think we have that mechanism already.
We have a mapping source -> universal-character-names (which for your
interest is specified both by IBM and Unicode), and
the universal-character-names -> execution mapping, which again is fully
specified.
I think that is enough to do, if desired, a direct source -> execution
which is bytes preserving, as it is not observable whether it was done or
not.



> I think this is similar to how raw string literals need some sort of
> mechanism.
>
>
>>
>> On Tue, Jun 9, 2020, 18:32 Hubert Tong <hubert.reinterpretcast_at_[hidden]>
>> wrote:
>>
>>> On Tue, Jun 9, 2020 at 5:21 PM Corentin Jabot <corentinjabot_at_[hidden]>
>>> wrote:
>>>
>>>>
>>>>
>>>> On Tue, 9 Jun 2020 at 23:06, Hubert Tong <
>>>> hubert.reinterpretcast_at_[hidden]> wrote:
>>>>
>>>>> On Tue, Jun 9, 2020 at 4:59 PM Corentin Jabot <corentinjabot_at_[hidden]>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, 9 Jun 2020 at 22:17, Hubert Tong <
>>>>>> hubert.reinterpretcast_at_[hidden]> wrote:
>>>>>>
>>>>>>> On Tue, Jun 9, 2020 at 1:01 PM Corentin Jabot via SG16 <
>>>>>>> sg16_at_[hidden]> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, 9 Jun 2020 at 18:45, Steve Downey <sdowney_at_[hidden]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> One thing I have realized while working on identifiers is that
>>>>>>>>> after conversion from whatever the sources are, lexing and parsing are
>>>>>>>>> symbolic. That is, 'a' doesn't have a value until it's rendered into a
>>>>>>>>> literal. That is " The values of the members of the execution
>>>>>>>>> character sets and the sets of additional members are locale-specific
>>>>>>>>> . <http://eel.is/c++draft/lex.charset#3.sentence-5>"
>>>>>>>>> http://eel.is/c++draft/lex.charset#3.sentence-5 really only comes
>>>>>>>>> into play when rendering the "execution character set" into a characters or
>>>>>>>>> strings. The execution character set and the source character set exist in
>>>>>>>>> the same logical space right now, and the "source character set" isn't what
>>>>>>>>> is in source files today.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Yep, and they don't have to have a value either. identifiers are
>>>>>>>> not sorted etc.
>>>>>>>> Everything in lex is symbolic anyway the phases don't exist in
>>>>>>>> practice.
>>>>>>>> However, the international representation being isomorphic to
>>>>>>>> Unicode, it would be possible to describe in term of unicode with no
>>>>>>>> observable behavior change.
>>>>>>>>
>>>>>>> I would like to allow characters not present in Unicode within
>>>>>>> character literals, string literals, comments, and header names. More
>>>>>>> abstractly, I would like to allow source -> encoding-used-for-output
>>>>>>> conversion.
>>>>>>>
>>>>>>
>>>>>> Do you have an example of a use case you want to support?
>>>>>>
>>>>> I am still evaluating the round-trip mapping for EBCDIC.
>>>>>
>>>>
>>>> I believe Unicode -> EBCDIC round trip perfectly using the process
>>>> described in https://www.unicode.org/reports/tr16/tr16-8.html
>>>> The tricky part is the control characters, which this TR maps to the C1
>>>> unicode control characters
>>>>
>>> I'm not questioning the ability to round-trip. I am questioning the
>>> ability to avoid conflating certain EBCDIC control characters with certain
>>> C1 control characters. For example, it seems clear to me that U+0096 START
>>> OF GUARDED AREA and U+0097 END OF GUARDED AREA are paired in the intended
>>> usage, but the mapping of these to, respectively, Numeric Backspace and
>>> Graphic Escape does not retain semantic meaning. If such EBCDIC characters
>>> appear within a literal that should be encoded in a Unicode encoding, I
>>> find it potentially questionable if the string is considered well-formed. I
>>> have similar thoughts for the case where a UCN escape for such a C1 control
>>> character appears in a string that is to be encoded in EBCDIC.
>>>
>>> In other words, I do not consider the mapping (which is useful if you
>>> track out-of-band whether the data was originally EBCDIC or not) to
>>> establish the presence of the EBCDIC control characters in Unicode. These
>>> opinions do not necessarily represent those of IBM.
>>>
>>> -- HT
>>>
>>