On Tue, Jun 9, 2020 at 10:03 PM Corentin Jabot <corentinjabot_at_[hidden]>
wrote:

>
>
> On Wed, Jun 10, 2020, 03:38 Hubert Tong <hubert.reinterpretcast_at_[hidden]>
> wrote:
>
>> On Tue, Jun 9, 2020 at 9:03 PM Corentin Jabot <corentinjabot_at_[hidden]>
>> wrote:
>>
>>>
>>>
>>> On Wed, 10 Jun 2020 at 01:39, Hubert Tong <
>>> hubert.reinterpretcast_at_[hidden]> wrote:
>>>
>>>> On Tue, Jun 9, 2020 at 7:12 PM Steve Downey <sdowney_at_[hidden]> wrote:
>>>>
>>>>> While I understand what you are asking for, and I agree it doesn't
>>>>> seem unreasonable, I also don't see how that it works with the machinery
>>>>> today?
>>>>>
>>>> I am not saying that the C++ wording today works for this by the letter
>>>> (except for heavy-handed interpretations of phase 1). I consider it to be a
>>>> bug that it doesn't.
>>>>
>>>>
>>>>> All characters outside the basic source character set are mapped to
>>>>> universal-character-names that are named by Unicode scalar values.
>>>>> We'd need a mechanism to get back to the completely untranslated
>>>>> original source.
>>>>>
>>>>
>>> I think we have that mechanism already.
>>> We have a mapping source -> universal-character-names (which for your
>>> interest is specified both by IBM and Unicode), and
>>> the universal-character-names -> execution mapping, which again is fully
>>> specified.
>>> I think that is enough to do, if desired, a direct source -> execution
>>> which is bytes preserving, as it is not observable whether it was done or
>>> not.
>>>
>> It is round-trippable but at the cost of one-way (during compilation)
>> conversions that are not semantically preserving. Even these are
>> justifiable, but I think they deserve to be called out. Which is to say
>> that the paper should document that these concerns were considered and not
>> simply dismiss the issue.
>>
>
> Definitely, I will document that better, thanks for the feedback!
>
> '\u0096' becoming '\x36': I suppose this could be justified for the case
>> where the user application is expected to have its output subjected to
>> automatic conversion, e.g., via SSH to a non-EBCDIC terminal.
>>
>
> I think the other (implementation defined) strategy is to make it Ill
> formed as non representable.
> Ideally, the wording should leave enough wiggle room for EBCDIC platforms
> to make these decisions!
>
Thanks. Perhaps we can work in a note for the wording that the mapping may
be based on established conventions that are not semantically-preserving?


>
>> For the much rarer case of u'<0x36>' (character literal that, in the
>> physical source file, contains the EBCDIC control character) becoming
>> u'\u0096': I suppose this could be justified for the case where the user
>> source was originally non-EBCDIC, but subjected to conversion into EBCDIC.
>>
>
>>
>>>
>>>
>>>> I think this is similar to how raw string literals need some sort of
>>>> mechanism.
>>>>
>>>>
>>>>>
>>>>> On Tue, Jun 9, 2020, 18:32 Hubert Tong <
>>>>> hubert.reinterpretcast_at_[hidden]> wrote:
>>>>>
>>>>>> On Tue, Jun 9, 2020 at 5:21 PM Corentin Jabot <
>>>>>> corentinjabot_at_[hidden]> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, 9 Jun 2020 at 23:06, Hubert Tong <
>>>>>>> hubert.reinterpretcast_at_[hidden]> wrote:
>>>>>>>
>>>>>>>> On Tue, Jun 9, 2020 at 4:59 PM Corentin Jabot <
>>>>>>>> corentinjabot_at_[hidden]> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, 9 Jun 2020 at 22:17, Hubert Tong <
>>>>>>>>> hubert.reinterpretcast_at_[hidden]> wrote:
>>>>>>>>>
>>>>>>>>>> On Tue, Jun 9, 2020 at 1:01 PM Corentin Jabot via SG16 <
>>>>>>>>>> sg16_at_[hidden]> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, 9 Jun 2020 at 18:45, Steve Downey <sdowney_at_[hidden]>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> One thing I have realized while working on identifiers is that
>>>>>>>>>>>> after conversion from whatever the sources are, lexing and parsing are
>>>>>>>>>>>> symbolic. That is, 'a' doesn't have a value until it's rendered into a
>>>>>>>>>>>> literal. That is " The values of the members of the execution
>>>>>>>>>>>> character sets and the sets of additional members are locale-specific
>>>>>>>>>>>> . <http://eel.is/c++draft/lex.charset#3.sentence-5>"
>>>>>>>>>>>> http://eel.is/c++draft/lex.charset#3.sentence-5 really only
>>>>>>>>>>>> comes into play when rendering the "execution character set" into a
>>>>>>>>>>>> characters or strings. The execution character set and the source character
>>>>>>>>>>>> set exist in the same logical space right now, and the "source character
>>>>>>>>>>>> set" isn't what is in source files today.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Yep, and they don't have to have a value either. identifiers are
>>>>>>>>>>> not sorted etc.
>>>>>>>>>>> Everything in lex is symbolic anyway the phases don't exist in
>>>>>>>>>>> practice.
>>>>>>>>>>> However, the international representation being isomorphic to
>>>>>>>>>>> Unicode, it would be possible to describe in term of unicode with no
>>>>>>>>>>> observable behavior change.
>>>>>>>>>>>
>>>>>>>>>> I would like to allow characters not present in Unicode within
>>>>>>>>>> character literals, string literals, comments, and header names. More
>>>>>>>>>> abstractly, I would like to allow source -> encoding-used-for-output
>>>>>>>>>> conversion.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Do you have an example of a use case you want to support?
>>>>>>>>>
>>>>>>>> I am still evaluating the round-trip mapping for EBCDIC.
>>>>>>>>
>>>>>>>
>>>>>>> I believe Unicode -> EBCDIC round trip perfectly using the process
>>>>>>> described in https://www.unicode.org/reports/tr16/tr16-8.html
>>>>>>> The tricky part is the control characters, which this TR maps to the
>>>>>>> C1 unicode control characters
>>>>>>>
>>>>>> I'm not questioning the ability to round-trip. I am questioning the
>>>>>> ability to avoid conflating certain EBCDIC control characters with certain
>>>>>> C1 control characters. For example, it seems clear to me that U+0096 START
>>>>>> OF GUARDED AREA and U+0097 END OF GUARDED AREA are paired in the intended
>>>>>> usage, but the mapping of these to, respectively, Numeric Backspace and
>>>>>> Graphic Escape does not retain semantic meaning. If such EBCDIC characters
>>>>>> appear within a literal that should be encoded in a Unicode encoding, I
>>>>>> find it potentially questionable if the string is considered well-formed. I
>>>>>> have similar thoughts for the case where a UCN escape for such a C1 control
>>>>>> character appears in a string that is to be encoded in EBCDIC.
>>>>>>
>>>>>> In other words, I do not consider the mapping (which is useful if you
>>>>>> track out-of-band whether the data was originally EBCDIC or not) to
>>>>>> establish the presence of the EBCDIC control characters in Unicode. These
>>>>>> opinions do not necessarily represent those of IBM.
>>>>>>
>>>>>> -- HT
>>>>>>
>>>>>