Thank you, Jens and Hubert for this further discussion.

I think these are important points for the paper to address. However, I don't think they materially affect the design intent, so I'm not inclined to revisit the SG16 consensus. Please let me know if you feel this is new information that warrants another trip through SG16.

Corentin, I suggest doing the following:

Apologies if some of this is already present in the latest revision. I haven't re-read the entire paper.

Tom.

On 10/7/21 11:30 AM, Hubert Tong via SG16 wrote:
On Thu, Oct 7, 2021 at 3:21 AM Jens Maurer <Jens.Maurer@gmx.net> wrote:
On 07/10/2021 03.24, Hubert Tong via SG16 wrote:
> For information, since interest was expressed in today's meeting.
>
> Wide characters are mostly a C/C++ invention. For EBCDIC encodings that do not have multibyte characters, the wide encoding of a character consists of the unsigned char value of the character in a wchar_t.
>
> EBCDIC also has multibyte encodings. These are formed by pairing single-byte encodings and double-byte encodings. The unification of single-byte and double-byte encodings into a multibyte, stateful "narrow" encoding is achieved using shift-out/shift-in.
>
> The wide encoding of a character from a multibyte EBCDIC encoding is as described above for a character from the single-byte component encoding. For a character from the double-byte component encoding, the wide encoding of a character consists of the value obtained by using the first byte of the double-byte character as the upper 8 bits of a 16-bit value and the second byte as the lower 8 bits.

So, we have a situation similar to UTF-16 here, I guess:

The EBCDIC wide encoding uses 16-bit code units (integer values of
type wchar_t).

Except wchar_t is 32 bits for 64-bit processes.
 
I'm reading your text that "upper bits" means
"bits in the 16-bit integer", which would then be mapped to
the object representation in some hardware endianness-dependent
way.

That's a point that I am not sure how to test. All implementations that I am aware of only produce programs that operate in big endian environments. It is potentially an open design decision whether program or file portability is more important. Note that a focus on file portability would also mean that the single-byte EBCDIC encodings would have the narrow and wide characters differ in value on little-endian systems. I believe it is safe to say that there are more programs in general than programs that try to interchange wide characters via byte streams/files.
 

Can you point to IANA entries that designate EBCDIC wide encodings?
(My guess is that the IANA entries designate multibyte shift-state
encodings, not wide encodings, so maybe we should invent a
recommendation what the derived naming convention should be.)

Your guess is correct. The paper's EBCDIC example is somewhat the invented derived naming convention (although the last version I saw in the paper did not indicate the width).
 

We want UTF-16 in wchar_t to be represented by "UTF16" without LE/BE.
By analogy, we should want wide-EBCDIC to be represented by some
name that does not offer an opinion on hardware endianness.

Meaning that the invented names are meant to represent "native endian". This appears consistent with the "program portability" direction.
 

Another question: What happens if wchar_t is 32-bits on a CHAR_BITS==8
platform and uses UTF-16, using two wchar_ts for some code points?
If we go for the "object representation iconv-compatible" model,
we can't say this is is UTF-16, because the object representation
has stray 0 bytes.

Yes.
 

So, it seems our implementation recommendations need to somehow
express that "UTF16" can only be returned if sizeof(wchar_t) == 2.
But it seems very plausible to also return "UTF16" if CHAR_BITS == 16
and sizeof(wchar_t) == 1.

We could try to go with non-normative explanations of intent regarding I/O serialization.


We're not yet at the bottom of this, sorry.

Jens