On 11/8/22 12:10 PM, Thomas Köppe wrote:
> Hi Tom,
>
> Corentin and Jens had already informed me of this issue, and I talked
> to a few people, and we briefly discussed this in Core yesterday.
Ah, another discussion I'm sorry to have missed!
>
> On Tue, 8 Nov 2022 at 04:51, Tom Honermann <tom_at_[hidden]> wrote:
>
> The following NB comments have to do with difficulties we're
> facing in referring to Unicode features in the C++ standard. See
> also a rejected attempt to resolve these matters editorially here
> <https://github.com/cplusplus/draft/pull/5826>.
>
> * FR-010-133 <https://github.com/cplusplus/nbballot/issues/412>
> [Bibliography] Unify references to Unicode
> * FR-021-013 <https://github.com/cplusplus/nbballot/issues/423>
> 5.3p5.2 [lex.charset] Codepoint names in identifiers
>
> The issue that we are facing is that
>
> 1. ISO/IEC 10646 specifies only a portion of the features
> specified in the Unicode Standard, and
> 2. The C++ standard has normative dependencies on features from
> the Unicode Standard that are not specified by ISO/IEC 10646, and
> 3. Use of an ISO/IEC 10646 standard that is not aligned with a
> use of a Unicode Standard results in problems like that
> reported in FR-021-013.
>
> The ISO requires normative references to refer to an ISO standard
> when one is available as stated in section 10.2, Permitted
> referenced documents, in part 2 of the ISO/IEC Directives
> <https://www.iso.org/sites/directives/current/part2/index.xhtml#_idTextAnchor130>.
> That has so far been applied such that, when a Unicode feature is
> available in ISO/IEC 10646, we refer to that standard for that
> functionality; when not, we refer to the Unicode Standard. This
> has resulted in the situation that, as FR-010-133 states, we
> arguably reference, directly or indirectly, up to four distinct
> versions of the Unicode Standard.
>
> Referring to both ISO/IEC 10646 and the Unicode Standard creates a
> burden with regard to how to align those references in order to
> reference a consistent set of Unicode features. The Unicode
> Standard is released once per year. ISO/IEC 10646 is released
> every three years, but amendments are issued to align the current
> release with new Unicode Standards as they occur. Some clever
> wording, thanks to section 10.4, Undated references, of the
> ISO/IEC Directives
> <https://www.iso.org/sites/directives/current/part2/index.xhtml#_idTextAnchor134>,
> would permit an undated reference to ISO/IEC 10646 to apply to the
> most recent amendment of that standard and enable a similarly
> undated reference to the Unicode Standard such that the references
> are aligned. That would solve the problem to a certain degree, but
> 1) requires us to provide wording that makes the relationship
> between the standards sufficiently clear, 2) retains a burden on
> implementors of having to consult both standards, and 3) puts us
> in the middle of any discrepancies found between the two standards.
>
> SG16 discussed these issues during its 2022-11-02 telecon
> <https://github.com/sg16-unicode/sg16-meetings#november-2nd-2022>.
> A summary pertaining to these two NB comments is present in a
> GitHub issue comment for FR-010-133
> <https://github.com/cplusplus/nbballot/issues/412#issuecomment-1304696400>.
> SG16 took two polls, one of which demonstrated consensus for
> exploring whether we could discontinue referring to ISO/IEC 10646
> in favor of referring only to the Unicode Standard. For ease of
> reference, here is the poll:
>
> * Poll 3: [FR-010-133][FR-021-013]: SG16 requests that the
> project editor discuss with the ISO the option of eschewing
> references to ISO/IEC 10646 in favor of the Unicode Standard
> both for technical consistency and release frequency.
> o Attendees: 9 (1 abstention)
> o
> SF
> F
> N
> A
> SA
> 3
> 3
> 0
> 1
> 1
>
> The consensus was not unanimous; there is some demand for more
> rigorous specification as demonstrated in ISO/IEC 10646 relative
> to the Unicode Standard.
>
> That brings us to you, Thomas. As project editor, what options do
> you see? Is an argument that ISO/IEC 10646 is not (currently)
> suitable for our purposes due to its limited scope sufficient to
> replace references to it with references to the Unicode Standard?
> Is this a question that we would have to take up with the ISO
> directly and, if so, can we do so?
>
> I think a reasonable position for us is to first and foremost do what
> is right for the Standard: if we need the references to the Unicode
> Standard for completeness and correctness, we will cite that
> reference, and if for consistency it means that we should also use the
> Unicode Standard for things that would also be available from ISO
> 10646, but it is easier to specify and for implementers to follow if
> we take everything from the same reference, then I think we have a
> good and defensible case for "the absence of appropriate ISO or IEC
> documents": ISO 10646 is simply not appropriate for our purposes.
> Given that we are already citing the Unicode Standard normatively, I
> think we can be confident that all the other conditions in 10.2 are
> already met, and it sounds like that a possible outcome would be that
> we simply _remove_ the reference to ISO 10646, which would also be an
> improvement for our users (who would need to obtain fewer documents).
>
> At this point I'd be happy for us to simply proceed in this direction
> and assume it is acceptable. Worst case ISO could send us a DIS
> comment about this, and we could revisit the decision at that point.
> If you prefer, I can contact the ISO secretariat proactively and ask
> about this, but I don't think that is necessary at this point and
> would only do it if you explicitly wanted me to.
Thank you! I'll leave it to your judgement regarding proactive
interaction with the ISO secretariat. The worst case is that we make a
change, have to revert it, and then live with what we have now until we
choose another way to clean it up (and perhaps adopt that as a DR).
>
> Please note that in the Core discussion yesterday we noticed some
> consequences on which I have no opinion, but which you should probably
> take into careful consideration:
>
> * Allegedly the Unicode Standard does not define terms in the same
> way and at the same level of precision as ISO 10646 does, so if by
> changing references we would lose some external definitions that
> we are currently relying on, we would need to find a suitable
> alternative. (I don't know anything about that detail regarding
> either the Unicode Standard or ISO 10646.)
> * We thought that the term "UCS scalar value" is something we
> currently use from the undated ISO 10646 reference, and allegedly
> that term does not exist in the Unicode Standard.
>
Thank you, yes, we'll need to perform some audits.

Referring to the Unicode Standard might also permit dropping the dated
ISO/IEC 10646 reference we have for "UCS-2". The current Unicode
Standard still retains a definition for that term in section C.2,
"Encoding Forms in ISO/IEC 10646"; though, the definition there might be
deemed unsatisfactory for our uses.

> I'm not familiar with the subject matter, so I would appreciate that
> if you do make changes in the proposed direction, then please check
> all this carefully. Do let me know if you would also like me to check
> at the end, but it'd be great if SG16 could make sure that all terms
> are properly defined once we lose the undated reference to ISO 10646.

Yes, we will. SG16 will discuss this further at our next telecon
scheduled for November 30th.

>
> Let me know if you'd like to discuss anything else!

Thank you for the quick and thorough response!

Tom.

>
> Thomas