Copying Eugene since, I think, he isn't subscribed to the SG16 mailing list. Eugene, feel free to contribute any thoughts you have; we're always happy to have yet another participant that has implemented UTF-8 encoding/decoding! :)

I agree that we should follow the Unicode/W3C/WhatWG guidance as a default behavior (possibly with additional opt-in error handling behaviors).

Corentin, with regard to the issue of reverse decoding, I'm not sure I understand the concern. This code:

equals(invalid_utf_view, reverse(reverse(invalid_utf_view)))

isn't valid since, assuming reverse is ranges::reverse(), the inner call returns an iterator and the outer call would then need a sentinel argument.

It also isn't clear to me whether invalid_utf_view is a range of (decoded) code points (in which case the reverse() calls operate on code points) or a range of (invalid) code units (in which case there seems to be no decoding taking place)

Can you clarify the example and the concern?

I would expect a forward decode and a reverse decode to implement the same substitution policy; basically, the reverse decode should behave as-if each code point is decoded by first iterating to the lead code unit of a well-formed sequence or to the first code unit of an invalid code unit sequence (either at the beginning of input, or immediately following a well-formed sequence), then decoding in the forward direction, and then reversing again to one before the initially identified code unit. That sounds inefficient, but I don't think it needs to be (at least, not for the happy path).

Perhaps I'm missing something, but I can't think of a scenario in which any of the three policy options described in PR-121 would (or should) produce different substitution results.

Tom.

Note that the current 'common practice' is noted by Unicode 15 https://www.unicode.org/versions/Unicode15.0.0/ch03.pdf  section 3.9  U+FFFD Substitution of Maximal Subparts which is also the Web encoding standard recommendation (which refers to an older rec from Unicode)  : http://www.w3.org/TR/encoding

We should probably follow the interop guidance. The Unicode doc has a few test cases for various flavors of ill-formed. 

On Fri, Feb 24, 2023 at 9:47 AM Zach Laine via SG16 <sg16@lists.isocpp.org> wrote:

I managed to get the same sequence forward and backward, even in error
cases.  I have tests that cover this exact situation:

https://github.com/tzlaine/text/blob/master/test/utf8.cpp#L784

I think it's appropriate for users to expect that a sequence be the
same, no matter which direction you traverse it.

Zach

On Fri, Feb 24, 2023 at 3:53 AM Corentin <corentin.jabot@gmail.com> wrote:
>
> Hey
>
> There is no technical challenge in decrementing a UTF-8 iterator that is well-formed.
>
> As eluded to in https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p2728r0.html,
> When the UTF-8 sequence is ill-formed, we need to keep track of the beginning of the range to not go before the start (note that views hide that complexity from users).
>
> But the other thing that we have not talked about in the case of ill-formed UTF-8, is that, if the error policy is to substitute the replacement character,
> then we (probably?) can't guarantee that  equals(invalid_utf_view, reverse(reverse(invalid_utf_view))), as different subsequences may be substituted.
> This leaves us with the options of either documenting it, and hoping nothing breaks, or adding a constraint on operator-- that the error policy is not the default substitution policy.
>
> (I don't think we should consider making decoding not bidirectional, it's just too useful, ie to search a grapheme from the end, for example)
>
> Anyway, that thought is now in the record :)
>
> (This was brought to my attention by Eugene Gershnik, the author of this decoding algorithm, which i use in my implementation https://gershnik.github.io/2021/03/24/reverse-utf8-decoding.html )
--
SG16 mailing list
SG16@lists.isocpp.org
https://lists.isocpp.org/mailman/listinfo.cgi/sg16