On 1/29/21 2:57 PM, Corentin wrote:
>
>
> On Fri, Jan 29, 2021 at 8:00 PM Tom Honermann <tom_at_[hidden]
> <mailto:tom_at_[hidden]>> wrote:
>
> On 1/29/21 4:27 AM, Corentin wrote:
>>
>>
>> On Fri, Jan 29, 2021 at 9:39 AM Jens Maurer <Jens.Maurer_at_[hidden]
>> <mailto:Jens.Maurer_at_[hidden]>> wrote:
>>
>> On 29/01/2021 08.20, Tom Honermann wrote:
>> > On 1/28/21 1:57 PM, Jens Maurer via SG16 wrote:
>> >> On 28/01/2021 19.37, Corentin via SG16 wrote:
>> >>> On Thu, Jan 28, 2021 at 7:22 PM Peter Brett
>> <pbrett_at_[hidden] <mailto:pbrett_at_[hidden]>
>> <mailto:pbrett_at_[hidden] <mailto:pbrett_at_[hidden]>>> wrote:
>> >>>
>> >>> I think the big problem here is trying to make it a
>> template.____
>> >>>
>> >>> __ __
>> >>>
>> >>> Make it named. It’s literally not possible to use
>> this correctly in generic code.
>> >>>
>> >>>
>> >>> Question then is do we want to solve the issue for wchar_t?
>> >>> Because having the name of the encoding in the function
>> kinda precludes that - the sizeof(wchar_t) being platform
>> dependant
>> >> You only get away with char* -> char8_t* because "char"
>> has special
>> >> aliasing exceptions.
>> >>
>> >> You'll get the full set of aliasing concerns for
>> >> wchar_t* -> char16_t* or char32_t*
>> >
>> > I think what we're looking for is a portable solution for
>> this ICU hack
>> <https://github.com/unicode-org/icu/blob/master/icu4c/source/common/unicode/char16ptr.h#L30-L36>
>> (generalized to make it work for [unsigned] char* conversion
>> to char8_t*); the goal being to enable some form of explicit
>> restricted pointer interconvertibility between same
>> sized/aligned types.
>>
>> Yes.
>>
>> > I don't understand the ICU hack sufficiently well to relate
>> it to a memory or object model. I'm also not sure that it
>> actually works (though it may suffice for the scenarios that
>> are encountered in practice).
>> >
>> > Perhaps something like this would suffice.
>> >
>> > template<typename To, typename From>
>> > requires requires {
>> > requires std::is_trivial_v<To>;
>> > requires std::is_trivial_v<From>;
>> > requires sizeof(To) == sizeof(From);
>> > requires alignof(To) == alignof(From);
>> > }
>> > To* alias_barrier_cast(From *p) {
>> > asm volatile("" : : "rm"(p) : "memory");
>> > return reinterpret_cast<To*>(p);
>> > }
>>
>> From a C++ memory model perspective, there is no difference
>> between,
>> say, char16_t and short or int: They form their own aliasing
>> domain.
>> Converting the pointer with reinterpret_cast or something is NOT
>> the problem; the problem is accessing the data before and after.
>>
>>
>> The solution I'm thinking of is that the data can only be
>> accessed through the returned pointer after the function call.
>> Do you think that is more workable?
>
> I was thinking in terms of a similar model as well, but if a
> returned pointer is required, then existing interfaces that don't
> return such a pointer won't be usable.
>
> Is there any?
> However, I've been thinking since then that some libraries may access
> (read) the memory through the original pointer, which would be
> undefined and the user wouldn't be able to tell.

Right. The issues aren't limited to just read access though either. I
had this kind of example in mind.

    // In lib X
    int fill_buffer(char *p, size_t n);

    // In lib Y
    void search_replace(char8_t *p, size_t n, const char8_t *search,
    const char8_t *replace);

    // Meanwhile, in lib Z.
    void f() {
       char8_t buffer[256];
       size_t n = fill_buffer(alias_barrier_cast<char>(buffer), 256);
       search_replace(buffer, 256, u8"bad text", u8"good text");
    }

The goal of the alias barrier here is to inform the compiler that it
needs to forget everything it thinks it knows about the contents of
buffer (which, in this case, means forgetting that it is uninitialized).

> I've been trying to construct a test case that demonstrates that
> reinterpret_cast doesn't suffice, but I have so far been unable to
> coerce an optimizer into taking advantage of anti-aliasing rules.
> I've tried gcc and clang with -O3 and -fstrict-aliasing, but it
> looks like if the optimizer's escape analysis finds that an
> address escapes, regardless of by which type, then it assumes the
> object may have been modified (unless declared const). I've been
> trying with various types, not just ones that involve the aliasing
> friendly types.
>
> If anyone can suggest or identify a test that demonstrates that
> reinterpret_cast doesn't suffice, that might be helpful for
> testing other solutions.
>
>>
>> There have been papers in the past that attempted to bless
>> regions of memory with a different data type (e.g. to deal with
>> mmapped file data); I think such a direction might be worthwhile
>> to investigate.
>>
>>
>> I certainly don't want to deal with a "solution" that covers
>> char8_t / char16_t / char32_t only, if the underlying concerns
>> are also applicable elsewhere.
>>
>>
>> Even if there was a use case for a generalized solution, we need
>> to do something specific for this as we have additional
>> preconditions, namely that the input is a well-formed sequence of
>> utf code units.
>
> I'm not sure where this requirement comes from. I think the only
> solution we need is one that permits access to a [unsigned] char
> object via a char8_t lvalue (in limited cases) without provoking
> UB. Whether the data is well-formed UTF-8 is an orthogonal
> concern that is more in the domain of contracts.
>
>
> Yes, there are two different concerns here, and I'm not sure how
> intertwined there are.
> Hopefully, we can agree that none of us want to walk back on char8_t's
> design.
> I think we (you) did the right call in C++ not to make it alias char
> or make it otherwise implicitly convertible.
>
> And I think we can also agree that the intent of char8_t, char16_t,
> char32_t is to store UTF data.
> We shouldn't weaken that by saying that we put no precondition
> whatsoever on conversion from char to char8_t.
> Libraries and users dealing in char8_t should be allowed to assume
> that char8_t is supposed to hold utf-8 data. otherwise char8_t is just
> a weird way to spell uint8_t.
Yes, I agree with the above.
>
> If that's true, then casting any random bytes to a sequence of char8_t
> would be very surprising for users.
> Therefore, I think it's reasonable to put that extra condition in. I
> don't think we want to give the impression that char and char8_t are
> interchangeable or represent the same platonic values.
I agree in principle, but I do think a distinct mechanism should be used
to affirm kind of data vs well-formedness of data. Take the example
above; the buffer is uninitialized, so nothing can be required of its
contents when passed to fill_buffer().
>
> To be clear, I absolutely agree this is indeed a very different
> concern of how the conversion can be made to work in the memory model.
>
> The two concerns then are:
>
> * How do we make this work
> * What interface do we present to the user, and with what preconditions
>
Those are indeed the right questions :)
>
> I had a very interesting chat with Peter this afternoon where he
> argued that we should only provide this kind of
> cast_from_utfN_unchecked function if there is also a cast_from_utfN
> function which would do validation.
>
> The conversion we are talking about, while necessary, is a giant foot
> gun and I think we should treat it as such.
> It would be very easily misused by people who don't fully understand
> Unicode or how C++ handles text.

I agree it is a potential foot gun, depending on what it turns out to be.

Perhaps it is useful to think about the ICU hack. Can it be misused?
Or, rather, can incorrect use of it create an illusion of safety where
there is none? I think the answer is probably yes.

Tom.