On 4/26/21 1:56 PM, Peter Brett via
SG16 wrote:
There has
been very little interest in char*_t overloads though.
I’ve been thinking about this quite a bit
over the weekend (esp. during our impromptu Twitter-based SG16
meeting).
I’ve come to the conclusion that char8_t is
actually a solution in search of a problem. All you know when
you see a u8string is that its associated encoding is UTF-8,
but this isn’t actually very useful for anyone; you still have
to treat it as a bag of bytes of indeterminate encoding
because, by (lack of) contract, a u8string can contain any old
nonsense.
Some of this is true, but not all of it.
It is correct that a char8_t sequence or a std::u8string
may contain garbage. This is not unique to char8_t; it
is true for all of the other character types as well. It is also
true of Rust's std::string (see from_utf8_unchecked());
it is possible to put garbage in Rust's string types too. The
Rust standard library has a precondition that std::string
contains well-formed UTF-8. Rust's string type is much better
designed to avoid such garbage than std::basic_string
is; no one is going to claim otherwise.
But that lack of a strict well-formed guarantee does not prevent
solving real problems. The introduction of char8_t
solved a real problem in the standard library itself; it enabled std::path
constructor overloads to support UTF-8 and eliminated the std::u8path()
factory function workaround.
char8_t was never intended to solve all UTF-8 related
problems. It is intended to enable type safe use of UTF-8 without
accidentally mixing UTF-8 text with text in other encodings
(stored in char).
Anyone using char8_t for anything other than UTF-8
(except for the special case of creating ill-formed UTF-8 text for
the purposes of testing that such text is correctly
rejected/handled) is abusing the type. Full stop.
We can certainly add a type that enforces well-formed UTF-8 using
either char or char8_t; I suspect it would
look a lot like Rust's string type, including unchecked functions
that enable construction with garbage because we don't want to pay
the overhead of constant re-validation. Functions that except
text will always have to have a wide contract or a precondition on
well-formed text. That cannot be avoided.
Therefore, if I have the option to build my
code with UTF-8 as the literal encoding (and nearly everyone
has that option), all char8_t does is to provide me with two
mutually-incompatible ways to express “some unknown bytes that
might be text.”
The standard library does not have that option. Nor does a lot
of commercial software, including nearly all software that runs in
some particular ecosystems. There are 3-5 million C++ developers
(or whatever the current estimation is) and experiences vary
considerably. Please don't assume that the options available to
you or that your experience translates widely throughout the
community. I have personally worked on code bases that, for both
migration cost and technical reasons, would find switching to
UTF-8 literal encoding infeasible.
I’ve therefore decided not to spend any
more of my limited available committee time on any language or
library changes related to charN_t. Instead I think it would
be more productive for me to focus on adding features that
provide text codecs and validation (thanks, JeanHeyd!)
That is certainly your choice and your contributions are of
course welcome.
Tom.
Thanks again for all your valuable
contributions to this discussion, Victor!
Peter
> Corentin:
> If our requirements are ... - all of which I agree
with individually - I don't see a path forward.
I don't see any problem. Since we are
in the beginning of the C++23 cycle it makes sense to
tackle more difficult problems such as locale first.
Adding format overloads is not hard and makes more
sense to do later when we have a better understanding
and hopefully solution for locale issues.
> A non-broken Unicode locale
support calls for research and implementation experience
outside of the standard.
That's exactly what I intend to do,
contributors are welcome.
> Providing something that is
consistent and *not more* broken than the narrow
overload seems useful in the short term.
First I don't think it's necessary per my comment above
and second it will likely severely constrain our design
space for fixing these issues in the future.
> Just for clarity, what does
{fmt} currently do?
Unfortunately {fmt} cannot fix the
standard library so it does something completely crazy
in char*_t overloads, namely uses a narrow locale and
casts characters. There has been very little interest in
char*_t overloads though.
Hi
Victor,
This
is helpful, thank you. I will put full ‘L’ handling
for UTF-8/16/32 as the preferred option in the
paper.
Just
for clarity, what does {fmt} currently do?
Obviously if it currently does something different
then I will have to do some work to demonstrate
implementability.
Best
wishes,
Peter
>
What should the following code do?
I
think (1) is the only acceptable option
because all the rest are inconsistent with
existing std::format overloads.
>
“std::locale in its current form is pretty
much useless,” may be a true statement but it
doesn’t help me make progress.
Maybe
we are trying to make "progress" in the wrong
direction? We don't have to quickly hack
something together for new std::format
overloads. We didn't have a chance to look at
locale in C++20 but now is a great time.
>
Converting between UTF-X and UTF-Y is a
lossless operation.
Only
valid ones. There is still a question of
handling transcoding errors.
>
what is it that we gain by not allowing
format(u8"{}", u""); and format(u8"{}", U"");?
We
gain consistency between all std::format
overloads, simple specification, not having to
deal with transcoding errors. I am not
suggesting that it shouldn't be possible but
that it should be explicit, e.g.
format(u8"{}",
xcode(u""))
With
explicit approach you can easily configure
error handling.
>
we could provide only the u8 overload
Sure
provided that we have transcoding facilities.
I don't think u16 and u32 overloads are
particularly useful since you can't do much
with the result.
>
mandates that the existing locale be
specialize for char8_t
If
this specialization inherits all existing
locale problems then I think it's not a good
idea.
Talking
with Peter, we realized we could provide
only the u8 overload and mandates that the
existing locale be specialize for char8_t
-
Satisfy Victor's excellent remark about
the need not to be
gratuitously inconsistent
-
Put minimum strain on implementers
-
Let us move forward with having a
Unicode overload in 23.
--
SG16 mailing list
SG16@lists.isocpp.org
https://lists.isocpp.org/mailman/listinfo.cgi/sg16