C++ Logo

sg16

Advanced search

Re: [isocpp-sg16] UTF-8 support status

From: Yongwei Wu <wuyongwei_at_[hidden]>
Date: Wed, 8 Jul 2026 22:23:03 +0800
On Wed, 8 Jul 2026 at 14:28, Tiago Freire <tmiguelf_at_[hidden]> wrote:

> > In which way may my approach not be supported?
>
>
>
> In whatever which way in the future a decision needs to be made that is
> incompatible with your usage.
>
>
>
> Ex.
>
> Should string literals with large Unicode code points be allowed? Nope,
> those are UB.
>

Are you kidding me? When does C++ begin to break working code this way?

And there is never UB. [lex.string] <https://eel.is/c++draft/lex.string>
says (my emphasis):

The sequence of characters denoted by each contiguous sequence of
*basic-s-char* <https://eel.is/c++draft/lex.string#nt:basic-s-char>*s*,
*r-char* <https://eel.is/c++draft/lex.string#nt:r-char>*s*,
*simple-escape-sequence*
<https://eel.is/c++draft/lex.ccon#nt:simple-escape-sequence>*s* ([lex.ccon]
<https://eel.is/c++draft/lex.ccon>), and *universal-character-name*
<https://eel.is/c++draft/lex.universal.char#nt:universal-character-name>*s*
([lex.charset] <https://eel.is/c++draft/lex.charset>) is encoded to a code
unit sequence using the *string-literal*
<https://eel.is/c++draft/lex.string#nt:string-literal>'s associated
character encoding. <https://eel.is/c++draft/lex.string#10.1.sentence-1>
If a character lacks representation in the associated character encoding,
then the program is *ill-formed*.
<https://eel.is/c++draft/lex.string#10.1.sentence-2>


 If I'm to create a plain transcoding function that converts from utf-8 to
> utf-16 which is the type used to represent utf-8? char8_t, char should not
> be supported. You want to use std::string? That's on you, you can
> reinterpret_cast it if you want, but you are ultimately responsible to
> figure out what the right thing is, and it has been discussed your approach
> will not work on every system.
>

Of course I honour objects, otherwise I would simply cast u8string& to
string& and call the job done. Exactly the fact that we should not do this
makes the lack of I/O support a show-stopper to me. What I said was simply
using string to store UTF-8 sequences (completely valid), and use I/O
functions to input/output them (completely valid).



> Essentially on any feature that we are yet to come up with in the future
> every time a decision needs to be made, char8_t is utf-8 and char isn't.
> What that is we may not know yet, but the decision is already made.
>

Where does the standard say char cannot be UTF-8? And mainstream
implementations can all make the working encoding be UTF-8.



> The introduction of char8_t has solved portability problems that char
> has. The wishy washiness of the arithmetic properties of char makes
> algorithms involving it not portable, char8_t was a god send for
> portability.
>

No, it is a pain in reality. It sounds like a good idea, but turns out to
be more trouble than benefits.

You may call it opinions, instead of facts. I do not claim it to be facts.
But you have made fact-like statements above that have no supporting
evidence at all.



> And I like that char8_t creates a clear type system distinction from char,
> so that I know which domain I’m operating at and what to expect.
>
> Aliasing u8string to string, it wouldn’t just be inconvenient, it would
> actually break code.
>

In what way is u8string not a string in real use cases? Of course, the
contrary may not be true.


> The lack of IO and formatting support in the standard makes for a
> usability problem. But that is because the formatting and IO in the
> standard are bad, it’s not because char8_t is bad.
>

I can live without type strictness for UTF-8-ness. I *cannot* live without
I/O support.

Lack of I/O support is a huge problem. It makes u8string pretty much
useless to me. And there are also huge compatibility problems with
char/string based code. It is much easier to require everyone to use UTF-8
than 1) dealing with string and u8string simultaneously, or 2) changing all
string code to use u8string.

Both 1) and 2) are very painful. Much more painful than the apparent beauty
of having UTF-8 type strictness.


> We should fix the bad parts, not throw way the good parts because they don
> ‘t work with the bad parts.
>

The car works with the "bad parts". The car does not work with the "good
parts". And you are requiring people to use the so-called "good parts"?

If the problems are not fixed, the "good parts" are not that "good".



> Maybe what we need is for a group of people to just go ahead make a
> library that works independent of the standard, have that be widely
> adopted, and when that happens then people will get it. And then people
> will go like “oh, that’s how you are supposed to use it, we want to do like
> library X” tand it clicks, and then you can adopt something in the
> standard because everyone gets it.
>

Even if the I/O problems are fixed, they are still a small part of the
problem. The reality remains that there is a huge amount of
char/string-based code, and char8_t/u8string simply does not work with it.

Type strictness turns out to be not that usable in this case.

-YW

Received on 2026-07-08 14:23:19