Please move this discussion to a different thread as it isn't related to
the original request.

Tom.

On 7/8/26 12:01 PM, Tiago Freire wrote:
>
> > Are you kidding me? When does C++ begin to break working code this way?
>
> > And there is never UB. [lex.string] <https://eel.is/c++draft/lex.string>
> says (my emphasis):
>
> > The sequence of characters denoted by each contiguous sequence of
> /basic-s-char/
> <https://eel.is/c++draft/lex.string#nt:basic-s-char>/s/, /r-char/
> <https://eel.is/c++draft/lex.string#nt:r-char>/s/,
> /simple-escape-sequence/
> <https://eel.is/c++draft/lex.ccon#nt:simple-escape-sequence>/s/
> ([lex.ccon] <https://eel.is/c++draft/lex.ccon>), and
> /universal-character-name/
> <https://eel.is/c++draft/lex.universal.char#nt:universal-character-name>/s/
> ([lex.charset] <https://eel.is/c++draft/lex.charset>) is encoded to a
> code unit sequence using the /string-literal/
> <https://eel.is/c++draft/lex.string#nt:string-literal>'s associated
> character encoding. <https://eel.is/c++draft/lex.string#10.1.sentence-1>
>
> > If a character lacks representation in the associated character encoding,
> then the program is *ill-formed*.
> <https://eel.is/c++draft/lex.string#10.1.sentence-2>
>
> Exactly!
>
> Read that last sentence again.
>
> But you are right, it’s not UB, it should be ill-formed.
>
> > Of course I honour objects, otherwise I would simply cast
> u8string& to string& and call the job done. Exactly the fact that we
> should not do this makes the lack of I/O support a show-stopper to me.
> What I said was simply using string to store UTF-8 sequences
> (completely valid), and use I/O functions to input/output them
> (completely valid).
>
> Nobody said that what you are doing right now on your system doesn’t
> work for you. Nobody is going to change your bits. Sure, you can write
> your bits from an std::string into a file and then have another text
> processing application open it and read perfectly valid utf-8.
>
> As for “the lack of I/O support”, if you mean being able to print
> utf-8 onto a terminal via your std::cout, that will never happen, and
> that is not a C++ problem, the problem is that it is not your
> application that dictates the encoding of cin/cout, it’s the terminal
> that does that, and your application can’t make the terminal do what
> it cannot do.
>
> If you are talking about std::format, then yes, something can be done
> about it, but then again if you want to guarantee utf-8 you would have
> to use std::u8string and you wouldn’t be able to ingest std::string,
> and there would be no requirement that a std::format working with
> std::string would have to assume char is utf-8, because it can’t. That
> ship has sailed a long time ago.
>
> > Where does the standard say char cannot be UTF-8? And mainstream
> implementations can all make the working encoding be UTF-8.
>
> The standard doesn’t say! And that is not a mistake.
>
> A maintainer could try to make the encoding utf-8, sure, but in
> practice it isn’t. Try to solve char_traits…
>
> Case in point what is the byte code for the EOF character again?...
> that’s not valid Unicode.
>
> > I can live without type strictness for UTF-8-ness. I /cannot/ live
> without I/O support.
>
> > Lack of I/O support is a huge problem. It makes u8string pretty much
> useless to me. And there are also huge compatibility problems with
> char/string based code. It is much easier to require everyone to use
> UTF-8 than 1) dealing with string and u8string simultaneously, or 2)
> changing all string code to use u8string.
>
> > Both 1) and 2) are very painful. Much more painful than the apparent
> beauty of having UTF-8 type strictness.
>
> It’s understandable. Yes, lack of formatting/IO support is a problem.
> Yes, it is a pain because you can not afford to have done the same
> thing that I have done.
>
> But I’m also not talking out of the void. I have done that! I have
> ditched the standard formatting and IO, and rebuild my own, and the
> benefit of having u8string is plainly obvious to me.
>
> I can deal with string and u8string simultaneously without problems,
> and I have changed “all string code to use u8string”. None of this is
> theoretical for me, its not just out of having a beautiful standard, I
> have solved this problem for my code base.
>
> You made a good analogy.
>
> > The car works with the "bad parts". The car does not work with the
> "good parts". And you are requiring people to use the so-called "good
> parts"?
>
> Ok, understandable. But nobody is taking std::string away, if your car
> only works with that part, then by all means use it.
>
> But I have a better car. We should build a new model car, not break
> the new car parts.
>
> Yes, you have a valid point. It is painful to work with the standard
> as it is right now. I get it!
>
> Yes, it is going to take time to migrate to doing things in a new way.
>
> But trust the process. As long as we keep moving in the right
> direction, things will eventually suck less.
>
> *From:*Yongwei Wu <wuyongwei_at_[hidden]>
> *Sent:* Wednesday, July 8, 2026 16:23
> *To:* Tiago Freire <tmiguelf_at_[hidden]>
> *Cc:* sg16_at_[hidden]; Christopher Nelson
> <nadiasvertex_at_[hidden]>; Tom Honermann <tom_at_[hidden]>
> *Subject:* Re: [isocpp-sg16] UTF-8 support status
>
> On Wed, 8 Jul 2026 at 14:28, Tiago Freire <tmiguelf_at_[hidden]> wrote:
>
> > In which way may my approach not be supported?
>
> In whatever which way in the future a decision needs to be made
> that is incompatible with your usage.
>
> Ex.
>
> Should string literals with large Unicode code points be allowed?
> Nope, those are UB.
>
> Are you kidding me? When does C++ begin to break working code this way?
>
> And there is never UB. [lex.string]
> <https://eel.is/c++draft/lex.string> says (my emphasis):
>
> The sequence of characters denoted by each contiguous sequence of
> /basic-s-char/
> <https://eel.is/c++draft/lex.string#nt:basic-s-char>/s/, /r-char/
> <https://eel.is/c++draft/lex.string#nt:r-char>/s/,
> /simple-escape-sequence/
> <https://eel.is/c++draft/lex.ccon#nt:simple-escape-sequence>/s/
> ([lex.ccon] <https://eel.is/c++draft/lex.ccon>), and
> /universal-character-name/
> <https://eel.is/c++draft/lex.universal.char#nt:universal-character-name>/s/
> ([lex.charset] <https://eel.is/c++draft/lex.charset>) is encoded to a
> code unit sequence using the /string-literal/
> <https://eel.is/c++draft/lex.string#nt:string-literal>'s associated
> character encoding. <https://eel.is/c++draft/lex.string#10.1.sentence-1>
>
> If a character lacks representation in the associated character
> encoding, then the program is *ill-formed*.
> <https://eel.is/c++draft/lex.string#10.1.sentence-2>
>
> If I'm to create a plain transcoding function that converts from
> utf-8 to utf-16 which is the type used to represent utf-8?
> char8_t, char should not be supported. You want to use
> std::string? That's on you, you can reinterpret_cast it if you
> want, but you are ultimately responsible to figure out what the
> right thing is, and it has been discussed your approach will not
> work on every system.
>
> Of course I honour objects, otherwise I would simply cast u8string& to
> string& and call the job done. Exactly the fact that we should not do
> this makes the lack of I/O support a show-stopper to me. What I said
> was simply using string to store UTF-8 sequences (completely valid),
> and use I/O functions to input/output them (completely valid).
>
> Essentially on any feature that we are yet to come up with in the
> future every time a decision needs to be made, char8_t is utf-8
> and char isn't. What that is we may not know yet, but the decision
> is already made.
>
> Where does the standard say char cannot be UTF-8? And mainstream
> implementations can all make the working encoding be UTF-8.
>
> The introduction of char8_t has solved portability problems that
> char has. The wishy washiness of the arithmetic properties of
> char makes algorithms involving it not portable, char8_t was a god
> send for portability.
>
> No, it is a pain in reality. It sounds like a good idea, but turns out
> to be more trouble than benefits.
>
> You may call it opinions, instead of facts. I do not claim it to be
> facts. But you have made fact-like statements above that have no
> supporting evidence at all.
>
> And I like that char8_t creates a clear type system distinction
> from char, so that I know which domain I’m operating at and what
> to expect.
>
> Aliasing u8string to string, it wouldn’t just be inconvenient, it
> would actually break code.
>
> In what way is u8string not a string in real use cases? Of course, the
> contrary may not be true.
>
> The lack of IO and formatting support in the standard makes for a
> usability problem. But that is because the formatting and IO in
> the standard are bad, it’s not because char8_t is bad.
>
> I can live without type strictness for UTF-8-ness. I /cannot/ live
> without I/O support.
>
> Lack of I/O support is a huge problem. It makes u8string pretty much
> useless to me. And there are also huge compatibility problems with
> char/string based code. It is much easier to require everyone to use
> UTF-8 than 1) dealing with string and u8string simultaneously, or 2)
> changing all string code to use u8string.
>
> Both 1) and 2) are very painful. Much more painful than the apparent
> beauty of having UTF-8 type strictness.
>
> We should fix the bad parts, not throw way the good parts because
> they don ‘t work with the bad parts.
>
> The car works with the "bad parts". The car does not work with the
> "good parts". And you are requiring people to use the so-called "good
> parts"?
>
> If the problems are not fixed, the "good parts" are not that "good".
>
> Maybe what we need is for a group of people to just go ahead make
> a library that works independent of the standard, have that be
> widely adopted, and when that happens then people will get it. And
> then people will go like “oh, that’s how you are supposed to use
> it, we want to do like library X” tand it clicks, and then you can
> adopt something in the standard because everyone gets it.
>
> Even if the I/O problems are fixed, they are still a small part of the
> problem. The reality remains that there is a huge amount of
> char/string-based code, and char8_t/u8string simply does not work with it.
>
> Type strictness turns out to be not that usable in this case.
>
> -YW
>