Date: Wed, 08 Jul 2026 07:21:02 -0700
On Tuesday, 7 July 2026 23:28:24 Pacific Daylight Time Tiago Freire via SG16
wrote:
> If I'm to create a plain transcoding function that converts from utf-8 to
> utf-16 which is the type used to represent utf-8? char8_t, char should not
> be supported.
That's not correct. Since C++11, char is required to support all UTF-8 code
units (that means the minimum size for a byte is 8 bits). The addition of
char8_t has not changed the requirement on char.
Therefore, it can't be UB.
Whether the code unit stored in the char makes sense when interpreted under
the narrow character set, it's a completely different story.
> The introduction of char8_t has solved portability problems that char has.
A problem that doesn't exist on Unix systems and a problem that isn't solved
elsewhere (especially Windows) because of the lack of APIs. Therefore, from
the *practical* perspective, either you have no problem, or char8_t isn't
sufficient for you. The lack of APIs is what Yongwei was complaining about and
until most of <string.h>, <locale.h>, <format>, etc. is duplicated for char8_t
support, char8_t will have very limited usefulness.
> Aliasing u8string to string, it wouldn’t just be inconvenient, it would
> actually break code.
That I agree with.
> The lack of IO and formatting support in the standard makes for a usability
> problem. But that is because the formatting and IO in the standard are bad,
> it’s not because char8_t is bad. We should fix the bad parts, not throw way
> the good parts because they don ‘t work with the bad parts.
Agreed.
I don't really care for iostreams. But <format> and all formatters in all
other headers need to support at least one of the char8_t/char16_t, and
ideally the other one be automatically transcoded as needed.
> Maybe what we need is for a group of people to just go ahead make a library
> that works independent of the standard, have that be widely adopted, and
> when that happens then people will get it. And then people will go like
> “oh, that’s how you are supposed to use it, we want to do like library X”
> and it clicks, and then you can adopt something in the standard because
> everyone gets it.
That's what happened with fmtlib, and it got adopted. That is good.
Except that fmtlib was designed with "char is UTF-8" in mind because it wasn't
part of the Standard and could reasonably make that assumption, and that has
never been revised.
wrote:
> If I'm to create a plain transcoding function that converts from utf-8 to
> utf-16 which is the type used to represent utf-8? char8_t, char should not
> be supported.
That's not correct. Since C++11, char is required to support all UTF-8 code
units (that means the minimum size for a byte is 8 bits). The addition of
char8_t has not changed the requirement on char.
Therefore, it can't be UB.
Whether the code unit stored in the char makes sense when interpreted under
the narrow character set, it's a completely different story.
> The introduction of char8_t has solved portability problems that char has.
A problem that doesn't exist on Unix systems and a problem that isn't solved
elsewhere (especially Windows) because of the lack of APIs. Therefore, from
the *practical* perspective, either you have no problem, or char8_t isn't
sufficient for you. The lack of APIs is what Yongwei was complaining about and
until most of <string.h>, <locale.h>, <format>, etc. is duplicated for char8_t
support, char8_t will have very limited usefulness.
> Aliasing u8string to string, it wouldn’t just be inconvenient, it would
> actually break code.
That I agree with.
> The lack of IO and formatting support in the standard makes for a usability
> problem. But that is because the formatting and IO in the standard are bad,
> it’s not because char8_t is bad. We should fix the bad parts, not throw way
> the good parts because they don ‘t work with the bad parts.
Agreed.
I don't really care for iostreams. But <format> and all formatters in all
other headers need to support at least one of the char8_t/char16_t, and
ideally the other one be automatically transcoded as needed.
> Maybe what we need is for a group of people to just go ahead make a library
> that works independent of the standard, have that be widely adopted, and
> when that happens then people will get it. And then people will go like
> “oh, that’s how you are supposed to use it, we want to do like library X”
> and it clicks, and then you can adopt something in the standard because
> everyone gets it.
That's what happened with fmtlib, and it got adopted. That is good.
Except that fmtlib was designed with "char is UTF-8" in mind because it wasn't
part of the Standard and could reasonably make that assumption, and that has
never been revised.
-- Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org Principal Engineer - Intel Data Center - Platform & Sys. Eng.
Received on 2026-07-08 14:21:16