Date: Thu, 9 Jul 2026 09:29:15 -0400
[Bit of a dump follows. I've been thinking about writing down my theory of
a plan for text in C++, and these are some of my current thoughts. ]
It's a little more nuanced. The associated encodings of char{8,16,32}_t are
always UTF8 etc. Now, it might not be well formed UTF-8, because a
primitive can't really make that promise, but the typelevel purpose is to
convey that intent.
Char (and all the related primitives) don't have as strong a relationship.
The closest is the "execution encoding", which is what you get with the
basic character classification functions. But, it's also entirely
reasonable to carry the encoding separately and explicitly, which is what,
for example, email or web systems need to do. Charset comes in with the
protocol to tell you how to interpret the next octets.
We have ancient support for this, posix provides `iconv`, but better
contemporary support is on my personal roadmap now that UTF transcoding has
mostly stabilized for '29, and some of the parts that haven't are exactly
where the questions of what extensions are in, for example, it's error
codes, vs what makes it less ergonomic. Draft code, not std reviewed, but
extensively tested: https://github.com/steve-downey/transcode
Now, the deep problem, and it's a wicked problem, is fundamental Unix
model, baked into C and C++, is that stdout can't tell you anything about
what it's connected to, or what the expectations are. We operate by
convention. This means that any attempts to "help" with automatic
re-encoding to the "right" output character set will break current existing
programs. We can't solve mojibake with technical solutions. We can't just
take the char8_t and "do the right thing" when we send it to stdout mixed
with other "char" data because we can't know what is the right thing. We
have to make the programmer express any intent that is above "stuff these
octets down that pipe". We also know that tied streams are a problem, a
frustration, and for many a pessimization.
Now. Pragmatics.
1. Easiest. Insist that everything, including char, is UTF-8. Set the
system locale to C.UTF-8. Python does this, others have picked it up, and
it's the new normal on posix ish systems. We've talked about standardizing
it, but you should do it yourself.
2. Windows easiest. The same thing, only you have to also sometimes work
with wchar_t API. Try to do that at the edges? And everything else keep in
UTF-8, and hope you can avoid everything that involves knowing about a
win32 Console. Although we made print and format a little aware, so, also
prefer those over iostreams. Write some genric adapters for <<. We didn't
for you because there are some really ugly edge cases, but sane output
doesn't. [Citation Needed]
3. Legacy encoding at perimeter, UTF-8 text at "rest", UTF-32 as codepoints
for Unicode algorithms. This is where we are headed, but you can do much of
it today, just not all from std. ICU4x, or the base ICU, are the tool in
the space right now. ICU has some baked in 16 but history, which is
unfortunate, but largely ignorable. And if/when you eventually need to
"tailor" the Unicode algorithms, like you need the Turkish, not Polish,
collation order, this is what you are going to need. Or defer to platform
utilities, and push these concerns out to UI, and make it someone else's
problem, like Apple or Google. We're not planning std support for
internationalization or localization any time soon. No one knows enough,
and unless it provides exactly the same answer as the platform, it would be
wrong, even if somehow technically correct. We should not standardize
another facility that we have to explain that it should never be used in
anger. [Std::async, std::thread, etc].
Now, if you are only processing UTF-8, and only producing UTF-8, we hope we
will have a much better story for you soon. It's still going to probably
involve char, char8_t, and char32_t, but those map still to perimeter,
internal, and codepoints. Possibly std::byte instead of char, but since
byte is modelling memory, and can alias, it can be a pessimization for IO
because the compiler often has to assume possible aliasing in the vicinity
of a std::byte pointer. When we get around to networking we may need an
additional non-aliasing non-arithmetic octet sized type.
On Tue, Jul 7, 2026, 13:53 Tiago Freire via SG16 <sg16_at_[hidden]>
wrote:
> I think the topic is meandering a little bit.
>
> The official position of the standard is char8_t for utf-8.
> How you decide to use it is of course up to you, but it doesn't mean that
> your usage is going to be supported in the future.
>
>
>
> ------------------------------
> *From:* Yongwei Wu <wuyongwei_at_[hidden]>
> *Sent:* Tuesday, July 7, 2026 6:41:13 PM
> *To:* sg16_at_[hidden] <sg16_at_[hidden]>
> *Cc:* Christopher Nelson <nadiasvertex_at_[hidden]>; Tom Honermann <
> tom_at_[hidden]>; Tiago Freire <tmiguelf_at_[hidden]>
> *Subject:* Re: [isocpp-sg16] UTF-8 support status
>
> No, I won't use std::u8string, as long as IO streams and std::print do not
> support it.
>
> EBCDIC is a non-issue, and I do not care about it: I have never touched
> such a system in my 30+ year career. I am not expecting to port a program
> to such a system.
>
> There are always trade-offs in life, and one chooses the lesser evil.
> Sticking to std::string and requiring everyone to use UTF-8 work the best.
>
> On Tue, 7 Jul 2026 at 23:04, Tiago Freire via SG16 <sg16_at_[hidden]>
> wrote:
>
> Correct me if I'm wrong but the internet of std::string is to be "system
> defined encoding" not even superset of ASCII. Ex. on an EBDIC system it's
> just EBDIC encoding.
> std::u8string is the utf-8 one, and you should just use that if your text
> is utf-8.
>
>
> ------------------------------
> *From:* Christopher Nelson <nadiasvertex_at_[hidden]>
> *Sent:* Tuesday, July 7, 2026 4:59:17 PM
> *To:* Tom Honermann <tom_at_[hidden]>
> *Cc:* sg16_at_[hidden] <sg16_at_[hidden]>; Tiago Freire <
> tmiguelf_at_[hidden]>
> *Subject:* Re: [isocpp-sg16] UTF-8 support status
>
> I think a significant detail that we'd like to understand is: what does
> the committee intend std::string to mean in the future? In many places it
> seems like it is becoming idiomatic to say that std::string is utf-8. Which
> makes us wonder what the design intent of the other string containers
> are. We're just trying to be good C++ citizens and not create more problems
> for ourselves in the future.
>
> On Fri, Jul 3, 2026 at 11:00 AM Tom Honermann <tom_at_[hidden]> wrote:
>
> I haven’t had a chance to review this thread yet and only a couple of
> regular attendees of SG16 meetings have weighed in so far. I suggest
> waiting at least a few more days for additional advice, especially
> considering the holiday weekend in the US.
>
> The summary below reflects one reasonable approach, but it is not the only
> one.
>
> Tom.
>
> On Jul 3, 2026, at 10:48 AM, Christopher Nelson via SG16 <
> sg16_at_[hidden]> wrote:
>
>
> Thanks folks, I really appreciate all of the guidance.
>
> The specific codebase we are working on is group of libraries that has
> bindings for several programming languages. C++(17/23/26) , C#, nodejs, and
> Rust. It runs on Linux, macOS, and Windows. We control all the applications
> it is embedded in.
>
> May I summarize what I understood from the conversation and you can tell
> me if I am on the right track?
>
> 1. Use std::string
> 2. Assume that all internal string data is UTF-8
> 3. Enforce that invariant at input boundaries using helper functions
> 4. Be explicit about output encoding if it matters, default to UTF-8
> 5. Make formatting helpers that understand intent (display/file/etc.)
> 6. For Windows, explicitly opt into UTF-8 with the manifest flag (where
> possible) and compiler flag
> 7. Realize that you are mostly on your own here, so investing in a small
> library that will "do the right thing" for your app makes sense
>
> Again, thank you so much for your time. I appreciate all the hard work you
> folks put into this.
>
> On Fri, Jul 3, 2026 at 2:50 AM Tiago Freire <tmiguelf_at_[hidden]> wrote:
>
> I can’t speak for everybody, but I can tell you what I do.
>
>
>
> On Windows apps I just set the console code page to be utf8. The exact
> method to do that has changed as some terminal applications have broken
> certain features. But you can do it in 3 ways:
>
> 1. There’s a global Windows setting that sets the default windows code
> page buried deep in the localization options.
> 2. You can embed the preferred code page in the manifest file for the
> application
> 3. You can set the code page at runtime using a function
>
> To be safe, you can do all 3, but ultimately it is going to depend on the
> terminal’s type setter to respect your preference (sometimes it doesn’t…
> *looks at windows terminal*).
>
>
>
> As for “filesystem::path” they are not Unicode and never will be no matter
> how many times people say otherwise and it is best to acknowledge that
> upfront. To deal with that there is not one but multiple strategies
> depending on the purpose of the “filesystem::path”.
>
> If the purpose is for printing you can create a transcoding function that
> makes certain invalid Unicode characters to be visibly printable indicating
> that “there’s a character there that has been altered for visibility but is
> not the actual character” and that makes “humans with eyes happy”.
>
> If the purpose is to pass trough code that uses [bytes] char8_t to store
> or transfer things, but you would really like to perfectly preserve the
> path, there is a different transcoding function that I use that makes the
> “path round-trip-able” that is “mostly utf8 compatible” but not truly utf8
> in order to encode the invalid code points (i.e. UTF-16 surrogates are
> representable).
>
> So, the right answer is “filesystem::path” is a path, not exactly text but
> text-like. And “what you do with it” depends on “what you want to achieve
> with it”, and Unicode in this context is only relevant if you want to print
> it for a user to see and it is otherwise just bits.
>
>
>
>
>
> As for “[it doesn’t] work with std::format and streams”, I’ve re-written
> my own output, formatting, and encoding library pipeline from scratch and I
> don’t suffer from such problems. The C++ standard has come a long way but
> is comparatively speaking “still just banging rocks together”.
>
> I can share more details regarding what I have done on this front, and
> even tough some concepts can be cannibalized to improve the standard, I’m
> afraid that the C++ status-quo has painted itself too much into a corner
> that is a monumental task to even just start to fix it.
>
>
>
> Best regards,
>
> Tiago
>
>
>
> *From:* SG16 <sg16-bounces_at_[hidden]> *On Behalf Of *Herb Sutter
> via SG16
> *Sent:* Friday, July 3, 2026 01:48
> *To:* sg16_at_[hidden]; 'Christopher Nelson' <nadiasvertex_at_[hidden]>
> *Cc:* Herb Sutter <herb.sutter_at_[hidden]>
> *Subject:* [isocpp-sg16] UTF-8 support status
>
>
>
> Hi C++ Unicode folks, literally asking for a friend (Christopher, on the
> To: line so please Reply-All)…
>
>
>
> His project team wants to use UTF-8 in their code base and they hoped to
> switch to u8string[_view] and char8_t throughout, but they encountered two
> sets of limitations:
>
>
>
> - In the standard, those types don’t seem to work with std::format and
> streams.
>
>
>
> - On Windows, platform APIs interpret narrow characters using the
> active code page (e.g., std::cout emits mojibake,
> filesystem::path{std::string} UTF-8 paths are mangled or throw).
>
>
>
> What’s the current best guidance for adopting UTF-8 in C++ code?
>
>
>
> Thank you,
>
>
>
> Herb
>
>
>
> --
> SG16 mailing list
> SG16_at_[hidden]
> https://lists.isocpp.org/mailman/listinfo.cgi/sg16
> Link to this post: http://lists.isocpp.org/sg16/2026/07/4786.php
>
>
> --
> SG16 mailing list
> SG16_at_[hidden]
> https://lists.isocpp.org/mailman/listinfo.cgi/sg16
> Link to this post: http://lists.isocpp.org/sg16/2026/07/4792.php
>
>
>
> --
> Yongwei Wu
> URL: http://wyw.dcweb.cn/
>
> --
> SG16 mailing list
> SG16_at_[hidden]
> https://lists.isocpp.org/mailman/listinfo.cgi/sg16
> Link to this post: http://lists.isocpp.org/sg16/2026/07/4806.php
>
a plan for text in C++, and these are some of my current thoughts. ]
It's a little more nuanced. The associated encodings of char{8,16,32}_t are
always UTF8 etc. Now, it might not be well formed UTF-8, because a
primitive can't really make that promise, but the typelevel purpose is to
convey that intent.
Char (and all the related primitives) don't have as strong a relationship.
The closest is the "execution encoding", which is what you get with the
basic character classification functions. But, it's also entirely
reasonable to carry the encoding separately and explicitly, which is what,
for example, email or web systems need to do. Charset comes in with the
protocol to tell you how to interpret the next octets.
We have ancient support for this, posix provides `iconv`, but better
contemporary support is on my personal roadmap now that UTF transcoding has
mostly stabilized for '29, and some of the parts that haven't are exactly
where the questions of what extensions are in, for example, it's error
codes, vs what makes it less ergonomic. Draft code, not std reviewed, but
extensively tested: https://github.com/steve-downey/transcode
Now, the deep problem, and it's a wicked problem, is fundamental Unix
model, baked into C and C++, is that stdout can't tell you anything about
what it's connected to, or what the expectations are. We operate by
convention. This means that any attempts to "help" with automatic
re-encoding to the "right" output character set will break current existing
programs. We can't solve mojibake with technical solutions. We can't just
take the char8_t and "do the right thing" when we send it to stdout mixed
with other "char" data because we can't know what is the right thing. We
have to make the programmer express any intent that is above "stuff these
octets down that pipe". We also know that tied streams are a problem, a
frustration, and for many a pessimization.
Now. Pragmatics.
1. Easiest. Insist that everything, including char, is UTF-8. Set the
system locale to C.UTF-8. Python does this, others have picked it up, and
it's the new normal on posix ish systems. We've talked about standardizing
it, but you should do it yourself.
2. Windows easiest. The same thing, only you have to also sometimes work
with wchar_t API. Try to do that at the edges? And everything else keep in
UTF-8, and hope you can avoid everything that involves knowing about a
win32 Console. Although we made print and format a little aware, so, also
prefer those over iostreams. Write some genric adapters for <<. We didn't
for you because there are some really ugly edge cases, but sane output
doesn't. [Citation Needed]
3. Legacy encoding at perimeter, UTF-8 text at "rest", UTF-32 as codepoints
for Unicode algorithms. This is where we are headed, but you can do much of
it today, just not all from std. ICU4x, or the base ICU, are the tool in
the space right now. ICU has some baked in 16 but history, which is
unfortunate, but largely ignorable. And if/when you eventually need to
"tailor" the Unicode algorithms, like you need the Turkish, not Polish,
collation order, this is what you are going to need. Or defer to platform
utilities, and push these concerns out to UI, and make it someone else's
problem, like Apple or Google. We're not planning std support for
internationalization or localization any time soon. No one knows enough,
and unless it provides exactly the same answer as the platform, it would be
wrong, even if somehow technically correct. We should not standardize
another facility that we have to explain that it should never be used in
anger. [Std::async, std::thread, etc].
Now, if you are only processing UTF-8, and only producing UTF-8, we hope we
will have a much better story for you soon. It's still going to probably
involve char, char8_t, and char32_t, but those map still to perimeter,
internal, and codepoints. Possibly std::byte instead of char, but since
byte is modelling memory, and can alias, it can be a pessimization for IO
because the compiler often has to assume possible aliasing in the vicinity
of a std::byte pointer. When we get around to networking we may need an
additional non-aliasing non-arithmetic octet sized type.
On Tue, Jul 7, 2026, 13:53 Tiago Freire via SG16 <sg16_at_[hidden]>
wrote:
> I think the topic is meandering a little bit.
>
> The official position of the standard is char8_t for utf-8.
> How you decide to use it is of course up to you, but it doesn't mean that
> your usage is going to be supported in the future.
>
>
>
> ------------------------------
> *From:* Yongwei Wu <wuyongwei_at_[hidden]>
> *Sent:* Tuesday, July 7, 2026 6:41:13 PM
> *To:* sg16_at_[hidden] <sg16_at_[hidden]>
> *Cc:* Christopher Nelson <nadiasvertex_at_[hidden]>; Tom Honermann <
> tom_at_[hidden]>; Tiago Freire <tmiguelf_at_[hidden]>
> *Subject:* Re: [isocpp-sg16] UTF-8 support status
>
> No, I won't use std::u8string, as long as IO streams and std::print do not
> support it.
>
> EBCDIC is a non-issue, and I do not care about it: I have never touched
> such a system in my 30+ year career. I am not expecting to port a program
> to such a system.
>
> There are always trade-offs in life, and one chooses the lesser evil.
> Sticking to std::string and requiring everyone to use UTF-8 work the best.
>
> On Tue, 7 Jul 2026 at 23:04, Tiago Freire via SG16 <sg16_at_[hidden]>
> wrote:
>
> Correct me if I'm wrong but the internet of std::string is to be "system
> defined encoding" not even superset of ASCII. Ex. on an EBDIC system it's
> just EBDIC encoding.
> std::u8string is the utf-8 one, and you should just use that if your text
> is utf-8.
>
>
> ------------------------------
> *From:* Christopher Nelson <nadiasvertex_at_[hidden]>
> *Sent:* Tuesday, July 7, 2026 4:59:17 PM
> *To:* Tom Honermann <tom_at_[hidden]>
> *Cc:* sg16_at_[hidden] <sg16_at_[hidden]>; Tiago Freire <
> tmiguelf_at_[hidden]>
> *Subject:* Re: [isocpp-sg16] UTF-8 support status
>
> I think a significant detail that we'd like to understand is: what does
> the committee intend std::string to mean in the future? In many places it
> seems like it is becoming idiomatic to say that std::string is utf-8. Which
> makes us wonder what the design intent of the other string containers
> are. We're just trying to be good C++ citizens and not create more problems
> for ourselves in the future.
>
> On Fri, Jul 3, 2026 at 11:00 AM Tom Honermann <tom_at_[hidden]> wrote:
>
> I haven’t had a chance to review this thread yet and only a couple of
> regular attendees of SG16 meetings have weighed in so far. I suggest
> waiting at least a few more days for additional advice, especially
> considering the holiday weekend in the US.
>
> The summary below reflects one reasonable approach, but it is not the only
> one.
>
> Tom.
>
> On Jul 3, 2026, at 10:48 AM, Christopher Nelson via SG16 <
> sg16_at_[hidden]> wrote:
>
>
> Thanks folks, I really appreciate all of the guidance.
>
> The specific codebase we are working on is group of libraries that has
> bindings for several programming languages. C++(17/23/26) , C#, nodejs, and
> Rust. It runs on Linux, macOS, and Windows. We control all the applications
> it is embedded in.
>
> May I summarize what I understood from the conversation and you can tell
> me if I am on the right track?
>
> 1. Use std::string
> 2. Assume that all internal string data is UTF-8
> 3. Enforce that invariant at input boundaries using helper functions
> 4. Be explicit about output encoding if it matters, default to UTF-8
> 5. Make formatting helpers that understand intent (display/file/etc.)
> 6. For Windows, explicitly opt into UTF-8 with the manifest flag (where
> possible) and compiler flag
> 7. Realize that you are mostly on your own here, so investing in a small
> library that will "do the right thing" for your app makes sense
>
> Again, thank you so much for your time. I appreciate all the hard work you
> folks put into this.
>
> On Fri, Jul 3, 2026 at 2:50 AM Tiago Freire <tmiguelf_at_[hidden]> wrote:
>
> I can’t speak for everybody, but I can tell you what I do.
>
>
>
> On Windows apps I just set the console code page to be utf8. The exact
> method to do that has changed as some terminal applications have broken
> certain features. But you can do it in 3 ways:
>
> 1. There’s a global Windows setting that sets the default windows code
> page buried deep in the localization options.
> 2. You can embed the preferred code page in the manifest file for the
> application
> 3. You can set the code page at runtime using a function
>
> To be safe, you can do all 3, but ultimately it is going to depend on the
> terminal’s type setter to respect your preference (sometimes it doesn’t…
> *looks at windows terminal*).
>
>
>
> As for “filesystem::path” they are not Unicode and never will be no matter
> how many times people say otherwise and it is best to acknowledge that
> upfront. To deal with that there is not one but multiple strategies
> depending on the purpose of the “filesystem::path”.
>
> If the purpose is for printing you can create a transcoding function that
> makes certain invalid Unicode characters to be visibly printable indicating
> that “there’s a character there that has been altered for visibility but is
> not the actual character” and that makes “humans with eyes happy”.
>
> If the purpose is to pass trough code that uses [bytes] char8_t to store
> or transfer things, but you would really like to perfectly preserve the
> path, there is a different transcoding function that I use that makes the
> “path round-trip-able” that is “mostly utf8 compatible” but not truly utf8
> in order to encode the invalid code points (i.e. UTF-16 surrogates are
> representable).
>
> So, the right answer is “filesystem::path” is a path, not exactly text but
> text-like. And “what you do with it” depends on “what you want to achieve
> with it”, and Unicode in this context is only relevant if you want to print
> it for a user to see and it is otherwise just bits.
>
>
>
>
>
> As for “[it doesn’t] work with std::format and streams”, I’ve re-written
> my own output, formatting, and encoding library pipeline from scratch and I
> don’t suffer from such problems. The C++ standard has come a long way but
> is comparatively speaking “still just banging rocks together”.
>
> I can share more details regarding what I have done on this front, and
> even tough some concepts can be cannibalized to improve the standard, I’m
> afraid that the C++ status-quo has painted itself too much into a corner
> that is a monumental task to even just start to fix it.
>
>
>
> Best regards,
>
> Tiago
>
>
>
> *From:* SG16 <sg16-bounces_at_[hidden]> *On Behalf Of *Herb Sutter
> via SG16
> *Sent:* Friday, July 3, 2026 01:48
> *To:* sg16_at_[hidden]; 'Christopher Nelson' <nadiasvertex_at_[hidden]>
> *Cc:* Herb Sutter <herb.sutter_at_[hidden]>
> *Subject:* [isocpp-sg16] UTF-8 support status
>
>
>
> Hi C++ Unicode folks, literally asking for a friend (Christopher, on the
> To: line so please Reply-All)…
>
>
>
> His project team wants to use UTF-8 in their code base and they hoped to
> switch to u8string[_view] and char8_t throughout, but they encountered two
> sets of limitations:
>
>
>
> - In the standard, those types don’t seem to work with std::format and
> streams.
>
>
>
> - On Windows, platform APIs interpret narrow characters using the
> active code page (e.g., std::cout emits mojibake,
> filesystem::path{std::string} UTF-8 paths are mangled or throw).
>
>
>
> What’s the current best guidance for adopting UTF-8 in C++ code?
>
>
>
> Thank you,
>
>
>
> Herb
>
>
>
> --
> SG16 mailing list
> SG16_at_[hidden]
> https://lists.isocpp.org/mailman/listinfo.cgi/sg16
> Link to this post: http://lists.isocpp.org/sg16/2026/07/4786.php
>
>
> --
> SG16 mailing list
> SG16_at_[hidden]
> https://lists.isocpp.org/mailman/listinfo.cgi/sg16
> Link to this post: http://lists.isocpp.org/sg16/2026/07/4792.php
>
>
>
> --
> Yongwei Wu
> URL: http://wyw.dcweb.cn/
>
> --
> SG16 mailing list
> SG16_at_[hidden]
> https://lists.isocpp.org/mailman/listinfo.cgi/sg16
> Link to this post: http://lists.isocpp.org/sg16/2026/07/4806.php
>
Received on 2026-07-09 13:29:31