Date: Tue, 7 Jul 2026 15:04:45 +0000
Correct me if I'm wrong but the internet of std::string is to be "system defined encoding" not even superset of ASCII. Ex. on an EBDIC system it's just EBDIC encoding.
std::u8string is the utf-8 one, and you should just use that if your text is utf-8.
________________________________
From: Christopher Nelson <nadiasvertex_at_[hidden]>
Sent: Tuesday, July 7, 2026 4:59:17 PM
To: Tom Honermann <tom_at_[hidden]>
Cc: sg16_at_[hidden] <sg16_at_lists.isocpp.org>; Tiago Freire <tmiguelf_at_[hidden]>
Subject: Re: [isocpp-sg16] UTF-8 support status
I think a significant detail that we'd like to understand is: what does the committee intend std::string to mean in the future? In many places it seems like it is becoming idiomatic to say that std::string is utf-8. Which makes us wonder what the design intent of the other string containers are. We're just trying to be good C++ citizens and not create more problems for ourselves in the future.
On Fri, Jul 3, 2026 at 11:00 AM Tom Honermann <tom_at_[hidden]<mailto:tom_at_[hidden]>> wrote:
I haven’t had a chance to review this thread yet and only a couple of regular attendees of SG16 meetings have weighed in so far. I suggest waiting at least a few more days for additional advice, especially considering the holiday weekend in the US.
The summary below reflects one reasonable approach, but it is not the only one.
Tom.
On Jul 3, 2026, at 10:48 AM, Christopher Nelson via SG16 <sg16_at_[hidden]<mailto:sg16_at_[hidden]>> wrote:
Thanks folks, I really appreciate all of the guidance.
The specific codebase we are working on is group of libraries that has bindings for several programming languages. C++(17/23/26) , C#, nodejs, and Rust. It runs on Linux, macOS, and Windows. We control all the applications it is embedded in.
May I summarize what I understood from the conversation and you can tell me if I am on the right track?
1. Use std::string
2. Assume that all internal string data is UTF-8
3. Enforce that invariant at input boundaries using helper functions
4. Be explicit about output encoding if it matters, default to UTF-8
5. Make formatting helpers that understand intent (display/file/etc.)
6. For Windows, explicitly opt into UTF-8 with the manifest flag (where possible) and compiler flag
7. Realize that you are mostly on your own here, so investing in a small library that will "do the right thing" for your app makes sense
Again, thank you so much for your time. I appreciate all the hard work you folks put into this.
On Fri, Jul 3, 2026 at 2:50 AM Tiago Freire <tmiguelf_at_[hidden]<mailto:tmiguelf_at_[hidden]>> wrote:
I can’t speak for everybody, but I can tell you what I do.
On Windows apps I just set the console code page to be utf8. The exact method to do that has changed as some terminal applications have broken certain features. But you can do it in 3 ways:
1. There’s a global Windows setting that sets the default windows code page buried deep in the localization options.
2. You can embed the preferred code page in the manifest file for the application
3. You can set the code page at runtime using a function
To be safe, you can do all 3, but ultimately it is going to depend on the terminal’s type setter to respect your preference (sometimes it doesn’t… *looks at windows terminal*).
As for “filesystem::path” they are not Unicode and never will be no matter how many times people say otherwise and it is best to acknowledge that upfront. To deal with that there is not one but multiple strategies depending on the purpose of the “filesystem::path”.
If the purpose is for printing you can create a transcoding function that makes certain invalid Unicode characters to be visibly printable indicating that “there’s a character there that has been altered for visibility but is not the actual character” and that makes “humans with eyes happy”.
If the purpose is to pass trough code that uses [bytes] char8_t to store or transfer things, but you would really like to perfectly preserve the path, there is a different transcoding function that I use that makes the “path round-trip-able” that is “mostly utf8 compatible” but not truly utf8 in order to encode the invalid code points (i.e. UTF-16 surrogates are representable).
So, the right answer is “filesystem::path” is a path, not exactly text but text-like. And “what you do with it” depends on “what you want to achieve with it”, and Unicode in this context is only relevant if you want to print it for a user to see and it is otherwise just bits.
As for “[it doesn’t] work with std::format and streams”, I’ve re-written my own output, formatting, and encoding library pipeline from scratch and I don’t suffer from such problems. The C++ standard has come a long way but is comparatively speaking “still just banging rocks together”.
I can share more details regarding what I have done on this front, and even tough some concepts can be cannibalized to improve the standard, I’m afraid that the C++ status-quo has painted itself too much into a corner that is a monumental task to even just start to fix it.
Best regards,
Tiago
From: SG16 <sg16-bounces_at_[hidden]<mailto:sg16-bounces_at_[hidden]>> On Behalf Of Herb Sutter via SG16
Sent: Friday, July 3, 2026 01:48
To: sg16_at_lists.isocpp.org<mailto:sg16_at_[hidden]>; 'Christopher Nelson' <nadiasvertex_at_[hidden]<mailto:nadiasvertex_at_[hidden]>>
Cc: Herb Sutter <herb.sutter_at_[hidden]<mailto:herb.sutter_at_[hidden]>>
Subject: [isocpp-sg16] UTF-8 support status
Hi C++ Unicode folks, literally asking for a friend (Christopher, on the To: line so please Reply-All)…
His project team wants to use UTF-8 in their code base and they hoped to switch to u8string[_view] and char8_t throughout, but they encountered two sets of limitations:
* In the standard, those types don’t seem to work with std::format and streams.
* On Windows, platform APIs interpret narrow characters using the active code page (e.g., std::cout emits mojibake, filesystem::path{std::string} UTF-8 paths are mangled or throw).
What’s the current best guidance for adopting UTF-8 in C++ code?
Thank you,
Herb
--
SG16 mailing list
SG16_at_[hidden]<mailto:SG16_at_[hidden]>
https://lists.isocpp.org/mailman/listinfo.cgi/sg16
Link to this post: http://lists.isocpp.org/sg16/2026/07/4786.php
std::u8string is the utf-8 one, and you should just use that if your text is utf-8.
________________________________
From: Christopher Nelson <nadiasvertex_at_[hidden]>
Sent: Tuesday, July 7, 2026 4:59:17 PM
To: Tom Honermann <tom_at_[hidden]>
Cc: sg16_at_[hidden] <sg16_at_lists.isocpp.org>; Tiago Freire <tmiguelf_at_[hidden]>
Subject: Re: [isocpp-sg16] UTF-8 support status
I think a significant detail that we'd like to understand is: what does the committee intend std::string to mean in the future? In many places it seems like it is becoming idiomatic to say that std::string is utf-8. Which makes us wonder what the design intent of the other string containers are. We're just trying to be good C++ citizens and not create more problems for ourselves in the future.
On Fri, Jul 3, 2026 at 11:00 AM Tom Honermann <tom_at_[hidden]<mailto:tom_at_[hidden]>> wrote:
I haven’t had a chance to review this thread yet and only a couple of regular attendees of SG16 meetings have weighed in so far. I suggest waiting at least a few more days for additional advice, especially considering the holiday weekend in the US.
The summary below reflects one reasonable approach, but it is not the only one.
Tom.
On Jul 3, 2026, at 10:48 AM, Christopher Nelson via SG16 <sg16_at_[hidden]<mailto:sg16_at_[hidden]>> wrote:
Thanks folks, I really appreciate all of the guidance.
The specific codebase we are working on is group of libraries that has bindings for several programming languages. C++(17/23/26) , C#, nodejs, and Rust. It runs on Linux, macOS, and Windows. We control all the applications it is embedded in.
May I summarize what I understood from the conversation and you can tell me if I am on the right track?
1. Use std::string
2. Assume that all internal string data is UTF-8
3. Enforce that invariant at input boundaries using helper functions
4. Be explicit about output encoding if it matters, default to UTF-8
5. Make formatting helpers that understand intent (display/file/etc.)
6. For Windows, explicitly opt into UTF-8 with the manifest flag (where possible) and compiler flag
7. Realize that you are mostly on your own here, so investing in a small library that will "do the right thing" for your app makes sense
Again, thank you so much for your time. I appreciate all the hard work you folks put into this.
On Fri, Jul 3, 2026 at 2:50 AM Tiago Freire <tmiguelf_at_[hidden]<mailto:tmiguelf_at_[hidden]>> wrote:
I can’t speak for everybody, but I can tell you what I do.
On Windows apps I just set the console code page to be utf8. The exact method to do that has changed as some terminal applications have broken certain features. But you can do it in 3 ways:
1. There’s a global Windows setting that sets the default windows code page buried deep in the localization options.
2. You can embed the preferred code page in the manifest file for the application
3. You can set the code page at runtime using a function
To be safe, you can do all 3, but ultimately it is going to depend on the terminal’s type setter to respect your preference (sometimes it doesn’t… *looks at windows terminal*).
As for “filesystem::path” they are not Unicode and never will be no matter how many times people say otherwise and it is best to acknowledge that upfront. To deal with that there is not one but multiple strategies depending on the purpose of the “filesystem::path”.
If the purpose is for printing you can create a transcoding function that makes certain invalid Unicode characters to be visibly printable indicating that “there’s a character there that has been altered for visibility but is not the actual character” and that makes “humans with eyes happy”.
If the purpose is to pass trough code that uses [bytes] char8_t to store or transfer things, but you would really like to perfectly preserve the path, there is a different transcoding function that I use that makes the “path round-trip-able” that is “mostly utf8 compatible” but not truly utf8 in order to encode the invalid code points (i.e. UTF-16 surrogates are representable).
So, the right answer is “filesystem::path” is a path, not exactly text but text-like. And “what you do with it” depends on “what you want to achieve with it”, and Unicode in this context is only relevant if you want to print it for a user to see and it is otherwise just bits.
As for “[it doesn’t] work with std::format and streams”, I’ve re-written my own output, formatting, and encoding library pipeline from scratch and I don’t suffer from such problems. The C++ standard has come a long way but is comparatively speaking “still just banging rocks together”.
I can share more details regarding what I have done on this front, and even tough some concepts can be cannibalized to improve the standard, I’m afraid that the C++ status-quo has painted itself too much into a corner that is a monumental task to even just start to fix it.
Best regards,
Tiago
From: SG16 <sg16-bounces_at_[hidden]<mailto:sg16-bounces_at_[hidden]>> On Behalf Of Herb Sutter via SG16
Sent: Friday, July 3, 2026 01:48
To: sg16_at_lists.isocpp.org<mailto:sg16_at_[hidden]>; 'Christopher Nelson' <nadiasvertex_at_[hidden]<mailto:nadiasvertex_at_[hidden]>>
Cc: Herb Sutter <herb.sutter_at_[hidden]<mailto:herb.sutter_at_[hidden]>>
Subject: [isocpp-sg16] UTF-8 support status
Hi C++ Unicode folks, literally asking for a friend (Christopher, on the To: line so please Reply-All)…
His project team wants to use UTF-8 in their code base and they hoped to switch to u8string[_view] and char8_t throughout, but they encountered two sets of limitations:
* In the standard, those types don’t seem to work with std::format and streams.
* On Windows, platform APIs interpret narrow characters using the active code page (e.g., std::cout emits mojibake, filesystem::path{std::string} UTF-8 paths are mangled or throw).
What’s the current best guidance for adopting UTF-8 in C++ code?
Thank you,
Herb
--
SG16 mailing list
SG16_at_[hidden]<mailto:SG16_at_[hidden]>
https://lists.isocpp.org/mailman/listinfo.cgi/sg16
Link to this post: http://lists.isocpp.org/sg16/2026/07/4786.php
Received on 2026-07-07 15:04:53