Date: Wed, 8 Jul 2026 09:26:11 +0800
In which way may my approach not be supported? I do not see such a
possibility. My approach does not in any way violate rules in the standard.
If neither IO streams nor std::print supports u8string, it is simply a
show-stopper to me. I know many people who think the same way, including
some committee members.
A wild thought: maybe we should make u8string a derived class of string,
and that will solve many usage problems. Of course, there are
many implementation/compatibility issues too, so it is just a thought at
this moment.
On Wed, 8 Jul 2026 at 01:53, Tiago Freire <tmiguelf_at_[hidden]> wrote:
> I think the topic is meandering a little bit.
>
> The official position of the standard is char8_t for utf-8.
> How you decide to use it is of course up to you, but it doesn't mean that
> your usage is going to be supported in the future.
>
>
>
> ------------------------------
> *From:* Yongwei Wu <wuyongwei_at_[hidden]>
> *Sent:* Tuesday, July 7, 2026 6:41:13 PM
> *To:* sg16_at_[hidden] <sg16_at_[hidden]>
> *Cc:* Christopher Nelson <nadiasvertex_at_[hidden]>; Tom Honermann <
> tom_at_[hidden]>; Tiago Freire <tmiguelf_at_[hidden]>
> *Subject:* Re: [isocpp-sg16] UTF-8 support status
>
> No, I won't use std::u8string, as long as IO streams and std::print do not
> support it.
>
> EBCDIC is a non-issue, and I do not care about it: I have never touched
> such a system in my 30+ year career. I am not expecting to port a program
> to such a system.
>
> There are always trade-offs in life, and one chooses the lesser evil.
> Sticking to std::string and requiring everyone to use UTF-8 work the best.
>
> On Tue, 7 Jul 2026 at 23:04, Tiago Freire via SG16 <sg16_at_[hidden]>
> wrote:
>
> Correct me if I'm wrong but the internet of std::string is to be "system
> defined encoding" not even superset of ASCII. Ex. on an EBDIC system it's
> just EBDIC encoding.
> std::u8string is the utf-8 one, and you should just use that if your text
> is utf-8.
>
>
> ------------------------------
> *From:* Christopher Nelson <nadiasvertex_at_[hidden]>
> *Sent:* Tuesday, July 7, 2026 4:59:17 PM
> *To:* Tom Honermann <tom_at_[hidden]>
> *Cc:* sg16_at_[hidden] <sg16_at_[hidden]>; Tiago Freire <
> tmiguelf_at_[hidden]>
> *Subject:* Re: [isocpp-sg16] UTF-8 support status
>
> I think a significant detail that we'd like to understand is: what does
> the committee intend std::string to mean in the future? In many places it
> seems like it is becoming idiomatic to say that std::string is utf-8. Which
> makes us wonder what the design intent of the other string containers
> are. We're just trying to be good C++ citizens and not create more problems
> for ourselves in the future.
>
> On Fri, Jul 3, 2026 at 11:00 AM Tom Honermann <tom_at_[hidden]> wrote:
>
> I haven’t had a chance to review this thread yet and only a couple of
> regular attendees of SG16 meetings have weighed in so far. I suggest
> waiting at least a few more days for additional advice, especially
> considering the holiday weekend in the US.
>
> The summary below reflects one reasonable approach, but it is not the only
> one.
>
> Tom.
>
> On Jul 3, 2026, at 10:48 AM, Christopher Nelson via SG16 <
> sg16_at_[hidden]> wrote:
>
>
> Thanks folks, I really appreciate all of the guidance.
>
> The specific codebase we are working on is group of libraries that has
> bindings for several programming languages. C++(17/23/26) , C#, nodejs, and
> Rust. It runs on Linux, macOS, and Windows. We control all the applications
> it is embedded in.
>
> May I summarize what I understood from the conversation and you can tell
> me if I am on the right track?
>
> 1. Use std::string
> 2. Assume that all internal string data is UTF-8
> 3. Enforce that invariant at input boundaries using helper functions
> 4. Be explicit about output encoding if it matters, default to UTF-8
> 5. Make formatting helpers that understand intent (display/file/etc.)
> 6. For Windows, explicitly opt into UTF-8 with the manifest flag (where
> possible) and compiler flag
> 7. Realize that you are mostly on your own here, so investing in a small
> library that will "do the right thing" for your app makes sense
>
> Again, thank you so much for your time. I appreciate all the hard work you
> folks put into this.
>
> On Fri, Jul 3, 2026 at 2:50 AM Tiago Freire <tmiguelf_at_[hidden]> wrote:
>
> I can’t speak for everybody, but I can tell you what I do.
>
>
>
> On Windows apps I just set the console code page to be utf8. The exact
> method to do that has changed as some terminal applications have broken
> certain features. But you can do it in 3 ways:
>
> 1. There’s a global Windows setting that sets the default windows code
> page buried deep in the localization options.
> 2. You can embed the preferred code page in the manifest file for the
> application
> 3. You can set the code page at runtime using a function
>
> To be safe, you can do all 3, but ultimately it is going to depend on the
> terminal’s type setter to respect your preference (sometimes it doesn’t…
> *looks at windows terminal*).
>
>
>
> As for “filesystem::path” they are not Unicode and never will be no matter
> how many times people say otherwise and it is best to acknowledge that
> upfront. To deal with that there is not one but multiple strategies
> depending on the purpose of the “filesystem::path”.
>
> If the purpose is for printing you can create a transcoding function that
> makes certain invalid Unicode characters to be visibly printable indicating
> that “there’s a character there that has been altered for visibility but is
> not the actual character” and that makes “humans with eyes happy”.
>
> If the purpose is to pass trough code that uses [bytes] char8_t to store
> or transfer things, but you would really like to perfectly preserve the
> path, there is a different transcoding function that I use that makes the
> “path round-trip-able” that is “mostly utf8 compatible” but not truly utf8
> in order to encode the invalid code points (i.e. UTF-16 surrogates are
> representable).
>
> So, the right answer is “filesystem::path” is a path, not exactly text but
> text-like. And “what you do with it” depends on “what you want to achieve
> with it”, and Unicode in this context is only relevant if you want to print
> it for a user to see and it is otherwise just bits.
>
>
>
>
>
> As for “[it doesn’t] work with std::format and streams”, I’ve re-written
> my own output, formatting, and encoding library pipeline from scratch and I
> don’t suffer from such problems. The C++ standard has come a long way but
> is comparatively speaking “still just banging rocks together”.
>
> I can share more details regarding what I have done on this front, and
> even tough some concepts can be cannibalized to improve the standard, I’m
> afraid that the C++ status-quo has painted itself too much into a corner
> that is a monumental task to even just start to fix it.
>
>
>
> Best regards,
>
> Tiago
>
>
>
> *From:* SG16 <sg16-bounces_at_[hidden]> *On Behalf Of *Herb Sutter
> via SG16
> *Sent:* Friday, July 3, 2026 01:48
> *To:* sg16_at_[hidden]; 'Christopher Nelson' <nadiasvertex_at_[hidden]>
> *Cc:* Herb Sutter <herb.sutter_at_[hidden]>
> *Subject:* [isocpp-sg16] UTF-8 support status
>
>
>
> Hi C++ Unicode folks, literally asking for a friend (Christopher, on the
> To: line so please Reply-All)…
>
>
>
> His project team wants to use UTF-8 in their code base and they hoped to
> switch to u8string[_view] and char8_t throughout, but they encountered two
> sets of limitations:
>
>
>
> - In the standard, those types don’t seem to work with std::format and
> streams.
>
>
>
> - On Windows, platform APIs interpret narrow characters using the
> active code page (e.g., std::cout emits mojibake,
> filesystem::path{std::string} UTF-8 paths are mangled or throw).
>
>
>
> What’s the current best guidance for adopting UTF-8 in C++ code?
>
>
>
> Thank you,
>
>
>
> Herb
>
>
>
> --
> SG16 mailing list
> SG16_at_[hidden]
> https://lists.isocpp.org/mailman/listinfo.cgi/sg16
> Link to this post: http://lists.isocpp.org/sg16/2026/07/4786.php
>
>
> --
> SG16 mailing list
> SG16_at_[hidden]
> https://lists.isocpp.org/mailman/listinfo.cgi/sg16
> Link to this post: http://lists.isocpp.org/sg16/2026/07/4792.php
>
>
>
> --
> Yongwei Wu
> URL: http://wyw.dcweb.cn/
>
>
possibility. My approach does not in any way violate rules in the standard.
If neither IO streams nor std::print supports u8string, it is simply a
show-stopper to me. I know many people who think the same way, including
some committee members.
A wild thought: maybe we should make u8string a derived class of string,
and that will solve many usage problems. Of course, there are
many implementation/compatibility issues too, so it is just a thought at
this moment.
On Wed, 8 Jul 2026 at 01:53, Tiago Freire <tmiguelf_at_[hidden]> wrote:
> I think the topic is meandering a little bit.
>
> The official position of the standard is char8_t for utf-8.
> How you decide to use it is of course up to you, but it doesn't mean that
> your usage is going to be supported in the future.
>
>
>
> ------------------------------
> *From:* Yongwei Wu <wuyongwei_at_[hidden]>
> *Sent:* Tuesday, July 7, 2026 6:41:13 PM
> *To:* sg16_at_[hidden] <sg16_at_[hidden]>
> *Cc:* Christopher Nelson <nadiasvertex_at_[hidden]>; Tom Honermann <
> tom_at_[hidden]>; Tiago Freire <tmiguelf_at_[hidden]>
> *Subject:* Re: [isocpp-sg16] UTF-8 support status
>
> No, I won't use std::u8string, as long as IO streams and std::print do not
> support it.
>
> EBCDIC is a non-issue, and I do not care about it: I have never touched
> such a system in my 30+ year career. I am not expecting to port a program
> to such a system.
>
> There are always trade-offs in life, and one chooses the lesser evil.
> Sticking to std::string and requiring everyone to use UTF-8 work the best.
>
> On Tue, 7 Jul 2026 at 23:04, Tiago Freire via SG16 <sg16_at_[hidden]>
> wrote:
>
> Correct me if I'm wrong but the internet of std::string is to be "system
> defined encoding" not even superset of ASCII. Ex. on an EBDIC system it's
> just EBDIC encoding.
> std::u8string is the utf-8 one, and you should just use that if your text
> is utf-8.
>
>
> ------------------------------
> *From:* Christopher Nelson <nadiasvertex_at_[hidden]>
> *Sent:* Tuesday, July 7, 2026 4:59:17 PM
> *To:* Tom Honermann <tom_at_[hidden]>
> *Cc:* sg16_at_[hidden] <sg16_at_[hidden]>; Tiago Freire <
> tmiguelf_at_[hidden]>
> *Subject:* Re: [isocpp-sg16] UTF-8 support status
>
> I think a significant detail that we'd like to understand is: what does
> the committee intend std::string to mean in the future? In many places it
> seems like it is becoming idiomatic to say that std::string is utf-8. Which
> makes us wonder what the design intent of the other string containers
> are. We're just trying to be good C++ citizens and not create more problems
> for ourselves in the future.
>
> On Fri, Jul 3, 2026 at 11:00 AM Tom Honermann <tom_at_[hidden]> wrote:
>
> I haven’t had a chance to review this thread yet and only a couple of
> regular attendees of SG16 meetings have weighed in so far. I suggest
> waiting at least a few more days for additional advice, especially
> considering the holiday weekend in the US.
>
> The summary below reflects one reasonable approach, but it is not the only
> one.
>
> Tom.
>
> On Jul 3, 2026, at 10:48 AM, Christopher Nelson via SG16 <
> sg16_at_[hidden]> wrote:
>
>
> Thanks folks, I really appreciate all of the guidance.
>
> The specific codebase we are working on is group of libraries that has
> bindings for several programming languages. C++(17/23/26) , C#, nodejs, and
> Rust. It runs on Linux, macOS, and Windows. We control all the applications
> it is embedded in.
>
> May I summarize what I understood from the conversation and you can tell
> me if I am on the right track?
>
> 1. Use std::string
> 2. Assume that all internal string data is UTF-8
> 3. Enforce that invariant at input boundaries using helper functions
> 4. Be explicit about output encoding if it matters, default to UTF-8
> 5. Make formatting helpers that understand intent (display/file/etc.)
> 6. For Windows, explicitly opt into UTF-8 with the manifest flag (where
> possible) and compiler flag
> 7. Realize that you are mostly on your own here, so investing in a small
> library that will "do the right thing" for your app makes sense
>
> Again, thank you so much for your time. I appreciate all the hard work you
> folks put into this.
>
> On Fri, Jul 3, 2026 at 2:50 AM Tiago Freire <tmiguelf_at_[hidden]> wrote:
>
> I can’t speak for everybody, but I can tell you what I do.
>
>
>
> On Windows apps I just set the console code page to be utf8. The exact
> method to do that has changed as some terminal applications have broken
> certain features. But you can do it in 3 ways:
>
> 1. There’s a global Windows setting that sets the default windows code
> page buried deep in the localization options.
> 2. You can embed the preferred code page in the manifest file for the
> application
> 3. You can set the code page at runtime using a function
>
> To be safe, you can do all 3, but ultimately it is going to depend on the
> terminal’s type setter to respect your preference (sometimes it doesn’t…
> *looks at windows terminal*).
>
>
>
> As for “filesystem::path” they are not Unicode and never will be no matter
> how many times people say otherwise and it is best to acknowledge that
> upfront. To deal with that there is not one but multiple strategies
> depending on the purpose of the “filesystem::path”.
>
> If the purpose is for printing you can create a transcoding function that
> makes certain invalid Unicode characters to be visibly printable indicating
> that “there’s a character there that has been altered for visibility but is
> not the actual character” and that makes “humans with eyes happy”.
>
> If the purpose is to pass trough code that uses [bytes] char8_t to store
> or transfer things, but you would really like to perfectly preserve the
> path, there is a different transcoding function that I use that makes the
> “path round-trip-able” that is “mostly utf8 compatible” but not truly utf8
> in order to encode the invalid code points (i.e. UTF-16 surrogates are
> representable).
>
> So, the right answer is “filesystem::path” is a path, not exactly text but
> text-like. And “what you do with it” depends on “what you want to achieve
> with it”, and Unicode in this context is only relevant if you want to print
> it for a user to see and it is otherwise just bits.
>
>
>
>
>
> As for “[it doesn’t] work with std::format and streams”, I’ve re-written
> my own output, formatting, and encoding library pipeline from scratch and I
> don’t suffer from such problems. The C++ standard has come a long way but
> is comparatively speaking “still just banging rocks together”.
>
> I can share more details regarding what I have done on this front, and
> even tough some concepts can be cannibalized to improve the standard, I’m
> afraid that the C++ status-quo has painted itself too much into a corner
> that is a monumental task to even just start to fix it.
>
>
>
> Best regards,
>
> Tiago
>
>
>
> *From:* SG16 <sg16-bounces_at_[hidden]> *On Behalf Of *Herb Sutter
> via SG16
> *Sent:* Friday, July 3, 2026 01:48
> *To:* sg16_at_[hidden]; 'Christopher Nelson' <nadiasvertex_at_[hidden]>
> *Cc:* Herb Sutter <herb.sutter_at_[hidden]>
> *Subject:* [isocpp-sg16] UTF-8 support status
>
>
>
> Hi C++ Unicode folks, literally asking for a friend (Christopher, on the
> To: line so please Reply-All)…
>
>
>
> His project team wants to use UTF-8 in their code base and they hoped to
> switch to u8string[_view] and char8_t throughout, but they encountered two
> sets of limitations:
>
>
>
> - In the standard, those types don’t seem to work with std::format and
> streams.
>
>
>
> - On Windows, platform APIs interpret narrow characters using the
> active code page (e.g., std::cout emits mojibake,
> filesystem::path{std::string} UTF-8 paths are mangled or throw).
>
>
>
> What’s the current best guidance for adopting UTF-8 in C++ code?
>
>
>
> Thank you,
>
>
>
> Herb
>
>
>
> --
> SG16 mailing list
> SG16_at_[hidden]
> https://lists.isocpp.org/mailman/listinfo.cgi/sg16
> Link to this post: http://lists.isocpp.org/sg16/2026/07/4786.php
>
>
> --
> SG16 mailing list
> SG16_at_[hidden]
> https://lists.isocpp.org/mailman/listinfo.cgi/sg16
> Link to this post: http://lists.isocpp.org/sg16/2026/07/4792.php
>
>
>
> --
> Yongwei Wu
> URL: http://wyw.dcweb.cn/
>
>
-- Yongwei Wu URL: http://wyw.dcweb.cn/
Received on 2026-07-08 01:26:29