In that case, then yes. No doubt about it. You should heavily invest in a library set to do all the formatting and encoding for you, what the standard can provide is not going to get you there.
I was talking in terms of char8_t, but that might not even be enough, if it goes far enough you might even have to deal with encodings that are not Unicode, you might need to bring in all of the char types and even that might not be enough.
From: Christopher Nelson <nadiasvertex@gmail.com>
Sent: Friday, July 3, 2026 17:22
To: Tiago Freire <tmiguelf@hotmail.com>
Cc: sg16@lists.isocpp.org; Herb Sutter <herb.sutter@gmail.com>
Subject: Re: [isocpp-sg16] UTF-8 support status
Thanks for the detail! Just to help the thread, the code is a publishing system that supports over 1200 human languages. We are very familiar with Unicode and have extensive custom Unicode support. We are trying to determine how to make
the best use of the language and standard library facilities in our own unicode library as we modernize the code.
We primarily want to remove footguns for new developers and make doing the right thing the easy thing.
On Fri, Jul 3, 2026 at 11:03 AM Tiago Freire <tmiguelf@hotmail.com> wrote:
> May I summarize what I understood from the conversation and you can tell me if I am on the right track?
> 1. Use std::string
Eh.. That is a very complex and controversial topic encompassed in 2 words. I’m going to have to spread into 2 categories:
1. Professionally I don’t recommend the usage of ‘std::string’, and devs should avoid it like the plague simply because of the over-usage of ‘std::string’ forces you to heap allocate even when you don’t need it. Prefer something like string_view instead. Sure, if you need to store it then fine use ‘std::string’, but avoid it when possible.
You have no idea how much difference this makes. I’ve profiled this in my company on applications that didn’t have this kind of discipline in their code base, and the top item consuming most cpu cycles out of everything by a long margin was ‘std::string::string()’. It was that bad.
But “std::string” and “std::string_view” is just an alias to “std::basic_string<char>” and “std::string_view<char>”, and “std::basic_string<char8_t>”/“std::string_view<char8_t>” (aka std::u8string, std::u8string_view) are also things that exist. It’s just best to talk in terms of ‘char’ family and ‘char8_t’ family of structures.
2. Which one is best ‘char’ or ‘char8_t’ family?
The short answer is, It depends.
If you need to support C++17 then you are stuck with ‘char’, because ‘char8_t’ is a C++20 or above feature.
But if you are at C++20 or above, I would recommend ‘char8_t’ but with asterisks. I regularly even use ‘char16_t’ and ‘char32_t’, but that depends on the purpose.
For logging (which involves mostly just dealing with printable text) then ‘char8_t’ all the way. Loading text from files, I like ‘char8_t’. For those I have moved on to std::u8string without a problem, even for text that is just ASCII I use char8_t over char, because char has a portability problem that its not really signed or unsigned and the result of things like `this_char < ‘A’` will vary depending on implementation, which was very annoying, and char8_t did genuinely solved that problem for me even if it was only originally intended to be used for utf-8.
But the “processing text” is a very small portion of the problems you might want to solve in software. File system paths are not Unicode, neither are environment variables, neither are command line arguments.
If your application is just reading those, then you can safely say “Everything is utf-8” I just validate to confirm that the strings are actually valid utf-8 encoded strings and if they aren’t I just reject them like I would reject any other form of invalid input. It gets a little bit trickier if you are dealing with paths, but depending on the application you can sometimes get away with “yeah I’m not going to support paths if they are not valid unicode”.
But sometimes you can not afford to do that, specially if you are getting something from a system that you don’t control and you are passing it to another system that you can’t control, then the solution I opt for that is to define a thing called os_char_t that is going to be a wchar_t on Windows and char pretty much everywhere else (and sometimes char16_t and char8_t), it’s a lot trickier to write code this way.
So, it really depends on what your application is, as with everything each solution has its trade-offs and it’s a matter of what you are willing to live with.
> 2. Assume that all internal string data is UTF-8
No!
Absolutely not, don’t make that mistake.
And again. It’s a big topic. If it’s really important, you should validate your text, it’s possible to store invalid utf-8 in a std::u8string, so check it if you are getting your strings from somewhere.
If you ask me if I make checks for valid utf-8 everywhere? Also no, absolutely not!
The right way to approach this is “I don’t care”. People coding on EBDCI systems will disagree, but if you are running a system that is a superset of ASCII, sometimes just saying “it’s a superset of ASCII” is good enough. utf-8 is a superset of ascii, so if you want to think that it is it doesn’t really matter, your computer doesn’t care. It’s something that doesn’t really matter until you try to print something on screen. You encode a valid utf-8 string, push into to a file, you print it to the console, you see Unicode characters, you are happy, did it really matter that it wasn’t really Unicode? I.e. that you could in theory created an invalid utf-8 sequence, everything will consume it just the same, you may not be able to see it, but you can still create that file with that sequence of bytes that is invalid in Unicode. You often sub-string the same, you concatenate the same way. The difference is only visible when you want to display it.
> 3. Enforce that invariant at input boundaries using helper functions
That’s a good approach for everything.
> 4. Be explicit about output encoding if it matters, default to UTF-8
Yes.
> 5. Make formatting helpers that understand intent (display/file/etc.)
Yes.
> 6. For Windows, explicitly opt into UTF-8 with the manifest flag (where possible) and compiler flag
Good practice. But also test your terminal application. What get’s displayed in your terminal is not actually controlled by your application, it’s controlled by your terminal.
Most terminals have good support for utf-8 now a days, but not every terminal you might encounter, and advanced terminal features (like figuring out the display widths) there’s no universal support for that.
> 7. Realize that you are mostly on your own here, so investing in a small library that will "do the right thing" for your app makes sense
If you want really good Unicode support, then yes. If you plan to deal with multiple languages, even more so. UI heavy libraries will often already come with those.
But to be honest for most applications you don’t really need it. As I might have hinted, it can be a very complex topic to deal with text correctly, if you are “just making it in english” you probably don’t need to waste all that effort, 90% would not recommend. If you want that last 10% it’s a massive effort increase, consider if your application is really worth it.
From: Christopher Nelson <nadiasvertex@gmail.com>
Sent: Friday, July 3, 2026 14:40
To: Tiago Freire <tmiguelf@hotmail.com>
Cc: sg16@lists.isocpp.org; Herb Sutter <herb.sutter@gmail.com>
Subject: Re: [isocpp-sg16] UTF-8 support status
Thanks folks, I really appreciate all of the guidance.
The specific codebase we are working on is group of libraries that has bindings for several programming languages. C++(17/23/26) , C#, nodejs, and Rust. It runs on Linux, macOS, and Windows. We control all the applications it is embedded in.
May I summarize what I understood from the conversation and you can tell me if I am on the right track?
1. Use std::string
2. Assume that all internal string data is UTF-83. Enforce that invariant at input boundaries using helper functions
4. Be explicit about output encoding if it matters, default to UTF-8
5. Make formatting helpers that understand intent (display/file/etc.)
6. For Windows, explicitly opt into UTF-8 with the manifest flag (where possible) and compiler flag
7. Realize that you are mostly on your own here, so investing in a small library that will "do the right thing" for your app makes sense
Again, thank you so much for your time. I appreciate all the hard work you folks put into this.
On Fri, Jul 3, 2026 at 2:50 AM Tiago Freire <tmiguelf@hotmail.com> wrote:
I can’t speak for everybody, but I can tell you what I do.
On Windows apps I just set the console code page to be utf8. The exact method to do that has changed as some terminal applications have broken certain features. But you can do it in 3 ways:
- There’s a global Windows setting that sets the default windows code page buried deep in the localization options.
- You can embed the preferred code page in the manifest file for the application
- You can set the code page at runtime using a function
To be safe, you can do all 3, but ultimately it is going to depend on the terminal’s type setter to respect your preference (sometimes it doesn’t… *looks at windows terminal*).
As for “filesystem::path” they are not Unicode and never will be no matter how many times people say otherwise and it is best to acknowledge that upfront. To deal with that there is not one but multiple strategies depending on the purpose of the “filesystem::path”.
If the purpose is for printing you can create a transcoding function that makes certain invalid Unicode characters to be visibly printable indicating that “there’s a character there that has been altered for visibility but is not the actual character” and that makes “humans with eyes happy”.
If the purpose is to pass trough code that uses [bytes] char8_t to store or transfer things, but you would really like to perfectly preserve the path, there is a different transcoding function that I use that makes the “path round-trip-able” that is “mostly utf8 compatible” but not truly utf8 in order to encode the invalid code points (i.e. UTF-16 surrogates are representable).
So, the right answer is “filesystem::path” is a path, not exactly text but text-like. And “what you do with it” depends on “what you want to achieve with it”, and Unicode in this context is only relevant if you want to print it for a user to see and it is otherwise just bits.
As for “[it doesn’t] work with std::format and streams”, I’ve re-written my own output, formatting, and encoding library pipeline from scratch and I don’t suffer from such problems. The C++ standard has come a long way but is comparatively speaking “still just banging rocks together”.
I can share more details regarding what I have done on this front, and even tough some concepts can be cannibalized to improve the standard, I’m afraid that the C++ status-quo has painted itself too much into a corner that is a monumental task to even just start to fix it.
Best regards,
Tiago
From: SG16 <sg16-bounces@lists.isocpp.org> On Behalf Of Herb Sutter via SG16
Sent: Friday, July 3, 2026 01:48
To: sg16@lists.isocpp.org; 'Christopher Nelson' <nadiasvertex@gmail.com>
Cc: Herb Sutter <herb.sutter@gmail.com>
Subject: [isocpp-sg16] UTF-8 support status
Hi C++ Unicode folks, literally asking for a friend (Christopher, on the To: line so please Reply-All)…
His project team wants to use UTF-8 in their code base and they hoped to switch to u8string[_view] and char8_t throughout, but they encountered two sets of limitations:
- In the standard, those types don’t seem to work with std::format and streams.
- On Windows, platform APIs interpret narrow characters using the active code page (e.g., std::cout emits mojibake, filesystem::path{std::string} UTF-8 paths are mangled or throw).
What’s the current best guidance for adopting UTF-8 in C++ code?
Thank you,
Herb