Thanks folks, I really appreciate all of the guidance.

The specific codebase we are working on is group of libraries that has
bindings for several programming languages. C++(17/23/26) , C#, nodejs, and
Rust. It runs on Linux, macOS, and Windows. We control all the applications
it is embedded in.

May I summarize what I understood from the conversation and you can tell me
if I am on the right track?

1. Use std::string
2. Assume that all internal string data is UTF-8
3. Enforce that invariant at input boundaries using helper functions
4. Be explicit about output encoding if it matters, default to UTF-8
5. Make formatting helpers that understand intent (display/file/etc.)
6. For Windows, explicitly opt into UTF-8 with the manifest flag (where
possible) and compiler flag
7. Realize that you are mostly on your own here, so investing in a small
library that will "do the right thing" for your app makes sense

Again, thank you so much for your time. I appreciate all the hard work you
folks put into this.

On Fri, Jul 3, 2026 at 2:50 AM Tiago Freire <tmiguelf_at_[hidden]> wrote:

> I can’t speak for everybody, but I can tell you what I do.
>
>
>
> On Windows apps I just set the console code page to be utf8. The exact
> method to do that has changed as some terminal applications have broken
> certain features. But you can do it in 3 ways:
>
> 1. There’s a global Windows setting that sets the default windows code
> page buried deep in the localization options.
> 2. You can embed the preferred code page in the manifest file for the
> application
> 3. You can set the code page at runtime using a function
>
> To be safe, you can do all 3, but ultimately it is going to depend on the
> terminal’s type setter to respect your preference (sometimes it doesn’t…
> *looks at windows terminal*).
>
>
>
> As for “filesystem::path” they are not Unicode and never will be no matter
> how many times people say otherwise and it is best to acknowledge that
> upfront. To deal with that there is not one but multiple strategies
> depending on the purpose of the “filesystem::path”.
>
> If the purpose is for printing you can create a transcoding function that
> makes certain invalid Unicode characters to be visibly printable indicating
> that “there’s a character there that has been altered for visibility but is
> not the actual character” and that makes “humans with eyes happy”.
>
> If the purpose is to pass trough code that uses [bytes] char8_t to store
> or transfer things, but you would really like to perfectly preserve the
> path, there is a different transcoding function that I use that makes the
> “path round-trip-able” that is “mostly utf8 compatible” but not truly utf8
> in order to encode the invalid code points (i.e. UTF-16 surrogates are
> representable).
>
> So, the right answer is “filesystem::path” is a path, not exactly text but
> text-like. And “what you do with it” depends on “what you want to achieve
> with it”, and Unicode in this context is only relevant if you want to print
> it for a user to see and it is otherwise just bits.
>
>
>
>
>
> As for “[it doesn’t] work with std::format and streams”, I’ve re-written
> my own output, formatting, and encoding library pipeline from scratch and I
> don’t suffer from such problems. The C++ standard has come a long way but
> is comparatively speaking “still just banging rocks together”.
>
> I can share more details regarding what I have done on this front, and
> even tough some concepts can be cannibalized to improve the standard, I’m
> afraid that the C++ status-quo has painted itself too much into a corner
> that is a monumental task to even just start to fix it.
>
>
>
> Best regards,
>
> Tiago
>
>
>
> *From:* SG16 <sg16-bounces_at_[hidden]> *On Behalf Of *Herb Sutter
> via SG16
> *Sent:* Friday, July 3, 2026 01:48
> *To:* sg16_at_[hidden]; 'Christopher Nelson' <nadiasvertex_at_[hidden]>
> *Cc:* Herb Sutter <herb.sutter_at_[hidden]>
> *Subject:* [isocpp-sg16] UTF-8 support status
>
>
>
> Hi C++ Unicode folks, literally asking for a friend (Christopher, on the
> To: line so please Reply-All)…
>
>
>
> His project team wants to use UTF-8 in their code base and they hoped to
> switch to u8string[_view] and char8_t throughout, but they encountered two
> sets of limitations:
>
>
>
> - In the standard, those types don’t seem to work with std::format and
> streams.
>
>
>
> - On Windows, platform APIs interpret narrow characters using the
> active code page (e.g., std::cout emits mojibake,
> filesystem::path{std::string} UTF-8 paths are mangled or throw).
>
>
>
> What’s the current best guidance for adopting UTF-8 in C++ code?
>
>
>
> Thank you,
>
>
>
> Herb
>
>
>