C++ Logo

sg16

Advanced search

Re: [isocpp-sg16] UTF-8 support status

From: Thiago Macieira <thiago_at_[hidden]>
Date: Thu, 02 Jul 2026 21:43:24 -0700
On Thursday, 2 July 2026 18:12:05 Pacific Daylight Time Tony V E via SG16
wrote:
> Isn't there a setting (ie on the compiler or in the exe manifest, etc) now
> under Windows to mark your exe as UTF8 such that your exe sees UTF8 as the
> active code page, regardless of what the OS has as the active code page?

There is, but only the application a.k.a. the owner of main() can set it. If
you're writing library code, you can't set it and you can't depend on it.

> > His project team wants to use UTF-8 in their code base and they hoped to
> > switch to u8string[_view] and char8_t throughout, but they encountered two
> > sets of limitations:
> >
> > - In the standard, those types don’t seem to work with std::format and
> > streams.

Right. I've also asked about support for u16string because it's more useful on
Windows, but right now there is no action on it. To me, it would be even more
useful than u8string_view, but that's not the same for everyone. Supporting
either u8 or u16 would solve most problems, so I'm eagerly awaiting conclusion
here too.

Just don't spend time on wstring_view and u32string_view. No one I know uses
those by choice.

> > - On Windows, platform APIs interpret narrow characters using the
> > active code page (e.g., std::cout emits mojibake,
> > filesystem::path{std::string} UTF-8 paths are mangled or throw).

std::cout is a nightmare on its own because there aren't two encodings on
Windows consoles, there are three: the ACP, the UTF-16 one, and the legacy DOS
one.

So yes it produces mojibake. Strictly speaking, the Standard is silent about
reading what gets printed, so printing mojibake is perfecty Standards-
compliant. But there are Win32 APIs to properly print (WriteConsoleW), so if
the implementation knows that std::cout is connected to the console, it can
properly print in a way that the user can read. Microsoft's UCRT knows stdout
is connected to a console, but it does not use WriteConsoleW. A hypothetical
std::print(utf8string_view) could DTRT.

This is only about the console, though. It breaks completely down if the I/O
is on something else like a file or a pipe. Should you encode it in UTF-8 or in
ANSI? It's unknowable and out of scope for the Standard (see my recommendation
below).

std::filesystem::path with u8string_view is supposed to DTRT and convert from
UTF-8 to UTF-16 and back. If it doesn't, it's a bug in the implementation
you're using. std::filesystem was designed with this Windows quirk in mind.

> > What’s the current best guidance for adopting UTF-8 in C++ code?

My recommendations after using Unicode for 20 years in cross-platform
environments are:
- if you are writing the application, set the manifest flag that Tony
  mentioned. That will make char == char8_t and make main()'s argv work.

- char is UTF-8 and forget char8_t for most uses. Just reinterpret_cast where
  needed to force compliance.

- on Windows, use only Unicode-aware API. If you set the flag above, great, all
  API calls are Unicode-aware. If not or if you don't know (library code),
  then use only those that end up calling the Win32 "W" functions:
  std::filesystem, _wopen(), _wgetenv(), etc..

- assume all byte I/O is UTF-8 (pipes, file encodings, etc.), unless an
  encoding marker is present (header, context, etc.). Open files in
  binary mode and perform CRLF conversion yourself (or no conversion).

- dealing with the C runtime's text conversion is a lost cause (though if
  you've set the UTF-8 application flag, the runtime will just perform a bunch
  of no-op UTF-8→UTF-16→UTF-8 conversions). Use WriteConsoleW if you can,
  restrict your text mode output to English and US-ASCII if you can't. For
  anything more than catastrophic errors or bare-bones information, use the
  GUI.

-- 
Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org
  Principal Engineer - Intel Data Center - Platform & Sys. Eng.

Received on 2026-07-03 04:43:32