C++ Logo

sg16

Advanced search

Re: [isocpp-sg16] UTF-8 support status

From: Thiago Macieira <thiago_at_[hidden]>
Date: Tue, 07 Jul 2026 10:00:30 -0700
On Tuesday, 7 July 2026 08:39:56 Pacific Daylight Time Tiago Freire wrote:
> There's lack of transcoding features for sure, but that doesn't make "what a
> user might do" correct. Visibility of this issue might be hard on popular
> systems (which are mostly ASCII supersets), but when the system diverges
> from this it becomes a problem.
>
> On an EBDIC system you wouldn't be able to reinterpret cast, as "string"
> would be bitwise different from u8"string". This would be very noticeably a
> problem. You will need to thread very gingerly let's say text that comes
> from the command line, which should obviously be std::string, and if you
> are reading from a file which you know is utf-8 it should be std: u8string.
>
> If you want portability to these potential systems then use std::u8string.

Strictly speaking, that's true, but the number of people who will ever touch
an EBCDIC system and don't work for IBM is virtually indistiguishable from
zero. You don't need UTF-8 to build in dependencies on ASCII; a simple
  if (c >= 'a' && c <= 'z')
is wrong under EBCDIC.

No, the biggest danger are assumptions that break with Windows, namely:
 1) the narrow character execution charset is UTF-8
which causes
 2) all file names and environment variables can be encoded under that charset

std::filesystem being UTF-16 aware on Windows introduces more issues, because
now it's possible to get a directory listing and get fs::path objects for files
whose names cannot be encoded in the narrow character charset.

-- 
Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org
  Principal Engineer - Intel Data Center - Platform & Sys. Eng.

Received on 2026-07-07 17:00:41