Date: Tue, 7 Jul 2026 18:49:47 +0200
Okay, so, a few things.
- std::string is a owning, null-terminated bag of bytes. It has no
implications of encoding, nor does it need one. all of the member functions
in std::string are encoding neutral.
In the wild, std::string, and other char-based types have been used to
represent binary content, text in the literal encoding, or execution
encoding, text in UTF-8, text in some other arbitrary encoding.
To use std::string in a way that assumes a specific encoding, you must be
able to understand what produced it, where it comes from.
If you use std::string with utf-8, you must understand that nothing is
going to prevent you from mishandling that utf8-sequences. ie you can split
code unit sequences etc.
The standard library usually assumes char-sequences and std::string are
in the execution-encoding, however different facilities make different
assumptions, based on where they were standardized.
Read the doc
- char8_t and u8string etc aim to solve some of these issues, by having the
intent carried through the type system that a u8string object does indeed
carry utf-8 data. Note however that there is no invariant enforcement in
std::u8string
or other standard u8 types. Their invariants are exactly the same as
those of std::string.
- Yes the support for char8_t is lacking in the standard library, it is
something we are actively working on, slowly. very slowly. In large part
because we keep having the same discussion that transpires in this thread.
- Yes you can, and should consider enabling utf-8 support for your windows
targeting application if you can. But having a utf8-locate on windows
certainly won't solve all of your problems.
- Until such time WG21 either fully supports char8_t in the library, or
gives you a way to convert char8_t to char or the other way around, you
might want to consider using, for example, unsigned char to represent utf-8
data.
You can also use char8_t and copy strings to char sequences when
operating with the standard library, or reinterpret_cast it and accept the
UB (which might manifest through weird pointer aliasing), or use
optimization barrier the way ICU does it.
On Tue, 7 Jul 2026 at 18:22, Alisdair Meredith via SG16 <
sg16_at_[hidden]> wrote:
>
> > On Jul 2, 2026, at 11:48 PM, Thiago Macieira via SG16 <
> sg16_at_[hidden]> wrote:
> >
> > On Thursday, 2 July 2026 20:01:55 Pacific Daylight Time Victor Zverovich
> via
> > SG16 wrote:
> >> Moreover, ACP is mostly irrelevant in modern code and it's sufficient to
> >> compile with /utf-8 on MSVC (and on most other systems UTF-8 is the
> default
> >> for char).
> >
> > That's not correct. That causes the compiler to interpret the source
> input as
> > UTF-8 and emit narrow character literals as UTF-8. It does not change
> how the
> > narrow character APIs behave at runtime.
> >
> > To do that, you need to set the manifest flag that Tony mentioned. And
> only the
> > application (a.k.a. the owner of main()) can do that. Library code
> cannot and
> > must therefore cope with being run with another ACP. It could declare
> other
> > ACPs are not supported, but that would limit the audience of the
> library, as
> > most applications do not set the flag for some reason or another,
> including but
> > not limited to simply not knowing about it.
>
> I remember a few years back some folks were advocating using
> `std::filesystem::path`
> as their basic string type, as its wrapped “native” string would be the
> most appropriate
> representation on a given platform at runtime.
>
> That is not necessarily an answer for Unicode, but an indicator of how
> creative users
> have become trying to work around some of the issues of portable strings.
>
> AlisdairM
>
> --
> SG16 mailing list
> SG16_at_[hidden]
> https://lists.isocpp.org/mailman/listinfo.cgi/sg16
> Link to this post: http://lists.isocpp.org/sg16/2026/07/4798.php
>
- std::string is a owning, null-terminated bag of bytes. It has no
implications of encoding, nor does it need one. all of the member functions
in std::string are encoding neutral.
In the wild, std::string, and other char-based types have been used to
represent binary content, text in the literal encoding, or execution
encoding, text in UTF-8, text in some other arbitrary encoding.
To use std::string in a way that assumes a specific encoding, you must be
able to understand what produced it, where it comes from.
If you use std::string with utf-8, you must understand that nothing is
going to prevent you from mishandling that utf8-sequences. ie you can split
code unit sequences etc.
The standard library usually assumes char-sequences and std::string are
in the execution-encoding, however different facilities make different
assumptions, based on where they were standardized.
Read the doc
- char8_t and u8string etc aim to solve some of these issues, by having the
intent carried through the type system that a u8string object does indeed
carry utf-8 data. Note however that there is no invariant enforcement in
std::u8string
or other standard u8 types. Their invariants are exactly the same as
those of std::string.
- Yes the support for char8_t is lacking in the standard library, it is
something we are actively working on, slowly. very slowly. In large part
because we keep having the same discussion that transpires in this thread.
- Yes you can, and should consider enabling utf-8 support for your windows
targeting application if you can. But having a utf8-locate on windows
certainly won't solve all of your problems.
- Until such time WG21 either fully supports char8_t in the library, or
gives you a way to convert char8_t to char or the other way around, you
might want to consider using, for example, unsigned char to represent utf-8
data.
You can also use char8_t and copy strings to char sequences when
operating with the standard library, or reinterpret_cast it and accept the
UB (which might manifest through weird pointer aliasing), or use
optimization barrier the way ICU does it.
On Tue, 7 Jul 2026 at 18:22, Alisdair Meredith via SG16 <
sg16_at_[hidden]> wrote:
>
> > On Jul 2, 2026, at 11:48 PM, Thiago Macieira via SG16 <
> sg16_at_[hidden]> wrote:
> >
> > On Thursday, 2 July 2026 20:01:55 Pacific Daylight Time Victor Zverovich
> via
> > SG16 wrote:
> >> Moreover, ACP is mostly irrelevant in modern code and it's sufficient to
> >> compile with /utf-8 on MSVC (and on most other systems UTF-8 is the
> default
> >> for char).
> >
> > That's not correct. That causes the compiler to interpret the source
> input as
> > UTF-8 and emit narrow character literals as UTF-8. It does not change
> how the
> > narrow character APIs behave at runtime.
> >
> > To do that, you need to set the manifest flag that Tony mentioned. And
> only the
> > application (a.k.a. the owner of main()) can do that. Library code
> cannot and
> > must therefore cope with being run with another ACP. It could declare
> other
> > ACPs are not supported, but that would limit the audience of the
> library, as
> > most applications do not set the flag for some reason or another,
> including but
> > not limited to simply not knowing about it.
>
> I remember a few years back some folks were advocating using
> `std::filesystem::path`
> as their basic string type, as its wrapped “native” string would be the
> most appropriate
> representation on a given platform at runtime.
>
> That is not necessarily an answer for Unicode, but an indicator of how
> creative users
> have become trying to work around some of the issues of portable strings.
>
> AlisdairM
>
> --
> SG16 mailing list
> SG16_at_[hidden]
> https://lists.isocpp.org/mailman/listinfo.cgi/sg16
> Link to this post: http://lists.isocpp.org/sg16/2026/07/4798.php
>
Received on 2026-07-07 16:50:09
