To piggyback on what both Peter and Eddie pointed out, there are very few
invariants you want to maintain over-time

- UTF-validity is something you want to check at the boundary of your
system and then preserve - I agree that a code-unit interface makes it easy
to botch things, but I don't think adding containers to the standard
library is a good remedy for that.
- Normalization is something you want to do as little as possible, because
it's expensive. So you do it once.
- Casing etc should not be considered properties, they are transformations
you do when you need.
- There are no locale-dependent properties in Unicode, there are
locale-dependent transformations or algorithms.

So... basically there are very invariants you want to maintain at all
times, you just want algorithms to do the right thing when you need it.
And because there are no invariants, the motivation for unicode-specific
types is somewhat limited to "how are we concerned about people inserting
things in the middle of code unit sequences"

I don't want to dismay outright the idea of codepoint - or even
grapheme-based storage (both Swift and Rust have tools of the sort) - but
they could be std::string-wrapper.
There may also be rationale for different sorts of storage for text, i.e
ropes, etc. But these have less to do with encoding and more with
optimizing specific usage patterns - so there again a layer approach is
best.

(and yes, we all agree char_traits is broken but part of the brokenness
came from these sort of layering violation and mixing of concerns)


It turns out the STL model of "containers contain and algorithms do
interesting things with iterators" works very well with Unicode. And in
that model a locale is just a parameter of the algorithm.

On Tue, 7 Jul 2026 at 19:15, Eddie Nolan via SG16 <sg16_at_[hidden]>
wrote:

> I agree with Peter’s points. I don’t think we should be moving in the
> direction of a new, Unicode-aware string type to provide Unicode support. I
> don’t think C++ is going to move away from a world where, as far as
> standard string types go, effectively the only thing that the type system
> tells you is what type it uses for its code units. char_traits is
> essentially legacy; we shouldn’t be thinking about making changes to it. If
> users want more features out of their text types— for example, a type like
> Rust’s String that enforces UTF-8 validity, or a string with an
> associated runtime collation order— they can implement their own libraries
> that use the building blocks the standard library gives them. (This is my
> own view, which may or may not reflect SG16’s current consensus.)
>
> I can also provide some additional details about my longer-term plans for
> standard library Unicode support.
>
> I want to use the ranges library as the vehicle for providing additional
> Unicode functionality that users of standard string types can leverage.
>
> So far my work on this has been scoped entirely to UTF transcoding (P2728
> <https://isocpp.org/files/papers/P2728R14.html>), but Zach Laine also
> wrote two additional papers that I’m planning to try to revive once my work
> on P2728 is complete.
>
> Those are P2729
> <https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p2729r0.html>,
> which will handle normalization, and the unpublished draft D2745
> <https://github.com/tzlaine/small_wg1_papers/blob/master/P2745_unicode_3_segmentation.md>,
> which will handle grapheme clustering and word/sentence/line breaking.
>
> In terms of how these proposals work with strings:
>
> - If you want to convert a std::string (or std::string_view) that
> contains UTF-8 text to “a container of code points,” you’d just convert it
> to a std::u32string. With P2728’s transcoding facilities, you’d spell
> that as std::u32string u32s = my_string | views::as_char8_t |
> views::to_utf32 | ranges::to<std::u32string>().
> - If you want to convert a std::string (or std::string_view) that
> contains UTF-8 text to NFC, once I update P2729, I expect the API to be
> shaped something like std::string normalized = my_string |
> views::as_char8_t | views::to_nfc<char8_t> | views::as_char |
> ranges::to<std::string>().
> - With respect to D2745’s text segmentation APIs, there’s a lot of
> design work that needs to be done, but one example use case might be
> extracting the first word from a string with something like std::string
> first_word = my_string | views::as_char8_t | views::to_unicode_words |
> views::take(1) | views::join | views::as_char | ranges::to<std::string>()
> .
>
> (Note that, for P2729 and P2745, my examples don’t reflect the current
> status of the paper, and are hypothetical APIs that will arrive in updated
> revisions.)
>
> Thanks,
>
> Eddie
>
>
> On Tue, Jul 7, 2026 at 12:44 PM Peter Bindels via SG16 <
> sg16_at_[hidden]> wrote:
>
>> Hi Alisdair,
>>
>> On Tuesday, July 7th, 2026 at 6:15 PM, Alisdair Meredith via SG16 <
>> sg16_at_[hidden]> wrote:
>>
>> > Raising a distinct concern from the thread on best practice for Unicode
>> today, does this group have a position on whether `basic_string` is an
>> appropriate type for text processing in Unicode?
>>
>> For text *storage*, it's the right type to use. It stores code units in
>> whatever underlying character type it uses, and for u8/u16/u32 it's a
>> Unicode type. All storage happens in terms of code units regardless.
>>
>> For text *processing*, we almost always want to view them as code points,
>> or higher-level constructs on top of that. Unicode properties that we want
>> to have aren't necessarily retained in simple code unit changes, such as "I
>> have a whole number of code points", "I have valid utf8", "my text is NFC
>> normalized" or "concatenating two strings of X + Y grapheme clusters
>> results in X+Y grapheme clusters". We need to have a view transform that
>> provides this code point view from the underlying code unit storage, and we
>> need to have a view transform that is usable for storing a code point range
>> of some kind into a code unit storage.
>>
>> With those fundamentals, we can build code point algorithms and grapheme
>> algorithms on top of them.
>>
>> > From my perspective, `basic_string` (and its type aliases) is a
>> container of code units, where for text processing we want a container of
>> code points. Likewise, is it parameterized on `char_traits` where we do
>> not provide a traits type following Unicode rules, and in fact some things
>> like collation order we would want to customize at runtime, but are baked
>> into a compile-time decision through this traits parameter. The various
>> functions that sound like processing text are not well suited to any
>> multi-byte encoding system, not just Unicode,but as long as we live in
>> ASCII plus the various codepage extensions native to our environment, we do
>> not run into issues.
>>
>> Char_traits is fundamentally broken and unsuitable for anything Unicode.
>> We need to think of it as ossified. Collation is a property of a collection
>> or sort invocation, not of a string type. The cstring_view proposal has
>> char_traits so that it can convert from string to string_view through it
>> without data loss, but beyond that SG16 agreed that it's useless.
>>
>> > The counterpoint is that `std::string` has been our established
>> vocabulary for the last 30 years, and `std::string_view` for the last 10.
>> Does that weight of that legacy make it intractable to provide a “better”
>> solution for a world that has slowly standardized on Unicode as its
>> preferred encoding scheme (even if adherence to semantics like collation
>> order may be less universal)?
>>
>> They're okay storage types. They have a bunch of functions that have no
>> Unicode meaning, and they have a lot of ossified features that we cannot
>> modify and that serve no purpose, but any replacement at best would do the
>> same thing modulo char_traits and string-like functions, and would
>> necessarily face an uphill battle against the existing types that work just
>> as well.
>>
>>
>> This is my understanding, please correct me if you disagree.
>> --
>> SG16 mailing list
>> SG16_at_[hidden]
>> https://lists.isocpp.org/mailman/listinfo.cgi/sg16
>> Link to this post: http://lists.isocpp.org/sg16/2026/07/4799.php
>>
> --
> SG16 mailing list
> SG16_at_[hidden]
> https://lists.isocpp.org/mailman/listinfo.cgi/sg16
> Link to this post: http://lists.isocpp.org/sg16/2026/07/4803.php
>