I disagree slightly on whether we should have a better concrete Text type
that maintains, for example, the shape necessary to produce a string that
python will consume, and not reject before getting started.

But, it's, I think, a lower priority than getting the rest of the Unicode
algorithm repetoire available.

For non UTF actual multibyte encodings, basic_string is actually worse than
it is for UTF-8 or 32, even for 16. The legacy encodings are forward-only.
You can not start at an arbitrary position and resync, forward or
backwards, and the representation self aliases. So every access method
other than sequential from the beginning and decoding is, at least
potentially, wrong.

Being a fancy char* is just barely enough to justify it. But barely.



On Tue, Jul 7, 2026, 13:15 Eddie Nolan via SG16 <sg16_at_[hidden]>
wrote:

> I agree with Peter’s points. I don’t think we should be moving in the
> direction of a new, Unicode-aware string type to provide Unicode support. I
> don’t think C++ is going to move away from a world where, as far as
> standard string types go, effectively the only thing that the type system
> tells you is what type it uses for its code units. char_traits is
> essentially legacy; we shouldn’t be thinking about making changes to it. If
> users want more features out of their text types— for example, a type like
> Rust’s String that enforces UTF-8 validity, or a string with an
> associated runtime collation order— they can implement their own libraries
> that use the building blocks the standard library gives them. (This is my
> own view, which may or may not reflect SG16’s current consensus.)
>
> I can also provide some additional details about my longer-term plans for
> standard library Unicode support.
>
> I want to use the ranges library as the vehicle for providing additional
> Unicode functionality that users of standard string types can leverage.
>
> So far my work on this has been scoped entirely to UTF transcoding (P2728
> <https://isocpp.org/files/papers/P2728R14.html>), but Zach Laine also
> wrote two additional papers that I’m planning to try to revive once my work
> on P2728 is complete.
>
> Those are P2729
> <https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p2729r0.html>,
> which will handle normalization, and the unpublished draft D2745
> <https://github.com/tzlaine/small_wg1_papers/blob/master/P2745_unicode_3_segmentation.md>,
> which will handle grapheme clustering and word/sentence/line breaking.
>
> In terms of how these proposals work with strings:
>
> - If you want to convert a std::string (or std::string_view) that
> contains UTF-8 text to “a container of code points,” you’d just convert it
> to a std::u32string. With P2728’s transcoding facilities, you’d spell
> that as std::u32string u32s = my_string | views::as_char8_t |
> views::to_utf32 | ranges::to<std::u32string>().
> - If you want to convert a std::string (or std::string_view) that
> contains UTF-8 text to NFC, once I update P2729, I expect the API to be
> shaped something like std::string normalized = my_string |
> views::as_char8_t | views::to_nfc<char8_t> | views::as_char |
> ranges::to<std::string>().
> - With respect to D2745’s text segmentation APIs, there’s a lot of
> design work that needs to be done, but one example use case might be
> extracting the first word from a string with something like std::string
> first_word = my_string | views::as_char8_t | views::to_unicode_words |
> views::take(1) | views::join | views::as_char | ranges::to<std::string>()
> .
>
> (Note that, for P2729 and P2745, my examples don’t reflect the current
> status of the paper, and are hypothetical APIs that will arrive in updated
> revisions.)
>
> Thanks,
>
> Eddie
>
>
> On Tue, Jul 7, 2026 at 12:44 PM Peter Bindels via SG16 <
> sg16_at_[hidden]> wrote:
>
>> Hi Alisdair,
>>
>> On Tuesday, July 7th, 2026 at 6:15 PM, Alisdair Meredith via SG16 <
>> sg16_at_[hidden]> wrote:
>>
>> > Raising a distinct concern from the thread on best practice for Unicode
>> today, does this group have a position on whether `basic_string` is an
>> appropriate type for text processing in Unicode?
>>
>> For text *storage*, it's the right type to use. It stores code units in
>> whatever underlying character type it uses, and for u8/u16/u32 it's a
>> Unicode type. All storage happens in terms of code units regardless.
>>
>> For text *processing*, we almost always want to view them as code points,
>> or higher-level constructs on top of that. Unicode properties that we want
>> to have aren't necessarily retained in simple code unit changes, such as "I
>> have a whole number of code points", "I have valid utf8", "my text is NFC
>> normalized" or "concatenating two strings of X + Y grapheme clusters
>> results in X+Y grapheme clusters". We need to have a view transform that
>> provides this code point view from the underlying code unit storage, and we
>> need to have a view transform that is usable for storing a code point range
>> of some kind into a code unit storage.
>>
>> With those fundamentals, we can build code point algorithms and grapheme
>> algorithms on top of them.
>>
>> > From my perspective, `basic_string` (and its type aliases) is a
>> container of code units, where for text processing we want a container of
>> code points. Likewise, is it parameterized on `char_traits` where we do
>> not provide a traits type following Unicode rules, and in fact some things
>> like collation order we would want to customize at runtime, but are baked
>> into a compile-time decision through this traits parameter. The various
>> functions that sound like processing text are not well suited to any
>> multi-byte encoding system, not just Unicode,but as long as we live in
>> ASCII plus the various codepage extensions native to our environment, we do
>> not run into issues.
>>
>> Char_traits is fundamentally broken and unsuitable for anything Unicode.
>> We need to think of it as ossified. Collation is a property of a collection
>> or sort invocation, not of a string type. The cstring_view proposal has
>> char_traits so that it can convert from string to string_view through it
>> without data loss, but beyond that SG16 agreed that it's useless.
>>
>> > The counterpoint is that `std::string` has been our established
>> vocabulary for the last 30 years, and `std::string_view` for the last 10.
>> Does that weight of that legacy make it intractable to provide a “better”
>> solution for a world that has slowly standardized on Unicode as its
>> preferred encoding scheme (even if adherence to semantics like collation
>> order may be less universal)?
>>
>> They're okay storage types. They have a bunch of functions that have no
>> Unicode meaning, and they have a lot of ossified features that we cannot
>> modify and that serve no purpose, but any replacement at best would do the
>> same thing modulo char_traits and string-like functions, and would
>> necessarily face an uphill battle against the existing types that work just
>> as well.
>>
>>
>> This is my understanding, please correct me if you disagree.
>> --
>> SG16 mailing list
>> SG16_at_[hidden]
>> https://lists.isocpp.org/mailman/listinfo.cgi/sg16
>> Link to this post: http://lists.isocpp.org/sg16/2026/07/4799.php
>>
> --
> SG16 mailing list
> SG16_at_[hidden]
> https://lists.isocpp.org/mailman/listinfo.cgi/sg16
> Link to this post: http://lists.isocpp.org/sg16/2026/07/4803.php
>