Date: Tue, 07 Jul 2026 16:44:42 +0000
Hi Alisdair,
On Tuesday, July 7th, 2026 at 6:15 PM, Alisdair Meredith via SG16 <sg16_at_[hidden]> wrote:
> Raising a distinct concern from the thread on best practice for Unicode today, does this group have a position on whether `basic_string` is an appropriate type for text processing in Unicode?
For text *storage*, it's the right type to use. It stores code units in whatever underlying character type it uses, and for u8/u16/u32 it's a Unicode type. All storage happens in terms of code units regardless.
For text *processing*, we almost always want to view them as code points, or higher-level constructs on top of that. Unicode properties that we want to have aren't necessarily retained in simple code unit changes, such as "I have a whole number of code points", "I have valid utf8", "my text is NFC normalized" or "concatenating two strings of X + Y grapheme clusters results in X+Y grapheme clusters". We need to have a view transform that provides this code point view from the underlying code unit storage, and we need to have a view transform that is usable for storing a code point range of some kind into a code unit storage.
With those fundamentals, we can build code point algorithms and grapheme algorithms on top of them.
> From my perspective, `basic_string` (and its type aliases) is a container of code units, where for text processing we want a container of code points. Likewise, is it parameterized on `char_traits` where we do not provide a traits type following Unicode rules, and in fact some things like collation order we would want to customize at runtime, but are baked into a compile-time decision through this traits parameter. The various functions that sound like processing text are not well suited to any multi-byte encoding system, not just Unicode,but as long as we live in ASCII plus the various codepage extensions native to our environment, we do not run into issues.
Char_traits is fundamentally broken and unsuitable for anything Unicode. We need to think of it as ossified. Collation is a property of a collection or sort invocation, not of a string type. The cstring_view proposal has char_traits so that it can convert from string to string_view through it without data loss, but beyond that SG16 agreed that it's useless.
> The counterpoint is that `std::string` has been our established vocabulary for the last 30 years, and `std::string_view` for the last 10. Does that weight of that legacy make it intractable to provide a “better” solution for a world that has slowly standardized on Unicode as its preferred encoding scheme (even if adherence to semantics like collation order may be less universal)?
They're okay storage types. They have a bunch of functions that have no Unicode meaning, and they have a lot of ossified features that we cannot modify and that serve no purpose, but any replacement at best would do the same thing modulo char_traits and string-like functions, and would necessarily face an uphill battle against the existing types that work just as well.
This is my understanding, please correct me if you disagree.
On Tuesday, July 7th, 2026 at 6:15 PM, Alisdair Meredith via SG16 <sg16_at_[hidden]> wrote:
> Raising a distinct concern from the thread on best practice for Unicode today, does this group have a position on whether `basic_string` is an appropriate type for text processing in Unicode?
For text *storage*, it's the right type to use. It stores code units in whatever underlying character type it uses, and for u8/u16/u32 it's a Unicode type. All storage happens in terms of code units regardless.
For text *processing*, we almost always want to view them as code points, or higher-level constructs on top of that. Unicode properties that we want to have aren't necessarily retained in simple code unit changes, such as "I have a whole number of code points", "I have valid utf8", "my text is NFC normalized" or "concatenating two strings of X + Y grapheme clusters results in X+Y grapheme clusters". We need to have a view transform that provides this code point view from the underlying code unit storage, and we need to have a view transform that is usable for storing a code point range of some kind into a code unit storage.
With those fundamentals, we can build code point algorithms and grapheme algorithms on top of them.
> From my perspective, `basic_string` (and its type aliases) is a container of code units, where for text processing we want a container of code points. Likewise, is it parameterized on `char_traits` where we do not provide a traits type following Unicode rules, and in fact some things like collation order we would want to customize at runtime, but are baked into a compile-time decision through this traits parameter. The various functions that sound like processing text are not well suited to any multi-byte encoding system, not just Unicode,but as long as we live in ASCII plus the various codepage extensions native to our environment, we do not run into issues.
Char_traits is fundamentally broken and unsuitable for anything Unicode. We need to think of it as ossified. Collation is a property of a collection or sort invocation, not of a string type. The cstring_view proposal has char_traits so that it can convert from string to string_view through it without data loss, but beyond that SG16 agreed that it's useless.
> The counterpoint is that `std::string` has been our established vocabulary for the last 30 years, and `std::string_view` for the last 10. Does that weight of that legacy make it intractable to provide a “better” solution for a world that has slowly standardized on Unicode as its preferred encoding scheme (even if adherence to semantics like collation order may be less universal)?
They're okay storage types. They have a bunch of functions that have no Unicode meaning, and they have a lot of ossified features that we cannot modify and that serve no purpose, but any replacement at best would do the same thing modulo char_traits and string-like functions, and would necessarily face an uphill battle against the existing types that work just as well.
This is my understanding, please correct me if you disagree.
Received on 2026-07-07 16:44:51