Date: Wed, 08 Jul 2026 06:57:20 +0000
On Wednesday, July 8th, 2026 at 12:39 AM, Steve Downey via SG16 <sg16_at_[hidden]> wrote:
> I disagree slightly on whether we should have a better concrete Text type that maintains, for example, the shape necessary to produce a string that python will consume, and not reject before getting started.
>
> But, it's, I think, a lower priority than getting the rest of the Unicode algorithm repetoire available.
>
> For non UTF actual multibyte encodings, basic_string is actually worse than it is for UTF-8 or 32, even for 16. The legacy encodings are forward-only. You can not start at an arbitrary position and resync, forward or backwards, and the representation self aliases. So every access method other than sequential from the beginning and decoding is, at least potentially, wrong.
>
> Being a fancy char* is just barely enough to justify it. But barely.
Note that the approach I outlined works fine with std::string, but does not rely on it being in a particular shape. It works equally well for any other storage type that can hold code units. For me std::vector<char8_t> comes first to mind, and would work fine, but any other string-holding-type would work equally well.
In priority terms, getting such a type is at the end of the list. We have a type that's widely used and accepted, and that serves the purpose well enough without any major drawbacks. The drawbacks that do exist are mostly in useless bits associated with it and error-prone functions, none of which do any harm unless used.
If you do want to make such a proposal, let's focus on the rest of it first as those are not currently being served, and leave these replacement storage types as a later concern.
> I disagree slightly on whether we should have a better concrete Text type that maintains, for example, the shape necessary to produce a string that python will consume, and not reject before getting started.
>
> But, it's, I think, a lower priority than getting the rest of the Unicode algorithm repetoire available.
>
> For non UTF actual multibyte encodings, basic_string is actually worse than it is for UTF-8 or 32, even for 16. The legacy encodings are forward-only. You can not start at an arbitrary position and resync, forward or backwards, and the representation self aliases. So every access method other than sequential from the beginning and decoding is, at least potentially, wrong.
>
> Being a fancy char* is just barely enough to justify it. But barely.
Note that the approach I outlined works fine with std::string, but does not rely on it being in a particular shape. It works equally well for any other storage type that can hold code units. For me std::vector<char8_t> comes first to mind, and would work fine, but any other string-holding-type would work equally well.
In priority terms, getting such a type is at the end of the list. We have a type that's widely used and accepted, and that serves the purpose well enough without any major drawbacks. The drawbacks that do exist are mostly in useless bits associated with it and error-prone functions, none of which do any harm unless used.
If you do want to make such a proposal, let's focus on the rest of it first as those are not currently being served, and leave these replacement storage types as a later concern.
Received on 2026-07-08 06:57:28