Dear Lyberta and Ansel,

     Thank you for the comments sent here! My responses will be below. Do let me know if I accidentally misunderstood or messed something up in my understanding, as I wasn't exactly sure I captured all of the concerns properly.

On Sat, Nov 2, 2019 at 12:11 PM Lyberta <> wrote:
Ansel Sermersheim:
> 1) There was some discussion about whether or not char32_t is guaranteed
> to be a Unicode Code Point. JeanHeyd pointed me to
>, which makes it clear that for string literals
> at least this is guaranteed.

Yes, char32_t is a bad type along with char8_t and char16_t. For that
reason I'm proposing strong types with proper guarantees:

You can put ill-formed Unicode in string literals via escape codes. This
is also bad.

Responding to both bites at once: you can do bad things with escape codes, but I am okay with that because you've written it using _escape codes_. Non-escaped text and \u and \U qualified text is mandated to be well-formed (and people are working on tightening the constraints here). I am perfectly okay with people having back doors and ways out, as long as the back doors and ways out are sufficiently grep-able / easy to identify.
> However, this is not sufficiently specified for all cases. For instance,
> a GB 18030 encoding *must* use codepoints in the PUA. If a string
> literal contains a PUA code point, how can you know the interpretation?
> Making this a compile error seems problematic, but the right answer is
> not clear to me.

Can probably be solved by custom instance of

I use char32_t solely out of the problem not being fully solved by SG16 at the moment. There was great interest in providing strong types for unicode_code_point; even in my implementation, I use the name unicode_code_point. Right now it defaults to an alias of char32_t since we have not fully decided which direction is worth taking here:
  • people with field experience and existing codebases want char32_t / uint32_t here;
  • previous implementations of text_view/text from Tom use an implementation where each encoding gets its own strong code point type;
  • other implementations just use char32_t and deem that to be fine enough.
I am heavily leaning towards char32_t representing a UTF32 code point, with the caveat that certain encodings which may use 32-bit types to represent its PUA characters will still be able to define a strong code_point type on their encoding and then use the various levers present in the paper and implementation to make it clear that their "code point" is different from the typical unicode code point, because it carries different semantic meaning. For example, a gb18030 and gb18030_web encoding object would use a gb_code_point type, which is more or less morally equivalent to a char32_t save for some different interpretations with PUA characters. This allows the type to clearly differentiate between "gb18030 code points" and "normal unicode code points". This may also be where -- as Lyberta has pointed out -- room for a custom character database in use with algorithms (normalization, collation, etc.) will come in handy.

All in all, Encoding Objects have a code_point type for exactly this reason. I suspect the majority will want to use char32_t for interopability and cohesion with the ecosystem at large. If someone uses special Private Use Area characters but still wants to present as a normal "unicode code point", they can do that. If they want a far more strict conversion policy, they can put a stronger type in there. The flexibility here is important to serve all use cases, and I think will work well here.
> 2) The issue of PUA usage also comes up in the implementation of
> Encoding Objects. It seems likely that the current direction will
> necessitate some third party library to handle encodings other than the
> main UTF ones. That seems reasonable. But without some sort of standard
> mechanism that at least enumerates other common interpretations, and
> allows third party libraries to declare their support for such, there
> will be a combinatorial explosion of mutually incompatible encodings.

I think providing conversions to and from Unicode scalar values is enough.

My long-term plan is that the entire slew of WHATWG encodings should end up in the Standard Library. But that is not feasible or rational as a "Step 0"; it is important to remember that as far as Unicode support goes, C++ and C are at Step -1 in the Standard. utf8/16/32, ascii, narrow_dynamic_encoding/wide_dynamic_encoding first, so we can get people from String Literal / Platform Strings -> Unicode, and then we can address the large body of encodings that exist outside of what is currently present and ships with "The System". The design, as pointed out in the last answer, is intentionally open to make sure we are not cutting CJK encodings or PUA encodings off at the legs here and asking them to walk when it comes time to make sure we include a broad suite of encodings.

> 3) By a similar construction and often overlapping concerns, the
> availability of a standardized way for encodings to declare which
> version of unicode they support is quite important. It's also not clear
> how some of the round trip encodings can possibly be fully specified in
> the type system. For example, how could I properly encode "UTF-8 Unicode
> version 10" text containing emoji into "UTF-16 Unicode version 5" text
> using the PUA for representation for display on OS X 10.7?

Different versions of Unicode and PUA are a job for
Perhaps it is a job for the database, but I want to be clear: what this proposal wants to deal with are encodings and -- potentially -- Normalization Forms. Encodings do not affect the interpretation of Emoji, and Normalization Forms have forward-compatibility guarantees since Unicode version 3.x. If emojis not defined in Unicode Version 5 are given to an application that only has a knowledge of Unicode Version 5, then the default answer is {however the application handles error state in its text}. For example, web browsers that do not understand X emoji display the codepoint value boxes. Other applications display "?". Some display the missing-value "�". It is up to the application to process characters that are within the 21 bits allotted by Unicode but have no understanding to process it how they see fit. It's on an application, its text renderer, etc.

The encoding layer is only to check for valid encodings (e.g., "fits in the 21-bit of Unicode and is not an invalid sequence according to the specification of this encoding"): subscribing meaning to that is a layer above this and something every application has different answers to. Some error and crash, some display replacement text, some ignore the text entirely, some tank the HTTP request, etc.

> 4) The behavior of std::basic_text with respect to null termination is
> valid but seems potentially risky. As I understand it, std::basic_text
> will be null terminated if the underlying container is the default
> std::basic_string. This seems likely to result in encoding
> implementations which inadvertently assume null termination on their
> operands. Our work on early versions of the CsString library persuaded
> us that optional null termination is the source of some really obscure
> bugs of the buffer overrun variety, and we eventually elected to force
> null termination for all strings.

I think null termination is just bad design. Pointer + length is the way
to go.

Whether or not Null Termination is bad design, C and C++ are about compatibility with enormous hulking old beasts of existing practice and operating system APIs, many of which work in null termination land.

The current design does optionally null terminate. Many people have problems with the null terminator and are desperately trying to design performance around it ( It may be dangerous, but the choice should be given to the user. std::vector<T> is as much a valid backing code unit storage as std::basic_string, and I do not think I should force users to only have basic_string-like semantics.

Absolutely, the default should be basic_string. I plan to make sure that `.c_str()` is a member of a text object if and only if the backing storage is basic_string. Otherwise, there will be no `.c_str()` argument, and code that uses .c_str() will stop compiling the moment someone changes the underlying container's string. Calls to .data() may still yet be problematic. I am glad for CsString's experience here; I will need to be very careful and bring this up to the Design Groups.