> Presumably one that produces a filled buffer? I'm always in a situation where I have the text sitting in memory already, and need to do some additional operations, and early benchmarks indicated that a view piped into `to<vector>` was not significantly different than a bulk API in performance. Some testing with Book of Genji and Journey to the West in legacy native encodings might be suggesting otherwise, but they still aren't awful, and there's more than a little room behind the API to improve some parts, I think.

My experience on this topic is that letting the user produce the buffer beats everything else in terms of performance, specially if the text is relatively small and you can prove that you can do it on the stack, and would be able to transcode several pieces of text before you are done allocating enough memory for the first one.

I know that saying it is “more performant” doesn’t necessarily make it true, and should prove that. But if you are willing, would you like to participate in a test?
We create a github repo, implement a couple of different algorithms, compile a set of use cases, make unit tests to make sure that the algorithms satisfy the requirements, and then we bench mark it to see what is best.
If we have an algorithm that consistently performs significantly better than the rest, then we should use that one. If we get mixed results then we can ask the community which uses cases should we prioritize.
At least, when an objection around this topic surface, we can avoid the discussion but just objectively pointing to the benchmarks.
Shall we do that?


From: Steve Downey <sdowney_at_gmail.com>
Sent: Tuesday, October 27, 2026 23:45
To: SG16 <sg16_at_[hidden]>
Cc: Tiago Freire <tmiguelf_at_hotmail.com>; Jonathan Wakely <cxx_at_[hidden]>
Subject: Re: [isocpp-sg16] P4246R0 Transcoding Text Views and Ranges for WhatWG and iconv


On Tue, Oct 27, 2026, 16:05 Tiago Freire via SG16 <sg16_at_[hidden]<mailto:sg16_at_[hidden]>> wrote:
> I also brought up that I don't see why this API needs to be based on views. This seems much more intended for transcoding a file or network packet once, at the program boundaries, in order to obtain Unicode. Views are useful for e.g. temporarily iterating over the code points of a UTF-8 string, without needing to allocate. > The API being view-based actually hurts its usability in my opinion.

How so? What alternative API would you use?

Presumably one that produces a filled buffer? I'm always in a situation where I have the text sitting in memory already, and need to do some additional operations, and early benchmarks indicated that a view piped into `to<vector>` was not significantly different than a bulk API in performance. Some testing with Book of Genji and Journey to the West in legacy native encodings might be suggesting otherwise, but they still aren't awful, and there's more than a little room behind the API to improve some parts, I think.



From: SG16 <sg16-bounces_at_lists.isocpp.org<mailto:sg16-bounces_at_[hidden]>> On Behalf Of Jan Schultke via SG16
Sent: Tuesday, October 27, 2026 20:40
To: sg16_at_[hidden]<mailto:sg16_at_[hidden]ocpp.org>
Cc: Jan Schultke <janschultke_at_[hidden]<mailto:janschultke_at_[hidden]>>; Jonathan Wakely <cxx_at_[hidden]<mailto:cxx_at_[hidden]>>
Subject: Re: [isocpp-sg16] P4246R0 Transcoding Text Views and Ranges for WhatWG and iconv

I have previously brought up very similar concerns on Mattermost: https://chat.isocpp.org/general/pl/k698q5qg93n3bnjq5renm859zr
The very first thing I brought up is why we are making a new enum. That concern was partially addressed by Steve: https://chat.isocpp.org/general/pl/861afkhr57dwmphrgjjz3kuy1a
The labels from text_encoding are a non-strict superset of what the WhatWG provides, and the IANA registry doesn't provide actual algorithms. It was one of the core reasons the browser ecosystem had to produce the standard. There was a lot of divergence on Big5 implementation, and in some places still is. You can sometimes get different answers out of iconv and whatwg, and for purposes of web interchange, iconv is then wrong.

I also brought up that I don't see why this API needs to be based on views. This seems much more intended for transcoding a file or network packet once, at the program boundaries, in order to obtain Unicode. Views are useful for e.g. temporarily iterating over the code points of a UTF-8 string, without needing to allocate. The API being view-based actually hurts its usability in my opinion.

On Tue, 27 Oct 2026 at 18:24, Jonathan Wakely via SG16 <sg16_at_[hidden]<mailto:sg16_at_[hidden]>> wrote:
I know the paper mentions Eddie's P2728 (Unicode in the library pt 1) and compares them, but I'm not convinced that a completely different API with entirely different names is needed. That seems likely to confuse users.

There seems to be a huge amount of overlap between the codec enum and std::text_encoding::id, why do we need a new enum? Won't that confuse users too?

Why do we need a novel mechanism for mapping encoding labels/names to IDs, isn't that exactly what std::text_encoding does?

Is 'codec' a suitable name? (I don't see that term anywhere in the WhatWG Encoding spec). The paper proposes 'codec' as an enum for identifying an encoding, but then decode_codec and encode_codec as concepts that are modelled by classers that encode and decode ... so is a codec an enum identifying an encoding or a class performing it? It seems to be both. The WhatWG spec uses "encoder" and "decoder", but not "codec".
--
SG16 mailing list
SG16_at_lists.isocpp.org<mailto:SG16_at_[hidden]>
https://lists.isocpp.org/mailman/listinfo.cgi/sg16
Link to this post: http://lists.isocpp.org/sg16/2026/10/4889.php
--
SG16 mailing list
SG16_at_[hidden]<mailto:SG16_at_lists.isocpp.org>
https://lists.isocpp.org/mailman/listinfo.cgi/sg16
Link to this post: http://lists.isocpp.org/sg16/2026/10/4895.php