We also should consider Unicode Technical Report #36 UNICODE SECURITY CONSIDERATIONS. Although my first thought was that allowing confusing characters in an identifier is just a developer causing problems for themselves, it is actually a problem in code review. Using punning names to disguise that a local `i` is not shadowing an outer scope `i` and using that to inject an exploitable buffer attack, for example. If I spend some black hat time, I could probably craft something even "better". 

TR31 has some discussion on normalization. I think canonicalization is probably the right thing to do, as anything else leads to tools lying to you without intending to. It should not matter how my editor decides to craft a letter with a diacritic, even if the source code takes a round trip through some rich text or word processor. This is an implementation burden, though. Really anything beyond the current white list is, in any case. 

We'd also probably need to clarify that this means an even stronger requirement on the internal representation of source code. The input text has to be converted into code points (universal character names) and all of the operations we are talking about apply to that representation. Representation of the code points is implementation specific. 

I'm going to end up writing this paper, aren't I. 

On Fri, Oct 25, 2019 at 3:10 AM Corentin <corentin.jabot_at_[hidden]> wrote:

On Fri, 25 Oct 2019 at 08:58, Corentin <corentin.jabot_at_[hidden]> wrote:

On Fri, Oct 25, 2019, 02:18 Zach Laine <whatwasthataddress_at_[hidden]> wrote:

Is this a real problem that is biting people right now?  Are people using these characters in identifiers and causing great upheaval?  This seems of the lowest possible priority to me, and not at all C++20-related.

Completely agree, with both of you.

I would be deeply unsatisfied with a solution that would:

* Not follow TR31 recommandations

* Not address the fact that you can only have Unicode identifiers if the compiler knows that your file id 

I sent the previous mail too fast, sorry about the noise.

As I was saying

Completely agree, with both of you.
I would be deeply unsatisfied with a solution that would:

* Not follow TR31 recommendations
* Not address the fact that you can only have Unicode identifiers if the compiler knows that your file is UTF encoded (same issue that u8 literals, we talked about that - P1880)

* Fail to address concerns related to mangling if a normalization form is not specified

* Fail to recognize that we will want to reflect on the name of these things (std::meta::name_of) and that would require reflection to be able to deal with that, both in terms of providing a uf8 api AND a specified normalization form 

All of that require careful consideration

Corentin

 

Zach

On Thu, Oct 24, 2019 at 5:25 PM Steve Downey <sdowney_at_[hidden]> wrote:

SG16 has an NB comment to deal with! Tom has already scheduled it for Belfast. It's basically that the list of allowed code points have some interesting control characters like zero width joiners and RTL modifiers. 

https://github.com/cplusplus/nbballot/issues/28

There's also an issue that JF raised earlier:
https://github.com/sg16-unicode/sg16/issues/48
Improve support for Unicode characters in identifiers  

Relevant unicode standard:
https://unicode.org/reports/tr31/ UNICODE IDENTIFIER AND PATTERN SYNTAX  

Which is complicated because it allows things like identifiers written in Farsi which requires zwj for disambiguation, and suggests regex to detect particular allowed identifiers. It's fairly dense, and I haven't digested it yet, but it looks like there might be allowed ways to exclude that. 

Plus tailoring would be needed because C++ disallows some characters such as '$' which might otherwise be allowed. This is also discussed in TR31. 


My feeling on the comment is that it's not a new issue for C++20, so it's not clear that it has to be fixed for C++20. I believe it should be fixed, but it ought to be fixed in a principled manner, and that likely means TR31.

We would also have to discuss if emoji are allowed in identifiers. TR31 does not strictly disallow them. The TonyTable shall be interesting. 

_______________________________________________
SG16 Unicode mailing list
Unicode_at_[hidden]
http://www.open-std.org/mailman/listinfo/unicode

_______________________________________________
SG16 Unicode mailing list
Unicode_at_[hidden]
http://www.open-std.org/mailman/listinfo/unicode

_______________________________________________
SG16 Unicode mailing list
Unicode_at_[hidden]
http://www.open-std.org/mailman/listinfo/unicode