We say that

If a U+0027 apostrophe, a U+0022 quotation mark, or any character not in
> the basic character set matches the last category, the program is
> ill-formed.


But no one implements that restriction
https://compiler-explorer.com/z/fe9jWcE1G

In non-pedantic mode, GCC accepts virtually anything as an identifier, and
in pedantic mode they consider that anything outside of the basic character
set is part of an identifier - and then reject them if they do not follow
the XID requirements.

Beside that oddity, no one rejects nonsensical pp-token.
The restriction itself is not consistent.

Either we want to allow codepoints to appear in source even if they are not
part of grammar elements... or we do not.
Whether they are part of the basic character set should have no impact.


So either we want to say
> *If any character matches the last category, the program is ill-formed.*
or
> *If a U+0027 apostrophe or a U+0022 quotation mark matches the last
category, the program is ill-formed.*

It seems to be that the use case for allowing these non-tokens tokens to
exist is to allow concatenation shenanigans.
But you can do token concatenation shenanigans with non-basic character set

#define __CONCAT(A,B) A ## B
#define CONCAT(A,B) __CONCAT(A, B)
#define ONE ፩ // not an identifier (xid_start = false, xid_continue = true)
int CONCAT(A, ONE);

Is that useful? Maybe not. Is that less useful than the shenanigans that
*_are_* allowed? Also no!


I'm curious what people think.

Cheers