Date: Sun, 13 Sep 2020 16:50:11 +0000
I'm going to bundle this feature test macro in with the other freestanding feature test macro discussions in https://wg21.link/P2198 (Freestanding Feature-Test Macros and Implementation-Defined Extensions).
Here's an extra oddity with this feature test macro. If I provide it in the positive (__cpp_lib_has_default_operator_new), it won't be useful for a very long time.
#if defined(__cpp_lib_has_default_operator_new) && __cpp_lib_has_default_operator_new >= 20200913
using my_container = my_vector;
#else
// spuriously triggers for C++20 and earlier code
using my_container = fixed_capacity_vector;
#endif
If I provide the macro in the negative (__cpp_lib_no_default_operator_new), then it will be the only feature test macro that isn't required to be set by any conforming implementation. This version of the macro is slightly more useful though, so I'll propose and discuss that approach in the next revision of P2198. That revision won't be in the September mailing, but will likely be in the October mailing.
________________________________
From: SG10 <sg10-bounces_at_[hidden]> on behalf of Ben Craig via SG10 <sg10_at_[hidden]>
Sent: Wednesday, September 9, 2020 1:56 PM
To: JF Bastien <cxx_at_[hidden]>; sg10_at_[hidden] <sg10_at_[hidden]>
Cc: Ben Craig <ben.craig_at_[hidden]>
Subject: [EXTERNAL] Re: [SG10] Intentionally omitted feature test macro in freestanding operator new (P2013)
> If there's no allocation, what does portable code do?
That question is part of the problem. A feature test macro for this paper doesn’t answer the question “can I use global operator new”. It answers the question “is a default implementation of global operator new provided”. In my companies code base, we have C++ code for our drivers. There is no default global operator new in any of the OS kernel spaces we support (Windows, Linux, OSX). Most of our drivers provide their own global operator new though.
So fine, this isn’t answering the _exact_ question the user wants answered. What if the user is fine with an overly conservative answer? A feature test macro for this feature could provide such a conservative answer. Now what do the users do with it?
Does the user use some alternative storage location? Well, why didn’t they just use that storage location in the first place? Just use that all the time and the problem is solved. Freestanding platforms can (and do) have multiple threads, so dumping the data into a global isn’t a portable solution. Freestanding platforms often have tiny stacks, so making a large stack allocation is usually inadvisable.
The closest legitimate use I can think of is for cases where you have a small stack buffer and a global fallback if the small buffer isn’t enough. You could remove the global fallback. I’m not particularly motivated by this case.
Do others find the small buffer + fallback use case motivating?
What is the degree of utility needed to add a feature test macro? Maybe these abstract use cases are enough?
Now, let’s assume we do want a feature test macro. Do we want it to be a core language feature test macro, or a library feature test macro?
If we go with a feature test macro, then I’m leaning towards making a library feature test macro (in <version> and <new>). I could have one compiler that targets my microcontroller or kernel or whatever, and a library and runtime that doesn’t have a default operator new, then later switch the library and runtime without changing the compiler.
From: JF Bastien <cxx_at_[hidden]>
Sent: Wednesday, September 9, 2020 10:56 AM
To: sg10_at_[hidden]
Cc: Louis Dionne <ldionne_at_[hidden]>; Ben Craig <ben.craig_at_[hidden]>
Subject: [EXTERNAL] Re: [SG10] Intentionally omitted feature test macro in freestanding operator new (P2013)
I wouldn't worry about implementation difficulties. The compiler knows about the target and the runtime, it might need to know more, but that's fine. It knows when it's targeting freestanding too, so it can conservatively say "this is a freestanding platform that I'm not sure about, let's assume there's no allocation". The library then pick up whatever the compiler says. This will be correct for implementations that care about this.
What's really important here is: what does a user do with this macro? Your argument is "shoot themselves in the foot", and I'm not sure that's right. If there's no allocation, what does portable code do? Not call new, and instead use the stack, or some side scratch space? That seems reasonable, and a good reason to have a feature test macro.
What other considerations are missing?
On Wed, Sep 9, 2020 at 6:33 AM Ben Craig via SG10 <sg10_at_[hidden]<mailto:sg10_at_[hidden]>> wrote:
The following is a conversation on this topic that Louis and I had on Slack, reproduced with his permission.
ldionne
The question you're asking is whether we should have a feature test macro that tells whether the replaceable operator new/operator delete have a default (weak) definition in the library, correct?
9:03
And you're saying "There should not be such a feature-test macro, because implementing it requires the headers to have a lot of information related to the runtime they will be used for". Is that correct?
9:04
If so, one additional bit of information is that on Apple platforms, we do have sufficient information to know whether a default definition of these replaceable allocation functions is provided, because we build libc++ and libc++abi all at the same time.
ben.craig 10:42 AM
You've got the right idea, yes. And I agree, that for Apple platforms, you have sufficient information.
10:43
I'm thinking that for homegrown OS's, it will be a harder thing to figure out. Whoever builds the toolchain would need to supply that information, which works, but it's a little hostile. Supporting your own OS is a hostile thing though.
ldionne 10:43 AM
I'm not sure I follow.
10:44
Why would whoever builds the toolchain have to supply that information?
10:44
That assumes libc++ is part of the toolchain, right?
ben.craig 10:44 AM
the default implementation of libc++abi is going to try to get memory from malloc, and malloc may not be there
10:45
and there isn't a way for the compiler front end to ask "is malloc there", and there isn't a good way for <version> to ask "is malloc there"
10:45
but the person that builds the toolchain should know if malloc is there, and therefore, whether operator new is there
ldionne 10:46 AM
So, on our platform, we ship libc++ as part of the system.
10:46
Not the toolchain
ben.craig 10:47 AM
fair. Your toolchain knows what OS it is targeting though
ldionne 10:48 AM
Yes, but that's not the point I'm trying to make. Even regardless of any toolchain, libc++ knows what it provides when it's built for <name some OS>. For example, when I build libc++ for e.g. macos 10.15, I build libc++, libc++abi and the libc++ headers all at once. The headers I provide can, in theory, have different contents based on that.
10:49
So for example if I were to build libc++ for some internal freestanding implementation, I would be able to avoid defining the feature-test-macro for operator new in that build
10:49
Does that make sense?
ben.craig 10:49 AM
let me refine what I said earlier.
Whoever builds the ~toolchain~ _standard library and runtime_ would need to supply that information, which works, but it's a little hostile. Supporting your own OS is a hostile thing though.
ldionne 10:50 AM
Right. I don't see how it's really that hostile?
ben.craig 10:50 AM
It's not that bad. It's nicer when I don't have to set a bunch of flags and the headers can just figure it out on their own
10:51
but things like MUSL already require some manual configuration, so it's still workable
10:51
I can't just take some binaries that someone else built and use those. I have to build them myself.
ldionne 10:52 AM
Here's how I'd implement that hypothetically in libc++ (if we supported freestanding better than not at all):
1. Add a CMake option whether to provide the default definition for operator new
2. Based on that CMake option, include (or don't) the definitions in libc++.a/libc++abi.a
3. Based on that CMake option, define (or don't) the feature test macro
The headers installed in include/c++/v1 would literally be different based on that CMake option. Then, when I build libc++ for my special internal freestanding, I'd define the CMake option and be done with it.
10:53
I agree this means the headers on their own can't figure it out.
10:53
IOW we can't have one set of headers that is used on both (e.g.) macosx and the freestanding implementation. They need to be *differently configured* headers
ben.craig 10:54 AM
I think this was definitely the weaker point against having a feature test macro, and it's weak enough at this point that I'm tempted to remove it. Do you have opinions on the second point (lack of a good use case)?
ldionne 10:55 AM
No, I don't have a use case off the top of my head. I don't have an opinion either.
10:55
I just wanted to let you know that having a feature-test macro wasn't a problem as far as Apple platforms were concerned, because of the way we ship libc++/libc++abi. This might not hold true for other vendors.
10:56
My point is: I don't care which way you go, but if you decide to add a macro, we'll be able to implement it.
10:56
Maybe not trivially, but it's doable.
10:57
Also, feel free to report this conversation to the mailing list. I signed up and should be able to reply inline next time.
From: Ben Craig
Sent: Friday, August 28, 2020 1:00 PM
To: Jonathan Wakely <cxx_at_[hidden]<mailto:cxx_at_[hidden]>>; sg10_at_[hidden]<mailto:sg10_at_[hidden]>
Subject: RE: [EXTERNAL] Re: [SG10] Intentionally omitted feature test macro in freestanding operator new (P2013)
> I don't really understand this, as I'm not sure what "the runtime" means. The macro would be provided by the <new> header, wouldn't it?
libsupc++ and libc++abi are some specific runtime pieces I had in mind with that statement.
> If the <new> header declares operator new, it would define the macro. If not, not.
My paper proposes that the declarations for operator new will always be present. The optional aspect is whether the definitions will be present.
<new> and <version> can’t really ask libsupc++ and libc++abi if they contain an operator new definition. It means that the information needs to be populated when the toolchain is built, and that the toolchain needs to know enough about the target environment to know whether operator new will be present or not.
It’s not a super strong argument, as toolchains will often know exactly what they are targeting, and can make a conservative choice in other circumstances. The lack of a good use case is my stronger argument. It’s still not clear to me whether we would make this a library or a language feature test macro though.
From: Jonathan Wakely <cxx_at_[hidden]<mailto:cxx_at_[hidden]>>
Sent: Friday, August 28, 2020 12:40 PM
To: sg10_at_[hidden]<mailto:sg10_at_[hidden]>
Cc: Ben Craig <ben.craig_at_[hidden]<mailto:ben.craig_at_[hidden]>>
Subject: [EXTERNAL] Re: [SG10] Intentionally omitted feature test macro in freestanding operator new (P2013)
On Thu, 20 Aug 2020, 15:15 Ben Craig via SG10, <sg10_at_[hidden]<mailto:sg10_at_[hidden]>> wrote:
Yesterday, EWG’s telecon requested that I run the feature test macro section of my design by SG10 to see if there were any objections.
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p2013r1.html#no_macro<https://urldefense.com/v3/__http:/www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p2013r1.html*no_macro__;Iw!!FbZ0ZwI3Qg!5M_20wW5k73BOnQQ4rUjjp-GzvRnAE-bu2kIJglmJRs8wiDkODj3eGhF0Gx4$>
Here’s that text reproduced in the email:
A feature test macro would be awkward to implement, and would encourage code that is more prone to ODR issues than other feature test macros.
In most toolchains, feature test macros can be exposed directly by the compiler (usually for core language features) and by the library (for library features). The presence or absence of ::operator new is dictated by the runtime though. In some implementations, neither the compiler nor the library headers necessarily know detailed information about the runtime. This hurdle is not intractable, but it is a hurdle nonetheless.
I don't really understand this, as I'm not sure what "the runtime" means. The macro would be provided by the <new> header, wouldn't it?
If the <new> header declares operator new, it would define the macro. If not, not.
The most likely usage of such a feature test macro is to conditionally define a custom ::operator new iff the implementation did not provide one by default. This is dangerous territory, as it encourages libraries to provide the one-and-only ::operator new definition. If two such libraries do this, then there is an ODR issue.
OK, I am convinced by that argument. Not having a valid use case is good justification.
Does SG10 have any concerns with this approach?
Here's an extra oddity with this feature test macro. If I provide it in the positive (__cpp_lib_has_default_operator_new), it won't be useful for a very long time.
#if defined(__cpp_lib_has_default_operator_new) && __cpp_lib_has_default_operator_new >= 20200913
using my_container = my_vector;
#else
// spuriously triggers for C++20 and earlier code
using my_container = fixed_capacity_vector;
#endif
If I provide the macro in the negative (__cpp_lib_no_default_operator_new), then it will be the only feature test macro that isn't required to be set by any conforming implementation. This version of the macro is slightly more useful though, so I'll propose and discuss that approach in the next revision of P2198. That revision won't be in the September mailing, but will likely be in the October mailing.
________________________________
From: SG10 <sg10-bounces_at_[hidden]> on behalf of Ben Craig via SG10 <sg10_at_[hidden]>
Sent: Wednesday, September 9, 2020 1:56 PM
To: JF Bastien <cxx_at_[hidden]>; sg10_at_[hidden] <sg10_at_[hidden]>
Cc: Ben Craig <ben.craig_at_[hidden]>
Subject: [EXTERNAL] Re: [SG10] Intentionally omitted feature test macro in freestanding operator new (P2013)
> If there's no allocation, what does portable code do?
That question is part of the problem. A feature test macro for this paper doesn’t answer the question “can I use global operator new”. It answers the question “is a default implementation of global operator new provided”. In my companies code base, we have C++ code for our drivers. There is no default global operator new in any of the OS kernel spaces we support (Windows, Linux, OSX). Most of our drivers provide their own global operator new though.
So fine, this isn’t answering the _exact_ question the user wants answered. What if the user is fine with an overly conservative answer? A feature test macro for this feature could provide such a conservative answer. Now what do the users do with it?
Does the user use some alternative storage location? Well, why didn’t they just use that storage location in the first place? Just use that all the time and the problem is solved. Freestanding platforms can (and do) have multiple threads, so dumping the data into a global isn’t a portable solution. Freestanding platforms often have tiny stacks, so making a large stack allocation is usually inadvisable.
The closest legitimate use I can think of is for cases where you have a small stack buffer and a global fallback if the small buffer isn’t enough. You could remove the global fallback. I’m not particularly motivated by this case.
Do others find the small buffer + fallback use case motivating?
What is the degree of utility needed to add a feature test macro? Maybe these abstract use cases are enough?
Now, let’s assume we do want a feature test macro. Do we want it to be a core language feature test macro, or a library feature test macro?
If we go with a feature test macro, then I’m leaning towards making a library feature test macro (in <version> and <new>). I could have one compiler that targets my microcontroller or kernel or whatever, and a library and runtime that doesn’t have a default operator new, then later switch the library and runtime without changing the compiler.
From: JF Bastien <cxx_at_[hidden]>
Sent: Wednesday, September 9, 2020 10:56 AM
To: sg10_at_[hidden]
Cc: Louis Dionne <ldionne_at_[hidden]>; Ben Craig <ben.craig_at_[hidden]>
Subject: [EXTERNAL] Re: [SG10] Intentionally omitted feature test macro in freestanding operator new (P2013)
I wouldn't worry about implementation difficulties. The compiler knows about the target and the runtime, it might need to know more, but that's fine. It knows when it's targeting freestanding too, so it can conservatively say "this is a freestanding platform that I'm not sure about, let's assume there's no allocation". The library then pick up whatever the compiler says. This will be correct for implementations that care about this.
What's really important here is: what does a user do with this macro? Your argument is "shoot themselves in the foot", and I'm not sure that's right. If there's no allocation, what does portable code do? Not call new, and instead use the stack, or some side scratch space? That seems reasonable, and a good reason to have a feature test macro.
What other considerations are missing?
On Wed, Sep 9, 2020 at 6:33 AM Ben Craig via SG10 <sg10_at_[hidden]<mailto:sg10_at_[hidden]>> wrote:
The following is a conversation on this topic that Louis and I had on Slack, reproduced with his permission.
ldionne
The question you're asking is whether we should have a feature test macro that tells whether the replaceable operator new/operator delete have a default (weak) definition in the library, correct?
9:03
And you're saying "There should not be such a feature-test macro, because implementing it requires the headers to have a lot of information related to the runtime they will be used for". Is that correct?
9:04
If so, one additional bit of information is that on Apple platforms, we do have sufficient information to know whether a default definition of these replaceable allocation functions is provided, because we build libc++ and libc++abi all at the same time.
ben.craig 10:42 AM
You've got the right idea, yes. And I agree, that for Apple platforms, you have sufficient information.
10:43
I'm thinking that for homegrown OS's, it will be a harder thing to figure out. Whoever builds the toolchain would need to supply that information, which works, but it's a little hostile. Supporting your own OS is a hostile thing though.
ldionne 10:43 AM
I'm not sure I follow.
10:44
Why would whoever builds the toolchain have to supply that information?
10:44
That assumes libc++ is part of the toolchain, right?
ben.craig 10:44 AM
the default implementation of libc++abi is going to try to get memory from malloc, and malloc may not be there
10:45
and there isn't a way for the compiler front end to ask "is malloc there", and there isn't a good way for <version> to ask "is malloc there"
10:45
but the person that builds the toolchain should know if malloc is there, and therefore, whether operator new is there
ldionne 10:46 AM
So, on our platform, we ship libc++ as part of the system.
10:46
Not the toolchain
ben.craig 10:47 AM
fair. Your toolchain knows what OS it is targeting though
ldionne 10:48 AM
Yes, but that's not the point I'm trying to make. Even regardless of any toolchain, libc++ knows what it provides when it's built for <name some OS>. For example, when I build libc++ for e.g. macos 10.15, I build libc++, libc++abi and the libc++ headers all at once. The headers I provide can, in theory, have different contents based on that.
10:49
So for example if I were to build libc++ for some internal freestanding implementation, I would be able to avoid defining the feature-test-macro for operator new in that build
10:49
Does that make sense?
ben.craig 10:49 AM
let me refine what I said earlier.
Whoever builds the ~toolchain~ _standard library and runtime_ would need to supply that information, which works, but it's a little hostile. Supporting your own OS is a hostile thing though.
ldionne 10:50 AM
Right. I don't see how it's really that hostile?
ben.craig 10:50 AM
It's not that bad. It's nicer when I don't have to set a bunch of flags and the headers can just figure it out on their own
10:51
but things like MUSL already require some manual configuration, so it's still workable
10:51
I can't just take some binaries that someone else built and use those. I have to build them myself.
ldionne 10:52 AM
Here's how I'd implement that hypothetically in libc++ (if we supported freestanding better than not at all):
1. Add a CMake option whether to provide the default definition for operator new
2. Based on that CMake option, include (or don't) the definitions in libc++.a/libc++abi.a
3. Based on that CMake option, define (or don't) the feature test macro
The headers installed in include/c++/v1 would literally be different based on that CMake option. Then, when I build libc++ for my special internal freestanding, I'd define the CMake option and be done with it.
10:53
I agree this means the headers on their own can't figure it out.
10:53
IOW we can't have one set of headers that is used on both (e.g.) macosx and the freestanding implementation. They need to be *differently configured* headers
ben.craig 10:54 AM
I think this was definitely the weaker point against having a feature test macro, and it's weak enough at this point that I'm tempted to remove it. Do you have opinions on the second point (lack of a good use case)?
ldionne 10:55 AM
No, I don't have a use case off the top of my head. I don't have an opinion either.
10:55
I just wanted to let you know that having a feature-test macro wasn't a problem as far as Apple platforms were concerned, because of the way we ship libc++/libc++abi. This might not hold true for other vendors.
10:56
My point is: I don't care which way you go, but if you decide to add a macro, we'll be able to implement it.
10:56
Maybe not trivially, but it's doable.
10:57
Also, feel free to report this conversation to the mailing list. I signed up and should be able to reply inline next time.
From: Ben Craig
Sent: Friday, August 28, 2020 1:00 PM
To: Jonathan Wakely <cxx_at_[hidden]<mailto:cxx_at_[hidden]>>; sg10_at_[hidden]<mailto:sg10_at_[hidden]>
Subject: RE: [EXTERNAL] Re: [SG10] Intentionally omitted feature test macro in freestanding operator new (P2013)
> I don't really understand this, as I'm not sure what "the runtime" means. The macro would be provided by the <new> header, wouldn't it?
libsupc++ and libc++abi are some specific runtime pieces I had in mind with that statement.
> If the <new> header declares operator new, it would define the macro. If not, not.
My paper proposes that the declarations for operator new will always be present. The optional aspect is whether the definitions will be present.
<new> and <version> can’t really ask libsupc++ and libc++abi if they contain an operator new definition. It means that the information needs to be populated when the toolchain is built, and that the toolchain needs to know enough about the target environment to know whether operator new will be present or not.
It’s not a super strong argument, as toolchains will often know exactly what they are targeting, and can make a conservative choice in other circumstances. The lack of a good use case is my stronger argument. It’s still not clear to me whether we would make this a library or a language feature test macro though.
From: Jonathan Wakely <cxx_at_[hidden]<mailto:cxx_at_[hidden]>>
Sent: Friday, August 28, 2020 12:40 PM
To: sg10_at_[hidden]<mailto:sg10_at_[hidden]>
Cc: Ben Craig <ben.craig_at_[hidden]<mailto:ben.craig_at_[hidden]>>
Subject: [EXTERNAL] Re: [SG10] Intentionally omitted feature test macro in freestanding operator new (P2013)
On Thu, 20 Aug 2020, 15:15 Ben Craig via SG10, <sg10_at_[hidden]<mailto:sg10_at_[hidden]>> wrote:
Yesterday, EWG’s telecon requested that I run the feature test macro section of my design by SG10 to see if there were any objections.
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p2013r1.html#no_macro<https://urldefense.com/v3/__http:/www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p2013r1.html*no_macro__;Iw!!FbZ0ZwI3Qg!5M_20wW5k73BOnQQ4rUjjp-GzvRnAE-bu2kIJglmJRs8wiDkODj3eGhF0Gx4$>
Here’s that text reproduced in the email:
A feature test macro would be awkward to implement, and would encourage code that is more prone to ODR issues than other feature test macros.
In most toolchains, feature test macros can be exposed directly by the compiler (usually for core language features) and by the library (for library features). The presence or absence of ::operator new is dictated by the runtime though. In some implementations, neither the compiler nor the library headers necessarily know detailed information about the runtime. This hurdle is not intractable, but it is a hurdle nonetheless.
I don't really understand this, as I'm not sure what "the runtime" means. The macro would be provided by the <new> header, wouldn't it?
If the <new> header declares operator new, it would define the macro. If not, not.
The most likely usage of such a feature test macro is to conditionally define a custom ::operator new iff the implementation did not provide one by default. This is dangerous territory, as it encourages libraries to provide the one-and-only ::operator new definition. If two such libraries do this, then there is an ODR issue.
OK, I am convinced by that argument. Not having a valid use case is good justification.
Does SG10 have any concerns with this approach?
-- SG10 mailing list SG10_at_[hidden]<mailto:SG10_at_[hidden]> https://lists.isocpp.org/mailman/listinfo.cgi/sg10<https://urldefense.com/v3/__https:/lists.isocpp.org/mailman/listinfo.cgi/sg10__;!!FbZ0ZwI3Qg!5M_20wW5k73BOnQQ4rUjjp-GzvRnAE-bu2kIJglmJRs8wiDkODj3eP9idKA7$> -- SG10 mailing list SG10_at_[hidden]<mailto:SG10_at_[hidden]> https://lists.isocpp.org/mailman/listinfo.cgi/sg10<https://urldefense.com/v3/__https:/lists.isocpp.org/mailman/listinfo.cgi/sg10__;!!FbZ0ZwI3Qg!5rC0lYk_TBsLAeQa6KPGG2r1wb2dARogrpGF5j61wHPxzI2wrsz1eXS0UXaB$>
Received on 2020-09-13 11:50:21