Date: Tue, 30 May 2023 21:00:38 +0000
[Mathias]
* BMIs for widely included modules will behave similarly, so it would be ideal if the BMI would only be touched if the interface changes, and not if just the implementation* changes.
The only algorithms we know today about whether the BMI would change requires computing the new BMI, and comparing with the existing one. The cost of that comparison is exceedingly miniscule compare to the computation, and is already well handled by the existing mechanisms of “move-if-change”. What is the new concrete algorithm that you are proposing?
-- Gaby
From: Mathias Stearn <redbeard0531+isocpp_at_[hidden]m>
Sent: Tuesday, May 30, 2023 1:21 PM
To: Gabriel Dos Reis <gdr_at_[hidden]>
Cc: sg15_at_[hidden]; Daniel Ruoso <daniel_at_[hidden]>
Subject: Re: [SG15] P2898R0: Importable Headers are Not Universally Implementable
On Tue, May 30, 2023 at 6:58 PM Gabriel Dos Reis <gdr_at_[hidden]<mailto:gdr_at_[hidden]>> wrote:
* It probably means that we need the compiler to support an option to only overwrite the file if the contents are different.
This is a build system issue general with any “source code” (the fact that BMI isn’t “source code” doesn’t change the fundamental issue) that is generated during build; compilers should be left with just the task of translating code, or occasionally providing some meta data they collected from performing some analysis on the input.
I often design source generators to avoid touching the file if its contents wouldn't change. This is very important for widely included headers, especially if the inputs to the generator are volatile. BMIs for widely included modules will behave similarly, so it would be ideal if the BMI would only be touched if the interface changes, and not if just the implementation* changes. Or alternatively, have a file next to the BMI with this behavior. Of course if the compiler doesn't do that, a build system can emulate this behavior, but it will be a lot simpler and probably more efficient for the compiler to do it itself.
* A very cool version of this that would be tricky for a build system to pull off completely would be to consider `inline` function bodies part of the interface only in optimizing builds. This would still get you perf in your release builds but also lightning fast incremental rebuilds in debug builds even if you change an inline function in a root module.
* The alternative being that the build rule has to generate an intermediate file which then only gets renamed over the old one if the contents are not the same.
This is the status quo.
#include is also the status quo. We are trying to improve the status quo. That will require compilers and build systems working together and considering each other's needs and requests, much more so than in the past.
-- Gaby
From: SG15 <sg15-bounces_at_lists.isocpp.org<mailto:sg15-bounces_at_[hidden]>> On Behalf Of Daniel Ruoso via SG15
Sent: Tuesday, May 30, 2023 7:45 AM
To: Mathias Stearn <redbeard0531+isocpp_at_[hidden]<mailto:redbeard0531%2Bisocpp_at_[hidden]>>
Cc: Daniel Ruoso <daniel_at_[hidden]<mailto:daniel_at_[hidden]>>; sg15_at_[hidden]<mailto:sg15_at_[hidden]>
Subject: Re: [SG15] P2898R0: Importable Headers are Not Universally Implementable
Em sex., 26 de mai. de 2023 às 11:57, Mathias Stearn <redbeard0531+isocpp_at_[hidden]<mailto:redbeard0531%2Bisocpp_at_[hidden]>> escreveu:
Sorry, there was a detail that I forgot to mention explicitly that is important to understanding how this answers your questions. The key is that this leans *very* heavily on ninja's restat=1 reature and a write_if_changed function<https://github.com/RedBeard0531/cxx_modules_builder/blob/e13174805992c7aa10e8fbc9b1452cbfc0f66ecc/modules_builder.py#L92-L99> in the scanner to be sure to only touch files if their contents change. This means that even if we need to redo a scan (which we would because all scans depend on the yaml file holding the list of importable headers), we *don't* need to recompile unless it actually would affect that compile.
This is a massive change in requirements for build systems.
Either the tool driving the build supports the `restat` feature (as ninja does), or the build system has to use the checksum of the contents to drive target invalidation (as scons does).
It probably means that we need the compiler to support an option to only overwrite the file if the contents are different. The alternative being that the build rule has to generate an intermediate file which then only gets renamed over the old one if the contents are not the same.
It essentially means that `make` is no longer a viable implementation to drive a C++ build (not even generated make, as CMake does). As I mentioned before, we use ninja for development builds, but it doesn't satisfy all our requirements for the production builds (e.g.: https://github.com/ninja-build/ninja/issues/1139 ).
And that's the crux of what the paper is pointing at: Only a subset of build systems in use today will be able to viably support Importable Headers, which will exclude systems where Named Modules could be implemented.
I guess the fundamental question is whether we have consensus on "GNU Make is no longer a viable C++ build driver". I, personally, strongly oppose that direction.
daniel
* BMIs for widely included modules will behave similarly, so it would be ideal if the BMI would only be touched if the interface changes, and not if just the implementation* changes.
The only algorithms we know today about whether the BMI would change requires computing the new BMI, and comparing with the existing one. The cost of that comparison is exceedingly miniscule compare to the computation, and is already well handled by the existing mechanisms of “move-if-change”. What is the new concrete algorithm that you are proposing?
-- Gaby
From: Mathias Stearn <redbeard0531+isocpp_at_[hidden]m>
Sent: Tuesday, May 30, 2023 1:21 PM
To: Gabriel Dos Reis <gdr_at_[hidden]>
Cc: sg15_at_[hidden]; Daniel Ruoso <daniel_at_[hidden]>
Subject: Re: [SG15] P2898R0: Importable Headers are Not Universally Implementable
On Tue, May 30, 2023 at 6:58 PM Gabriel Dos Reis <gdr_at_[hidden]<mailto:gdr_at_[hidden]>> wrote:
* It probably means that we need the compiler to support an option to only overwrite the file if the contents are different.
This is a build system issue general with any “source code” (the fact that BMI isn’t “source code” doesn’t change the fundamental issue) that is generated during build; compilers should be left with just the task of translating code, or occasionally providing some meta data they collected from performing some analysis on the input.
I often design source generators to avoid touching the file if its contents wouldn't change. This is very important for widely included headers, especially if the inputs to the generator are volatile. BMIs for widely included modules will behave similarly, so it would be ideal if the BMI would only be touched if the interface changes, and not if just the implementation* changes. Or alternatively, have a file next to the BMI with this behavior. Of course if the compiler doesn't do that, a build system can emulate this behavior, but it will be a lot simpler and probably more efficient for the compiler to do it itself.
* A very cool version of this that would be tricky for a build system to pull off completely would be to consider `inline` function bodies part of the interface only in optimizing builds. This would still get you perf in your release builds but also lightning fast incremental rebuilds in debug builds even if you change an inline function in a root module.
* The alternative being that the build rule has to generate an intermediate file which then only gets renamed over the old one if the contents are not the same.
This is the status quo.
#include is also the status quo. We are trying to improve the status quo. That will require compilers and build systems working together and considering each other's needs and requests, much more so than in the past.
-- Gaby
From: SG15 <sg15-bounces_at_lists.isocpp.org<mailto:sg15-bounces_at_[hidden]>> On Behalf Of Daniel Ruoso via SG15
Sent: Tuesday, May 30, 2023 7:45 AM
To: Mathias Stearn <redbeard0531+isocpp_at_[hidden]<mailto:redbeard0531%2Bisocpp_at_[hidden]>>
Cc: Daniel Ruoso <daniel_at_[hidden]<mailto:daniel_at_[hidden]>>; sg15_at_[hidden]<mailto:sg15_at_[hidden]>
Subject: Re: [SG15] P2898R0: Importable Headers are Not Universally Implementable
Em sex., 26 de mai. de 2023 às 11:57, Mathias Stearn <redbeard0531+isocpp_at_[hidden]<mailto:redbeard0531%2Bisocpp_at_[hidden]>> escreveu:
Sorry, there was a detail that I forgot to mention explicitly that is important to understanding how this answers your questions. The key is that this leans *very* heavily on ninja's restat=1 reature and a write_if_changed function<https://github.com/RedBeard0531/cxx_modules_builder/blob/e13174805992c7aa10e8fbc9b1452cbfc0f66ecc/modules_builder.py#L92-L99> in the scanner to be sure to only touch files if their contents change. This means that even if we need to redo a scan (which we would because all scans depend on the yaml file holding the list of importable headers), we *don't* need to recompile unless it actually would affect that compile.
This is a massive change in requirements for build systems.
Either the tool driving the build supports the `restat` feature (as ninja does), or the build system has to use the checksum of the contents to drive target invalidation (as scons does).
It probably means that we need the compiler to support an option to only overwrite the file if the contents are different. The alternative being that the build rule has to generate an intermediate file which then only gets renamed over the old one if the contents are not the same.
It essentially means that `make` is no longer a viable implementation to drive a C++ build (not even generated make, as CMake does). As I mentioned before, we use ninja for development builds, but it doesn't satisfy all our requirements for the production builds (e.g.: https://github.com/ninja-build/ninja/issues/1139 ).
And that's the crux of what the paper is pointing at: Only a subset of build systems in use today will be able to viably support Importable Headers, which will exclude systems where Named Modules could be implemented.
I guess the fundamental question is whether we have consensus on "GNU Make is no longer a viable C++ build driver". I, personally, strongly oppose that direction.
daniel
Received on 2023-05-30 21:00:41