Date: Wed, 24 May 2023 11:06:54 -0400
On 5/24/23 10:14 AM, Daniel Ruoso via SG15 wrote:
> Em ter., 23 de mai. de 2023 às 21:54, Tom Honermann
> <tom_at_[hidden]> escreveu:
>> What subset of environments is it not implementable in? The paper doesn't actually say as far as I can tell. What is a concrete example?
> It is not implementable in open-ended build systems, such as what we
> have at Bloomberg.
I still don't see an issue. It would be very helpful if, as Jens
suggested, you provide a concrete example of what you perceive to be the
problem.
>
> Concretely, what would happen is that if I do an `apt-get update &&
> apt-get dist-upgrade` it could result in a full clean build in my
> incremental build because the list of importable headers or the
> arguments to those could change.
We seem to have a clear disagreement regarding this. Let's try to
separate the possibilities:
_1: A header file that was not previously importable becomes importable._
You have been claiming that this situation requires that dependency
information for all TUs in the build be re-generated. I claim that only
the TUs that previously included that header file need to be re-scanned
(and rebuilt if dependencies actually changed). We aren't both right.
What is your argument for why I am wrong? Please provide an example that
illustrates the problem. In particular, please show an example of a TU
that did not previously include the header file but that requires
re-scanning because that header file is now importable.
_2: A previously importable header becomes non-importable._
The issues here are similar to the situation in which a previously
deprecated/obsoleted header file is removed. At least some build systems
will report an error in this case because a dependent file (e.g., a BMI
file for the no longer importable header file) might no longer be found
(and no rules are present to generate it). The solution in these cases
is to regenerate dependency information (for the affected TUs). If such
an error does not occur (e.g., because a BMI file is still present),
then build systems that use the common approach of using dependency
information that is generated as a side effect of compilation will be
fine (because prior dependencies will be satisfied and re-compilation
will produce dependencies that no longer reference the BMI).
Note that I am assuming that the BMI file corresponding to an importable
header is tracked with the dependency information (a build system that
does not track BMI files is not sound because it won't behave correctly
when BMI files are modified from outside the build system).
_3: A new (public) header file (that might or might not be importable)
is added._
This technically requires a full re-scan of the project because of the
possibility of conditional inclusion based on __has_include. And TUs
that gain a new dependency must be rebuilt.
I readily admit that many build systems do not handle these kind of
changes as efficiently as they could. But these are not new problems; as
I've previously stated, these situations are very similar to those for
generated header files.
I'll also state that I don't find the example of an incremental rebuild
following an apt-get update or apt-get dist-upgrade compelling. I have
yet to use a build system that I would trust to handle such situations
(primarily because timestamp information between the distribution
provided header files and previously built objects are no longer
correlated in a meaningful way; assuming the package manager installs
header files with a preserved timestamp; a system that matches file
contents is necessary in that case).
>> The paper doesn't quantify costs in any way. The closest it comes is (correctly) noting that dependency scanning for header units requires computing imported macros in a bottom up way that is not required for named modules. But the paper doesn't quantify that cost. Is it a 5% hit? A 95% hit? Linear with respect to the number of header units? Other papers have offered quantification; see P1441 (Are modules fast?) for example.
> The cost is quite binary, actually. Every time the list of importable
> headers or the arguments to those change, it results in a full clean
> build.
Why a clean build? If the dependency information didn't actually change
for a given TU, why would its previously built objects have to be
rebuilt (assuming its dependent header files weren't actually modified)?
Tom.
>
> daniel
> _______________________________________________
> SG15 mailing list
> SG15_at_[hidden]
> https://lists.isocpp.org/mailman/listinfo.cgi/sg15
> Em ter., 23 de mai. de 2023 às 21:54, Tom Honermann
> <tom_at_[hidden]> escreveu:
>> What subset of environments is it not implementable in? The paper doesn't actually say as far as I can tell. What is a concrete example?
> It is not implementable in open-ended build systems, such as what we
> have at Bloomberg.
I still don't see an issue. It would be very helpful if, as Jens
suggested, you provide a concrete example of what you perceive to be the
problem.
>
> Concretely, what would happen is that if I do an `apt-get update &&
> apt-get dist-upgrade` it could result in a full clean build in my
> incremental build because the list of importable headers or the
> arguments to those could change.
We seem to have a clear disagreement regarding this. Let's try to
separate the possibilities:
_1: A header file that was not previously importable becomes importable._
You have been claiming that this situation requires that dependency
information for all TUs in the build be re-generated. I claim that only
the TUs that previously included that header file need to be re-scanned
(and rebuilt if dependencies actually changed). We aren't both right.
What is your argument for why I am wrong? Please provide an example that
illustrates the problem. In particular, please show an example of a TU
that did not previously include the header file but that requires
re-scanning because that header file is now importable.
_2: A previously importable header becomes non-importable._
The issues here are similar to the situation in which a previously
deprecated/obsoleted header file is removed. At least some build systems
will report an error in this case because a dependent file (e.g., a BMI
file for the no longer importable header file) might no longer be found
(and no rules are present to generate it). The solution in these cases
is to regenerate dependency information (for the affected TUs). If such
an error does not occur (e.g., because a BMI file is still present),
then build systems that use the common approach of using dependency
information that is generated as a side effect of compilation will be
fine (because prior dependencies will be satisfied and re-compilation
will produce dependencies that no longer reference the BMI).
Note that I am assuming that the BMI file corresponding to an importable
header is tracked with the dependency information (a build system that
does not track BMI files is not sound because it won't behave correctly
when BMI files are modified from outside the build system).
_3: A new (public) header file (that might or might not be importable)
is added._
This technically requires a full re-scan of the project because of the
possibility of conditional inclusion based on __has_include. And TUs
that gain a new dependency must be rebuilt.
I readily admit that many build systems do not handle these kind of
changes as efficiently as they could. But these are not new problems; as
I've previously stated, these situations are very similar to those for
generated header files.
I'll also state that I don't find the example of an incremental rebuild
following an apt-get update or apt-get dist-upgrade compelling. I have
yet to use a build system that I would trust to handle such situations
(primarily because timestamp information between the distribution
provided header files and previously built objects are no longer
correlated in a meaningful way; assuming the package manager installs
header files with a preserved timestamp; a system that matches file
contents is necessary in that case).
>> The paper doesn't quantify costs in any way. The closest it comes is (correctly) noting that dependency scanning for header units requires computing imported macros in a bottom up way that is not required for named modules. But the paper doesn't quantify that cost. Is it a 5% hit? A 95% hit? Linear with respect to the number of header units? Other papers have offered quantification; see P1441 (Are modules fast?) for example.
> The cost is quite binary, actually. Every time the list of importable
> headers or the arguments to those change, it results in a full clean
> build.
Why a clean build? If the dependency information didn't actually change
for a given TU, why would its previously built objects have to be
rebuilt (assuming its dependent header files weren't actually modified)?
Tom.
>
> daniel
> _______________________________________________
> SG15 mailing list
> SG15_at_[hidden]
> https://lists.isocpp.org/mailman/listinfo.cgi/sg15
Received on 2023-05-24 15:06:55