Date: Tue, 19 Apr 2022 11:22:09 +0000
[Tom]
> If by this you mean that cached module artifacts may become out of date
> with respect to the source files used to construct them, I don't believe
> that to be a fundamental limitation of the implicit module approach.
I wasn't stating belief, but merely reporting on the experience of the people who either introduced or worked on the implicit build in Clang, and then used them.
> Implementations can index artifacts by hashes of their inputs (including
> command line options, file and path names, file timestamps, file
> contents, etc...), observe when a cached artifact is older than one of
> its inputs, and then generate a replacement. Cached artifacts can be
> garbage collected to reduce cache size. In cases where many source files
> contribute to a module, that implies a need to frequently stat() a
> potentially large set of source files, but the costs of doing so can
> likely be mitigated by a service that listens for and caches filesystem
> updates for a select set of files. Avoiding violations of the ODR would
> still require that a build system ensure that source files that
> contribute to module artifacts appear as dependencies for TUs that
> consume those modules (or be assumed to be static as is often the case
> for system headers); there is no avoiding that requirement.
At which point, the compiler has now implemented a possibly new build system. It is a question to be resolved whether that build system logic is best built into a separate tool that calls onto the compiler to just to traditional compiler stuff, agnostic to build system logics. Such separate tools can then be used by teachers or hobbyists or experienced programmers trying examples in the small. I still don't see the connection to the $Subject of this thread though.
> I'm much more concerned about the ability to compile a single TU that imports (portions of) the standard library.
While an important use case, I am not a concerned, as the experiments that MSVC has been running since 2015 contains those scenarios as basics.
> If, going
> forward, a build system is required to compile the following example
> because it is necessary to prebuild modules for the standard library
> using whatever compiler options are being used to compile a single
> source file into an executable, then I think we'll have done the C++
> community quite a disservice.
In the experience that MSVC has been offering its users since 2015, you don't need a particular build system (if you replace "std" with "std.io"; but that will be adjusted once we know for certain that "std" is in C++23 draft) to compiler that example.
-- Gaby
-----Original Message-----
From: Tom Honermann <tom_at_[hidden]>
Sent: Monday, April 18, 2022 8:24 PM
To: Gabriel Dos Reis <gdr_at_[hidden]>; ext_at_[hidden]; Boris Kolpackov <boris_at_[hidden]>
Cc: Nathan Sidwell <nathan_at_[hidden]>; sg15_at_[hidden]
Subject: Re: [isocpp-ext] Can we expect that all C++ source files can have the same suffix?
On 4/17/22 10:13 AM, Gabriel Dos Reis wrote:
> [Tom]
>> But I don't think that implies that implicit
>> modules causes problems for build systems and dependency management at
>> scale; there are other related factors.
> Cache invalidation.
If by this you mean that cached module artifacts may become out of date
with respect to the source files used to construct them, I don't believe
that to be a fundamental limitation of the implicit module approach.
Implementations can index artifacts by hashes of their inputs (including
command line options, file and path names, file timestamps, file
contents, etc...), observe when a cached artifact is older than one of
its inputs, and then generate a replacement. Cached artifacts can be
garbage collected to reduce cache size. In cases where many source files
contribute to a module, that implies a need to frequently stat() a
potentially large set of source files, but the costs of doing so can
likely be mitigated by a service that listens for and caches filesystem
updates for a select set of files. Avoiding violations of the ODR would
still require that a build system ensure that source files that
contribute to module artifacts appear as dependencies for TUs that
consume those modules (or be assumed to be static as is often the case
for system headers); there is no avoiding that requirement.
With respect to Nico's concerns, I'm not very concerned about the
ability to compile a bunch of source files together in a single command
line invocation. I'm much more concerned about the ability to compile a
single TU that imports (portions of) the standard library. If, going
forward, a build system is required to compile the following example
because it is necessary to prebuild modules for the standard library
using whatever compiler options are being used to compile a single
source file into an executable, then I think we'll have done the C++
community quite a disservice.
import std;
int main(int argc, char **argv) {
std::cout << "Hello!\n";
}
Tom.
>
> -- Gaby
>
>
> -----Original Message-----
> From: Tom Honermann <tom_at_[hidden]>
> Sent: Sunday, April 17, 2022 6:02 AM
> To: ext_at_[hidden]; Boris Kolpackov <boris_at_[hidden]>
> Cc: Gabriel Dos Reis <gdr_at_[hidden]>; Nathan Sidwell <nathan_at_[hidden]>; WG21 Tooling Study Group SG15 <tooling_at_[hidden]>
> Subject: Re: [isocpp-ext] Can we expect that all C++ source files can have the same suffix?
>
> On 4/16/22 2:57 PM, Gabriel Dos Reis via Ext wrote:
>> [Nathan]
>>> True, but IIUC that is how clang's implicit module machinery already
>>> works. using the module map, the compiler will fork to compile an
>>> implicit module (equivalent of a header unit). I don't know how
>>> concurrency concerns are addressed. (I don't have data on how this
>>> scheme behaves in practice)
>> "Implicit build of module" a la Clang (which was originally implemented for Clang Modules) is known to cause problems for build systems and dependency management at scale, which is why they introduced an "explicit build". Last time I heard, organizations using the existing "implicit build" system would love to move away from it, and move towards the "explicit build" approach.
>>
>> I fully appreciate that "implicit build" works for nice little example. I am hoping that issue can be solved by other means (either using an IDE or a pre-scanner to construct a sort of module map) than moving build system logics into the compilers.
> My impression is a little different. When working on Coverity, I saw
> many examples of implicit modules being successfully used by Coverity
> customers (primarily with Objective-C, primarily in conjunction with
> Apple provided frameworks). I think the desire for explicit builds is
> driven by a desire to improve efficiency over what implicit builds can
> provide (implicit builds often result in the same interface units being
> built multiple times subject to implementation ability to recognize when
> certain command line options are irrelevant to the module being built;
> an admittedly difficult problem). Thus there is a tradeoff; willingness
> to accept a less efficient solution vs expending additional effort to
> modify the build system to support explicit module builds. I acknowledge
> that providing an explicit build solution that is easy to deploy is a
> win-win for everyone. But I don't think that implies that implicit
> modules causes problems for build systems and dependency management at
> scale; there are other related factors.
>
> Tom.
>
>> -- Gaby
>>
>> -----Original Message-----
>> From: Ext <ext-bounces_at_[hidden]> On Behalf Of Nathan Sidwell via Ext
>> Sent: Friday, April 15, 2022 4:51 AM
>> To: ext_at_[hidden]; Boris Kolpackov <boris_at_[hidden]>
>> Cc: Nathan Sidwell <nathan_at_[hidden]>; WG21 Tooling Study Group SG15 <tooling_at_[hidden]>
>> Subject: Re: [isocpp-ext] Can we expect that all C++ source files can have the same suffix?
>>
>> On 4/14/22 16:18, Ben Boeckel via Ext wrote:
>>> On Thu, Apr 14, 2022 at 15:09:35 +0200, Boris Kolpackov wrote:
>>>> Ben Boeckel via Ext <ext_at_[hidden]> writes:
>>>>> Need? No. Even with different suffixes, a scanner is still necessary to
>>>>> know what order files need to be compiled in.
>>>> A (pre-)scanner is not the only way to do it. Another approach is a
>>>> module mapper (implemented in GCC, being implemented in Clang, used
>>>> in build2).
>>> Well, an active module mapper. CMake just creates the static files which
>>> handle the GCC's module mapper protocol.
>>>
>>>> With the mapper approach you don't need to pre-scan everything but
>>>> you do need to map module names to module interface files. Having
>>>> a separate extension for interface files helps narrow down the
>>>> candidate pool.
>>> While true, I have doubts about the scalability of holding umpteen
>>> compiler instances around while executing more to find the bottleneck
>>> TU. Unsatisifiable requests also end up taking way more resources
>>> without smarter caching mechanisms to store that information across
>> True, but IIUC that is how clang's implicit module machinery already
>> works. using the module map, the compiler will fork to compile an
>> implicit module (equivalent of a header unit). I don't know how
>> concurrency concerns are addressed. (I don't have data on how this
>> scheme behaves in practice)
>>
>>> attempts. Which is fine if you're implementing your own build executor I
>>> suppose. Getting Make or Ninja to understand it without complicated
>>> stamp juggling doesn't sound fun though.
>>>
>>> --Ben
>>> _______________________________________________
>>> Ext mailing list
>>> Ext_at_[hidden]
>>> Subscription: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.isocpp.org%2Fmailman%2Flistinfo.cgi%2Fext&data=05%7C01%7Cgdr%40microsoft.com%7Cd16b0376753b4c37091e08da21b40b1d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637859354422028725%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3kctZNp7Kc%2FvV54%2Fu8zC1MXuqwK63OmrjNL7GFlrpyw%3D&reserved=0
>>> Link to this post: https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.isocpp.org%2Fext%2F2022%2F04%2F19012.php&data=05%7C01%7Cgdr%40microsoft.com%7Cd16b0376753b4c37091e08da21b40b1d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637859354422028725%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=GguTcSpRpEE0LlDPPUf9GBxmNh2BfOoO6oYuJYZXgQA%3D&reserved=0
>>>
> If by this you mean that cached module artifacts may become out of date
> with respect to the source files used to construct them, I don't believe
> that to be a fundamental limitation of the implicit module approach.
I wasn't stating belief, but merely reporting on the experience of the people who either introduced or worked on the implicit build in Clang, and then used them.
> Implementations can index artifacts by hashes of their inputs (including
> command line options, file and path names, file timestamps, file
> contents, etc...), observe when a cached artifact is older than one of
> its inputs, and then generate a replacement. Cached artifacts can be
> garbage collected to reduce cache size. In cases where many source files
> contribute to a module, that implies a need to frequently stat() a
> potentially large set of source files, but the costs of doing so can
> likely be mitigated by a service that listens for and caches filesystem
> updates for a select set of files. Avoiding violations of the ODR would
> still require that a build system ensure that source files that
> contribute to module artifacts appear as dependencies for TUs that
> consume those modules (or be assumed to be static as is often the case
> for system headers); there is no avoiding that requirement.
At which point, the compiler has now implemented a possibly new build system. It is a question to be resolved whether that build system logic is best built into a separate tool that calls onto the compiler to just to traditional compiler stuff, agnostic to build system logics. Such separate tools can then be used by teachers or hobbyists or experienced programmers trying examples in the small. I still don't see the connection to the $Subject of this thread though.
> I'm much more concerned about the ability to compile a single TU that imports (portions of) the standard library.
While an important use case, I am not a concerned, as the experiments that MSVC has been running since 2015 contains those scenarios as basics.
> If, going
> forward, a build system is required to compile the following example
> because it is necessary to prebuild modules for the standard library
> using whatever compiler options are being used to compile a single
> source file into an executable, then I think we'll have done the C++
> community quite a disservice.
In the experience that MSVC has been offering its users since 2015, you don't need a particular build system (if you replace "std" with "std.io"; but that will be adjusted once we know for certain that "std" is in C++23 draft) to compiler that example.
-- Gaby
-----Original Message-----
From: Tom Honermann <tom_at_[hidden]>
Sent: Monday, April 18, 2022 8:24 PM
To: Gabriel Dos Reis <gdr_at_[hidden]>; ext_at_[hidden]; Boris Kolpackov <boris_at_[hidden]>
Cc: Nathan Sidwell <nathan_at_[hidden]>; sg15_at_[hidden]
Subject: Re: [isocpp-ext] Can we expect that all C++ source files can have the same suffix?
On 4/17/22 10:13 AM, Gabriel Dos Reis wrote:
> [Tom]
>> But I don't think that implies that implicit
>> modules causes problems for build systems and dependency management at
>> scale; there are other related factors.
> Cache invalidation.
If by this you mean that cached module artifacts may become out of date
with respect to the source files used to construct them, I don't believe
that to be a fundamental limitation of the implicit module approach.
Implementations can index artifacts by hashes of their inputs (including
command line options, file and path names, file timestamps, file
contents, etc...), observe when a cached artifact is older than one of
its inputs, and then generate a replacement. Cached artifacts can be
garbage collected to reduce cache size. In cases where many source files
contribute to a module, that implies a need to frequently stat() a
potentially large set of source files, but the costs of doing so can
likely be mitigated by a service that listens for and caches filesystem
updates for a select set of files. Avoiding violations of the ODR would
still require that a build system ensure that source files that
contribute to module artifacts appear as dependencies for TUs that
consume those modules (or be assumed to be static as is often the case
for system headers); there is no avoiding that requirement.
With respect to Nico's concerns, I'm not very concerned about the
ability to compile a bunch of source files together in a single command
line invocation. I'm much more concerned about the ability to compile a
single TU that imports (portions of) the standard library. If, going
forward, a build system is required to compile the following example
because it is necessary to prebuild modules for the standard library
using whatever compiler options are being used to compile a single
source file into an executable, then I think we'll have done the C++
community quite a disservice.
import std;
int main(int argc, char **argv) {
std::cout << "Hello!\n";
}
Tom.
>
> -- Gaby
>
>
> -----Original Message-----
> From: Tom Honermann <tom_at_[hidden]>
> Sent: Sunday, April 17, 2022 6:02 AM
> To: ext_at_[hidden]; Boris Kolpackov <boris_at_[hidden]>
> Cc: Gabriel Dos Reis <gdr_at_[hidden]>; Nathan Sidwell <nathan_at_[hidden]>; WG21 Tooling Study Group SG15 <tooling_at_[hidden]>
> Subject: Re: [isocpp-ext] Can we expect that all C++ source files can have the same suffix?
>
> On 4/16/22 2:57 PM, Gabriel Dos Reis via Ext wrote:
>> [Nathan]
>>> True, but IIUC that is how clang's implicit module machinery already
>>> works. using the module map, the compiler will fork to compile an
>>> implicit module (equivalent of a header unit). I don't know how
>>> concurrency concerns are addressed. (I don't have data on how this
>>> scheme behaves in practice)
>> "Implicit build of module" a la Clang (which was originally implemented for Clang Modules) is known to cause problems for build systems and dependency management at scale, which is why they introduced an "explicit build". Last time I heard, organizations using the existing "implicit build" system would love to move away from it, and move towards the "explicit build" approach.
>>
>> I fully appreciate that "implicit build" works for nice little example. I am hoping that issue can be solved by other means (either using an IDE or a pre-scanner to construct a sort of module map) than moving build system logics into the compilers.
> My impression is a little different. When working on Coverity, I saw
> many examples of implicit modules being successfully used by Coverity
> customers (primarily with Objective-C, primarily in conjunction with
> Apple provided frameworks). I think the desire for explicit builds is
> driven by a desire to improve efficiency over what implicit builds can
> provide (implicit builds often result in the same interface units being
> built multiple times subject to implementation ability to recognize when
> certain command line options are irrelevant to the module being built;
> an admittedly difficult problem). Thus there is a tradeoff; willingness
> to accept a less efficient solution vs expending additional effort to
> modify the build system to support explicit module builds. I acknowledge
> that providing an explicit build solution that is easy to deploy is a
> win-win for everyone. But I don't think that implies that implicit
> modules causes problems for build systems and dependency management at
> scale; there are other related factors.
>
> Tom.
>
>> -- Gaby
>>
>> -----Original Message-----
>> From: Ext <ext-bounces_at_[hidden]> On Behalf Of Nathan Sidwell via Ext
>> Sent: Friday, April 15, 2022 4:51 AM
>> To: ext_at_[hidden]; Boris Kolpackov <boris_at_[hidden]>
>> Cc: Nathan Sidwell <nathan_at_[hidden]>; WG21 Tooling Study Group SG15 <tooling_at_[hidden]>
>> Subject: Re: [isocpp-ext] Can we expect that all C++ source files can have the same suffix?
>>
>> On 4/14/22 16:18, Ben Boeckel via Ext wrote:
>>> On Thu, Apr 14, 2022 at 15:09:35 +0200, Boris Kolpackov wrote:
>>>> Ben Boeckel via Ext <ext_at_[hidden]> writes:
>>>>> Need? No. Even with different suffixes, a scanner is still necessary to
>>>>> know what order files need to be compiled in.
>>>> A (pre-)scanner is not the only way to do it. Another approach is a
>>>> module mapper (implemented in GCC, being implemented in Clang, used
>>>> in build2).
>>> Well, an active module mapper. CMake just creates the static files which
>>> handle the GCC's module mapper protocol.
>>>
>>>> With the mapper approach you don't need to pre-scan everything but
>>>> you do need to map module names to module interface files. Having
>>>> a separate extension for interface files helps narrow down the
>>>> candidate pool.
>>> While true, I have doubts about the scalability of holding umpteen
>>> compiler instances around while executing more to find the bottleneck
>>> TU. Unsatisifiable requests also end up taking way more resources
>>> without smarter caching mechanisms to store that information across
>> True, but IIUC that is how clang's implicit module machinery already
>> works. using the module map, the compiler will fork to compile an
>> implicit module (equivalent of a header unit). I don't know how
>> concurrency concerns are addressed. (I don't have data on how this
>> scheme behaves in practice)
>>
>>> attempts. Which is fine if you're implementing your own build executor I
>>> suppose. Getting Make or Ninja to understand it without complicated
>>> stamp juggling doesn't sound fun though.
>>>
>>> --Ben
>>> _______________________________________________
>>> Ext mailing list
>>> Ext_at_[hidden]
>>> Subscription: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.isocpp.org%2Fmailman%2Flistinfo.cgi%2Fext&data=05%7C01%7Cgdr%40microsoft.com%7Cd16b0376753b4c37091e08da21b40b1d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637859354422028725%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3kctZNp7Kc%2FvV54%2Fu8zC1MXuqwK63OmrjNL7GFlrpyw%3D&reserved=0
>>> Link to this post: https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.isocpp.org%2Fext%2F2022%2F04%2F19012.php&data=05%7C01%7Cgdr%40microsoft.com%7Cd16b0376753b4c37091e08da21b40b1d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637859354422028725%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=GguTcSpRpEE0LlDPPUf9GBxmNh2BfOoO6oYuJYZXgQA%3D&reserved=0
>>>
Received on 2022-04-19 11:22:14