Date: Tue, 19 Apr 2022 12:42:54 -0400
On 4/19/22 7:22 AM, Gabriel Dos Reis via SG15 wrote:
> [Tom]
>
>> Implementations can index artifacts by hashes of their inputs (including
>> command line options, file and path names, file timestamps, file
>> contents, etc...), observe when a cached artifact is older than one of
>> its inputs, and then generate a replacement. Cached artifacts can be
>> garbage collected to reduce cache size. In cases where many source files
>> contribute to a module, that implies a need to frequently stat() a
>> potentially large set of source files, but the costs of doing so can
>> likely be mitigated by a service that listens for and caches filesystem
>> updates for a select set of files. Avoiding violations of the ODR would
>> still require that a build system ensure that source files that
>> contribute to module artifacts appear as dependencies for TUs that
>> consume those modules (or be assumed to be static as is often the case
>> for system headers); there is no avoiding that requirement.
> At which point, the compiler has now implemented a possibly new build system. It is a question to be resolved whether that build system logic is best built into a separate tool that calls onto the compiler to just to traditional compiler stuff, agnostic to build system logics. Such separate tools can then be used by teachers or hobbyists or experienced programmers trying examples in the small. I still don't see the connection to the $Subject of this thread though.
Compiler drivers have effectively been mini build systems for a long
time now. They aren't the best at what they do and that is ok; they have
traditionally served the simple interactive use cases well. The adoption
of a more capable build system comes with the need for automation and
performance; but those needs are not universal, nor without significant
cost.
The connection to the subject of this thread is Nico's underlying
motivation. He wants to continue to be able to compiler simple programs
with something like $CC module.cppm module_impl.cpp main.cpp -o
prog.exe. I agree with that motivation; I compile small programs like
that nearly every day. My strong concerns with support of a modularized
standard library are similar; interactive command line compiler
invocations are an important concern.
>
>> I'm much more concerned about the ability to compile a single TU that imports (portions of) the standard library.
> While an important use case, I am not a concerned, as the experiments that MSVC has been running since 2015 contains those scenarios as basics.
Have those experiments involved interactive use of command line compilers?
>
>> If, going
>> forward, a build system is required to compile the following example
>> because it is necessary to prebuild modules for the standard library
>> using whatever compiler options are being used to compile a single
>> source file into an executable, then I think we'll have done the C++
>> community quite a disservice.
> In the experience that MSVC has been offering its users since 2015, you don't need a particular build system (if you replace "std" with "std.io"; but that will be adjusted once we know for certain that "std" is in C++23 draft) to compiler that example.
That is true for a limited number of configurations because MSVC
distributes prebuilt BMIs for those configurations, but to my knowledge
is not true in general. For example, if I wanted to enable specific
Microsoft extensions like support for iterator debugging, then I would
have to build my own variants of the std module.
Tom.
>
> -- Gaby
>
> -----Original Message-----
> From: Tom Honermann <tom_at_[hidden]>
> Sent: Monday, April 18, 2022 8:24 PM
> To: Gabriel Dos Reis <gdr_at_[hidden]>; ext_at_[hidden]; Boris Kolpackov <boris_at_[hidden]>
> Cc: Nathan Sidwell <nathan_at_[hidden]>; sg15_at_[hidden]
> Subject: Re: [isocpp-ext] Can we expect that all C++ source files can have the same suffix?
>
> On 4/17/22 10:13 AM, Gabriel Dos Reis wrote:
>> [Tom]
>>> But I don't think that implies that implicit
>>> modules causes problems for build systems and dependency management at
>>> scale; there are other related factors.
>> Cache invalidation.
> If by this you mean that cached module artifacts may become out of date
> with respect to the source files used to construct them, I don't believe
> that to be a fundamental limitation of the implicit module approach.
> Implementations can index artifacts by hashes of their inputs (including
> command line options, file and path names, file timestamps, file
> contents, etc...), observe when a cached artifact is older than one of
> its inputs, and then generate a replacement. Cached artifacts can be
> garbage collected to reduce cache size. In cases where many source files
> contribute to a module, that implies a need to frequently stat() a
> potentially large set of source files, but the costs of doing so can
> likely be mitigated by a service that listens for and caches filesystem
> updates for a select set of files. Avoiding violations of the ODR would
> still require that a build system ensure that source files that
> contribute to module artifacts appear as dependencies for TUs that
> consume those modules (or be assumed to be static as is often the case
> for system headers); there is no avoiding that requirement.
>
> With respect to Nico's concerns, I'm not very concerned about the
> ability to compile a bunch of source files together in a single command
> line invocation. I'm much more concerned about the ability to compile a
> single TU that imports (portions of) the standard library. If, going
> forward, a build system is required to compile the following example
> because it is necessary to prebuild modules for the standard library
> using whatever compiler options are being used to compile a single
> source file into an executable, then I think we'll have done the C++
> community quite a disservice.
>
> import std;
> int main(int argc, char **argv) {
> std::cout << "Hello!\n";
> }
>
> Tom.
>
>> -- Gaby
>>
>>
>> -----Original Message-----
>> From: Tom Honermann <tom_at_[hidden]>
>> Sent: Sunday, April 17, 2022 6:02 AM
>> To: ext_at_[hidden]; Boris Kolpackov <boris_at_[hidden]>
>> Cc: Gabriel Dos Reis <gdr_at_[hidden]>; Nathan Sidwell <nathan_at_[hidden]>; WG21 Tooling Study Group SG15 <tooling_at_[hidden]>
>> Subject: Re: [isocpp-ext] Can we expect that all C++ source files can have the same suffix?
>>
>> On 4/16/22 2:57 PM, Gabriel Dos Reis via Ext wrote:
>>> [Nathan]
>>>> True, but IIUC that is how clang's implicit module machinery already
>>>> works. using the module map, the compiler will fork to compile an
>>>> implicit module (equivalent of a header unit). I don't know how
>>>> concurrency concerns are addressed. (I don't have data on how this
>>>> scheme behaves in practice)
>>> "Implicit build of module" a la Clang (which was originally implemented for Clang Modules) is known to cause problems for build systems and dependency management at scale, which is why they introduced an "explicit build". Last time I heard, organizations using the existing "implicit build" system would love to move away from it, and move towards the "explicit build" approach.
>>>
>>> I fully appreciate that "implicit build" works for nice little example. I am hoping that issue can be solved by other means (either using an IDE or a pre-scanner to construct a sort of module map) than moving build system logics into the compilers.
>> My impression is a little different. When working on Coverity, I saw
>> many examples of implicit modules being successfully used by Coverity
>> customers (primarily with Objective-C, primarily in conjunction with
>> Apple provided frameworks). I think the desire for explicit builds is
>> driven by a desire to improve efficiency over what implicit builds can
>> provide (implicit builds often result in the same interface units being
>> built multiple times subject to implementation ability to recognize when
>> certain command line options are irrelevant to the module being built;
>> an admittedly difficult problem). Thus there is a tradeoff; willingness
>> to accept a less efficient solution vs expending additional effort to
>> modify the build system to support explicit module builds. I acknowledge
>> that providing an explicit build solution that is easy to deploy is a
>> win-win for everyone. But I don't think that implies that implicit
>> modules causes problems for build systems and dependency management at
>> scale; there are other related factors.
>>
>> Tom.
>>
>>> -- Gaby
>>>
>>> -----Original Message-----
>>> From: Ext <ext-bounces_at_[hidden]> On Behalf Of Nathan Sidwell via Ext
>>> Sent: Friday, April 15, 2022 4:51 AM
>>> To: ext_at_[hidden]; Boris Kolpackov <boris_at_[hidden]>
>>> Cc: Nathan Sidwell <nathan_at_[hidden]>; WG21 Tooling Study Group SG15 <tooling_at_[hidden]>
>>> Subject: Re: [isocpp-ext] Can we expect that all C++ source files can have the same suffix?
>>>
>>> On 4/14/22 16:18, Ben Boeckel via Ext wrote:
>>>> On Thu, Apr 14, 2022 at 15:09:35 +0200, Boris Kolpackov wrote:
>>>>> Ben Boeckel via Ext <ext_at_[hidden]> writes:
>>>>>> Need? No. Even with different suffixes, a scanner is still necessary to
>>>>>> know what order files need to be compiled in.
>>>>> A (pre-)scanner is not the only way to do it. Another approach is a
>>>>> module mapper (implemented in GCC, being implemented in Clang, used
>>>>> in build2).
>>>> Well, an active module mapper. CMake just creates the static files which
>>>> handle the GCC's module mapper protocol.
>>>>
>>>>> With the mapper approach you don't need to pre-scan everything but
>>>>> you do need to map module names to module interface files. Having
>>>>> a separate extension for interface files helps narrow down the
>>>>> candidate pool.
>>>> While true, I have doubts about the scalability of holding umpteen
>>>> compiler instances around while executing more to find the bottleneck
>>>> TU. Unsatisifiable requests also end up taking way more resources
>>>> without smarter caching mechanisms to store that information across
>>> True, but IIUC that is how clang's implicit module machinery already
>>> works. using the module map, the compiler will fork to compile an
>>> implicit module (equivalent of a header unit). I don't know how
>>> concurrency concerns are addressed. (I don't have data on how this
>>> scheme behaves in practice)
>>>
>>>> attempts. Which is fine if you're implementing your own build executor I
>>>> suppose. Getting Make or Ninja to understand it without complicated
>>>> stamp juggling doesn't sound fun though.
>>>>
>>>> --Ben
>>>> _______________________________________________
>>>> Ext mailing list
>>>> Ext_at_[hidden]
>>>> Subscription: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.isocpp.org%2Fmailman%2Flistinfo.cgi%2Fext&data=05%7C01%7Cgdr%40microsoft.com%7Cd16b0376753b4c37091e08da21b40b1d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637859354422028725%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3kctZNp7Kc%2FvV54%2Fu8zC1MXuqwK63OmrjNL7GFlrpyw%3D&reserved=0
>>>> Link to this post: https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.isocpp.org%2Fext%2F2022%2F04%2F19012.php&data=05%7C01%7Cgdr%40microsoft.com%7Cd16b0376753b4c37091e08da21b40b1d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637859354422028725%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=GguTcSpRpEE0LlDPPUf9GBxmNh2BfOoO6oYuJYZXgQA%3D&reserved=0
>>>>
> _______________________________________________
> SG15 mailing list
> SG15_at_[hidden]
> https://lists.isocpp.org/mailman/listinfo.cgi/sg15
> [Tom]
>
>> Implementations can index artifacts by hashes of their inputs (including
>> command line options, file and path names, file timestamps, file
>> contents, etc...), observe when a cached artifact is older than one of
>> its inputs, and then generate a replacement. Cached artifacts can be
>> garbage collected to reduce cache size. In cases where many source files
>> contribute to a module, that implies a need to frequently stat() a
>> potentially large set of source files, but the costs of doing so can
>> likely be mitigated by a service that listens for and caches filesystem
>> updates for a select set of files. Avoiding violations of the ODR would
>> still require that a build system ensure that source files that
>> contribute to module artifacts appear as dependencies for TUs that
>> consume those modules (or be assumed to be static as is often the case
>> for system headers); there is no avoiding that requirement.
> At which point, the compiler has now implemented a possibly new build system. It is a question to be resolved whether that build system logic is best built into a separate tool that calls onto the compiler to just to traditional compiler stuff, agnostic to build system logics. Such separate tools can then be used by teachers or hobbyists or experienced programmers trying examples in the small. I still don't see the connection to the $Subject of this thread though.
Compiler drivers have effectively been mini build systems for a long
time now. They aren't the best at what they do and that is ok; they have
traditionally served the simple interactive use cases well. The adoption
of a more capable build system comes with the need for automation and
performance; but those needs are not universal, nor without significant
cost.
The connection to the subject of this thread is Nico's underlying
motivation. He wants to continue to be able to compiler simple programs
with something like $CC module.cppm module_impl.cpp main.cpp -o
prog.exe. I agree with that motivation; I compile small programs like
that nearly every day. My strong concerns with support of a modularized
standard library are similar; interactive command line compiler
invocations are an important concern.
>
>> I'm much more concerned about the ability to compile a single TU that imports (portions of) the standard library.
> While an important use case, I am not a concerned, as the experiments that MSVC has been running since 2015 contains those scenarios as basics.
Have those experiments involved interactive use of command line compilers?
>
>> If, going
>> forward, a build system is required to compile the following example
>> because it is necessary to prebuild modules for the standard library
>> using whatever compiler options are being used to compile a single
>> source file into an executable, then I think we'll have done the C++
>> community quite a disservice.
> In the experience that MSVC has been offering its users since 2015, you don't need a particular build system (if you replace "std" with "std.io"; but that will be adjusted once we know for certain that "std" is in C++23 draft) to compiler that example.
That is true for a limited number of configurations because MSVC
distributes prebuilt BMIs for those configurations, but to my knowledge
is not true in general. For example, if I wanted to enable specific
Microsoft extensions like support for iterator debugging, then I would
have to build my own variants of the std module.
Tom.
>
> -- Gaby
>
> -----Original Message-----
> From: Tom Honermann <tom_at_[hidden]>
> Sent: Monday, April 18, 2022 8:24 PM
> To: Gabriel Dos Reis <gdr_at_[hidden]>; ext_at_[hidden]; Boris Kolpackov <boris_at_[hidden]>
> Cc: Nathan Sidwell <nathan_at_[hidden]>; sg15_at_[hidden]
> Subject: Re: [isocpp-ext] Can we expect that all C++ source files can have the same suffix?
>
> On 4/17/22 10:13 AM, Gabriel Dos Reis wrote:
>> [Tom]
>>> But I don't think that implies that implicit
>>> modules causes problems for build systems and dependency management at
>>> scale; there are other related factors.
>> Cache invalidation.
> If by this you mean that cached module artifacts may become out of date
> with respect to the source files used to construct them, I don't believe
> that to be a fundamental limitation of the implicit module approach.
> Implementations can index artifacts by hashes of their inputs (including
> command line options, file and path names, file timestamps, file
> contents, etc...), observe when a cached artifact is older than one of
> its inputs, and then generate a replacement. Cached artifacts can be
> garbage collected to reduce cache size. In cases where many source files
> contribute to a module, that implies a need to frequently stat() a
> potentially large set of source files, but the costs of doing so can
> likely be mitigated by a service that listens for and caches filesystem
> updates for a select set of files. Avoiding violations of the ODR would
> still require that a build system ensure that source files that
> contribute to module artifacts appear as dependencies for TUs that
> consume those modules (or be assumed to be static as is often the case
> for system headers); there is no avoiding that requirement.
>
> With respect to Nico's concerns, I'm not very concerned about the
> ability to compile a bunch of source files together in a single command
> line invocation. I'm much more concerned about the ability to compile a
> single TU that imports (portions of) the standard library. If, going
> forward, a build system is required to compile the following example
> because it is necessary to prebuild modules for the standard library
> using whatever compiler options are being used to compile a single
> source file into an executable, then I think we'll have done the C++
> community quite a disservice.
>
> import std;
> int main(int argc, char **argv) {
> std::cout << "Hello!\n";
> }
>
> Tom.
>
>> -- Gaby
>>
>>
>> -----Original Message-----
>> From: Tom Honermann <tom_at_[hidden]>
>> Sent: Sunday, April 17, 2022 6:02 AM
>> To: ext_at_[hidden]; Boris Kolpackov <boris_at_[hidden]>
>> Cc: Gabriel Dos Reis <gdr_at_[hidden]>; Nathan Sidwell <nathan_at_[hidden]>; WG21 Tooling Study Group SG15 <tooling_at_[hidden]>
>> Subject: Re: [isocpp-ext] Can we expect that all C++ source files can have the same suffix?
>>
>> On 4/16/22 2:57 PM, Gabriel Dos Reis via Ext wrote:
>>> [Nathan]
>>>> True, but IIUC that is how clang's implicit module machinery already
>>>> works. using the module map, the compiler will fork to compile an
>>>> implicit module (equivalent of a header unit). I don't know how
>>>> concurrency concerns are addressed. (I don't have data on how this
>>>> scheme behaves in practice)
>>> "Implicit build of module" a la Clang (which was originally implemented for Clang Modules) is known to cause problems for build systems and dependency management at scale, which is why they introduced an "explicit build". Last time I heard, organizations using the existing "implicit build" system would love to move away from it, and move towards the "explicit build" approach.
>>>
>>> I fully appreciate that "implicit build" works for nice little example. I am hoping that issue can be solved by other means (either using an IDE or a pre-scanner to construct a sort of module map) than moving build system logics into the compilers.
>> My impression is a little different. When working on Coverity, I saw
>> many examples of implicit modules being successfully used by Coverity
>> customers (primarily with Objective-C, primarily in conjunction with
>> Apple provided frameworks). I think the desire for explicit builds is
>> driven by a desire to improve efficiency over what implicit builds can
>> provide (implicit builds often result in the same interface units being
>> built multiple times subject to implementation ability to recognize when
>> certain command line options are irrelevant to the module being built;
>> an admittedly difficult problem). Thus there is a tradeoff; willingness
>> to accept a less efficient solution vs expending additional effort to
>> modify the build system to support explicit module builds. I acknowledge
>> that providing an explicit build solution that is easy to deploy is a
>> win-win for everyone. But I don't think that implies that implicit
>> modules causes problems for build systems and dependency management at
>> scale; there are other related factors.
>>
>> Tom.
>>
>>> -- Gaby
>>>
>>> -----Original Message-----
>>> From: Ext <ext-bounces_at_[hidden]> On Behalf Of Nathan Sidwell via Ext
>>> Sent: Friday, April 15, 2022 4:51 AM
>>> To: ext_at_[hidden]; Boris Kolpackov <boris_at_[hidden]>
>>> Cc: Nathan Sidwell <nathan_at_[hidden]>; WG21 Tooling Study Group SG15 <tooling_at_[hidden]>
>>> Subject: Re: [isocpp-ext] Can we expect that all C++ source files can have the same suffix?
>>>
>>> On 4/14/22 16:18, Ben Boeckel via Ext wrote:
>>>> On Thu, Apr 14, 2022 at 15:09:35 +0200, Boris Kolpackov wrote:
>>>>> Ben Boeckel via Ext <ext_at_[hidden]> writes:
>>>>>> Need? No. Even with different suffixes, a scanner is still necessary to
>>>>>> know what order files need to be compiled in.
>>>>> A (pre-)scanner is not the only way to do it. Another approach is a
>>>>> module mapper (implemented in GCC, being implemented in Clang, used
>>>>> in build2).
>>>> Well, an active module mapper. CMake just creates the static files which
>>>> handle the GCC's module mapper protocol.
>>>>
>>>>> With the mapper approach you don't need to pre-scan everything but
>>>>> you do need to map module names to module interface files. Having
>>>>> a separate extension for interface files helps narrow down the
>>>>> candidate pool.
>>>> While true, I have doubts about the scalability of holding umpteen
>>>> compiler instances around while executing more to find the bottleneck
>>>> TU. Unsatisifiable requests also end up taking way more resources
>>>> without smarter caching mechanisms to store that information across
>>> True, but IIUC that is how clang's implicit module machinery already
>>> works. using the module map, the compiler will fork to compile an
>>> implicit module (equivalent of a header unit). I don't know how
>>> concurrency concerns are addressed. (I don't have data on how this
>>> scheme behaves in practice)
>>>
>>>> attempts. Which is fine if you're implementing your own build executor I
>>>> suppose. Getting Make or Ninja to understand it without complicated
>>>> stamp juggling doesn't sound fun though.
>>>>
>>>> --Ben
>>>> _______________________________________________
>>>> Ext mailing list
>>>> Ext_at_[hidden]
>>>> Subscription: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.isocpp.org%2Fmailman%2Flistinfo.cgi%2Fext&data=05%7C01%7Cgdr%40microsoft.com%7Cd16b0376753b4c37091e08da21b40b1d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637859354422028725%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3kctZNp7Kc%2FvV54%2Fu8zC1MXuqwK63OmrjNL7GFlrpyw%3D&reserved=0
>>>> Link to this post: https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.isocpp.org%2Fext%2F2022%2F04%2F19012.php&data=05%7C01%7Cgdr%40microsoft.com%7Cd16b0376753b4c37091e08da21b40b1d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637859354422028725%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=GguTcSpRpEE0LlDPPUf9GBxmNh2BfOoO6oYuJYZXgQA%3D&reserved=0
>>>>
> _______________________________________________
> SG15 mailing list
> SG15_at_[hidden]
> https://lists.isocpp.org/mailman/listinfo.cgi/sg15
Received on 2022-04-19 16:42:56