Date: Fri, 18 Jun 2021 16:57:13 -0700
On Fri, Jun 18, 2021 at 4:48 PM Bjarne Stroustrup <bjarne_at_[hidden]>
wrote:
> Let's try not to confuse familiarity with simplicity. Modules are far
> simpler than header files and will make life simpler for all C++ developers
> for decades to come. However, we have lived with the fundamentally messy
> header files for almost 5 decades and have pretty good ideas of how to deal
> with them, so they can seem simple and fit with many tool chains. We should
> not compromise the future by burdening it with backwards-looking standard
> rules that will become pure overhead.
>
I'm not quite connecting this commentary to the rest of the thread - what
sort of things are you suggesting would be/are "backwards-looking standard
rules"?
>
> On 6/18/2021 7:19 PM, David Blaikie via SG15 wrote:
>
> I agree that the N+1 problem is a real concern - hence bringing up whether
> pkg-config might be the wagon we could hitch our cart to in this regard (or
> some other existing solution that could be fleshed out and more strongly
> encouraged (than in the past) as "the" tool to use here, with some
> expectation/desire/buy-in from other tool vendors (like cmake) that they'd
> be on board with that direction).
>
> Be good to get some of the relevant tooling folks (the cmakes, etc, of the
> world) to see what they think would be a good path forward for them.
>
> On Fri, Jun 18, 2021 at 3:41 PM Gabriel Dos Reis via SG15 <
> sg15_at_[hidden]> wrote:
>
>>
>> - Module interfaces on paper fit this model perfectly, except for the
>> part of actually getting builds to work.
>>
>>
>>
>> Module interfaces have not yet had the many decades afforded to header
>> files to work with the many build systems. From experience with trying to
>> stay above the “build systems garden variety” and yet have sufficient
>> hands-on experience/involvement to understand each one’s challenged
>> assumptions, I don’t think we’ve hit a point where they (modules) can’t
>> work with existing build systems. The real thorny issue is whether we
>> continue with ad-hoc solutions as we’ve always had in the last 50 years, or
>> if we shoot for a common solution which hits the “N+1 standards problem”.
>>
>> For example, we know from empirical data and experience that C++ Modules
>> require less of build systems than Fortran does – this is not a contest of
>> who can require more.
>>
>>
>>
>> -- Gaby
>>
>>
>>
>> *From:* SG15 <sg15-bounces_at_[hidden]> *On Behalf Of *Steve Downey
>> via SG15
>> *Sent:* Friday, June 18, 2021 3:31 PM
>> *To:* ISO C++ Tooling Study Group <sg15_at_[hidden]>
>> *Cc:* Steve Downey <sdowney_at_[hidden]>
>> *Subject:* Re: [SG15] Draft: Requirements for Usage of C++ Modules at
>> Bloomberg
>>
>>
>>
>> Also, although Daniel's written this from the standpoint of Bloomberg,
>> that's because we know for certain we can speak for our requirements. We
>> really do think we share requirements with many other development
>> organizations, though. We ship packages to each other, literally in dpkg,
>> on all our posixy platforms. We can see each other's source code, but don't
>> usually build it. Our shared model is basically a debian unstable, `sid`,
>> shipped as fast as we can turn the crank. We're managing large scale reuse,
>> with dependency graph depths of 40 or more, involving hundreds of packages,
>> in thousands of applications. Module interfaces on paper fit this model
>> perfectly, except for the part of actually getting builds to work.
>>
>>
>>
>> We really want modules to work for us, and we think if they work for us,
>> they work for a large class of developer orgs.
>>
>>
>>
>> On Fri, Jun 18, 2021 at 5:25 PM Steve Downey <sdowney_at_[hidden]> wrote:
>>
>> With header files (except for generated ones) there's not the requirement
>> of a build DAG, just out-of-date information. That's why we can generate
>> the dependencies as part of the compilation step. With modules, at least
>> with the current physical model, we have to make sure the interface has
>> been translated before it can be consumed. Incremental builds are going to
>> break for a lot of people in subtle ways. And yes, this is very much like
>> the past and using tools like `makedeps`.
>>
>> What we learned then is that it's very very difficult to precisely
>> reproduce the preprocessor and include searching. There's a lot of poorly
>> specified and undefined behavior in the early phases of translation. If the
>> tools disagree, build problems ensue.
>>
>> I don't believe we have made the situation impossible. A tool should be
>> able to proceed sufficiently with preprocessing in the face of missing
>> named modules and be able to report all of the requirements of a TU. It
>> wouldn't for header imports, but that's a more well understood problem, as
>> those map to filenames in a well agreed on way.
>>
>> What module an interface provides is also discoverable, but that may not
>> be as useful in an open universe situation. The question of which instance
>> of the primary module interface out of many is hard. We have that today
>> with headers, of course, and we try to deal with it via ordering rules for
>> search paths, but it's still a source of error. Duplicates and collisions
>> are harder for named modules, even if figuring out what an interface
>> provides is straight forward.
>>
>>
>>
>> On Fri, Jun 18, 2021 at 2:12 PM David Blaikie via SG15 <
>> sg15_at_[hidden]> wrote:
>>
>>
>>
>>
>>
>> On Fri, Jun 18, 2021 at 11:01 AM Gabriel Dos Reis <gdr_at_[hidden]>
>> wrote:
>>
>>
>> - but you cannot discover the graph with a simple lexer
>>
>>
>>
>> Yes, and I have not found that to be an actual problem in practice. In
>> fact there is really nothing new here (other than recognizing the ‘import’
>> directives and friends) in the non-module world: the ol’ C and C++
>> standards allow
>>
>>
>>
>> #include HEADER
>>
>>
>>
>> where HEADER comes from macro expansions and all the other stuff that you
>> mention about inability to use “simple lexer”, conditional inclusion, etc.
>> As a PSA, I would like to disabuse anyone who ever thought that grep was
>> sufficient to compute dependency of C or C++ source files pre-C++20.
>>
>>
>>
>> Maybe it is my dinosaurus status showing bleeding out here but are we
>> rehashing the same arguments from the past?
>>
>>
>>
>> The one thing we can do for the community is to have a library readily
>> available to anyone who ever thought that a simple lexer is sufficient to
>> compute C++ source file dependency. That would be a major step forward.
>>
>>
>> I don't think Daniel is suggesting/has mentioned the idea that a simple
>> lexer is sufficient for either headers or modules. But expected/would like
>> it if we could simplify the situation further than module files alone
>> currently allow in this regard.
>>
>> I do lean towards "maybe it's simple enough, even if it doesn't allow a
>> simple lexer to do this operation" - at least I don't think that's an
>> ironclad proof that we need the dependency information in a more
>> immediately legible format such as in the pkg-config files or equivalent. I
>> wouldn't rule it out either, though.
>>
>> Like, I wouldn't be averse to pkg-config or equivalent files supporting
>> this as an optional feature - so that folks could explore whether it
>> significantly improves performance. If it's more a matter of software
>> complexity (since build tools that don't know anything about parsing C++
>> are going to want to discover this dependency graph) - yeah, maybe
>> compilers could vend a command line mode that reports the dependencies as
>> quickly as possible - ala clang-scan-deps & for library level uses
>> (llvm-buildozer, background running build systems/IDE type situations)
>> there could be a library for that.
>>
>> But, going on past experience, those kinds of utilities/dependencies
>> don't necessarily eventuate and I can understand the concern about
>> proliferation/variation/lack of ubiquity in such things & so a preference
>> to bake it into a common config file. I guess all I have on that is that I
>> think it'd be unfortunate for developers to have to repeat this information
>> in multiple places like that.
>>
>> - Dave
>>
>>
>>
>>
>> -- Gaby
>>
>>
>>
>> *From:* Daniel Ruoso <daniel_at_[hidden]>
>> *Sent:* Friday, June 18, 2021 10:30 AM
>> *To:* Gabriel Dos Reis <gdr_at_[hidden]>
>> *Cc:* sg15_at_[hidden]; David Blaikie <dblaikie_at_[hidden]>
>> *Subject:* Re: [SG15] Draft: Requirements for Usage of C++ Modules at
>> Bloomberg
>>
>>
>>
>> On Fri, Jun 18, 2021 at 1:20 PM Gabriel Dos Reis <gdr_at_[hidden]>
>> wrote:
>>
>> Also notice that while a dependency scanner needs to chase #included or
>> imported files (via the appropriate implementation-defined mapping), it
>> actually does not need to fully preprocess the source code found therein;
>>
>> the spec has been carefully crafted to that effect.
>>
>>
>>
>> It doesn't need to preprocess the complete file, but it needs to
>> significantly preprocess the early sections of the file. While the spec has
>> carefully prevented conditional specification of the keywords, it still
>> allows macros to be used, which requires support for conditional
>> evaluation, variable expansion, and string concatenation.
>>
>> In other words, the only optimization granted by the standard today is on
>> the discovery of whether or not a source file needs to participate on the
>> construction of the DAG for processing modules, but you cannot discover the
>> graph with a simple lexer, you need the full expression evaluation logic of
>> the preprocessor.
>>
>>
>>
>> daniel
>>
>> _______________________________________________
>> SG15 mailing list
>> SG15_at_[hidden]
>> https://lists.isocpp.org/mailman/listinfo.cgi/sg15
>> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.isocpp.org%2Fmailman%2Flistinfo.cgi%2Fsg15&data=04%7C01%7Cgdr%40microsoft.com%7Ca844c0bda0b14e2bb92108d932a8e322%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637596523230847476%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=GjMkkE2j7fhda%2FCjyc1G2H%2Bhx5Mcnqzxvs6bcJMm%2BDk%3D&reserved=0>
>> _______________________________________________
>> SG15 mailing list
>> SG15_at_[hidden]
>> https://lists.isocpp.org/mailman/listinfo.cgi/sg15
>>
>
> _______________________________________________
> SG15 mailing listSG15_at_[hidden]://lists.isocpp.org/mailman/listinfo.cgi/sg15
>
>
wrote:
> Let's try not to confuse familiarity with simplicity. Modules are far
> simpler than header files and will make life simpler for all C++ developers
> for decades to come. However, we have lived with the fundamentally messy
> header files for almost 5 decades and have pretty good ideas of how to deal
> with them, so they can seem simple and fit with many tool chains. We should
> not compromise the future by burdening it with backwards-looking standard
> rules that will become pure overhead.
>
I'm not quite connecting this commentary to the rest of the thread - what
sort of things are you suggesting would be/are "backwards-looking standard
rules"?
>
> On 6/18/2021 7:19 PM, David Blaikie via SG15 wrote:
>
> I agree that the N+1 problem is a real concern - hence bringing up whether
> pkg-config might be the wagon we could hitch our cart to in this regard (or
> some other existing solution that could be fleshed out and more strongly
> encouraged (than in the past) as "the" tool to use here, with some
> expectation/desire/buy-in from other tool vendors (like cmake) that they'd
> be on board with that direction).
>
> Be good to get some of the relevant tooling folks (the cmakes, etc, of the
> world) to see what they think would be a good path forward for them.
>
> On Fri, Jun 18, 2021 at 3:41 PM Gabriel Dos Reis via SG15 <
> sg15_at_[hidden]> wrote:
>
>>
>> - Module interfaces on paper fit this model perfectly, except for the
>> part of actually getting builds to work.
>>
>>
>>
>> Module interfaces have not yet had the many decades afforded to header
>> files to work with the many build systems. From experience with trying to
>> stay above the “build systems garden variety” and yet have sufficient
>> hands-on experience/involvement to understand each one’s challenged
>> assumptions, I don’t think we’ve hit a point where they (modules) can’t
>> work with existing build systems. The real thorny issue is whether we
>> continue with ad-hoc solutions as we’ve always had in the last 50 years, or
>> if we shoot for a common solution which hits the “N+1 standards problem”.
>>
>> For example, we know from empirical data and experience that C++ Modules
>> require less of build systems than Fortran does – this is not a contest of
>> who can require more.
>>
>>
>>
>> -- Gaby
>>
>>
>>
>> *From:* SG15 <sg15-bounces_at_[hidden]> *On Behalf Of *Steve Downey
>> via SG15
>> *Sent:* Friday, June 18, 2021 3:31 PM
>> *To:* ISO C++ Tooling Study Group <sg15_at_[hidden]>
>> *Cc:* Steve Downey <sdowney_at_[hidden]>
>> *Subject:* Re: [SG15] Draft: Requirements for Usage of C++ Modules at
>> Bloomberg
>>
>>
>>
>> Also, although Daniel's written this from the standpoint of Bloomberg,
>> that's because we know for certain we can speak for our requirements. We
>> really do think we share requirements with many other development
>> organizations, though. We ship packages to each other, literally in dpkg,
>> on all our posixy platforms. We can see each other's source code, but don't
>> usually build it. Our shared model is basically a debian unstable, `sid`,
>> shipped as fast as we can turn the crank. We're managing large scale reuse,
>> with dependency graph depths of 40 or more, involving hundreds of packages,
>> in thousands of applications. Module interfaces on paper fit this model
>> perfectly, except for the part of actually getting builds to work.
>>
>>
>>
>> We really want modules to work for us, and we think if they work for us,
>> they work for a large class of developer orgs.
>>
>>
>>
>> On Fri, Jun 18, 2021 at 5:25 PM Steve Downey <sdowney_at_[hidden]> wrote:
>>
>> With header files (except for generated ones) there's not the requirement
>> of a build DAG, just out-of-date information. That's why we can generate
>> the dependencies as part of the compilation step. With modules, at least
>> with the current physical model, we have to make sure the interface has
>> been translated before it can be consumed. Incremental builds are going to
>> break for a lot of people in subtle ways. And yes, this is very much like
>> the past and using tools like `makedeps`.
>>
>> What we learned then is that it's very very difficult to precisely
>> reproduce the preprocessor and include searching. There's a lot of poorly
>> specified and undefined behavior in the early phases of translation. If the
>> tools disagree, build problems ensue.
>>
>> I don't believe we have made the situation impossible. A tool should be
>> able to proceed sufficiently with preprocessing in the face of missing
>> named modules and be able to report all of the requirements of a TU. It
>> wouldn't for header imports, but that's a more well understood problem, as
>> those map to filenames in a well agreed on way.
>>
>> What module an interface provides is also discoverable, but that may not
>> be as useful in an open universe situation. The question of which instance
>> of the primary module interface out of many is hard. We have that today
>> with headers, of course, and we try to deal with it via ordering rules for
>> search paths, but it's still a source of error. Duplicates and collisions
>> are harder for named modules, even if figuring out what an interface
>> provides is straight forward.
>>
>>
>>
>> On Fri, Jun 18, 2021 at 2:12 PM David Blaikie via SG15 <
>> sg15_at_[hidden]> wrote:
>>
>>
>>
>>
>>
>> On Fri, Jun 18, 2021 at 11:01 AM Gabriel Dos Reis <gdr_at_[hidden]>
>> wrote:
>>
>>
>> - but you cannot discover the graph with a simple lexer
>>
>>
>>
>> Yes, and I have not found that to be an actual problem in practice. In
>> fact there is really nothing new here (other than recognizing the ‘import’
>> directives and friends) in the non-module world: the ol’ C and C++
>> standards allow
>>
>>
>>
>> #include HEADER
>>
>>
>>
>> where HEADER comes from macro expansions and all the other stuff that you
>> mention about inability to use “simple lexer”, conditional inclusion, etc.
>> As a PSA, I would like to disabuse anyone who ever thought that grep was
>> sufficient to compute dependency of C or C++ source files pre-C++20.
>>
>>
>>
>> Maybe it is my dinosaurus status showing bleeding out here but are we
>> rehashing the same arguments from the past?
>>
>>
>>
>> The one thing we can do for the community is to have a library readily
>> available to anyone who ever thought that a simple lexer is sufficient to
>> compute C++ source file dependency. That would be a major step forward.
>>
>>
>> I don't think Daniel is suggesting/has mentioned the idea that a simple
>> lexer is sufficient for either headers or modules. But expected/would like
>> it if we could simplify the situation further than module files alone
>> currently allow in this regard.
>>
>> I do lean towards "maybe it's simple enough, even if it doesn't allow a
>> simple lexer to do this operation" - at least I don't think that's an
>> ironclad proof that we need the dependency information in a more
>> immediately legible format such as in the pkg-config files or equivalent. I
>> wouldn't rule it out either, though.
>>
>> Like, I wouldn't be averse to pkg-config or equivalent files supporting
>> this as an optional feature - so that folks could explore whether it
>> significantly improves performance. If it's more a matter of software
>> complexity (since build tools that don't know anything about parsing C++
>> are going to want to discover this dependency graph) - yeah, maybe
>> compilers could vend a command line mode that reports the dependencies as
>> quickly as possible - ala clang-scan-deps & for library level uses
>> (llvm-buildozer, background running build systems/IDE type situations)
>> there could be a library for that.
>>
>> But, going on past experience, those kinds of utilities/dependencies
>> don't necessarily eventuate and I can understand the concern about
>> proliferation/variation/lack of ubiquity in such things & so a preference
>> to bake it into a common config file. I guess all I have on that is that I
>> think it'd be unfortunate for developers to have to repeat this information
>> in multiple places like that.
>>
>> - Dave
>>
>>
>>
>>
>> -- Gaby
>>
>>
>>
>> *From:* Daniel Ruoso <daniel_at_[hidden]>
>> *Sent:* Friday, June 18, 2021 10:30 AM
>> *To:* Gabriel Dos Reis <gdr_at_[hidden]>
>> *Cc:* sg15_at_[hidden]; David Blaikie <dblaikie_at_[hidden]>
>> *Subject:* Re: [SG15] Draft: Requirements for Usage of C++ Modules at
>> Bloomberg
>>
>>
>>
>> On Fri, Jun 18, 2021 at 1:20 PM Gabriel Dos Reis <gdr_at_[hidden]>
>> wrote:
>>
>> Also notice that while a dependency scanner needs to chase #included or
>> imported files (via the appropriate implementation-defined mapping), it
>> actually does not need to fully preprocess the source code found therein;
>>
>> the spec has been carefully crafted to that effect.
>>
>>
>>
>> It doesn't need to preprocess the complete file, but it needs to
>> significantly preprocess the early sections of the file. While the spec has
>> carefully prevented conditional specification of the keywords, it still
>> allows macros to be used, which requires support for conditional
>> evaluation, variable expansion, and string concatenation.
>>
>> In other words, the only optimization granted by the standard today is on
>> the discovery of whether or not a source file needs to participate on the
>> construction of the DAG for processing modules, but you cannot discover the
>> graph with a simple lexer, you need the full expression evaluation logic of
>> the preprocessor.
>>
>>
>>
>> daniel
>>
>> _______________________________________________
>> SG15 mailing list
>> SG15_at_[hidden]
>> https://lists.isocpp.org/mailman/listinfo.cgi/sg15
>> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.isocpp.org%2Fmailman%2Flistinfo.cgi%2Fsg15&data=04%7C01%7Cgdr%40microsoft.com%7Ca844c0bda0b14e2bb92108d932a8e322%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637596523230847476%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=GjMkkE2j7fhda%2FCjyc1G2H%2Bhx5Mcnqzxvs6bcJMm%2BDk%3D&reserved=0>
>> _______________________________________________
>> SG15 mailing list
>> SG15_at_[hidden]
>> https://lists.isocpp.org/mailman/listinfo.cgi/sg15
>>
>
> _______________________________________________
> SG15 mailing listSG15_at_[hidden]://lists.isocpp.org/mailman/listinfo.cgi/sg15
>
>
Received on 2021-06-18 18:57:28