Date: Mon, 21 Jun 2021 13:00:26 +0000
Hey all,
I’ll preface this post with the fact that I’m coming from a place of ignorance, I’m very new to the SG, huge grain of salt on all of this.
But I’ve rarely seen pkg-config solutions work with a lot of software I work on, we tend to use CMake stuff. For a novice user, pkg-config is often pretty good for “say where the headers are” but less effective for “I have a build using multiple languages, of a software package with a bunch of configuration options and targets.” To a novice user, pkg-config mostly supports “here are the cppflags and ldflags.” With that in mind, I wanted to talk about some requirements we’d have
To my mind, there are a few things we need to make sure still work with modules. I have no knowledge of whether the things being proposed support these. My rough guess is that they are orthogonal to the proposal, and won’t be specifically supported or hampered by it.
1. Encoding how the build was configured (did I enable certain configure-time options). My money here is on that working exactly as it always has, with CMake or pkg-config artifacts
2. Dependency info, how do I say I rely on a given library or set of source artifacts (previously I would have said “header files”). This is where I think a lot of the work comes in. I think the question here is whether a) targets (in a CMake sense) can be specified in terms of modules (“I depend on the Kokkos Core module and the Kokkos containers module”) and b) whether CMake can parse whatever module files we output in order to support this. Do we have a CMake guru in the group who can tell me “David, don’t worry, we’ve been planning for this?” My guess is that all of this is fine, but my guess is worth a lot less than somebody at Kitware saying “we’re not freaking out about this.”
3. Can we interoperate between header dependencies and module dependencies? If APP depends on Trilinos depends on Kokkos, and Kokkos does modules, but Trilinos does headers, do we have support for this? If APP depends on Kokkos and Caliper, and Caliper supports modules but Kokkos doesn’t, do we support this?
Those are the questions I’ll have in mind, more than “pkg-config” or “CMake,” is just whether we provide the functionality for the very smart people who do build systems to do at least what they’ve always done. I’d bet that it turns out to be “hey, where before you talked about software artifacts as sets of headers, now you talk about sets of modules.”
I don’t know, am I asking people in the group things they’ve thought about months/years ago and realized aren’t a problem, or is any of this worth discussing further?
Best,
David P
From: SG15 <sg15-bounces_at_[hidden]> on behalf of David Blaikie via SG15 <sg15_at_[hidden]>
Reply-To: "sg15_at_[hidden]" <sg15_at_[hidden]>
Date: Friday, June 18, 2021 at 5:19 PM
To: "sg15_at_[hidden]" <sg15_at_[hidden]>
Cc: David Blaikie <dblaikie_at_gmail.com>
Subject: [EXTERNAL] Re: [SG15] Draft: Requirements for Usage of C++ Modules at Bloomberg
I agree that the N+1 problem is a real concern - hence bringing up whether pkg-config might be the wagon we could hitch our cart to in this regard (or some other existing solution that could be fleshed out and more strongly encouraged (than in the past) as "the" tool to use here, with some expectation/desire/buy-in from other tool vendors (like cmake) that they'd be on board with that direction).
Be good to get some of the relevant tooling folks (the cmakes, etc, of the world) to see what they think would be a good path forward for them.
On Fri, Jun 18, 2021 at 3:41 PM Gabriel Dos Reis via SG15 <sg15_at_[hidden]<mailto:sg15_at_[hidden]>> wrote:
* Module interfaces on paper fit this model perfectly, except for the part of actually getting builds to work.
Module interfaces have not yet had the many decades afforded to header files to work with the many build systems. From experience with trying to stay above the “build systems garden variety” and yet have sufficient hands-on experience/involvement to understand each one’s challenged assumptions, I don’t think we’ve hit a point where they (modules) can’t work with existing build systems. The real thorny issue is whether we continue with ad-hoc solutions as we’ve always had in the last 50 years, or if we shoot for a common solution which hits the “N+1 standards problem”.
For example, we know from empirical data and experience that C++ Modules require less of build systems than Fortran does – this is not a contest of who can require more.
-- Gaby
From: SG15 <sg15-bounces_at_lists.isocpp.org<mailto:sg15-bounces_at_[hidden]>> On Behalf Of Steve Downey via SG15
Sent: Friday, June 18, 2021 3:31 PM
To: ISO C++ Tooling Study Group <sg15_at_[hidden]<mailto:sg15_at_[hidden]>>
Cc: Steve Downey <sdowney_at_[hidden]<mailto:sdowney_at_[hidden]>>
Subject: Re: [SG15] Draft: Requirements for Usage of C++ Modules at Bloomberg
Also, although Daniel's written this from the standpoint of Bloomberg, that's because we know for certain we can speak for our requirements. We really do think we share requirements with many other development organizations, though. We ship packages to each other, literally in dpkg, on all our posixy platforms. We can see each other's source code, but don't usually build it. Our shared model is basically a debian unstable, `sid`, shipped as fast as we can turn the crank. We're managing large scale reuse, with dependency graph depths of 40 or more, involving hundreds of packages, in thousands of applications. Module interfaces on paper fit this model perfectly, except for the part of actually getting builds to work.
We really want modules to work for us, and we think if they work for us, they work for a large class of developer orgs.
On Fri, Jun 18, 2021 at 5:25 PM Steve Downey <sdowney_at_gmail.com<mailto:sdowney_at_[hidden]>> wrote:
With header files (except for generated ones) there's not the requirement of a build DAG, just out-of-date information. That's why we can generate the dependencies as part of the compilation step. With modules, at least with the current physical model, we have to make sure the interface has been translated before it can be consumed. Incremental builds are going to break for a lot of people in subtle ways. And yes, this is very much like the past and using tools like `makedeps`.
What we learned then is that it's very very difficult to precisely reproduce the preprocessor and include searching. There's a lot of poorly specified and undefined behavior in the early phases of translation. If the tools disagree, build problems ensue.
I don't believe we have made the situation impossible. A tool should be able to proceed sufficiently with preprocessing in the face of missing named modules and be able to report all of the requirements of a TU. It wouldn't for header imports, but that's a more well understood problem, as those map to filenames in a well agreed on way.
What module an interface provides is also discoverable, but that may not be as useful in an open universe situation. The question of which instance of the primary module interface out of many is hard. We have that today with headers, of course, and we try to deal with it via ordering rules for search paths, but it's still a source of error. Duplicates and collisions are harder for named modules, even if figuring out what an interface provides is straight forward.
On Fri, Jun 18, 2021 at 2:12 PM David Blaikie via SG15 <sg15_at_[hidden]<mailto:sg15_at_[hidden]>> wrote:
On Fri, Jun 18, 2021 at 11:01 AM Gabriel Dos Reis <gdr_at_[hidden]<mailto:gdr_at_[hidden]>> wrote:
* but you cannot discover the graph with a simple lexer
Yes, and I have not found that to be an actual problem in practice. In fact there is really nothing new here (other than recognizing the ‘import’ directives and friends) in the non-module world: the ol’ C and C++ standards allow
#include HEADER
where HEADER comes from macro expansions and all the other stuff that you mention about inability to use “simple lexer”, conditional inclusion, etc. As a PSA, I would like to disabuse anyone who ever thought that grep was sufficient to compute dependency of C or C++ source files pre-C++20.
Maybe it is my dinosaurus status showing bleeding out here but are we rehashing the same arguments from the past?
The one thing we can do for the community is to have a library readily available to anyone who ever thought that a simple lexer is sufficient to compute C++ source file dependency. That would be a major step forward.
I don't think Daniel is suggesting/has mentioned the idea that a simple lexer is sufficient for either headers or modules. But expected/would like it if we could simplify the situation further than module files alone currently allow in this regard.
I do lean towards "maybe it's simple enough, even if it doesn't allow a simple lexer to do this operation" - at least I don't think that's an ironclad proof that we need the dependency information in a more immediately legible format such as in the pkg-config files or equivalent. I wouldn't rule it out either, though.
Like, I wouldn't be averse to pkg-config or equivalent files supporting this as an optional feature - so that folks could explore whether it significantly improves performance. If it's more a matter of software complexity (since build tools that don't know anything about parsing C++ are going to want to discover this dependency graph) - yeah, maybe compilers could vend a command line mode that reports the dependencies as quickly as possible - ala clang-scan-deps & for library level uses (llvm-buildozer, background running build systems/IDE type situations) there could be a library for that.
But, going on past experience, those kinds of utilities/dependencies don't necessarily eventuate and I can understand the concern about proliferation/variation/lack of ubiquity in such things & so a preference to bake it into a common config file. I guess all I have on that is that I think it'd be unfortunate for developers to have to repeat this information in multiple places like that.
- Dave
-- Gaby
From: Daniel Ruoso <daniel_at_[hidden]<mailto:daniel_at_[hidden]>>
Sent: Friday, June 18, 2021 10:30 AM
To: Gabriel Dos Reis <gdr_at_[hidden]<mailto:gdr_at_[hidden]>>
Cc: sg15_at_[hidden]<mailto:sg15_at_[hidden]>; David Blaikie <dblaikie_at_[hidden]<mailto:dblaikie_at_[hidden]>>
Subject: Re: [SG15] Draft: Requirements for Usage of C++ Modules at Bloomberg
On Fri, Jun 18, 2021 at 1:20 PM Gabriel Dos Reis <gdr_at_[hidden]<mailto:gdr_at_[hidden]>> wrote:
Also notice that while a dependency scanner needs to chase #included or imported files (via the appropriate implementation-defined mapping), it actually does not need to fully preprocess the source code found therein;
the spec has been carefully crafted to that effect.
It doesn't need to preprocess the complete file, but it needs to significantly preprocess the early sections of the file. While the spec has carefully prevented conditional specification of the keywords, it still allows macros to be used, which requires support for conditional evaluation, variable expansion, and string concatenation.
In other words, the only optimization granted by the standard today is on the discovery of whether or not a source file needs to participate on the construction of the DAG for processing modules, but you cannot discover the graph with a simple lexer, you need the full expression evaluation logic of the preprocessor.
daniel
_______________________________________________
SG15 mailing list
SG15_at_[hidden]<mailto:SG15_at_[hidden]>
https://lists.isocpp.org/mailman/listinfo.cgi/sg15<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.isocpp.org%2Fmailman%2Flistinfo.cgi%2Fsg15&data=04%7C01%7Cgdr%40microsoft.com%7Ca844c0bda0b14e2bb92108d932a8e322%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637596523230847476%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=GjMkkE2j7fhda%2FCjyc1G2H%2Bhx5Mcnqzxvs6bcJMm%2BDk%3D&reserved=0>
_______________________________________________
SG15 mailing list
SG15_at_[hidden]<mailto:SG15_at_lists.isocpp.org>
https://lists.isocpp.org/mailman/listinfo.cgi/sg15
I’ll preface this post with the fact that I’m coming from a place of ignorance, I’m very new to the SG, huge grain of salt on all of this.
But I’ve rarely seen pkg-config solutions work with a lot of software I work on, we tend to use CMake stuff. For a novice user, pkg-config is often pretty good for “say where the headers are” but less effective for “I have a build using multiple languages, of a software package with a bunch of configuration options and targets.” To a novice user, pkg-config mostly supports “here are the cppflags and ldflags.” With that in mind, I wanted to talk about some requirements we’d have
To my mind, there are a few things we need to make sure still work with modules. I have no knowledge of whether the things being proposed support these. My rough guess is that they are orthogonal to the proposal, and won’t be specifically supported or hampered by it.
1. Encoding how the build was configured (did I enable certain configure-time options). My money here is on that working exactly as it always has, with CMake or pkg-config artifacts
2. Dependency info, how do I say I rely on a given library or set of source artifacts (previously I would have said “header files”). This is where I think a lot of the work comes in. I think the question here is whether a) targets (in a CMake sense) can be specified in terms of modules (“I depend on the Kokkos Core module and the Kokkos containers module”) and b) whether CMake can parse whatever module files we output in order to support this. Do we have a CMake guru in the group who can tell me “David, don’t worry, we’ve been planning for this?” My guess is that all of this is fine, but my guess is worth a lot less than somebody at Kitware saying “we’re not freaking out about this.”
3. Can we interoperate between header dependencies and module dependencies? If APP depends on Trilinos depends on Kokkos, and Kokkos does modules, but Trilinos does headers, do we have support for this? If APP depends on Kokkos and Caliper, and Caliper supports modules but Kokkos doesn’t, do we support this?
Those are the questions I’ll have in mind, more than “pkg-config” or “CMake,” is just whether we provide the functionality for the very smart people who do build systems to do at least what they’ve always done. I’d bet that it turns out to be “hey, where before you talked about software artifacts as sets of headers, now you talk about sets of modules.”
I don’t know, am I asking people in the group things they’ve thought about months/years ago and realized aren’t a problem, or is any of this worth discussing further?
Best,
David P
From: SG15 <sg15-bounces_at_[hidden]> on behalf of David Blaikie via SG15 <sg15_at_[hidden]>
Reply-To: "sg15_at_[hidden]" <sg15_at_[hidden]>
Date: Friday, June 18, 2021 at 5:19 PM
To: "sg15_at_[hidden]" <sg15_at_[hidden]>
Cc: David Blaikie <dblaikie_at_gmail.com>
Subject: [EXTERNAL] Re: [SG15] Draft: Requirements for Usage of C++ Modules at Bloomberg
I agree that the N+1 problem is a real concern - hence bringing up whether pkg-config might be the wagon we could hitch our cart to in this regard (or some other existing solution that could be fleshed out and more strongly encouraged (than in the past) as "the" tool to use here, with some expectation/desire/buy-in from other tool vendors (like cmake) that they'd be on board with that direction).
Be good to get some of the relevant tooling folks (the cmakes, etc, of the world) to see what they think would be a good path forward for them.
On Fri, Jun 18, 2021 at 3:41 PM Gabriel Dos Reis via SG15 <sg15_at_[hidden]<mailto:sg15_at_[hidden]>> wrote:
* Module interfaces on paper fit this model perfectly, except for the part of actually getting builds to work.
Module interfaces have not yet had the many decades afforded to header files to work with the many build systems. From experience with trying to stay above the “build systems garden variety” and yet have sufficient hands-on experience/involvement to understand each one’s challenged assumptions, I don’t think we’ve hit a point where they (modules) can’t work with existing build systems. The real thorny issue is whether we continue with ad-hoc solutions as we’ve always had in the last 50 years, or if we shoot for a common solution which hits the “N+1 standards problem”.
For example, we know from empirical data and experience that C++ Modules require less of build systems than Fortran does – this is not a contest of who can require more.
-- Gaby
From: SG15 <sg15-bounces_at_lists.isocpp.org<mailto:sg15-bounces_at_[hidden]>> On Behalf Of Steve Downey via SG15
Sent: Friday, June 18, 2021 3:31 PM
To: ISO C++ Tooling Study Group <sg15_at_[hidden]<mailto:sg15_at_[hidden]>>
Cc: Steve Downey <sdowney_at_[hidden]<mailto:sdowney_at_[hidden]>>
Subject: Re: [SG15] Draft: Requirements for Usage of C++ Modules at Bloomberg
Also, although Daniel's written this from the standpoint of Bloomberg, that's because we know for certain we can speak for our requirements. We really do think we share requirements with many other development organizations, though. We ship packages to each other, literally in dpkg, on all our posixy platforms. We can see each other's source code, but don't usually build it. Our shared model is basically a debian unstable, `sid`, shipped as fast as we can turn the crank. We're managing large scale reuse, with dependency graph depths of 40 or more, involving hundreds of packages, in thousands of applications. Module interfaces on paper fit this model perfectly, except for the part of actually getting builds to work.
We really want modules to work for us, and we think if they work for us, they work for a large class of developer orgs.
On Fri, Jun 18, 2021 at 5:25 PM Steve Downey <sdowney_at_gmail.com<mailto:sdowney_at_[hidden]>> wrote:
With header files (except for generated ones) there's not the requirement of a build DAG, just out-of-date information. That's why we can generate the dependencies as part of the compilation step. With modules, at least with the current physical model, we have to make sure the interface has been translated before it can be consumed. Incremental builds are going to break for a lot of people in subtle ways. And yes, this is very much like the past and using tools like `makedeps`.
What we learned then is that it's very very difficult to precisely reproduce the preprocessor and include searching. There's a lot of poorly specified and undefined behavior in the early phases of translation. If the tools disagree, build problems ensue.
I don't believe we have made the situation impossible. A tool should be able to proceed sufficiently with preprocessing in the face of missing named modules and be able to report all of the requirements of a TU. It wouldn't for header imports, but that's a more well understood problem, as those map to filenames in a well agreed on way.
What module an interface provides is also discoverable, but that may not be as useful in an open universe situation. The question of which instance of the primary module interface out of many is hard. We have that today with headers, of course, and we try to deal with it via ordering rules for search paths, but it's still a source of error. Duplicates and collisions are harder for named modules, even if figuring out what an interface provides is straight forward.
On Fri, Jun 18, 2021 at 2:12 PM David Blaikie via SG15 <sg15_at_[hidden]<mailto:sg15_at_[hidden]>> wrote:
On Fri, Jun 18, 2021 at 11:01 AM Gabriel Dos Reis <gdr_at_[hidden]<mailto:gdr_at_[hidden]>> wrote:
* but you cannot discover the graph with a simple lexer
Yes, and I have not found that to be an actual problem in practice. In fact there is really nothing new here (other than recognizing the ‘import’ directives and friends) in the non-module world: the ol’ C and C++ standards allow
#include HEADER
where HEADER comes from macro expansions and all the other stuff that you mention about inability to use “simple lexer”, conditional inclusion, etc. As a PSA, I would like to disabuse anyone who ever thought that grep was sufficient to compute dependency of C or C++ source files pre-C++20.
Maybe it is my dinosaurus status showing bleeding out here but are we rehashing the same arguments from the past?
The one thing we can do for the community is to have a library readily available to anyone who ever thought that a simple lexer is sufficient to compute C++ source file dependency. That would be a major step forward.
I don't think Daniel is suggesting/has mentioned the idea that a simple lexer is sufficient for either headers or modules. But expected/would like it if we could simplify the situation further than module files alone currently allow in this regard.
I do lean towards "maybe it's simple enough, even if it doesn't allow a simple lexer to do this operation" - at least I don't think that's an ironclad proof that we need the dependency information in a more immediately legible format such as in the pkg-config files or equivalent. I wouldn't rule it out either, though.
Like, I wouldn't be averse to pkg-config or equivalent files supporting this as an optional feature - so that folks could explore whether it significantly improves performance. If it's more a matter of software complexity (since build tools that don't know anything about parsing C++ are going to want to discover this dependency graph) - yeah, maybe compilers could vend a command line mode that reports the dependencies as quickly as possible - ala clang-scan-deps & for library level uses (llvm-buildozer, background running build systems/IDE type situations) there could be a library for that.
But, going on past experience, those kinds of utilities/dependencies don't necessarily eventuate and I can understand the concern about proliferation/variation/lack of ubiquity in such things & so a preference to bake it into a common config file. I guess all I have on that is that I think it'd be unfortunate for developers to have to repeat this information in multiple places like that.
- Dave
-- Gaby
From: Daniel Ruoso <daniel_at_[hidden]<mailto:daniel_at_[hidden]>>
Sent: Friday, June 18, 2021 10:30 AM
To: Gabriel Dos Reis <gdr_at_[hidden]<mailto:gdr_at_[hidden]>>
Cc: sg15_at_[hidden]<mailto:sg15_at_[hidden]>; David Blaikie <dblaikie_at_[hidden]<mailto:dblaikie_at_[hidden]>>
Subject: Re: [SG15] Draft: Requirements for Usage of C++ Modules at Bloomberg
On Fri, Jun 18, 2021 at 1:20 PM Gabriel Dos Reis <gdr_at_[hidden]<mailto:gdr_at_[hidden]>> wrote:
Also notice that while a dependency scanner needs to chase #included or imported files (via the appropriate implementation-defined mapping), it actually does not need to fully preprocess the source code found therein;
the spec has been carefully crafted to that effect.
It doesn't need to preprocess the complete file, but it needs to significantly preprocess the early sections of the file. While the spec has carefully prevented conditional specification of the keywords, it still allows macros to be used, which requires support for conditional evaluation, variable expansion, and string concatenation.
In other words, the only optimization granted by the standard today is on the discovery of whether or not a source file needs to participate on the construction of the DAG for processing modules, but you cannot discover the graph with a simple lexer, you need the full expression evaluation logic of the preprocessor.
daniel
_______________________________________________
SG15 mailing list
SG15_at_[hidden]<mailto:SG15_at_[hidden]>
https://lists.isocpp.org/mailman/listinfo.cgi/sg15<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.isocpp.org%2Fmailman%2Flistinfo.cgi%2Fsg15&data=04%7C01%7Cgdr%40microsoft.com%7Ca844c0bda0b14e2bb92108d932a8e322%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637596523230847476%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=GjMkkE2j7fhda%2FCjyc1G2H%2Bhx5Mcnqzxvs6bcJMm%2BDk%3D&reserved=0>
_______________________________________________
SG15 mailing list
SG15_at_[hidden]<mailto:SG15_at_lists.isocpp.org>
https://lists.isocpp.org/mailman/listinfo.cgi/sg15
Received on 2021-06-21 08:00:35