Date: Fri, 18 Jun 2021 18:55:07 +0000
David - Daniel is explicitly stating that you can't discover dependency tree with a simple lexer.
I also don't think we should do something in this space based on "fear".
-- Gaby
________________________________
From: David Blaikie <dblaikie_at_[hidden]>
Sent: Friday, June 18, 2021, 11:12 AM
To: Gabriel Dos Reis
Cc: Daniel Ruoso; sg15_at_[hidden]
Subject: Re: [SG15] Draft: Requirements for Usage of C++ Modules at Bloomberg
On Fri, Jun 18, 2021 at 11:01 AM Gabriel Dos Reis <gdr_at_[hidden]<mailto:gdr_at_[hidden]>> wrote:
* but you cannot discover the graph with a simple lexer
Yes, and I have not found that to be an actual problem in practice. In fact there is really nothing new here (other than recognizing the ‘import’ directives and friends) in the non-module world: the ol’ C and C++ standards allow
#include HEADER
where HEADER comes from macro expansions and all the other stuff that you mention about inability to use “simple lexer”, conditional inclusion, etc. As a PSA, I would like to disabuse anyone who ever thought that grep was sufficient to compute dependency of C or C++ source files pre-C++20.
Maybe it is my dinosaurus status showing bleeding out here but are we rehashing the same arguments from the past?
The one thing we can do for the community is to have a library readily available to anyone who ever thought that a simple lexer is sufficient to compute C++ source file dependency. That would be a major step forward.
I don't think Daniel is suggesting/has mentioned the idea that a simple lexer is sufficient for either headers or modules. But expected/would like it if we could simplify the situation further than module files alone currently allow in this regard.
I do lean towards "maybe it's simple enough, even if it doesn't allow a simple lexer to do this operation" - at least I don't think that's an ironclad proof that we need the dependency information in a more immediately legible format such as in the pkg-config files or equivalent. I wouldn't rule it out either, though.
Like, I wouldn't be averse to pkg-config or equivalent files supporting this as an optional feature - so that folks could explore whether it significantly improves performance. If it's more a matter of software complexity (since build tools that don't know anything about parsing C++ are going to want to discover this dependency graph) - yeah, maybe compilers could vend a command line mode that reports the dependencies as quickly as possible - ala clang-scan-deps & for library level uses (llvm-buildozer, background running build systems/IDE type situations) there could be a library for that.
But, going on past experience, those kinds of utilities/dependencies don't necessarily eventuate and I can understand the concern about proliferation/variation/lack of ubiquity in such things & so a preference to bake it into a common config file. I guess all I have on that is that I think it'd be unfortunate for developers to have to repeat this information in multiple places like that.
- Dave
-- Gaby
From: Daniel Ruoso <daniel_at_[hidden]<mailto:daniel_at_[hidden]>>
Sent: Friday, June 18, 2021 10:30 AM
To: Gabriel Dos Reis <gdr_at_[hidden]<mailto:gdr_at_[hidden]>>
Cc: sg15_at_[hidden]<mailto:sg15_at_[hidden]>; David Blaikie <dblaikie_at_[hidden]<mailto:dblaikie_at_[hidden]>>
Subject: Re: [SG15] Draft: Requirements for Usage of C++ Modules at Bloomberg
On Fri, Jun 18, 2021 at 1:20 PM Gabriel Dos Reis <gdr_at_[hidden]<mailto:gdr_at_[hidden]>> wrote:
Also notice that while a dependency scanner needs to chase #included or imported files (via the appropriate implementation-defined mapping), it actually does not need to fully preprocess the source code found therein;
the spec has been carefully crafted to that effect.
It doesn't need to preprocess the complete file, but it needs to significantly preprocess the early sections of the file. While the spec has carefully prevented conditional specification of the keywords, it still allows macros to be used, which requires support for conditional evaluation, variable expansion, and string concatenation.
In other words, the only optimization granted by the standard today is on the discovery of whether or not a source file needs to participate on the construction of the DAG for processing modules, but you cannot discover the graph with a simple lexer, you need the full expression evaluation logic of the preprocessor.
daniel
I also don't think we should do something in this space based on "fear".
-- Gaby
________________________________
From: David Blaikie <dblaikie_at_[hidden]>
Sent: Friday, June 18, 2021, 11:12 AM
To: Gabriel Dos Reis
Cc: Daniel Ruoso; sg15_at_[hidden]
Subject: Re: [SG15] Draft: Requirements for Usage of C++ Modules at Bloomberg
On Fri, Jun 18, 2021 at 11:01 AM Gabriel Dos Reis <gdr_at_[hidden]<mailto:gdr_at_[hidden]>> wrote:
* but you cannot discover the graph with a simple lexer
Yes, and I have not found that to be an actual problem in practice. In fact there is really nothing new here (other than recognizing the ‘import’ directives and friends) in the non-module world: the ol’ C and C++ standards allow
#include HEADER
where HEADER comes from macro expansions and all the other stuff that you mention about inability to use “simple lexer”, conditional inclusion, etc. As a PSA, I would like to disabuse anyone who ever thought that grep was sufficient to compute dependency of C or C++ source files pre-C++20.
Maybe it is my dinosaurus status showing bleeding out here but are we rehashing the same arguments from the past?
The one thing we can do for the community is to have a library readily available to anyone who ever thought that a simple lexer is sufficient to compute C++ source file dependency. That would be a major step forward.
I don't think Daniel is suggesting/has mentioned the idea that a simple lexer is sufficient for either headers or modules. But expected/would like it if we could simplify the situation further than module files alone currently allow in this regard.
I do lean towards "maybe it's simple enough, even if it doesn't allow a simple lexer to do this operation" - at least I don't think that's an ironclad proof that we need the dependency information in a more immediately legible format such as in the pkg-config files or equivalent. I wouldn't rule it out either, though.
Like, I wouldn't be averse to pkg-config or equivalent files supporting this as an optional feature - so that folks could explore whether it significantly improves performance. If it's more a matter of software complexity (since build tools that don't know anything about parsing C++ are going to want to discover this dependency graph) - yeah, maybe compilers could vend a command line mode that reports the dependencies as quickly as possible - ala clang-scan-deps & for library level uses (llvm-buildozer, background running build systems/IDE type situations) there could be a library for that.
But, going on past experience, those kinds of utilities/dependencies don't necessarily eventuate and I can understand the concern about proliferation/variation/lack of ubiquity in such things & so a preference to bake it into a common config file. I guess all I have on that is that I think it'd be unfortunate for developers to have to repeat this information in multiple places like that.
- Dave
-- Gaby
From: Daniel Ruoso <daniel_at_[hidden]<mailto:daniel_at_[hidden]>>
Sent: Friday, June 18, 2021 10:30 AM
To: Gabriel Dos Reis <gdr_at_[hidden]<mailto:gdr_at_[hidden]>>
Cc: sg15_at_[hidden]<mailto:sg15_at_[hidden]>; David Blaikie <dblaikie_at_[hidden]<mailto:dblaikie_at_[hidden]>>
Subject: Re: [SG15] Draft: Requirements for Usage of C++ Modules at Bloomberg
On Fri, Jun 18, 2021 at 1:20 PM Gabriel Dos Reis <gdr_at_[hidden]<mailto:gdr_at_[hidden]>> wrote:
Also notice that while a dependency scanner needs to chase #included or imported files (via the appropriate implementation-defined mapping), it actually does not need to fully preprocess the source code found therein;
the spec has been carefully crafted to that effect.
It doesn't need to preprocess the complete file, but it needs to significantly preprocess the early sections of the file. While the spec has carefully prevented conditional specification of the keywords, it still allows macros to be used, which requires support for conditional evaluation, variable expansion, and string concatenation.
In other words, the only optimization granted by the standard today is on the discovery of whether or not a source file needs to participate on the construction of the DAG for processing modules, but you cannot discover the graph with a simple lexer, you need the full expression evaluation logic of the preprocessor.
daniel
Received on 2021-06-18 13:55:16