Date: Tue, 13 Aug 2019 13:41:39 -0700
Michael Spencer will be talking about clang-scan-deps, and this
(originally posted to the Clang developer's list):
[clang][RFC] C++20 modules dependency discovery
C++20 is coming and we need to decide how clang will handle dependency
discovery for modules. In the following, module means compiled C++20
module interface unit, and I will use header unit to refer to the
thing generated by a clang module map.
There are two different modes we care about when it comes to module
dependencies: implicit and explicit.
Implicit Modules
================
For implicit modules the build system doesn’t know anything about
them, and thus can’t care about any intermediate files. It needs to
know about all source files that if changed should cause a rebuild of
this translation unit.
For this case clang needs to output the full transitive set of
dependencies, excluding any intermediate temporaries. This also means
that we can’t get the full set of dependencies without actually at
least preprocessing every module transitively referenced. This means
that -E -MD should fail if it can’t find a module or header unit.
Explicit Modules
================
For explicit modules we only need to know the direct dependencies, as
the build system will handle the transitive set.
For preprocessing we still need to import header units (but only their
preprocessor state), but not normal modules. For this case it’s ok if
-E -MD fails to find a module. But it does still need to be able to
find header units and module maps. Additionally the normal Make output
syntax is not sufficient to represent the needed information unless
the driver decides how modules and header units should be built and
where intermediate files should go. There’s currently a json format
working its way through the tooling subgroup of the standards
committee that I think we should adopt for this.
I think we need separate modes in clang for these along with support
for scanning through header units without actually building a clang
module for them. clang-scan-deps will make use of the explicit mode.
The question I have is how should we select this mode, and what clang
options do we need to add?
Proposal
========
As a rough idea I propose the following:
-M? means output the json format which can correctly represent
dependencies on a module for which we don’t know what the final file
path will be.
clang++ -std=c++20 -E -MD -fimplicit-header-units should implicitly
find header unit sources, but not modules (as we've not given it any
way to look up how to build modules).
This means that the dep file will contain a bunch of .hs,
.modulemaps, and any .pcms explicitly listed on the command line.
This also means erroring on unknown imported modules as we don't
know what to put in the dep file for them.
clang++ -std=c++20 -E -MD -fimplicit-header-units -fimplicit-module-lookup=?
should do the same as the above, except that it does know how to find
modules, and should list all of the transitive dependencies of any
modules it finds.
clang++ -std=c++20 -E -MD should fail if it hits a module or header
unit, and should never do implicit lookup.
clang++ -std=c++20 -E -M? should scan through header units without
actually building clang modules for them (to get the macros it needs),
and should note all module imports.
This means that the dep file will contain only .hs that it includes,
and use the json representation of header units and modules.
It will also be shallow, with only direct dependencies.
Additionally, we should (eventually) make:
$ clang++ -std=c++20 a.cpp b.cpp c.cpp a.cppm -o program
Work without a build system, even in the presence of modules. To do
this we will need to prescan the files to determine the module
dependencies between them and then build them in dependency order.
This does mean adding a (simple) build system to the driver (maybe
[llbuild](https://github.com/apple/swift-llbuild)?), but I think it’s
worth it to make simple cases simple. It may also make sense to
actually push this work out to a real build system. For example have
clang write a temporary ninja file and invoke ninja to perform the
build.
(originally posted to the Clang developer's list):
[clang][RFC] C++20 modules dependency discovery
C++20 is coming and we need to decide how clang will handle dependency
discovery for modules. In the following, module means compiled C++20
module interface unit, and I will use header unit to refer to the
thing generated by a clang module map.
There are two different modes we care about when it comes to module
dependencies: implicit and explicit.
Implicit Modules
================
For implicit modules the build system doesn’t know anything about
them, and thus can’t care about any intermediate files. It needs to
know about all source files that if changed should cause a rebuild of
this translation unit.
For this case clang needs to output the full transitive set of
dependencies, excluding any intermediate temporaries. This also means
that we can’t get the full set of dependencies without actually at
least preprocessing every module transitively referenced. This means
that -E -MD should fail if it can’t find a module or header unit.
Explicit Modules
================
For explicit modules we only need to know the direct dependencies, as
the build system will handle the transitive set.
For preprocessing we still need to import header units (but only their
preprocessor state), but not normal modules. For this case it’s ok if
-E -MD fails to find a module. But it does still need to be able to
find header units and module maps. Additionally the normal Make output
syntax is not sufficient to represent the needed information unless
the driver decides how modules and header units should be built and
where intermediate files should go. There’s currently a json format
working its way through the tooling subgroup of the standards
committee that I think we should adopt for this.
I think we need separate modes in clang for these along with support
for scanning through header units without actually building a clang
module for them. clang-scan-deps will make use of the explicit mode.
The question I have is how should we select this mode, and what clang
options do we need to add?
Proposal
========
As a rough idea I propose the following:
-M? means output the json format which can correctly represent
dependencies on a module for which we don’t know what the final file
path will be.
clang++ -std=c++20 -E -MD -fimplicit-header-units should implicitly
find header unit sources, but not modules (as we've not given it any
way to look up how to build modules).
This means that the dep file will contain a bunch of .hs,
.modulemaps, and any .pcms explicitly listed on the command line.
This also means erroring on unknown imported modules as we don't
know what to put in the dep file for them.
clang++ -std=c++20 -E -MD -fimplicit-header-units -fimplicit-module-lookup=?
should do the same as the above, except that it does know how to find
modules, and should list all of the transitive dependencies of any
modules it finds.
clang++ -std=c++20 -E -MD should fail if it hits a module or header
unit, and should never do implicit lookup.
clang++ -std=c++20 -E -M? should scan through header units without
actually building clang modules for them (to get the macros it needs),
and should note all module imports.
This means that the dep file will contain only .hs that it includes,
and use the json representation of header units and modules.
It will also be shallow, with only direct dependencies.
Additionally, we should (eventually) make:
$ clang++ -std=c++20 a.cpp b.cpp c.cpp a.cppm -o program
Work without a build system, even in the presence of modules. To do
this we will need to prescan the files to determine the module
dependencies between them and then build them in dependency order.
This does mean adding a (simple) build system to the driver (maybe
[llbuild](https://github.com/apple/swift-llbuild)?), but I think it’s
worth it to make simple cases simple. It may also make sense to
actually push this work out to a real build system. For example have
clang write a temporary ninja file and invoke ninja to perform the
build.
-- Bryce Adelstein Lelbach aka wash CUDA Core C++ Libraries Lead @ NVIDIA ISO C++ Library Evolution Incubator Chair ISO C++ Tooling Chair CppCon and C++Now Program Chair CUDA Convert and Reformed AVX Junkie Sleep is for the weak --
Received on 2019-08-13 15:44:06