Gabriel Dos Reis <gdr_at_[hidden]> writes:

> You, Boris, actually wen through the trouble of modularizing your tools
> (Modules TS semantics). I would like to hear from your concrete experience.

Sure. More specifically, we did the following:

1. Added support for Modules TS (GCC, Clang, and VC) to the build2 build
   system.

2. Modularized the build2 utility library (libbutl[1]; about 40 modules)
   as a first step in modularizing the entire toolchain. In the process
   we also created the std modules library (libstd-modules[2]) along the
   lines of the P0581R1[3] proposal and documented our experience in a
   set of modularization guidelines[4].

Now to the experience:

1. While support for Modules TS in a build system is sure complex, most
   of this complexity felt inherent rather than accidental.

   In particular, we were able to extract module dependency information
   without any support from the compilers by preprocessing the translation
   unit and then shallow-parsing the result to find module-related
   declarations. This shallow-parsing is possible thanks to all such
   declarations being top-level (which, I was told, was not by accident).
   See the code[5] if you are interested in details.

   Also, Modules TS fitted naturally into the overall "dependency extraction
   and compilation pipeline", which in build2 looks like this:

   1. Partially preprocess the translation unit (GCC's -fdirectives-only,
      Clang's -frewrite-includes; hopefully VC's new preprocessor will
      have something similar) and at the same time extract header
      dependency information (GCC/Clang's -M* options, VC's /showIncludes).

   2. Fully preprocess the translation unit and extract module dependency
      information. Here the user can also signal to the build system that
      no module declarations depend on the preprocessor (e.g., #ifdef'ed
      out, etc) in which case the full preprocessing is unnecessary (there
      is also a way to signal that the translation unit doesn't have any
      #include's to avoid partial preprocessing on step (1)).

   3. Compile the partially preprocessed translation unit. Interestingly,
      doing it this way (partially preprocessing all the translation units
      at the beginning and then compiling them can result[6] in faster
      overall builds).

   Also note that this pipeline is ready for distributed compilation where
   on step (3) we ship the partially preprocessed source and the imported
   BMIs to a remote host for compilation.

2. An attempt to modularize a real library that uses modern C++ didn't go
   smoothly. Currently, compiler support for Modules TS ranges from
   incomplete to buggy. I've only managed to build libbutl with modules
   enabled using Clang but had to hack both the compiler and libstdc++.
   The good news is there is a steady progress on the VC front with every
   release there are fewer and fewer remaining issues.

   We did, however, get an early glimpse at the kind of speedup we can
   expect -- in our case it was 3x compared to using headers.

If anyone is interested in any details, I would happy to elaborate.

[1] https://git.build2.org/cgit/libbutl/tree/
[2] https://git.build2.org/cgit/libstd-modules/tree/
[3] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0581r1.pdf
[4] https://build2.org/build2/doc/build2-build-system-manual.xhtml#cxx-modules-guidelines
[5] https://git.build2.org/cgit/build2/tree/build2/cc/parser.cxx
[6] https://build2.org/article/preprocess-compile-performance.xhtml

Boris