Agenda, as was presented in the meeting...

ISO C++ SG15 Tooling
Pre-Cologne Modules Tooling Interactions Telecon
2019-03-22

Chairing: Ben Craig

Minute Taker: Tom Honermann

Today's Goal: Work on fleshing out an outline for the TR and start assigning homework to people.

Please discuss the following, suggest changes/improvements, and attempt to build consensus.

Timing: The TR will be feature-driven not deadline-driven; we'll ship it when it's ready.

Proposed Outline (based on Bryce's pre-Kona notes and Rene/Corentin's Kona slides, see end of document for details)


  1. Usage: Explains the requirements and expected usage of modules across the C++ ecosystem. Raises questions which need to be addressed later in the document.

     * Stakeholders: Different types of users of modules (more details below).
     * Archetypes: Concrete examples based on expected usage (more details below).
  1. Findings: Focused technical sections that explore open questions in detail and present the results of field experience. Unopinionated.
. Module Mapping: What approaches and formats are effective for communicating module name <-> module-interface/header file mappings? Module name + configuration <-> BMI mappings?
a. Module Naming: How should module names be structured? How do we avoid conflicts between different projects? How do we deal with versioning?
b. Module Granularity: What size should modules be to maximize performance and usability? Does the cost of an import scale with the size of the module?
c. Module ABI: How do we maintain stable ABIs in a modular world?
d. Codebase Transition Path: How should projects transition from headers to modules? How should projects support both pre-C++20 headers and C++20 modules?
e. BMI Configuration: How do we find the BMI that was compiled in the same way as the current TU? What defines the configuration of a BMI?
f. BMI Distribution: How effective is the distribution of BMIs alongside module-interface/header files?
g. Dependency Scanning: How do we do dependency scanning in a modular world? Can we make it fast?
h. Build Performance: How do modules impact build performance? What impact does modules have on parallelism in C++ builds?

  1. Guidance: Concise set of guidelines for the C++ ecosystem. Addresses questions raised in Usage and draws conclusions based on results from Findings.

Do we agree to use this outline as a starting point?

What's missing (Findings sections in particular)? How can this be improved?

Who is interested in working on a particular section of the proposed outline? Please collect a list of names. Don't volunteer for everything - just for what you care about and can commit to working on.
Stakeholders:


  * C++ Implementations
     * GCC, Clang/LLVM, Visual Studio, EDG, PGI, ICC, xlC
  * Build Systems
     * CMake, Make, Boost Build, autoconf, shell scripts, Meson, Ninja, Scons, Waf, Bazel, Buck, Cargo, Gulp, Webpack, Ant, llbuild, Evoke, qmake, MSBuild, internal company build systems, mixed build systems, distributed build systems (icecc, ccache, sccache, incredibuild, distcc, fastbuild, Bazel remote build execution)
  * Tools
     * IDEs, Clang-based tools, CastXML, static analysis tools (Coverity, Clang Static Analyzer, Grammatech, cppcheck), code generation tools (QT Moc, Protobuf), test frameworks (Google Test), test case reduction tools (creduce, delta)
  * Libraries
     * Boost, header only libraries
  * Distributions
     * vcpkg, conan.io, Linux distributions (RPM-based, Debian-based)
  * Other Languages
     * CUDA, OpenCL/SYCL, C, Python, Rust, Java, SWIG
  * End Users

Who is interested in working on a particular stakeholder group? Collect a list of names.

For each stakeholder group, we need a short description of the group (e.g. what things are in the group) and a bullet list of the issues that matter for that group. Volunteers? Collect a list of names.
Archetypes:


  * Hello world with modules
  * Header only library
  * Incremental build
  * Distributed build
  * Building BMIs only for tooling consumption
  * Dependency scanning vs explicit module dependencies build

We need more concrete examples. Who volunteers to go write some up? Collect list of names.
Format: The TR will likely need to be in Latex, using a fresh fork of the IS Latex that has been customized, similar to the Coroutines TS Latex. We have had very painful issues with non-Latex formal documents in the past. Ex: The Parallelism TS v2 was originally written in HTML which was converted to PDF for publication. It had to be completely rewritten in Latex after we voted to publish it because of typesetting issues raised by ISO that could not be resolved.

https://github.com/cpp-tooling has been created for collaboration. Not sure how we'll use it yet; feel free to create a repository for examples and/or brainstorming. JF Bastien can add people while Bryce is away.
Appendix

List from Bryce's pre-Kona notes

* Module Map Format
o Name of Module + ABI Hash -> Physical Location
* Module Mappers
* Module ABI Hashing
* Module Versioning
* Dependency Scanning
o How should tools use a dependency scanner? Command line? Programmatic API?

List from Rene/Corentin presented at the Kona evening session:

* Module name <-> Module header unit name mapping
* Module name <-> BMI mapping
* Module naming
* Guidelines for BMI implementations strategies
* Guidelines (maybe format) for shipping modularized closed source libraries
* Guidelines for Linux distributions (maybe)
* Guidelines/format for handling legacy header units
* Guidelines for using modules:
o ABI concerns/hashing
o Not authoring modules for 3rd party code



From: Modules <modules-bounces_at_[hidden]> On Behalf Of Ben Craig
Sent: Friday, March 22, 2019 8:36 PM
To: modules_at_[hidden]; WG21 Tooling Study Group SG15 <tooling_at_[hidden]>
Subject: [EXTERNAL] [isocpp-modules] March 22 telecon minutes

The agenda that was in the telecon will come in a later email, here's the minutes. Note that some people that couldn't make the meeting have been assigned homework. Many thanks to Tom for taking minutes today!

===============================
Tom's notes
===============================

Attendees:
- Anna Gringauze (AG)
- Ben Boeckel (BB)
- Ben Craig (BC)
- Bruno Cardoso Lopes (BCL)
- Colby Pike (CP)
- Gor Nishanov (GN)
- JF Bastien (JF)
- Mark Zeren (MZ)
- Mathew Woehlke (MW)
- Mathias Stern (MS)
- Michael Spencer (MSp)
- Nathan Sidwell (NS)
- Olga Arkhipova (OA)
- Peter Bindels (PB)
- Rene Rivera (RR)
- Steve Downey (SD)
- Stephen Kelly (SK)
- Tom Honermann (TH)

BC: Introduces agenda; agreement on the outline of the TR; get volunteers; minimal tech talk.
BC: Any objection to the findings as an outline?
JF: Need to base this on code at some point. Would like to see a code base that uses modules and multiple build systems. We can talk all day about theoretical concerns, but need to base work on reality.
BC: As an example, modularize Boost?
JF: Need to focus on applications.
BC: Need to distinguish modularizing a library and consuming a library.
PB: Perhaps try the kitchen sink example from CopperSpice?
TH: JF, do you want real projects or exemplary projects?
JF: Real projects.
SD: Don't think we need really real projects, just exemplary ones. POSIX demonstrates how to do compiles, link; use case oriented.
TH: Agree, and it would be nice to have examples in the TR demonstrating usage.
BC: Bryce has a hello world with modules that could be an example in the TR.
PB: I volunteer to make a hello world example.
BC: Back to the outline, who is working on what? Corentin and PB are working on module mapping?
PB: We have P1484.
MS: I volunteer to help with module mapping.
TH: Which aspect of module mapping are we discussing here?
PB: Mapping to source.
MS: Also want mapping to BMIs.
BC: Sounds like this covers mapping to source, BMI, and indication of header units.
BC: Michael Spencer is working on dependency scanning. So is Ben Boeckel. Can I record them as volunteers to work on this?
MSp: Yes.
JF: Dependency scanning is part of build system implementation. What is the goal of discussing dependency scanning (and other features we're
discussing) as part of the TR?
MS: It is a contract between stakeholders.
TH: Not concerned about implementation details; concerned about ensuring meta data is represented in ways usable by multiple tools, buld systems, etc..
MSp: The best/fastest way to build modules are relevant for implementation.
PB: Perhaps worth discussing trade offs between fast and accurate?
MS: Let's take inaccurate off the table.
BC: Moving on to build performance, can I sign Rene and Mathias up for that?
MS: Yes, questions to address: can BMI and object files be built concurrently? What gets built and how? These are worth researching.
RR: Happy to work on performance related issues and testing. There is a chicken/egg problem of needing working compilers. We can't tackle distributed builds without additional work.
MS: Would like feedback on how reasonable it is to look at performance of current compiler incarnations.
NS: In three years time, performance profile will probably be quite different; focus now is correctness, not speed.
MSp: Clang has some inefficiencies around finding modules now. I think overhead of modules will go down over time. Dependencies will remain.
MS: Wondering about relative performance, scanning vs code gen vs BMI gen, etc... Perhaps a TR2 would be a good focus for performance.
MSp: Some performance sensitive things will change, some things won't.
BC: Looking at code base transition path now. BC volunteers.
TH: Interested in transition path.
SK: Also interested.
SD: Also interested. Will bring a Lacos and Bloomberg informed focus.
BC: Olga, will you sign up for BMI distribution?
OA: Yes. We've had internal discussions about sharing BMIs.
BCL: Volunteers to work on BMI distribution as well.
SD: This overlaps with sharing of object files as well. What is the range of IFNDR when sharing BMIs? If a BMI isn't suitable, how does it get recompiled.
TH: Is MSp interested in volunteering with regard to BMI distribution?
JF: We can volunteer to write a section that says "don't".
BC: I don't think we should have a section that raises questions for all stackeholders. Instead, each area under findings should raise questions and explore them from the standpoint of each stakeholder.
MS: I agree, though not particularly productive to discuss now until we have stuff to put in the doc.
BC: Makes sense.
BC: PB volunteered for hello world, volunteers for distributed build?
MS: I volunteer. Would be nice to have someone from Google due to difference in approaches.
GN: Volunteers Gaby to contribute to hello world examples.
BC: Would like to sign up Gaby for explicit module dependencies as the Microsoft Edge team purportedly used them.
MS: Do we want to encourage explicit module dependencies?
??: No.
TH: Matches existing PCH usage in Microsoft ecosystems.
BC: We should discuss.
OA: For dependency scanning, we are planning to do work to support this, but haven't started yet. Mixed mode dependency scanning and explicit dependencies may happen.
BC: Moving on to header only libraries. Done today to avoid build system pain. Any volunteers?
PB: Catch2 considering moving away from header-only for technical reasons (e.g., build speed).
BC: No volunteers for header-only.
BC: Volunteers for incremental-build? Kind of inherent to builds in general...
MS: I volunteer to writeup something for incremental builds.
BC: On to building BMIs for tooling consumption.
OA: I work on static analysis, so interested in special tools.
Interested in saving information useful for tools in BMIs.
TH: Will contribute to discussion on sharing BMIs across compilers/tools.
BC: No assignments for module naming, module granularity, BMI configuration. Got the rest. Stakeholders to be covered on a per item basis.
TH: Perhaps next meeting we can have everyone vote about their highest priority concerns to be addressed in the TR.

===============================
Ben's notes
===============================

Possibly interleave findings and guidance?

Need concrete code base / code bases, need multiple concrete build systems
   Tie things to specific examples
      For multiple stakeholders, use cases
      Kitchen Sink CopperSpice example?


Peter Bindels + Gaby: Volunteering for hello world with modules section

Corentin Jabot, Peter Bindels, Mathias Stearn: Module Mapping P1484
   mapping to source files. mapping to BMI. Where to find a header unit

Michael Spencer and Ben Boeckel: Dependency Scanning
   What we've found, and what others might want to do. This is from the impl's perspective.
   Does the "can we make it fast?" belong in the TR?

Mathias Stearn + Rene Rivera: Build Performance?
   Can't get recommendations yet based off of current work
   Different models of building? Concurrent BMI and .o vs. distinct builds. Needs research
   Chicken and Egg

Ben Craig + Tom Honermann + Steve Downey + Stephen Kelly: Codebase Transition Path

Olga Arkhipova + Bruno Cardoso Lopes (maybe?): BMI Distribution

Mathias + Ben + Googler TBD: Distributed build


Microsoft (Gaby + Olga): Dependency scanning vs explicit module dependencies build

Michael Spencer: Incremental Builds (information on performance and build theory)

Anna Gringauze, Tom Honermann: Building BMIs only for tooling consumption:
   Probably going to look at source. Explain how tools are different from compilers.
   Sharing BMIs between compilers and tools.

Bruno Cardoso Lopes (maybe?): BMI Configuration:

TBD:
   Header only library
   Module Naming
   Module Granularity

Stakeholders to be covered on a per item basis


Hope that another meeting we can establish priorities