Thanks to Ben Craig for taking them.

ISO C++ SG15 Modules Tooling Interactions
2019-02-07 Telecon Minutes

Attendees:
Bryce Adelstein Lelbach (Chairing), NVIDIA
Ben Craig (Minute Taker)
JF Bastien, Apple
Michael Spencer, Apple
Bruno Cardoso Lopes, Apple
Hana Dusíková, Avast and Czech Republic
Steve Downey, Bloomberg
Rene Rivera, Boost Build
Boris Kolpackov, build2
Richard Smith, Google
David Blaikie, Google
Manuel Klimek, Google
Dmitri Gribenko, Google
Robert Maynard, Kitware
Gabriel Dos Reis (GDR), Microsoft
Michał Dominiak, NVIDIA and Poland
Tom Honermann, Synopsys
Tomasz Kaminski
Christof Meerwald
Corentin Jabot
Thierry Lavoie
Isabella Muerte (Izzy)
Antony Peacock
Colby Pike

Goals:
Finish informative discussion of modules naming/lookup concerns.
Informative discussion of the impact on non-build-system tools.
Start building consensus for concrete solutions/steps we can take to
resolve concerns, specifically something in the shape of a standing
document on module mapping/lookup.

Agenda:
Isabella Muerte - P1302
Apple - D1482
Kitware - D1483 (if we have time)

Izzy presents P1302R1

Izzy: Provide a fast path for module lookup. Gets us closer to a
standard directory layout. Currently, have to support all the project
layouts which is a nightmare. Splayed support will be minimum,
heirarchical for most OSes. One directory = one module. Module entry
point (mod.cxx or module.cxx, we don't enforce that because of zOS)

Tom: Don't worry about zOS

Izzy: The name of the partition doesn't matter, we can feed a
directory into the compiler, and the compiler will create the
interface parts. Doesn't involve dependent module BMIs. The compiler
does not automatically import sub directories, we don't recurse. That
is the responsibility of the build system. Doesn't involve
preprocessing and macro names. Compilers don't become a build system.
Dependency management effort isn't thrown out. By using
per-directory, we can lower the number of processes launched. We can
rely on directory timestamps.

Ben: should the splayed directory be named "core.io" instead of "io"

Izzy: directory name doesn't matter, could be independent of module name

Downey: Using the name of the bmi as the name of a directory... colon
is a problem and is there as a partition. Might need to figure out a
file system safe character to map that to.

Izzy: The colon portion of that is not part of the name of the file,
the way you currently declare a module partition is with a colon and
name.

Downey: One of the ways we've been trying to figure out how to convert
module to partition is to make the module name the file name. If a
module name isn't a legal file name, we have problems.

Izzy: That's not really an issue in this case
Izzy: This doesn't preclude a build system from mapping module name to
file name.

Richard Smith: You effectively build the bmi for an entire module
interface at once. With this scheme, you likely wouldn't ever want to
build a partition BMI.

Izzy: This takes some wording from the filesystem library, and puts
that in core wording. A module container is a directory.

Ben: I don't think a freestanding environment needs to be able to run
a compiler

Richard Smith: Conversely, the places where a compiler runs need not
*even* satisfy the requirements on a freestanding implementation and
not all implementations are compilers

Boris: This is all about getting a BMI as fast as possible, object
files are a separate manner. Doesn't that mean that the build system
needs to know that those files are part of a partition? The build
system needs to know what they are before it even starts.

Izzy: Didn't know about the module:private thing that was written.
Build system already needs to know that or not.

Boris: If build systems need to know it and don't already know it,
that could be an expensive operation.

Tom: I'm shaky on module partitions. Does the idea of module
extensions done by somebody as a partition make sense. If I consume a
package from somewhere, might I want to add a partition to someone
else's module? Would that require me to put my file in their
directory?

Izzy: It would require you to mess with their directory. That use
case scares me though. Force injecting PCHs have the same kind of
scare factors.

Richard: All of the module interface partitions have to be mentioned
in the primary interface unit without agreeing with the module author
to do so. Nothing is going to stop you from exporting a module with
someone else's name, but it isn't terribly useful. Modules are
basically closed.

Corentin: It solves module partition issue, but I think there should
be few module partitions. They are a way to hide implementation
details, doesn't solve the larger issue of module mapping globally.

Richard: Modules design does not require a size of modules. If you
make lots of small modules, this doesn't help much. If you larger
modules, then partitions help. This paper doesn't try to solve
anything outside of partitions. The heirarchical part of this
solution might solve the larger module issue. This is certainly a
step along the path of translating from module name to file name.

Izzy: Trying to keep the scope of this paper small, so I don't want to
tack on the larger module mapping problem onto this.

JF: How would this paper fit towards a standing document. What would
that mean, what's the timeline, what kind of teeth does it have, how
do people implement it?

Bryce: ISO standing documents are the weakest form of a document that
ISO puts out. They are fairly easy to publish. They aren't
normative. It is a way for us to put out an auxilliary document that
has wg21's stamp on it. No national body balloting on standing
documents.

JF: Have you heard from implementers that something like this would be
the root of a bigger standard.

Izzy: Vote in san diego SG15 evening session was to not put some of
these things in a standing document.

GDR: To get wg21 to agree to something requires a lot of
socialization. Doesn't mean that it is a hard no. Needs more
explanation.

Corentin: For the mapping particularly it is important that the
compiler and build systems agree and have a set of rules. Don't want
to leave it as non-portable implementation defined. More normative is
better.

JF: We're clang, so if clang does something, we'll look at it and see
if that is what we are going to do. If it's in the IS, we'll have it.
My concern is that I don't want to rush through something if some
implementers aren't comfortable with. I like the idea of doing quick
things with a standing document because you can change it easy.

GDR: If it were to go first into the IS, that would be problematic.

Izzy: The C standard has details on what a vendor has to do, so we are
borrowing some precedent from them. The compiler interface is
standardized on POSIX. We need to start standardizing behavior of
compilers. When they diverge, it causes build system problems.

Corentin: Renaming modules is messy. We won't be able to update a
standing document afterwards because of legacy code.

Richard Smith: Need to make sure we aren't too short sighted or
unimaginative. There are C++ interpreters. Need to be careful not to
preclude C++ implementations. Calculating C++ by hand is legal. Must
be an extension to the core language. It could still be in the IS
though.

Corentin: Not asking for a module to a file, but to be able to map a
module name without opening the target file.

GDR: Let's not pushed by lack of imagination. We have implementations
that take input from non-files. Wasn't saying that standardizing
compiler behavior is against the ISO rules.

Apple presents D1482R0 (Unpublished)

JF: Clang modules are useful data points, even though they aren't the
current modules.

Bruno: Should leave modules out of scope of package management.

Corentin: While modules do not solve everything , isolation and the
fact that modules are... well, modularized can make paxkage management
MUCH easier

Michael: current includes need to have completely correct preprocessor.

Corentin: includes are usually extracted during the compilation, not before

Richard Smith and David Blaikie: citation needed, corentin
there are lots of build systems and many of them do include scanning in advance

Izzy: You can (typically) run each compiler in the preprocessor mode
with the -M options to get the actual headers. Unsure if this would
cause issues.

Steve: import in non-modular code may be an issue, but I don't believe
it makes scanning any worse.

Richard: if you have generated headers, extracting includes during
compilation doesn’t work very well, which is at least one reason why
some build systems don’t work that way

Steve: The ones that do scanning for dependencies up front often have
bugs, and add overhead. Updating the header dependencies during
compilation has been the norm since the late 90s. Citation:
http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/

JF: One batch of module code can turn into multiple module artifacts,
likely because of -D. Going to need to recompile things. Nobody has
needed to do fine grained memoization

Michael: dependency scanner prototype. single process to scan over
entire build.

JF: Dependency scanning is still suboptimal, but it's not that bad. A
lot of room to make it fast.

Steve: Scons is a known counter-example

Richard: I’m not saying no-one does it, just “usually” and “the norm”
are unhelpful characterizations. none of us has sufficient information
to make such a claim. Also bazel, also perforce jam.

Boris Kolpackov: also build2

Rene: b2 (aka boost build) does pre-scanning.

Richard: If all you have is a make, every problem looks like
generating a Makefile :)

Rene: And note.. the “overhead” in b2 of pre-scanning is essentially
zero because of file system caching.

Izzy: I understand why people want to distribute BMIs with their
packages, but I think that's like 10 years down the road *maybe*.

Rene: I live in a world where binary distribution is the norm. With
libraries from vendors. We can't produce something that will make
those use cases harder. Want to make it as easy as possible to adopt.

JF: If you use our platform to develop, we also produce headers with
binaries. We don't think this is a problem modules need to solve.

Steve: I would like to use the bmi that I built here, and use it over
there. I think getting them from anyone else is going to be a problem.

Izzy: Is there anything that prevents a vendor from distributing
interface modules in source form next to a static or shared library?
And if not, could we get vendors to permit generating *just* the
'source' interface side of things?

Ben Craig leaves, Bryce takes over for minutes

Tom: I think D1482 is great, it echos a lot of my experience deploying
clang modules with Coverity. I believe the reason for the success of
Clang Modules is: (0) Built on #include; the design doesn't break
existing tools, and (2) Modules artifacts are implicitly built and
therefore don't require build system updates. They are an opt-in tool
specific optimization. I think we should look carefully at the reasons
 that Clang Modules has been successful and ensure that we replicate
that experience in the standard. The currently proposed design and
implementation direction (import declarations, explicitly built module
artifacts managed by build systems) contrasts with the above.

Corentin: I think it would be a mistake to have modules and not a
modularized standard library.

Gaby: It’s my understanding that implicit modules required modifying
build systems and putting some build system logic into the compiler.
Corentin - I'm a proponent of having some modular version of the
standard library. At the same time, I think it may be possible to
have two modules. One big one, and one freestanding.

Bryce: I agree with Gaby. I think “one big std module” or no modular
standard library are the only options for C++20. P1453R0 was not
optimistic about the “one big std module” approach when I wrote it,
but subsequently I’ve spoken with Billy O’Neal from the Visual Studio
standard library team, and he thinks it might be viable. There were
two concerns that made me pessimistic about the “one big std module”
approach: (0) global dynamic constructors and (1) the module migration
ABI question (e.g. can you move something from one module to another
without breaking the ABI of the thing). Billy doesn’t think (0) is a
concern, because most standard libraries have gotten rid of their
global dynamic constructors (through constexpr constructors, etc). He
thinks the only real challenge is some free function overloaded
operators that conflict, which could be solved with hidden friend
functions (example: system_error and future_error operator==). So if
the answer to (1) is that migrating something between modules isn’t an
ABI break, then “one big std module” is feasible.

-- 
Bryce Adelstein Lelbach aka wash
Thrust Lead @ NVIDIA
ISO C++ Library Evolution Incubator Chair
CppCon and C++Now Program Chair
CUDA Convert and Reformed AVX Junkie
Sleep is for the weak
--