Hi, Tom, and thank you for starting the discussion.

In CLion, we're looking closely at how modules are developing and
sharing some of your concerns and ideas.

> Many tools can't require explicit integration with build systems or
> environment configuration.

Currently, tools are already required to communicate with build
systems in one way or another (at least via a compilation database, or
more sophisticated mechanisms). While there is a lot of room for
improvement in this area, making sure that modules won't make things
much more complicated from what it is now would be a great initial
step.

> Many tools can't depend on compiler specific module support (e.g., won't be
> able to consume module artifacts produced by other tools).

Here I fully agree with you and want to emphasize: even in a world
where each tool is backed by a compiler frontend (or an equally
capable engine), requiring them to use the _same_ compiler as the user
is building the project with would be unfortunate (and, probably,
mostly unachievable). There are some cases when it's possible, though
(i.e. when the toolchain and the tools/IDE are provided by the same
vendor and bundled in a single distribution, like Visual C++ or
Xcode).

> For example, it strikes me
> as unreasonable to require build systems to be augmented with explicit
> support for each of Vim, Emacs, Visual C++, VS Code, Xcode, CLion, Cevelop,
> Eclipse, etc... in order for the maintainers of any particular code base to
> use their preferred editor with advanced features like code completion.
> Likewise, it seems unreasonable to require tools like editors to be able to
> query any particular build system.

Build system could list the modules with its names, sources and
compilation flags as it does with sources today. Alternatively, there
could be a standardized way to query it from a compiled module unit.
Either way, while the taken approach important to agree on, it's not
much different from what tools are going through now.

> Should we expect editors like Vim, Emacs, CLion,
> Cevelop, etc... to be able to consume module artifacts? If so, for which
> (versions of which) compilers?

Here, if the only thing the IDE got is the compiled module interface,
I expect to be able to query:
- the file(s) it was compiled from, and
- compilation command that was used for it.

Note that getting back the source files (paths or otherwise), and not
the source code reconstructed back from the binary module, is
important, as it'll allow proper navigation, project-wide
refactorings, etc. Also, note that that's not a restriction for
single-vendor toolchain/IDE/tools combo - in this setup, they could
avoid going the "source file" route and take advantage on knowing the
module internal representation.

> Some modules proponents have argued for a standardized module format that
> all tools could consume.

I obviously miss some context, since I wasn't present in Jacksonville,
but, intuitively, I have some concerts if it's ever implementable.
However, it's a very interesting topic to explore. Still, it would
take years to achieve and a massive collaboration between all involved
parties. I think it would be crucial for module adoption that we have
a shorter and easier route for tooling/IDEs, while this common format
is still discussed/developed/adopted.

> The LSP (language server protocol; https://langserver.org) provides a tool
> agnostic approach to avoiding the parsing question altogether by providing a
> protocol by which a client can request some semantic information such as
> code completion, hover text, and location information. The server (likely
> closely tied to a particular compiler) responds with information collected
> during a build (whether cached or on demand). Vim, Emacs, VS Code, CLion,
> and other editors have added or are adding support for it.

Note that it's just a protocol (one of the many possible), and all the
problems being discussed are still applicable when looking at a C++
LSP server.

> The Modules TS specifies (5.2
> [lex.phases] p7) "It is implementation-defined whether the source for module
> interface units for modules on which the current translation unit has an
> interface dependency (10.7.3) is required to be available".

This is a concerning clause for us, is it expected to land in the
final standard version?

> Historically, we've taken the individual tool configuration approach for
> support of header files and, despite limitations, it has sufficed. However,
> modules changes one critical aspect of such configuration. Previously,
> header files needed to be consumable with the same set of include
> paths and macro definitions as is used for the primary source file.
> Translating module interface unit source code may require different,
> even conflicting, include paths and macro definitions. Thus,
> configuration will become more challenging.
> <...>
> I think we need an industry standard, tool agnostic solution that works
> for common environments
> <...>
> A set of requirements for translating the module interface unit source code
> (for one or more variations or build modes). This includes preprocessor
> information (include paths, macro definitions, macro undefinitions), and,
> potentially, language dialect requirements (specified in a generic form and,
> perhaps, with the ability to customize for specific tools).

A tool/IDE/frontend would probably:
1. take a module source code
2. translate it into its internal representation (using the provided
compiler options - as it's already does for normal translation units)
3. After that it becomes mostly compiler- and environment-
independent, and could be used to analyse the primary source file (or
another module).

I'm not sure if making step 2 compiler-independent would change much currently.

I don't immediately see how this problem (a tool has to be aware of
the used compiler's flags and quirks) is specific for modules, plain
old TUs also lack this information (which should be extracted from
compiler flags, which is sometimes non-trivial). I've submitted a mail
recently about this
(http://www.open-std.org/pipermail/tooling/2018-August/000148.html),
admittedly, a bit chaotic.

So, TLDR:
1. There should be a way to get original source files for each module
used in a compilation (at least initially, to allow the tooling to
catch up quickly).
2. Given that's true, implementing module support in tools/IDEs is
probably not that much different from the existing TUs support,
regardless of the way modules are implemented and integrated into the
build process (not considering the actual language changes: name
lookup rules, etc, which have to be made anyway, and would arrive for
free to the tools backed by a modern compiler frontend)
3. Simplifying of the tools <-> build systems/compilers communications
is very welcome, but is not unique to the modules support, and is not
a blocker for the modules support.

Best regards,
Dmitry Kozhevnikov
CLion team