On 08/30/2018 06:34 AM, Dmitry Kozhevnikov wrote:
> Hi, Tom, and thank you for starting the discussion.
>
> In CLion, we're looking closely at how modules are developing and
> sharing some of your concerns and ideas.
>
>> Many tools can't require explicit integration with build systems or
>> environment configuration.
> Currently, tools are already required to communicate with build
> systems in one way or another (at least via a compilation database, or
> more sophisticated mechanisms). While there is a lot of room for
> improvement in this area, making sure that modules won't make things
> much more complicated from what it is now would be a great initial
> step.

Agreed, and nicely said. This aligns with my "I think we should strive
for a better solution for modules" statement.

>
>> Many tools can't depend on compiler specific module support (e.g., won't be
>> able to consume module artifacts produced by other tools).
> Here I fully agree with you and want to emphasize: even in a world
> where each tool is backed by a compiler frontend (or an equally
> capable engine), requiring them to use the _same_ compiler as the user
> is building the project with would be unfortunate (and, probably,
> mostly unachievable). There are some cases when it's possible, though
> (i.e. when the toolchain and the tools/IDE are provided by the same
> vendor and bundled in a single distribution, like Visual C++ or
> Xcode).

Yes, exactly. The design I'm advocating for is one where such
integration is an optimization rather than a requirement.

>
>> For example, it strikes me
>> as unreasonable to require build systems to be augmented with explicit
>> support for each of Vim, Emacs, Visual C++, VS Code, Xcode, CLion, Cevelop,
>> Eclipse, etc... in order for the maintainers of any particular code base to
>> use their preferred editor with advanced features like code completion.
>> Likewise, it seems unreasonable to require tools like editors to be able to
>> query any particular build system.
> Build system could list the modules with its names, sources and
> compilation flags as it does with sources today.

Yes, this matches my "They could be generated directly by a build
system, or as a side effect of compilation" statement. Few build
systems I've worked with currently have the ability to generate such
lists without major enhancements.

> Alternatively, there
> could be a standardized way to query it from a compiled module unit.
> Either way, while the taken approach important to agree on, it's not
> much different from what tools are going through now.

I've thought about this approach too, but have come to the conclusion
that it substitutes one set of difficulties for another. A standardized
way to query properties of a precompiled module artifact could be
useful, but it still leaves unresolved the question of which (variant of
which) module artifact to query.

>
>> Should we expect editors like Vim, Emacs, CLion,
>> Cevelop, etc... to be able to consume module artifacts? If so, for which
>> (versions of which) compilers?
> Here, if the only thing the IDE got is the compiled module interface,
> I expect to be able to query:
> - the file(s) it was compiled from, and
> - compilation command that was used for it.

This implies that the tool knows how to interpret the command line. For
preprocessor information, such interpretation hasn't traditionally been
particularly difficult since most compilers implement mostly equivalent
forms of -I, -D, and -U options (substituting '/' for '-' in some
cases). However, it so far isn't looking like defacto standard options
will appear for modules related options. For example, clang has
'-fmodule-file=<filename>' where Microsoft has '/module:reference
<filename>'. Gcc doesn't (as far as I can tell) have a corresponding
option. Note that neither Clang nor Microsoft's option indicates the
actual module name (the module name need not match the filename).

>
> Note that getting back the source files (paths or otherwise), and not
> the source code reconstructed back from the binary module, is
> important, as it'll allow proper navigation, project-wide
> refactorings, etc. Also, note that that's not a restriction for
> single-vendor toolchain/IDE/tools combo - in this setup, they could
> avoid going the "source file" route and take advantage on knowing the
> module internal representation.

+1

>
>> Some modules proponents have argued for a standardized module format that
>> all tools could consume.
> I obviously miss some context, since I wasn't present in Jacksonville,
> but, intuitively, I have some concerts if it's ever implementable.
> However, it's a very interesting topic to explore. Still, it would
> take years to achieve and a massive collaboration between all involved
> parties. I think it would be crucial for module adoption that we have
> a shorter and easier route for tooling/IDEs, while this common format
> is still discussed/developed/adopted.

Very much agreed.

>> The Modules TS specifies (5.2
>> [lex.phases] p7) "It is implementation-defined whether the source for module
>> interface units for modules on which the current translation unit has an
>> interface dependency (10.7.3) is required to be available".
> This is a concerning clause for us, is it expected to land in the
> final standard version?

I'm not sure this addition really has much significance. In one sense,
it just describes reality: if a given tool doesn't need the source, then
the source need not be present. It would be strange to add wording like
"An implementation must diagnose the absence of source for module
interface units" if the implementation doesn't actually need the source
to be present. Some pre-compiled header implementations don't require
header sources to be available, and standard library headers need not
exist outside of compiler internals at all, so this addition doesn't
really represent a change from status quo.

>
>> Historically, we've taken the individual tool configuration approach for
>> support of header files and, despite limitations, it has sufficed. However,
>> modules changes one critical aspect of such configuration. Previously,
>> header files needed to be consumable with the same set of include
>> paths and macro definitions as is used for the primary source file.
>> Translating module interface unit source code may require different,
>> even conflicting, include paths and macro definitions. Thus,
>> configuration will become more challenging.
>> <...>
>> I think we need an industry standard, tool agnostic solution that works
>> for common environments
>> <...>
>> A set of requirements for translating the module interface unit source code
>> (for one or more variations or build modes). This includes preprocessor
>> information (include paths, macro definitions, macro undefinitions), and,
>> potentially, language dialect requirements (specified in a generic form and,
>> perhaps, with the ability to customize for specific tools).
> A tool/IDE/frontend would probably:
> 1. take a module source code
> 2. translate it into its internal representation (using the provided
> compiler options - as it's already does for normal translation units)
> 3. After that it becomes mostly compiler- and environment-
> independent, and could be used to analyse the primary source file (or
> another module).

Agreed.

> I'm not sure if making step 2 compiler-independent would change much currently.

Indeed, and that isn't what I was suggesting. My suggestion was that
there be a tool independent way of specifying requirements for
translation. E.g., what macros must (not) be defined, what include
paths to use, what language dialect options are required (think of
options that don't affect ABI, like Microsoft's /source-charset, /Za,
/Zc, and /Ze options).

>
> I don't immediately see how this problem (a tool has to be aware of
> the used compiler's flags and quirks) is specific for modules, plain
> old TUs also lack this information (which should be extracted from
> compiler flags, which is sometimes non-trivial). I've submitted a mail
> recently about this
> (http://www.open-std.org/pipermail/tooling/2018-August/000148.html),
> admittedly, a bit chaotic.

The assumption of extracting information from a compiler applies to
some, but not all, tools. For example, vim users may need to add
include paths to their .vimrc file. I agree that this is certainly not
a new problem. What is new with modules is that the set of preprocessor
information and/or dialect options needed to translate the primary
source file vs imported modules may now differ. In other words, a
single source of preprocessor and language options no longer
(necessarily) suffices.

>
> So, TLDR:
> 1. There should be a way to get original source files for each module
> used in a compilation (at least initially, to allow the tooling to
> catch up quickly).

Yes. Or rather, there must be a way to do so, otherwise tools (that are
unable to consume some precompiled artifact) can't successfully parse
the primary source file.

> 2. Given that's true, implementing module support in tools/IDEs is
> probably not that much different from the existing TUs support,
> regardless of the way modules are implemented and integrated into the
> build process (not considering the actual language changes: name
> lookup rules, etc, which have to be made anyway, and would arrive for
> free to the tools backed by a modern compiler frontend)

It isn't *much* different, but I think it is different in critical ways
as discussed above.

> 3. Simplifying of the tools <-> build systems/compilers communications
> is very welcome, but is not unique to the modules support, and is not
> a blocker for the modules support.

Agreed, though I think this is tangential. I see build system/compiler
integration as an optimization opportunity distinct from module
support. What is a blocker for modules support in tooling is if tools
*must* be integrated with a build system/compiler in order to resolve
module imports. That is the potential future I want to avoid.

Tom.