Hi all,

We keep treating the compiler as *end* target in the chain of tools to
build code. We are trying to prepare everything for it to make it be able
to compile the code.

But what if the compiler is not the last tool in the chain? What if we
could standardize a tool, that, for example, given a required module to
import, and the required compilation flags it needs to be build with,
simply returns the BMI for it? And if the tool can't find it it returns the
sources that the module can be built from.

Such a tool can be invoked by the compiler whenever there is a need to
import a module, any module, not just std.

Few advantages for such a tool:
- no need to scan the code and know which imports are required, which flags
were used, the compiler does it.
- file locations, detection of changes in sources, sensitivity to
compilation flags - all are left to the implementation of the tool. It is
QoI if it can manage the complexity or always return just the sources.


In the past the compiler used to invoke a few tools to accomplish its
mission, so it is not a new idea to run things "after" the compiler.



Ran.


On Tue, Dec 12, 2023, 05:29 Gabriel Dos Reis via SG15 <sg15_at_[hidden]>
wrote:

> Yes. Now get a roomful of experts define "small project" :-)
>
> -- Gaby
>
>
> ------------------------------
> *From:* SG15 <sg15-bounces_at_[hidden]> on behalf of Chuanqi Xu via
> SG15 <sg15_at_[hidden]>
> *Sent:* Monday, December 11, 2023 7:04:11 PM
> *To:* SG15 <sg15_at_[hidden]>
> *Cc:* Chuanqi Xu <chuanqi.xcq_at_[hidden]>; Tom Honermann <
> tom_at_[hidden]>
> *Subject:* Re: [SG15] [Modules] Should the compiler try to build std
> module implicitly?
>
> Hi Olga and Boris,
>
> The idea of making build systems acting more like a compiler is
> interesting. It reminds me of Rust. In the tutorial of rust, the appearance
> of `rustc` is really rare. All the examples invoke `cargo` simply. And I
> am wondering if we can make things more simpler. e.g., we can skip to
> provide all the source files (may with a special pattern) by treating all
> files in the workdir as source files. Also even if we can skip a module map
> for such simple tools. For example, we can assume athe source file `X.cppm`
> will declare a module `X`.
>
> Let's call the tool as std-tool or std-build. Then we're talking about the
> conventions in the tool. This won't a convention that require all tools to
> follow.
>
> The goal of the tool should be helpful to beginners and small projects. It
> should not be a goal to take the job of cmake for complex and big projects.
>
> Thanks,
> Chuanqi
>
>
> Thanks,
> Chuanqi
>
> ------------------------------------------------------------------
> From: SG15 <sg15_at_[hidden]>
> Send Time:2023 Dec. 12 (Tue.) 10:26
> To: SG15 <sg15_at_[hidden]>
> Cc:Olga Arkhipova <olgaark_at_[hidden]>; Tom Honermann <
> tom_at_[hidden]>
> Subject:Re: [SG15] [Modules] Should the compiler try to build std module
> implicitly?
>
> >> Perhaps we are approaching this from the wrong end? Maybe instead of
> the compiler acting more and more like a build system we can make the build
> system act more like a compiler? That is, a build system's driver can
> support the "direct compilation" mode that mimics the compiler invocation
> while under the hood taking care of the above "build system" issues (which
> would naturally be quite easy).
>
> I think it can be even simpler - what if there is a "simple cpp build"
> exe which would be able to build cpp sources with modules by invoking the
> compiler (or whatever other tools) multiple times like build does?
>
> At some point I wrote an experimental command line tool to be used instead
> of compiler for simple builds to provide similar experience people have
> today with direct invocation of the compiler, something like
>
> cppbuild [tool switches] sources=module.ixx;source.cpp [compiler
> switches]
>
> The command line of this tool might be a little different from the real
> compiler so the tool does not need to know about all compiler switches to
> be able to identify the sources, but I think it is simple enough.
>
> The tool invokes the compiler to scan the sources, constructs the
> dependency graph and builds all given c++ sources and header units.
> The idea also was to allow some extensibility to be able to call any
> compiler assuming this compiler (or related tools) can produce standardized
> scan data, build a BMI and have a way pass those BMIs as references. At the
> time of my experimentations only MSVC had modules support, but I believe
> such extensibility is possible. If we have common/standardized compiler
> features (and or mapping to compiler specific switches) it will make this
> extensibility much easier.
>
> I don't think it would be a big problem extending such a tool to build std
> modules as soon as it knows where to find them (say, via standardized
> library manifest with well-known location).
> And if the tool can work with library manifests, it means that it will
> work with other libraries too (as soon as it can find their manifests),
> which I guess is a plus.
>
> It is a good question who will own/maintain such a tool and how people get
> it.
> I think the most logical is to include it into compiler toolset, but it
> could be other options as well.
>
> Thanks,
> Olga
>
>
> -----Original Message-----
> From: SG15 <sg15-bounces_at_[hidden]> On Behalf Of Boris Kolpackov
> via SG15
> Sent: Monday, December 11, 2023 5:00 AM
> To: Chuanqi Xu via SG15 <sg15_at_[hidden]>
> Cc: Boris Kolpackov <boris_at_[hidden]>; Tom Honermann <
> tom_at_[hidden]>
> Subject: Re: [SG15] [Modules] Should the compiler try to build std module
> implicitly?
>
> Chuanqi Xu via SG15 <sg15_at_[hidden]> writes:
>
> > In case we would love to do the same thing when writing the object to
> > a known location, we need to handle/answer the following things:
> >
> > (1) Where should be the position for the produced BMI of the std module?
> >
> > (2) Where should be the position of the produced object file of the std
> > module?
> >
> > (3) Do we want to share the BMIs and object files with other invocations
> > in the same system?
> >
> > (4) If (3) is yes, how can we find a BMI and the object file that match
> the
> > flags and compilers we're using. For example, a BMI for std module
> > produced with `-std=c++23` can't be consumed with an invocation with
> > `-std=c++20`. Also a BMI produced with clang18 can't be consumed
> with an
> > invocation with clang19. And then how can we record that? This should
> > not be an easy problem.
> >
> > (5) Once the `std.cppm` and the corresponding header files changes, how
> > can we know that to remove the existing BMIs and object files.
> >
> > If we can answer all of the issues, I think we're inventing a new tiny
> > build systems. But this is not our goal.
>
> Agree and would also add that if you choose to share BMIs, you will also
> have to deal with concurrent invocations/file locking.
>
> Perhaps we are approaching this from the wrong end? Maybe instead of the
> compiler acting more and more like a build system we can make the build
> system act more like a compiler? That is, a build system's driver can
> support the "direct compilation" mode that mimics the compiler invocation
> while under the hood taking care of the above "build system" issues (which
> would naturally be quite easy).
>
> Specifically, instead of, say:
>
> $ clang++ -c hello.cpp
>
> One would run (using the build2 driver as an example):
>
> $ b -m cxx clang++ -c hello.cpp
>
> (I am just riff'ing on the idea; here `-m cxx` means "ask the build system
> module that provides the C++ compilation support to perform direct
> compilation using the arguments that follow".)
>
> Some extra things that immediately follow from this:
>
>
> - Sensible default so you don't need to specify /EHsc /nologo or
> -fPIC manually.
>
>
> - We could support change tracking:
>
> $ b -m cxx clang++ hello.cpp impl.cpp
> $ touch hello.cpp
> $ b -m cxx clang++ hello.cpp impl.cpp # Only recompiles hello.cpp.
>
>
> - We could support additional operations, such as clean, test,
> install/uninstall:
>
> $ b clean -m cxx clang++ hello.cpp impl.cpp
>
>
> - We could invent an imaginary compile option for producing static
> libraries:
>
> $ b -m cxx clang++ hello.cpp # Executable.
> $ b -m cxx clang++ -shared hello.cpp # Shared library.
> $ b -m cxx clang++ -archive hello.cpp # Static library.
>
>
> In a sense it becomes a form of "build by convention" support where the
> convention is not how we lay source files on disk and name them but how we
> invoke the compiler.
>
> I could probably prototype something like this in build2 if there is
> interest.
> _______________________________________________
> SG15 mailing list
> SG15_at_[hidden]
> https://lists.isocpp.org/mailman/listinfo.cgi/sg15
> _______________________________________________
> SG15 mailing list
> SG15_at_[hidden]
> https://lists.isocpp.org/mailman/listinfo.cgi/sg15
>
> _______________________________________________
> SG15 mailing list
> SG15_at_[hidden]
> https://lists.isocpp.org/mailman/listinfo.cgi/sg15
>