Chuanqi,

At the end of you last message you mentioned that solving the issues you
raised means building a new build system, which is something
standardisation does not do - I agree.

However, if common practices do not solve the problem, we have to find a
way that does.

I think that all the questions below are answered with a tool that does
exactly this:
When queried with compilation flags it returns all the required
information: BMIs, sources, if the BMIs match the source or not, and in
general all the information the compiler needs.


Ran.



On Thu, Dec 7, 2023, 04:10 Chuanqi Xu via SG15 <sg15_at_[hidden]>
wrote:

> Hi Tom,
>
> > My motivation is that I would still like to be able compile (and
> not link) simple example/test code without having to use a build system.
>
> My main concern is that how can the compiler identify the invocation
> from build systems and directly invoked from humans for **small** projects.
> Since the compiler has no idea about what is small project or not.
>
> > That complexity would still have to be present in the driver. The only
> difference is whether an object file is written in a temporary location and
> then deleted vs writing that object file to a known location and then
> stopping.
>
> In case we would love to do the same thing when writing the object to a
> known location, we need to handle/answer the following things:
> (1) Where should be the position for the produced BMI of the std module?
> (2) Where should be the position of the produced object file of the std
> module?
> (3) Do we want to share the BMIs and object files with other invocations
> in the same system?
> (4) If (3) is yes, how can we find a BMI and the object file that match
> the flags and compilers we're using. For example, a BMI for std module
> produced with `-std=c++23` can't be consumed with an invocation with
> `-std=c++20`. Also a BMI produced with clang18 can't be consumed with an
> invocation with clang19. And then how can we record that? This should not
> be an easy problem.
> (5) Once the `std.cppm` and the corresponding header files changes, how
> can we know that to remove the existing BMIs and object files.
>
> If we can answer all of the issues, I think we're inventing a new tiny
> build systems. But this is not our goal.
>
> Thanks,
> Chuanqi
>
>
> Thanks,
> Chuanqi
>
> ------------------------------------------------------------------
> From:Tom Honermann <tom_at_[hidden]>
> Send Time:2023 Dec. 7 (Thu.) 00:44
> To:Chuanqi <chuanqi.xcq_at_[hidden]>; SG15 <sg15_at_[hidden]>
> Subject:Re: [SG15] [Modules] Should the compiler try to build std module
> implicitly?
>
> On 12/5/23 9:59 PM, Chuanqi Xu wrote:
> Hi Tom,
>
> > I agree with this conception of a small project. Both of the above
> examples generate an executable, but I'm assuming you don't intend that to
> be a requirement. I would expect the std module to be built if needed for
> an invocation that generates a static or shared library or a single object
> file as well.
>
> I intended to make that a requirement. It looks incorrect to me to extend
> this for an invocation that generates single objects. Since the output of
> build systems are basically a list of invocations that generating single
> objects. Then we're stepping on the toes of build systems. I don't like to
> do that since it adds the complexity in the compiler side which should be
> done in the build systems side.
>
> I'm not following. That complexity would still have to be present in the
> driver. The only difference is whether an object file is written in a
> temporary location and then deleted vs writing that object file to a known
> location and then stopping.
>
> It doesn't make sense to me that I could use the driver for this:
>
> - clang hello-world.cpp -o hello-world
>
> but not for this:
>
> - clang -c hello-world.cpp -o hello-world.o
>
>
> But now I think we can do that if the output of the invocation is in text
> form to read (e.g., `-S`, `-S -emit-llvm`) so that the output won't be used
> to produce other things further more. The motivation comes from supporting
> std module in compiler-explorer:
> https://github.com/compiler-explorer/compiler-explorer/issues/5404.
>
> I see no reason to limit this to just textual output. Support for Compiler
> Explorer is just one use case.
>
> My motivation is that I would still like to be able compile (and not link)
> simple example/test code without having to use a build system.
>
> Tom.
>
> Thanks,
> Chuanqi
>
>
> Thanks,
> Chuanqi
>
> ------------------------------------------------------------------
> From:Tom Honermann <tom_at_[hidden]> <tom_at_[hidden]>
> Send Time:2023 Dec. 6 (Wed.) 10:45
> To: SG15 <sg15_at_[hidden]> <sg15_at_[hidden]>; Ran Regev
> <regev.ran_at_[hidden]> <regev.ran_at_[hidden]>
> Cc:Chuanqi <chuanqi.xcq_at_[hidden]> <chuanqi.xcq_at_[hidden]>
> Subject:Re: [SG15] [Modules] Should the compiler try to build std module
> implicitly?
>
> On 12/4/23 9:43 PM, Chuanqi Xu via SG15 wrote:
> Hi,
>
> It looks like we're interested in the direction to provide a simpler
> command line interfaces for "small" projects. I'll try to summarize this in
> a paper later.
>
> A key point here is definition of "small" projects. I'll state my idea in
> the later section of the mail.
>
> > If this can be done without too much difficulty, great.
> >
> > However, would you expect this to work for both -stdlib=libc++ and
> -stdlib=libstdc++? How about when Clang is used with Microsoft's standard
> library (for which there is no corresponding -stdliboption)?
>
> If you were asking me (instead of the ideas of *wrapper*), I think it
> should be easy to find the locations of std module units according to the
> flags and platforms. Then it is an open question that if clang understands
> how to compile the std module from libstdc++ and MSVC. Since we don't know
> that with build systems yet. But from my experience in implementing a fake
> std module in libstdc++, it won't be a problem for the std module in
> libstdc++ if they follows the same style as libc++ did.
> I guess it depends on how complicated the standard library implementations
> get. They can become arbitrarily complicated of course. But if the metadata
> we've been discussing is provided, then a minimal built in build system
> should be able to identify the source files to be compiled, scan them, and
> then build the BMIs.
>
> > I think a **wrapper** around the compiler that generates the
> dependencies and builds the needed BMI might be a more promising and better
> route.
>
> Good idea. It reduces the work to implement the strategy I described. It
> should be easier to implement this within a yet-another clang tools. It may
> be problematic to implement this in a shell scripts or python scripts.
> Since we have to implement a tiny driver that way and it is also
> problematic to find the compiler and clang-scan-deps.
>
> ---
>
> Hi Wyatt, I feel your reply may beyond the scope of the motivation of the
> post a little bit. I don't talk about building modules implicitly nor
> standardizing the GCC's module mapper. All I want to talk about is the std
> module. I don't want to generalize it too much. Otherwise the compiler are
> trying to do the jobs of build systems.
>
> For the definition of "small" project mentioned above. In my mind, the
> "small" project should be able to be compiled within a single command line.
>
> e.g.,
>
> ```
> clang++ -std=c++20 hello.cpp -o hello
> ```
>
> or if there are several input:
>
> ```
> clang++ -std=c++20 a.cpp b.cpp c.cpp -o a.out
> ```
>
> Then if a project is not suitable to be compiled in this way, please reach
> out build systems. This is the model in my mind.
>
> I agree with this conception of a small project. Both of the above
> examples generate an executable, but I'm assuming you don't intend that to
> be a requirement. I would expect the std module to be built if needed for
> an invocation that generates a static or shared library or a single object
> file as well.
>
> Tom.
>
> Thanks,
> Chuanqi
>
>
> Thanks,
> Chuanqi
>
> ------------------------------------------------------------------
> From: SG15 <sg15_at_[hidden]> <sg15_at_[hidden]>
> Send Time:2023 Dec. 5 (Tue.) 03:01
> To:Ran Regev <regev.ran_at_[hidden]> <regev.ran_at_[hidden]>; SG15
> <sg15_at_[hidden]> <sg15_at_[hidden]>
> Cc:Tom Honermann <tom_at_[hidden]> <tom_at_[hidden]>
> Subject:Re: [SG15] [Modules] Should the compiler try to build std module
> implicitly?
>
> On 12/4/23 12:43 PM, Ran Regev wrote:
>
>
> On Mon, Dec 4, 2023, 19:31 Tom Honermann via SG15 <sg15_at_[hidden]>
> wrote:
>
> If this can be done without too much difficulty, great.
>
> However, would you expect this to work for both -stdlib=libc++ and
> -stdlib=libstdc++? How about when Clang is used with Microsoft's standard
> library (for which there is no corresponding -stdlib option)?
>
> I think defaults should be used when no value is supplied. Like, -O0 is
> the default in all compilers unless otherwise specified.
>
> The same for default std lib and any other parameter.
>
> The target is to enable portable:
> # std-tool -o hello hello.cpp
>
> Even when hello.cpp has import std; directive inside it.
>
> Sure, but the default for Clang differs based on platform. So support for
> Clang in general would require support for -std=libc++ (the default on
> Apple), -stdlib=libstdc++ (the default on Linux), and the MSVC standard
> library (the default for the *-windows-msvc target triples). Thus, Clang
> (or a wrapper) would presumably need to be taught how to build the BMIs for
> each (supported) version of each implementation. A wrapper tool would need
> to know how to identify which standard library implementation (and version)
> the compiler expects (preferably via some form of compiler introspection).
>
> Tom.
>
>
> Ran.
>
> Tom.
> On 12/4/23 1:00 AM, Chuanqi Xu via SG15 wrote:
> Hi Guys,
>
> This post is mainly about some random thoughts about teachability of
> std modules. Also [P2412R0: Minimal module support for the standard
> library] (
> https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p2412r0.pdf)
> mentions that the std module is important for educations and beginners. If
> there are more interests, I can try to summarize them into a paper.
>
> When I start to learn C++, I only need to copy a hello world example
> and run:
> ```
> clang++ hello.cpp -o hello
> ```
>
> However, with the current direction of std modules, the beginner need
> to install the build systems and try to copy the build scripts that them
> can hardly understand. Possibly some simpler form of
> https://libcxx.llvm.org/Modules.html. But it still seems scaring to
> beginners.
>
> And I am wondering if we can simplify the process. For example, the
> beginners can compile a hello world example with std module:
> ```
> import std;
> int main() { std::cout << "Hello modular world\n"; }
> ```
> with a single command line:
> ```
> clang++ -std=c++23 hello.cpp -o hello
> ```
>
> The compiler may achieve that when:
> - the std module is required but not provided.
> - the path to std module's BMI is not specified. (implied that the
> invocation doesn't from build systems)
> - the final output is an executable.
> - the std module's source exists in the installed path (being discussed in
> https://github.com/llvm/llvm-project/issues/73089).
>
> then the compiler can try to compile the std module's source to BMI and
> object file. Then we can import the BMI and link the object file. There are
> some details. e.g., where should we put the BMI? Should we try to reuse the
> BMIs? Can we extend the process to other modules? They are open questions
> and my answers are:
> - By default, they should be in `/tmp` and if `-save-temps` is specified,
> they'll be in the same directory with the temporaries.
> - The compiler shouldn't try to reuse the BMIs. That is the job of the
> build systems.
> - No, we can't. It is possible for std module since we'll try to
> standardize the locations of std modules sources. So that the tools are
> able to find the source of std modules. But it is not the case for other
> generalized modules. Also I think this is the job of build systems too.
>
> The motivation is primarily for beginners and educations. How do you feel
> about the idea?
>
> Thanks,
> Chuanqi
>
>
> _______________________________________________
> SG15 mailing listSG15_at_[hidden]://lists.isocpp.org/mailman/listinfo.cgi/sg15
>
> _______________________________________________
> SG15 mailing list
> SG15_at_[hidden]
> https://lists.isocpp.org/mailman/listinfo.cgi/sg15
>
> _______________________________________________
> SG15 mailing listSG15_at_[hidden]://lists.isocpp.org/mailman/listinfo.cgi/sg15
>
> _______________________________________________
> SG15 mailing list
> SG15_at_[hidden]
> https://lists.isocpp.org/mailman/listinfo.cgi/sg15
>