Date: Thu, 07 Dec 2023 16:30:11 +0800
Hi Ran,
> I think that all the questions below are answered with a tool that does exactly this:When queried with compilation flags it returns all the required information: BMIs, sources, if the BMIs match the source or not, and in general all the information the compiler needs.
Such a tool sounds wonderful. But what do we need to do to achieve that? We need a format to present the structure of modules. e.g, where is the source for module `x`?
Also we need another format to present the result you described as `in general all the information the compiler needs`. To achieve this, we need a scanner to process the source files. Also we need some form to track whether or not the BMIs are out-of-date due to the input files changes.
I feel like such a tool is more and more like a build system... The format to record the structure of modules sounds like a configuration file of a build system. Also another format to present everything sounds like the internal data structures in the build systems... Then I just feel we're talking about standardizing such things that meant to be implementation details of build system vendors...
Thanks,
Chuanqi
Thanks,
Chuanqi
------------------------------------------------------------------
From:Ran Regev <regev.ran_at_[hidden]>
Send Time:2023 Dec. 7 (Thu.) 15:44
To: SG15 <sg15_at_[hidden]>
Cc:Tom Honermann <tom_at_honermann.net>; Chuanqi <chuanqi.xcq_at_[hidden]>
Subject:Re: [SG15] [Modules] Should the compiler try to build std module implicitly?
Chuanqi,
At the end of you last message you mentioned that solving the issues you raised means building a new build system, which is something standardisation does not do - I agree.
However, if common practices do not solve the problem, we have to find a way that does.
I think that all the questions below are answered with a tool that does exactly this:
When queried with compilation flags it returns all the required information: BMIs, sources, if the BMIs match the source or not, and in general all the information the compiler needs.
Ran.
On Thu, Dec 7, 2023, 04:10 Chuanqi Xu via SG15 <sg15_at_[hidden] <mailto:sg15_at_[hidden] >> wrote:
Hi Tom,
> My motivation is that I would still like to be able compile (and not link) simple example/test code without having to use a build system.
My main concern is that how can the compiler identify the invocation from build systems and directly invoked from humans for **small** projects. Since the compiler has no idea about what is small project or not.
> That complexity would still have to be present in the driver. The only difference is whether an object file is written in a temporary location and then deleted vs writing that object file to a known location and then stopping.
In case we would love to do the same thing when writing the object to a known location, we need to handle/answer the following things:
(1) Where should be the position for the produced BMI of the std module?
(2) Where should be the position of the produced object file of the std module?
(3) Do we want to share the BMIs and object files with other invocations in the same system?
(4) If (3) is yes, how can we find a BMI and the object file that match the flags and compilers we're using. For example, a BMI for std module produced with `-std=c++23` can't be consumed with an invocation with `-std=c++20`. Also a BMI produced with clang18 can't be consumed with an invocation with clang19. And then how can we record that? This should not be an easy problem.
(5) Once the `std.cppm` and the corresponding header files changes, how can we know that to remove the existing BMIs and object files.
If we can answer all of the issues, I think we're inventing a new tiny build systems. But this is not our goal.
Thanks,
Chuanqi
Thanks,
Chuanqi
------------------------------------------------------------------
From:Tom Honermann <tom_at_honermann.net <mailto:tom_at_[hidden] >>
Send Time:2023 Dec. 7 (Thu.) 00:44
To:Chuanqi <chuanqi.xcq_at_[hidden]om <mailto:chuanqi.xcq_at_[hidden] >>; SG15 <sg15_at_[hidden] <mailto:sg15_at_[hidden] >>
Subject:Re: [SG15] [Modules] Should the compiler try to build std module implicitly?
On 12/5/23 9:59 PM, Chuanqi Xu wrote:
Hi Tom,
> I agree with this conception of a small project. Both of the above examples generate an executable, but I'm assuming you don't intend that to be a requirement. I would expect the std module to be built if needed for an invocation that generates a static or shared library or a single object file as well.
I intended to make that a requirement. It looks incorrect to me to extend this for an invocation that generates single objects. Since the output of build systems are basically a list of invocations that generating single objects. Then we're stepping on the toes of build systems. I don't like to do that since it adds the complexity in the compiler side which should be done in the build systems side.
I'm not following. That complexity would still have to be present in the driver. The only difference is whether an object file is written in a temporary location and then deleted vs writing that object file to a known location and then stopping.
It doesn't make sense to me that I could use the driver for this:
* clang hello-world.cpp -o hello-world
but not for this:
* clang -c hello-world.cpp -o hello-world.o
But now I think we can do that if the output of the invocation is in text form to read (e.g., `-S`, `-S -emit-llvm`) so that the output won't be used to produce other things further more. The motivation comes from supporting std module in compiler-explorer: https://github.com/compiler-explorer/compiler-explorer/issues/5404 <https://github.com/compiler-explorer/compiler-explorer/issues/5404 >.
I see no reason to limit this to just textual output. Support for Compiler Explorer is just one use case.
My motivation is that I would still like to be able compile (and not link) simple example/test code without having to use a build system.
Tom.
Thanks,
Chuanqi
Thanks,
Chuanqi
------------------------------------------------------------------
From:Tom Honermann <tom_at_[hidden]> <mailto:tom_at_[hidden] >
Send Time:2023 Dec. 6 (Wed.) 10:45
To: SG15 <sg15_at_[hidden]> <mailto:sg15_at_[hidden] >; Ran Regev <regev.ran_at_[hidden]> <mailto:regev.ran_at_[hidden] >
Cc:Chuanqi <chuanqi.xcq_at_[hidden]> <mailto:chuanqi.xcq_at_[hidden] >
Subject:Re: [SG15] [Modules] Should the compiler try to build std module implicitly?
On 12/4/23 9:43 PM, Chuanqi Xu via SG15 wrote:
Hi,
It looks like we're interested in the direction to provide a simpler command line interfaces for "small" projects. I'll try to summarize this in a paper later.
A key point here is definition of "small" projects. I'll state my idea in the later section of the mail.
> If this can be done without too much difficulty, great.
>
> However, would you expect this to work for both -stdlib=libc++ and -stdlib=libstdc++? How about when Clang is used with Microsoft's standard library (for which there is no corresponding -stdliboption)?
If you were asking me (instead of the ideas of *wrapper*), I think it should be easy to find the locations of std module units according to the flags and platforms. Then it is an open question that if clang understands how to compile the std module from libstdc++ and MSVC. Since we don't know that with build systems yet. But from my experience in implementing a fake std module in libstdc++, it won't be a problem for the std module in libstdc++ if they follows the same style as libc++ did.
I guess it depends on how complicated the standard library implementations get. They can become arbitrarily complicated of course. But if the metadata we've been discussing is provided, then a minimal built in build system should be able to identify the source files to be compiled, scan them, and then build the BMIs.
> I think a *wrapper* around the compiler that generates the dependencies and builds the needed BMI might be a more promising and better route.
Good idea. It reduces the work to implement the strategy I described. It should be easier to implement this within a yet-another clang tools. It may be problematic to implement this in a shell scripts or python scripts. Since we have to implement a tiny driver that way and it is also problematic to find the compiler and clang-scan-deps.
> I think that all the questions below are answered with a tool that does exactly this:When queried with compilation flags it returns all the required information: BMIs, sources, if the BMIs match the source or not, and in general all the information the compiler needs.
Such a tool sounds wonderful. But what do we need to do to achieve that? We need a format to present the structure of modules. e.g, where is the source for module `x`?
Also we need another format to present the result you described as `in general all the information the compiler needs`. To achieve this, we need a scanner to process the source files. Also we need some form to track whether or not the BMIs are out-of-date due to the input files changes.
I feel like such a tool is more and more like a build system... The format to record the structure of modules sounds like a configuration file of a build system. Also another format to present everything sounds like the internal data structures in the build systems... Then I just feel we're talking about standardizing such things that meant to be implementation details of build system vendors...
Thanks,
Chuanqi
Thanks,
Chuanqi
------------------------------------------------------------------
From:Ran Regev <regev.ran_at_[hidden]>
Send Time:2023 Dec. 7 (Thu.) 15:44
To: SG15 <sg15_at_[hidden]>
Cc:Tom Honermann <tom_at_honermann.net>; Chuanqi <chuanqi.xcq_at_[hidden]>
Subject:Re: [SG15] [Modules] Should the compiler try to build std module implicitly?
Chuanqi,
At the end of you last message you mentioned that solving the issues you raised means building a new build system, which is something standardisation does not do - I agree.
However, if common practices do not solve the problem, we have to find a way that does.
I think that all the questions below are answered with a tool that does exactly this:
When queried with compilation flags it returns all the required information: BMIs, sources, if the BMIs match the source or not, and in general all the information the compiler needs.
Ran.
On Thu, Dec 7, 2023, 04:10 Chuanqi Xu via SG15 <sg15_at_[hidden] <mailto:sg15_at_[hidden] >> wrote:
Hi Tom,
> My motivation is that I would still like to be able compile (and not link) simple example/test code without having to use a build system.
My main concern is that how can the compiler identify the invocation from build systems and directly invoked from humans for **small** projects. Since the compiler has no idea about what is small project or not.
> That complexity would still have to be present in the driver. The only difference is whether an object file is written in a temporary location and then deleted vs writing that object file to a known location and then stopping.
In case we would love to do the same thing when writing the object to a known location, we need to handle/answer the following things:
(1) Where should be the position for the produced BMI of the std module?
(2) Where should be the position of the produced object file of the std module?
(3) Do we want to share the BMIs and object files with other invocations in the same system?
(4) If (3) is yes, how can we find a BMI and the object file that match the flags and compilers we're using. For example, a BMI for std module produced with `-std=c++23` can't be consumed with an invocation with `-std=c++20`. Also a BMI produced with clang18 can't be consumed with an invocation with clang19. And then how can we record that? This should not be an easy problem.
(5) Once the `std.cppm` and the corresponding header files changes, how can we know that to remove the existing BMIs and object files.
If we can answer all of the issues, I think we're inventing a new tiny build systems. But this is not our goal.
Thanks,
Chuanqi
Thanks,
Chuanqi
------------------------------------------------------------------
From:Tom Honermann <tom_at_honermann.net <mailto:tom_at_[hidden] >>
Send Time:2023 Dec. 7 (Thu.) 00:44
To:Chuanqi <chuanqi.xcq_at_[hidden]om <mailto:chuanqi.xcq_at_[hidden] >>; SG15 <sg15_at_[hidden] <mailto:sg15_at_[hidden] >>
Subject:Re: [SG15] [Modules] Should the compiler try to build std module implicitly?
On 12/5/23 9:59 PM, Chuanqi Xu wrote:
Hi Tom,
> I agree with this conception of a small project. Both of the above examples generate an executable, but I'm assuming you don't intend that to be a requirement. I would expect the std module to be built if needed for an invocation that generates a static or shared library or a single object file as well.
I intended to make that a requirement. It looks incorrect to me to extend this for an invocation that generates single objects. Since the output of build systems are basically a list of invocations that generating single objects. Then we're stepping on the toes of build systems. I don't like to do that since it adds the complexity in the compiler side which should be done in the build systems side.
I'm not following. That complexity would still have to be present in the driver. The only difference is whether an object file is written in a temporary location and then deleted vs writing that object file to a known location and then stopping.
It doesn't make sense to me that I could use the driver for this:
* clang hello-world.cpp -o hello-world
but not for this:
* clang -c hello-world.cpp -o hello-world.o
But now I think we can do that if the output of the invocation is in text form to read (e.g., `-S`, `-S -emit-llvm`) so that the output won't be used to produce other things further more. The motivation comes from supporting std module in compiler-explorer: https://github.com/compiler-explorer/compiler-explorer/issues/5404 <https://github.com/compiler-explorer/compiler-explorer/issues/5404 >.
I see no reason to limit this to just textual output. Support for Compiler Explorer is just one use case.
My motivation is that I would still like to be able compile (and not link) simple example/test code without having to use a build system.
Tom.
Thanks,
Chuanqi
Thanks,
Chuanqi
------------------------------------------------------------------
From:Tom Honermann <tom_at_[hidden]> <mailto:tom_at_[hidden] >
Send Time:2023 Dec. 6 (Wed.) 10:45
To: SG15 <sg15_at_[hidden]> <mailto:sg15_at_[hidden] >; Ran Regev <regev.ran_at_[hidden]> <mailto:regev.ran_at_[hidden] >
Cc:Chuanqi <chuanqi.xcq_at_[hidden]> <mailto:chuanqi.xcq_at_[hidden] >
Subject:Re: [SG15] [Modules] Should the compiler try to build std module implicitly?
On 12/4/23 9:43 PM, Chuanqi Xu via SG15 wrote:
Hi,
It looks like we're interested in the direction to provide a simpler command line interfaces for "small" projects. I'll try to summarize this in a paper later.
A key point here is definition of "small" projects. I'll state my idea in the later section of the mail.
> If this can be done without too much difficulty, great.
>
> However, would you expect this to work for both -stdlib=libc++ and -stdlib=libstdc++? How about when Clang is used with Microsoft's standard library (for which there is no corresponding -stdliboption)?
If you were asking me (instead of the ideas of *wrapper*), I think it should be easy to find the locations of std module units according to the flags and platforms. Then it is an open question that if clang understands how to compile the std module from libstdc++ and MSVC. Since we don't know that with build systems yet. But from my experience in implementing a fake std module in libstdc++, it won't be a problem for the std module in libstdc++ if they follows the same style as libc++ did.
I guess it depends on how complicated the standard library implementations get. They can become arbitrarily complicated of course. But if the metadata we've been discussing is provided, then a minimal built in build system should be able to identify the source files to be compiled, scan them, and then build the BMIs.
> I think a *wrapper* around the compiler that generates the dependencies and builds the needed BMI might be a more promising and better route.
Good idea. It reduces the work to implement the strategy I described. It should be easier to implement this within a yet-another clang tools. It may be problematic to implement this in a shell scripts or python scripts. Since we have to implement a tiny driver that way and it is also problematic to find the compiler and clang-scan-deps.
--- Hi Wyatt, I feel your reply may beyond the scope of the motivation of the post a little bit. I don't talk about building modules implicitly nor standardizing the GCC's module mapper. All I want to talk about is the std module. I don't want to generalize it too much. Otherwise the compiler are trying to do the jobs of build systems. For the definition of "small" project mentioned above. In my mind, the "small" project should be able to be compiled within a single command line. e.g., ``` clang++ -std=c++20 hello.cpp -o hello ``` or if there are several input: ``` clang++ -std=c++20 a.cpp b.cpp c.cpp -o a.out ``` Then if a project is not suitable to be compiled in this way, please reach out build systems. This is the model in my mind. I agree with this conception of a small project. Both of the above examples generate an executable, but I'm assuming you don't intend that to be a requirement. I would expect the std module to be built if needed for an invocation that generates a static or shared library or a single object file as well. Tom. Thanks, Chuanqi Thanks, Chuanqi ------------------------------------------------------------------ From: SG15 <sg15_at_[hidden]> <mailto:sg15_at_lists.isocpp.org > Send Time:2023 Dec. 5 (Tue.) 03:01 To:Ran Regev <regev.ran_at_[hidden]> <mailto:regev.ran_at_[hidden] >; SG15 <sg15_at_[hidden]> <mailto:sg15_at_[hidden] > Cc:Tom Honermann <tom_at_[hidden]> <mailto:tom_at_honermann.net > Subject:Re: [SG15] [Modules] Should the compiler try to build std module implicitly? On 12/4/23 12:43 PM, Ran Regev wrote: On Mon, Dec 4, 2023, 19:31 Tom Honermann via SG15 <sg15_at_[hidden] <mailto:sg15_at_[hidden] >> wrote: If this can be done without too much difficulty, great. However, would you expect this to work for both -stdlib=libc++ and -stdlib=libstdc++? How about when Clang is used with Microsoft's standard library (for which there is no corresponding -stdlib option)? I think defaults should be used when no value is supplied. Like, -O0 is the default in all compilers unless otherwise specified. The same for default std lib and any other parameter. The target is to enable portable: # std-tool -o hello hello.cpp Even when hello.cpp has import std; directive inside it. Sure, but the default for Clang differs based on platform. So support for Clang in general would require support for -std=libc++ (the default on Apple), -stdlib=libstdc++ (the default on Linux), and the MSVC standard library (the default for the *-windows-msvc target triples). Thus, Clang (or a wrapper) would presumably need to be taught how to build the BMIs for each (supported) version of each implementation. A wrapper tool would need to know how to identify which standard library implementation (and version) the compiler expects (preferably via some form of compiler introspection). Tom. Ran. Tom. On 12/4/23 1:00 AM, Chuanqi Xu via SG15 wrote: Hi Guys, This post is mainly about some random thoughts about teachability of std modules. Also [P2412R0: Minimal module support for the standard library] (https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p2412r0.pdf <https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p2412r0.pdf >) mentions that the std module is important for educations and beginners. If there are more interests, I can try to summarize them into a paper. When I start to learn C++, I only need to copy a hello world example and run: ``` clang++ hello.cpp -o hello ``` However, with the current direction of std modules, the beginner need to install the build systems and try to copy the build scripts that them can hardly understand. Possibly some simpler form of https://libcxx.llvm.org/Modules.html <https://libcxx.llvm.org/Modules.html >. But it still seems scaring to beginners. And I am wondering if we can simplify the process. For example, the beginners can compile a hello world example with std module: ``` import std; int main() { std::cout << "Hello modular world\n"; } ``` with a single command line: ``` clang++ -std=c++23 hello.cpp -o hello ``` The compiler may achieve that when: - the std module is required but not provided. - the path to std module's BMI is not specified. (implied that the invocation doesn't from build systems) - the final output is an executable. - the std module's source exists in the installed path (being discussed in https://github.com/llvm/llvm-project/issues/73089 <https://github.com/llvm/llvm-project/issues/73089 >). then the compiler can try to compile the std module's source to BMI and object file. Then we can import the BMI and link the object file. There are some details. e.g., where should we put the BMI? Should we try to reuse the BMIs? Can we extend the process to other modules? They are open questions and my answers are: - By default, they should be in `/tmp` and if `-save-temps` is specified, they'll be in the same directory with the temporaries. - The compiler shouldn't try to reuse the BMIs. That is the job of the build systems. - No, we can't. It is possible for std module since we'll try to standardize the locations of std modules sources. So that the tools are able to find the source of std modules. But it is not the case for other generalized modules. Also I think this is the job of build systems too. The motivation is primarily for beginners and educations. How do you feel about the idea? Thanks, Chuanqi _______________________________________________ SG15 mailing list SG15_at_[hidden] <mailto:SG15_at_[hidden] > https://lists.isocpp.org/mailman/listinfo.cgi/sg15 <https://lists.isocpp.org/mailman/listinfo.cgi/sg15 > _______________________________________________ SG15 mailing list SG15_at_[hidden] <mailto:SG15_at_lists.isocpp.org > https://lists.isocpp.org/mailman/listinfo.cgi/sg15 <https://lists.isocpp.org/mailman/listinfo.cgi/sg15 > _______________________________________________ SG15 mailing list SG15_at_[hidden] <mailto:SG15_at_[hidden] > https://lists.isocpp.org/mailman/listinfo.cgi/sg15 <https://lists.isocpp.org/mailman/listinfo.cgi/sg15 > _______________________________________________ SG15 mailing list SG15_at_[hidden] <mailto:SG15_at_lists.isocpp.org > https://lists.isocpp.org/mailman/listinfo.cgi/sg15 <https://lists.isocpp.org/mailman/listinfo.cgi/sg15 >
Received on 2023-12-07 08:30:17