>
> How do we know where the "top" or "bottom" is, without pre-scanning the
> source files?
>

In the case of drop-in header-files to header units replacement, If an
executable depends on a library, then the library and executable source
files can be compiled in parallel. But the library source files can be
preferred before the executable. Only if more threads are available, the
executable source files will be compiled, which generally also means that
the user has more RAM installed. This way when compiling the library, stl
header-units and library header-units are compiled and their states are
kept during the process. executable header units only come into the
picture when executable source files are compiled. As soon as all source
files of the library are compiled, private header-units of the library are
cleared. If we had preferred executable source files over library source
files or there were no preferences, then there will be more states that
need to be kept, states of executable header-units plus states of library
header-units plus states of stl header-units.

In the case of modules if an executable depends on library A that depends
privately on library B. This means that module/s in exe can only import
module/s in A but not in B. module/s of library A however can import
module/s in B in its implementation-units. Order of preference is B > A >
exe. Once all module files of A are built, the B module ifc files are
cleared.

A correction in my above email:

> I checked aggregated .obj file size in llvm cmake-build-debug directory.
> It is around 5GB. I think the maximum size of the ifc files needs to be
> kept around at any time will be around that as well.
>
> Because of this, I feel on a machine with 32GB RAM and 12 threads, 2 llvm
> configurations can be compiled simultaneously without any problem. 12GB of
> ifc files + compiler states will exist in memory. The remaining 18GB will
> be available for 12 compiler threads. And before linking, all of these will
> be written back to the disk. If this takes more memory, the user will
> always have the option to fall back to the conventional solution.
>

In 2 phase model, I guess the ifc file size will be double as it also
contains the object-file in it. So, only one configuration will be
possible. But a simple trick would be to start building the second
configuration after the first configuration. My build-system currently
simultaneously starts building the configurations. This way any number
could be built. Because by the time of the second, the memory of the first
would be cleared.

I have done slight modifications and added comments about 2 phase model
support.
https://github.com/HassanSajjad-302/HMake/wiki/HMake-Flaw-And-Its-Possible-Fixes#5

Best,
Hassan Sajjad


On Thu, Aug 17, 2023, 02:20 Jens Maurer <jens.maurer_at_[hidden]> wrote:

>
> On 16/08/2023 12.15, Hassan Sajjad wrote:
> > For deeply nested dependency graphs, that seems to mean that
> compilation
> > state of many translation units is kept in memory while the
> respective
> > compile waits for a dependency to be built. That feels like
> uncontrolled
> > resource consumption and doesn't seem to work on small-ish machines
> > (compared to the project to be built).
> >
> >
> > Yes. Memory consumption of such an approach will be more than the
> traditional approach. But we can alleviate it in 2, 3 ways. If we start
> from the bottom and call newCompile() for a source-file, it might depend on
> a header-file which might depend on another header-file, and so on till the
> standard header files. So, a lot of compiler-states will need to be kept.
> But if start from the top, it can only depend on standard headers, hence
> fewer compiler-states will need to be preserved. However, as we move down,
> we will have more and more ifc files.
>
> How do we know where the "top" or "bottom" is, without pre-scanning the
> source files?
>
> Jens
>
>
> > As we compile we can free ifc file resources if there is no other
> compilation-unit left that can depend on that ifc. Linking can be moved to
> the end and before that, we can clear all the ifc. I checked aggregated
> .obj file size in llvm cmake-build-debug directory. It is around 5GB. I
> think the maximum size of the ifc files needs to be kept around at any time
> will be around that as well.
> >
> > Because of this, I feel on a machine with 32GB RAM and 12 threads, 2
> llvm configurations can be compiled simultaneously without any problem.
> 12GB of ifc files + compiler states will exist in memory. The remaining
> 18GB will be available for 12 compiler threads. And before linking, all of
> these will be written back to the disk. If this takes more memory, the user
> will always have the option to fall back to the conventional solution.
> >
> > Best,
> > Hassan Sajjad
> >
> > On Wed, Aug 16, 2023 at 11:01 AM Jens Maurer <jens.maurer_at_[hidden]
> <mailto:jens.maurer_at_[hidden]>> wrote:
> >
> >
> >
> > On 16/08/2023 01.54, Hassan Sajjad via SG15 wrote:
> > > Please share your thoughts on this.
> > >
> > > Best,
> > > Hassan Sajjad
> > >
> > > On Mon, Aug 14, 2023, 04:51 Hassan Sajjad <
> hassan.sajjad069_at_[hidden] <mailto:hassan.sajjad069_at_[hidden]> <mailto:
> hassan.sajjad069_at_[hidden] <mailto:hassan.sajjad069_at_[hidden]>>> wrote:
> > >
> > > Hi.
> > >
> > > Thank you so much for explaining the solution. It is a happy
> news. With this consensus adoption, my build-system achieves full standard
> compliance without any modifications required. This is however less optimal
> compared to solution 5 that I proposed. Depending on different factors, I
> conjecture that my solution will be up to *40%* faster compared to the
> current consensus. The compilation speed-up will be more visible in clean
> builds using C++20 modules / header-units. Here I explain the reasoning
> behind this: https://github.com/HassanSajjad-302/solution5 <
> https://github.com/HassanSajjad-302/solution5> <
> https://github.com/HassanSajjad-302/solution5 <
> https://github.com/HassanSajjad-302/solution5>>.
> >
> > I've skimmed this. It seems you want to run the compiler as a
> library,
> > intercept any loads of header and/or module files, and then kick off
> > the compilation of those as a "subroutine".
> >
> > For deeply nested dependency graphs, that seems to mean that
> compilation
> > state of many translation units is kept in memory while the
> respective
> > compile waits for a dependency to be built. That feels like
> uncontrolled
> > resource consumption and doesn't seem to work on small-ish machines
> > (compared to the project to be built).
> >
> > Jens
> >
>