>
> For deeply nested dependency graphs, that seems to mean that compilation
> state of many translation units is kept in memory while the respective
> compile waits for a dependency to be built. That feels like uncontrolled
> resource consumption and doesn't seem to work on small-ish machines
> (compared to the project to be built).
>

Yes. Memory consumption of such an approach will be more than the
traditional approach. But we can alleviate it in 2, 3 ways. If we start
from the bottom and call newCompile() for a source-file, it might depend on
a header-file which might depend on another header-file, and so on till the
standard header files. So, a lot of compiler-states will need to be kept.
But if start from the top, it can only depend on standard headers, hence
fewer compiler-states will need to be preserved. However, as we move down,
we will have more and more ifc files. As we compile we can free ifc file
resources if there is no other compilation-unit left that can depend on
that ifc. Linking can be moved to the end and before that, we can clear all
the ifc. I checked aggregated .obj file size in llvm cmake-build-debug
directory. It is around 5GB. I think the maximum size of the ifc files
needs to be kept around at any time will be around that as well.

Because of this, I feel on a machine with 32GB RAM and 12 threads, 2 llvm
configurations can be compiled simultaneously without any problem. 12GB of
ifc files + compiler states will exist in memory. The remaining 18GB will
be available for 12 compiler threads. And before linking, all of these will
be written back to the disk. If this takes more memory, the user will
always have the option to fall back to the conventional solution.

Best,
Hassan Sajjad

On Wed, Aug 16, 2023 at 11:01 AM Jens Maurer <jens.maurer_at_[hidden]> wrote:

>
>
> On 16/08/2023 01.54, Hassan Sajjad via SG15 wrote:
> > Please share your thoughts on this.
> >
> > Best,
> > Hassan Sajjad
> >
> > On Mon, Aug 14, 2023, 04:51 Hassan Sajjad <hassan.sajjad069_at_[hidden]
> <mailto:hassan.sajjad069_at_[hidden]>> wrote:
> >
> > Hi.
> >
> > Thank you so much for explaining the solution. It is a happy news.
> With this consensus adoption, my build-system achieves full standard
> compliance without any modifications required. This is however less optimal
> compared to solution 5 that I proposed. Depending on different factors, I
> conjecture that my solution will be up to *40%* faster compared to the
> current consensus. The compilation speed-up will be more visible in clean
> builds using C++20 modules / header-units. Here I explain the reasoning
> behind this: https://github.com/HassanSajjad-302/solution5 <
> https://github.com/HassanSajjad-302/solution5>.
>
> I've skimmed this. It seems you want to run the compiler as a library,
> intercept any loads of header and/or module files, and then kick off
> the compilation of those as a "subroutine".
>
> For deeply nested dependency graphs, that seems to mean that compilation
> state of many translation units is kept in memory while the respective
> compile waits for a dependency to be built. That feels like uncontrolled
> resource consumption and doesn't seem to work on small-ish machines
> (compared to the project to be built).
>
> Jens
>
>