How do we know where the "top" or "bottom" is, without pre-scanning the source files?
In the case of drop-in header-files to header units replacement, If an executable depends on a library, then the library and executable source files can be compiled in parallel. But the library source files can be preferred before the executable. Only if more threads are available, the executable source files will be compiled, which generally also means that the user has more RAM installed. This way when compiling the library, stl header-units and library header-units are compiled and their states are kept during the process. executable header units only come into the picture when executable source files are compiled. As soon as all source files of the library are compiled, private header-units of the library are cleared. If we had preferred executable source files over library source files or there were no preferences, then there will be more states that need to be kept, states of executable header-units plus states of library header-units plus states of stl header-units.
In the case of modules if an executable depends on library A that depends privately on library B. This means that module/s in exe can only import module/s in A but not in B. module/s of library A however can import module/s in B in its implementation-units. Order of preference is B > A > exe. Once all module files of A are built, the B module ifc files are cleared.
A correction in my above email:
I checked aggregated .obj file size in llvm cmake-build-debug directory. It is around 5GB. I think the maximum size of the ifc files needs to be kept around at any time will be around that as well.
Because of this, I feel on a machine with 32GB RAM and 12 threads, 2 llvm configurations can be compiled simultaneously without any problem. 12GB of ifc files + compiler states will exist in memory. The remaining 18GB will be available for 12 compiler threads. And before linking, all of these will be written back to the disk. If this takes more memory, the user will always have the option to fall back to the conventional solution.
In 2 phase model, I guess the ifc file size will be double as it also contains the object-file in it. So, only one configuration will be possible. But a simple trick would be to start building the second configuration after the first configuration. My build-system currently simultaneously starts building the configurations. This way any number could be built. Because by the time of the second, the memory of the first would be cleared.
I have done slight modifications and added comments about 2 phase model support.
https://github.com/HassanSajjad-302/HMake/wiki/HMake-Flaw-And-Its-Possible-Fixes#5Best,
Hassan Sajjad