Date: Fri, 29 Nov 2024 11:35:05 +0100
On Thu, Nov 28, 2024 at 11:37 AM Jussi Pakkanen via SG15 <
sg15_at_[hidden]> wrote:
> On Thu, 28 Nov 2024 at 10:06, Boris Kolpackov <boris_at_[hidden]>
> wrote:
>
> > What is the point of placing the output to this top-level target
> > directory rather than the per-TU directory as suggested by Mathias?
>
> If you have a project with 10k compilations then you need to create
> 10k extra subdirectories in the file system. This is both wasteful and
> taxing on slow file systems.
>
We are talking about compiling C++ here. Just to be clear, are you saying
that making a directory per TU is a significant overhead relative to
actually doing the compile? I could _maybe_ see that if you have 10k very
small TUs and you are using a remote filesystem from halfway around the
world. But in any realistic scenario, I would expect an mkdir to be well
under 1% or even 0.1% of the cost of the compile. Plus, you only need to
mkdir on clean builds. This should have no impact on incremental build
times.
If the problem is more about filesystems that do linear scans over
directories (are there any left?), A) Don't you already have a problem
dumping all .o files in the same directory? B) putting multiple outputs in
a subdir should reduce that by a factor of the number of outputs, C) You
could always nest them btree style with say 100 directories per level.
>
> > My feeling is so that the additional outputs are aggregated on the
> > per end-product basis so that, for example, one can easily find all
> > the SARIF files that describe a library build. If that's the case,
> > then creating a separate directory for utility libraries would break
> > this aggregation.
>
> It is in fact the exact opposite. Having the utility directory
> separate is _exactly what you want_, because then you can tell how
> many errors _that_ specific utility library has. This is important in
> case, for example, the utility library is maintained by team A whereas
> the rest of the project is maintained by project B. If you then want
> to aggregate everything together it is simple, because the build
> system already knows that the utility library is part of the whole
> build. It can even provide a hierarchical view of the project. If one
> lumps everything together then this information is lost and it is
> burdensome to separate it back out.
>
> There are other things you might want to group per build target rather
> than some higher level thing such as build timings.
>
> > I would expect this to be racy in most build systems. It would
> > definitely be in build2: depending on which executable-object
> > dependency caused the compilation, the additional outputs would
> > end up in one executable's top-level directory or another.
>
> I don't understand what you are trying to say here. Let's use an
> example instead for concreteness.
>
> Suppose we have source file foo.cc, two executables e1 and e2 and a
> utility library l.
>
> Foo.cc goes directly to e1 and also to l which is linked to e2.
>
> When foo.cc is compiled for e1, the extra files go to e1's private
> directory (it has to be compiled a second time as the flags might be
> different).
>
> When it is compiled for l, the extra files go to l's private directory.
>
> When linking, no extra files are created by the obj file (the target
> itself might, but it is unrelated).
>
> There is no race condition. There is no overlap. Every file goes to
> its own place with a unique name.
>
> There is one special case which is that the end user can tell (in
> Meson at least, dunno about other build systems) that they want to
> "steal" the object file from l and use it directly (not via the
> library) in target e3 at their own risk. Even in this case nothing
> changes. The extra files go in l's private dir because that is what
> actually builds the object file.
>
> > If this facility is to be added, I would suggest going with the
> > semantics proposed by Mathias: output directory per TU.
>
> I would suggest the opposite.
> _______________________________________________
> SG15 mailing list
> SG15_at_[hidden]
> https://lists.isocpp.org/mailman/listinfo.cgi/sg15
>
sg15_at_[hidden]> wrote:
> On Thu, 28 Nov 2024 at 10:06, Boris Kolpackov <boris_at_[hidden]>
> wrote:
>
> > What is the point of placing the output to this top-level target
> > directory rather than the per-TU directory as suggested by Mathias?
>
> If you have a project with 10k compilations then you need to create
> 10k extra subdirectories in the file system. This is both wasteful and
> taxing on slow file systems.
>
We are talking about compiling C++ here. Just to be clear, are you saying
that making a directory per TU is a significant overhead relative to
actually doing the compile? I could _maybe_ see that if you have 10k very
small TUs and you are using a remote filesystem from halfway around the
world. But in any realistic scenario, I would expect an mkdir to be well
under 1% or even 0.1% of the cost of the compile. Plus, you only need to
mkdir on clean builds. This should have no impact on incremental build
times.
If the problem is more about filesystems that do linear scans over
directories (are there any left?), A) Don't you already have a problem
dumping all .o files in the same directory? B) putting multiple outputs in
a subdir should reduce that by a factor of the number of outputs, C) You
could always nest them btree style with say 100 directories per level.
>
> > My feeling is so that the additional outputs are aggregated on the
> > per end-product basis so that, for example, one can easily find all
> > the SARIF files that describe a library build. If that's the case,
> > then creating a separate directory for utility libraries would break
> > this aggregation.
>
> It is in fact the exact opposite. Having the utility directory
> separate is _exactly what you want_, because then you can tell how
> many errors _that_ specific utility library has. This is important in
> case, for example, the utility library is maintained by team A whereas
> the rest of the project is maintained by project B. If you then want
> to aggregate everything together it is simple, because the build
> system already knows that the utility library is part of the whole
> build. It can even provide a hierarchical view of the project. If one
> lumps everything together then this information is lost and it is
> burdensome to separate it back out.
>
> There are other things you might want to group per build target rather
> than some higher level thing such as build timings.
>
> > I would expect this to be racy in most build systems. It would
> > definitely be in build2: depending on which executable-object
> > dependency caused the compilation, the additional outputs would
> > end up in one executable's top-level directory or another.
>
> I don't understand what you are trying to say here. Let's use an
> example instead for concreteness.
>
> Suppose we have source file foo.cc, two executables e1 and e2 and a
> utility library l.
>
> Foo.cc goes directly to e1 and also to l which is linked to e2.
>
> When foo.cc is compiled for e1, the extra files go to e1's private
> directory (it has to be compiled a second time as the flags might be
> different).
>
> When it is compiled for l, the extra files go to l's private directory.
>
> When linking, no extra files are created by the obj file (the target
> itself might, but it is unrelated).
>
> There is no race condition. There is no overlap. Every file goes to
> its own place with a unique name.
>
> There is one special case which is that the end user can tell (in
> Meson at least, dunno about other build systems) that they want to
> "steal" the object file from l and use it directly (not via the
> library) in target e3 at their own risk. Even in this case nothing
> changes. The extra files go in l's private dir because that is what
> actually builds the object file.
>
> > If this facility is to be added, I would suggest going with the
> > semantics proposed by Mathias: output directory per TU.
>
> I would suggest the opposite.
> _______________________________________________
> SG15 mailing list
> SG15_at_[hidden]
> https://lists.isocpp.org/mailman/listinfo.cgi/sg15
>
Received on 2024-11-29 10:35:22