> Firstly, there won't be just Gamma and Zeta, there will be a handful of
> build systems involved. Say, my project needs ICU (custom build system,
> requires Python), OpenSSL (custom build system, requires Perl), Curl
> (automake), Boost (Boost Build), and Qt6 (CMake plus who knows what
> for a long list of dependencies). That's at least 5 different build
> systems.
Of course there won't just be Gamma and Zeta.

Particularly on the point of custom build systems: custom build systems
are never going to support standardized anything unless the authors
actually go out of their way to add support. Furthermore, we cannot as a
committee or study group expect to mandate "standards" and everyone just
follow them.

Mandating a standard build file format (or "project description format")
is (IMO) exactly how we get into this (ridiculed) situation:

https://xkcd.com/927/

The authors of these projects and their custom build systems are
arguably /less likely/ to move to a "standard build system format" (that
they have to port their build to, relearn, and provides them nothing
other than "being standard") than they are to add support to those build
systems for build system interop.

> 1. If things go wrong (and they always go wrong in this area), you
> need to be knowledgeable in multiple build systems.
> 2. When you need support for a new platform or compiler, you are
> dependent on other build systems to provide the same support.

I don't foresee us being able to solve these problem; they're a cost of
having a diverse ecosystem.

The way I see it, the problem is equivalent to having C++, C#, and Rust.
They all do (on some level) "the same job", some trivial cases could be
automatically converted (to a new "standard execution description
format"), however the widely accepted practical and scalable solution is
... interop.

> 3. With different build systems you have to build things one-by-one
> in the dependency order. This can substantially reduce build
> parallelism.
I don't think this is /necessarily/ true. An interop specification could
provide access to a shared "build parallelism semaphore" that allows all
the build systems to coordinate their parallelism against the available
threads on the system.

I think any such feature would /have to/ be an optional "capability"
within the specification. If we want to deliver maximum value for the
C++ community, I think we need to focus on the breadth of the problem
(i.e., getting as many build systems being able to work together as
possible) more than any specific efficiency (a complex, hard to
understand, and hard to implement specification would make that harder).

> 4. With different build systems you would often have to build the
> entire project even though you may only need a small part. For
> example, ICU is actually three libraries but you may only need
> some subset of them.
Sure, and an interop specification could say "your build system should
show annotated targets you can build", e.g.:

{
   "shared_libraries": [
     {
       "name": "abc",
       "version": "3.1"
     }
   ],
   "static_libraries": [
     {
       "name": "abc",
       "version": "3.1"
     }
   ],
   "executables": []
}

If we specify the process of:

 1. User provides location of a third party build system and list of
    libraries from it that they depend on to Zeta
 2. Zeta queries the 3rd-party build system (Gamma) for what it supports
    (to verify the user's request is do-able)
 3. Zeta can have Gamma build the libraries
 4. Zeta can retrieve the libraries

... that seems very valuable "1.0". If 1.0 gets adoption, "1.1" could be
"okay, now here's this semaphore-like-thing you should [ideally] support
and if you do, you can build in parallel with other build systems."

For some build systems, like CMake I think adopting the "1.0" wouldn't
be bad at all, but the optional 1.1 feature would likely depend on the
features of the actual build generator used (which is part of why I
think a "protocol" with optional sub-features is a better approach than
trying to come up with some kind of "universal representation").

Notably, there's also a migration path provided by this approach. You
don't need to replace your build system, you just need to be able to
have your build system (or some -- ideally simple -- wrapper) be able to
speak on its behalf. Pieces and parts that don't work or aren't
efficient (e.g., you use a Ninja build generator with CMake and Ninja
for whatever reason refuses to support the features required for CMake
to implement "spec 1.1") can be replaced.

- Wyatt


On 11/26/24 08:08, Boris Kolpackov wrote:
> Wyatt Childers via SG15<sg15_at_[hidden]> writes:
>
>> It seems much more beneficial to specify the standard interop where Zeta
>> passes information to Gamma, Gamma builds the external project and then
>> gives the build artifacts back to Zeta. Then I can in my Zeta build system
>> just say "I want to use library X, here are the source files" (and then we
>> can later specify some kind of package management that builds on that with
>> source/binary fetch).
> I agree, this sounds sensible in theory but in practice this approach
> doesn't scale very well. Specifically:
>
> Firstly, there won't be just Gamma and Zeta, there will be a handful of
> build systems involved. Say, my project needs ICU (custom build system,
> requires Python), OpenSSL (custom build system, requires Perl), Curl
> (automake), Boost (Boost Build), and Qt6 (CMake plus who knows what
> for a long list of dependencies). That's at least 5 different build
> systems.
>
> This has some unfortunate implications:
>
> 1. If things go wrong (and they always go wrong in this area), you
> need to be knowledgeable in multiple build systems.
>
> 2. When you need support for a new platform or compiler, you are
> dependent on other build systems to provide the same support.
>
> 3. With different build systems you have to build things one-by-one
> in the dependency order. This can substantially reduce build
> parallelism.
>
> 4. With different build systems you would often have to build the
> entire project even though you may only need a small part. For
> example, ICU is actually three libraries but you may only need
> some subset of them.