If anyone has power to mandate anything, it's the standard. Again, the
purpose is not to force people to leave whatever tool or file format they
are using or not to force authors of existing build systems to support the
new thing. Like everything the standard does, we can't force people to
switch. Existing projects are most likely to stick to whatever they are
using and they won't be affected at all unless they opt-in. It doesn't mean
we shouldn't do something, just because existing projects won't use it.
If we are gonna go down this road to do something meaningful for
specifying/building projects, we might as well create a new file format
(replacing existing build system file format). It's less complex to do it
and we get to create something better in terms of complexity/dev experience.
Specifics of how it works can be discussed, maybe we let build systems
handle everything or there could be a middleware tool (could be compilers)
that goes through everything and creates final information for build tools
to use.
I'm aware of problems with "Yet another X" or "There were 10 solutions, so
we created our own. Now there are 11 solutions". If I wasn't aware of that,
I would go implement a new one. That's why I'm here. Only way to avoid this
is, "standard says it". This is not %100 solution to that, but at least it
wouldn't be yet another solution nobody uses since the industry standard is
CMake. People pretty much hate CMake, at least parts of it. But they can't
do anything about it because it's the industry standard. Some does, but
they don't get to benefit using a solution which is industry standard


On Tue, Nov 26, 2024 at 4:51 PM Wyatt Childers <wcc_at_[hidden]> wrote:

> Firstly, there won't be just Gamma and Zeta, there will be a handful of
> build systems involved. Say, my project needs ICU (custom build system,
> requires Python), OpenSSL (custom build system, requires Perl), Curl
> (automake), Boost (Boost Build), and Qt6 (CMake plus who knows what
> for a long list of dependencies). That's at least 5 different build
> systems.
>
> Of course there won't just be Gamma and Zeta.
>
> Particularly on the point of custom build systems: custom build systems
> are never going to support standardized anything unless the authors
> actually go out of their way to add support. Furthermore, we cannot as a
> committee or study group expect to mandate "standards" and everyone just
> follow them.
>
> Mandating a standard build file format (or "project description format")
> is (IMO) exactly how we get into this (ridiculed) situation:
>
> https://xkcd.com/927/
>
> The authors of these projects and their custom build systems are arguably *less
> likely* to move to a "standard build system format" (that they have to
> port their build to, relearn, and provides them nothing other than "being
> standard") than they are to add support to those build systems for build
> system interop.
>
> 1. If things go wrong (and they always go wrong in this area), you
> need to be knowledgeable in multiple build systems.
>
> 2. When you need support for a new platform or compiler, you are
> dependent on other build systems to provide the same support.
>
>
> I don't foresee us being able to solve these problem; they're a cost of
> having a diverse ecosystem.
>
> The way I see it, the problem is equivalent to having C++, C#, and Rust.
> They all do (on some level) "the same job", some trivial cases could be
> automatically converted (to a new "standard execution description format"),
> however the widely accepted practical and scalable solution is ... interop.
>
> 3. With different build systems you have to build things one-by-one
> in the dependency order. This can substantially reduce build
> parallelism.
>
> I don't think this is *necessarily* true. An interop specification could
> provide access to a shared "build parallelism semaphore" that allows all
> the build systems to coordinate their parallelism against the available
> threads on the system.
>
> I think any such feature would *have to* be an optional "capability"
> within the specification. If we want to deliver maximum value for the C++
> community, I think we need to focus on the breadth of the problem (i.e.,
> getting as many build systems being able to work together as possible) more
> than any specific efficiency (a complex, hard to understand, and hard to
> implement specification would make that harder).
>
> 4. With different build systems you would often have to build the
> entire project even though you may only need a small part. For
> example, ICU is actually three libraries but you may only need
> some subset of them.
>
> Sure, and an interop specification could say "your build system should
> show annotated targets you can build", e.g.:
>
> {
> "shared_libraries": [
> {
> "name": "abc",
> "version": "3.1"
> }
> ],
> "static_libraries": [
> {
> "name": "abc",
> "version": "3.1"
> }
> ],
> "executables": []
> }
>
> If we specify the process of:
>
> 1. User provides location of a third party build system and list of
> libraries from it that they depend on to Zeta
> 2. Zeta queries the 3rd-party build system (Gamma) for what it
> supports (to verify the user's request is do-able)
> 3. Zeta can have Gamma build the libraries
> 4. Zeta can retrieve the libraries
>
> ... that seems very valuable "1.0". If 1.0 gets adoption, "1.1" could be
> "okay, now here's this semaphore-like-thing you should [ideally] support
> and if you do, you can build in parallel with other build systems."
>
> For some build systems, like CMake I think adopting the "1.0" wouldn't be
> bad at all, but the optional 1.1 feature would likely depend on the
> features of the actual build generator used (which is part of why I think a
> "protocol" with optional sub-features is a better approach than trying to
> come up with some kind of "universal representation").
>
> Notably, there's also a migration path provided by this approach. You
> don't need to replace your build system, you just need to be able to have
> your build system (or some -- ideally simple -- wrapper) be able to speak
> on its behalf. Pieces and parts that don't work or aren't efficient (e.g.,
> you use a Ninja build generator with CMake and Ninja for whatever reason
> refuses to support the features required for CMake to implement "spec 1.1")
> can be replaced.
> - Wyatt
>
>
> On 11/26/24 08:08, Boris Kolpackov wrote:
>
> Wyatt Childers via SG15 <sg15_at_[hidden]> <sg15_at_[hidden]> writes:
>
>
> It seems much more beneficial to specify the standard interop where Zeta
> passes information to Gamma, Gamma builds the external project and then
> gives the build artifacts back to Zeta. Then I can in my Zeta build system
> just say "I want to use library X, here are the source files" (and then we
> can later specify some kind of package management that builds on that with
> source/binary fetch).
>
> I agree, this sounds sensible in theory but in practice this approach
> doesn't scale very well. Specifically:
>
> Firstly, there won't be just Gamma and Zeta, there will be a handful of
> build systems involved. Say, my project needs ICU (custom build system,
> requires Python), OpenSSL (custom build system, requires Perl), Curl
> (automake), Boost (Boost Build), and Qt6 (CMake plus who knows what
> for a long list of dependencies). That's at least 5 different build
> systems.
>
> This has some unfortunate implications:
>
> 1. If things go wrong (and they always go wrong in this area), you
> need to be knowledgeable in multiple build systems.
>
> 2. When you need support for a new platform or compiler, you are
> dependent on other build systems to provide the same support.
>
> 3. With different build systems you have to build things one-by-one
> in the dependency order. This can substantially reduce build
> parallelism.
>
> 4. With different build systems you would often have to build the
> entire project even though you may only need a small part. For
> example, ICU is actually three libraries but you may only need
> some subset of them.
>
>
>