I tried to write something quick, as CppCon is going on, but this conversation is making me want to work some of this into my talk on Friday.

Rene’s answer below resonates with my experience on Spack (https://github.com/spack/spack), and also with what I know of other projects like Spack (nix, guix, Julia’s Yggdrasil and BinaryBuilder, CondaForge, and to some extent Conan and vcpkg).

In Spack, we represent package metadata for on-disk installations and for binary packages with a DAG that includes the nearly complete provenance of the build.  Each node in the DAG has:


This is what we have right now — it goes in a canonical JSON format and you can use the hash of this metadata to identify a binary.  We use hashes to identify things instead of some combinatorial name because every such name I’ve ever seen eventually runs out of room for new options.  The format is designed to be extensible — you can add fields and get new hashes for your new artifacts, and they don’t conflict with old ones (if you chose a good hash).  This is inspired by systems like Nix — more on the origins of the scheme is in this paper: https://tgamblin.github.io/pubs/spack-sc15.pdf, but we really should update that — a lot has changed since then.

Even with all this information, we do not (yet) have all the information we would like for ABI compatibility. The most glaring issues at the moment are:


We are, this fall-ish, making a few changes:

All of this metadata is used as input to Spack’s dependency solver.  That used to be an ad-hoc thing, but we are now using Clingo (it looks like prolog but boils down to CDCL SAT + optimization — see https://github.com/spack/spack/blob/develop/lib/spack/spack/solver/concretize.lp for the logic program we currently use to manage these constraints).

There is an example of how we’ve architected a solver that can process metadata for lots of installed packages here:
https://github.com/spack/spack/pull/25310 

The cool thing about this is that you could express constraints on matching semantics for flags, etc.  e.g., you could say that certain flags are ABI-breaking, and that you require them to match across packages, but not others.

I think this is the kind of use case motivating the metadata we’re talking about here — what exactly needs to be checked to ensure that what you’re consuming is compatible with what you are building.

I could say more — particularly around bootstrapping and cross-compiling, but I think this probably should give folks a feeling for what needs to be in the ultimate specification.

-Todd





On Oct 27, 2021, at 11:54 AM, René Ferdinand Rivera Morell via SG15 <sg15@lists.isocpp.org> wrote:

Those are all good starts. But the short answer, since I don't have time ATM, for what you need to communicate to produce and consume prebuilt libraries are:

* All the inputs that went into producing the library. This includes what's been mentioned and also the environment (yes, as in ENV vars and everything else) they were produced in that might affect the result.
* All the usage requirements for the library.
* The entire n-dimensional matrix of compiler flag ABI, and general compatibility for the compilers in question.

It should be obvious that this is an NP complete problem to solve. And that we will only get partial / good-enough solutions to. Just like we do for dealing with ABI.

On Wed, Oct 27, 2021 at 1:38 PM Steve Downey via SG15 <sg15@lists.isocpp.org> wrote:
Include directory(ies) [-I flags], library directories [-L flags], library names [-l flags] . Required preprocessor defines to be consistent with the as-built binary. 
What C++ features were used in the pre-built library, particularly needed if we share dependencies. Library polyfills and alternatives are a problem here. For example, boost.json can use std:: types if they are available, or use boost components. A consumer needs to match. The feature test macros might be a tool to expose those. 
Names of dependencies to get from the package manager, so I can get their flags and usage. 
On Wed, Oct 27, 2021 at 2:06 PM Ben Craig via SG15 <sg15@lists.isocpp.org> wrote:

How do I invoke your code generator (like with gRPC, Apache Thrift, Microsoft RPC, QT)?  How do I deal with dependencies with your code generator?

 

What preprocessor flags and linker flags do I need to set to consume your library?  What configuration options are there?

 

What restrictions are your libraries placing on my binary?  For example, am I required to use a static CRT to consume your library?  A particular compiler version?

 

From: SG15 <sg15-bounces@lists.isocpp.org> On Behalf Of Daniel Ruoso via SG15
Sent: Wednesday, October 27, 2021 11:09 AM
To: sg15@lists.isocpp.org
Cc: Daniel Ruoso <daniel@ruoso.com>
Subject: [EXTERNAL] [SG15] RFC: Requirements to consume a prebuilt library from arbitrary build systems

 

Hello,

Thanks to the opportunity of being on site for cppcon (yay), me, Bret, Gabriel and Cameron had an opportunity to talk about the direction we're going for how to distribute libraries with C++ modules.

 

In that conversation we got to an agreement that we should also explore the possibility of making this a solution that is not exclusive for modules, and take a step back for a more generic solution.

 

I'm volunteering to drive the coordination work for a paper framing the requirements for a specification that would allow package managers and build systems to communicate on how to consume a prebuilt library.

 

I should clarify up front that the goal here is not to specify the format of library packages, the mechanism to resolve versions and how to fetch the package. In fact, the goal is that this should be something that fills existing gaps in the interactions between different package managers and different build systems, hopefully allowing more interoperability amongst the players that are already in this space.

 

So, I'm starting this effort with a request for comments on:

 

What are things that your build system needs to know when consuming a prebuilt library? What are the non-obvious cases? What are the architecture-specific details?

 

daniel

 

 

_______________________________________________
SG15 mailing list
SG15@lists.isocpp.org
https://lists.isocpp.org/mailman/listinfo.cgi/sg15
_______________________________________________
SG15 mailing list
SG15@lists.isocpp.org
https://lists.isocpp.org/mailman/listinfo.cgi/sg15


--
-- René Ferdinand Rivera Morell
-- Don't Assume Anything  -- No Supone Nada
-- Robot Dreams - http://robot-dreams.net

_______________________________________________
SG15 mailing list
SG15@lists.isocpp.org
https://urldefense.us/v3/__https://lists.isocpp.org/mailman/listinfo.cgi/sg15__;!!G2kpM7uM-TzIFchu!glNqWjHQASu4E6dggud0j1_7BWTxf6HUihTO-LiS7VeWwzgBUEZGVUh3QGa5R_qBjA$