Date: Thu, 28 Oct 2021 20:30:09 +0000
On Oct 28, 2021, at 12:39 PM, Daniel Ruoso <daniel_at_ruoso.com<mailto:daniel_at_[hidden]>> wrote:
On Wed, Oct 27, 2021 at 4:16 PM Gamblin, Todd via SG15 <sg15_at_[hidden]<mailto:sg15_at_[hidden]>> wrote:
The glibc version used is not a complete description of the dependency, because it presumes that glibc will never drop versioned symbols that are only needed for older binaries. RedHat has been communicating on their ABI guidelines that even glibc should only be expected to be compatible across three major versions. Likewise, while it's true that most Linux distributions have a binary-compatible glibc, I don't think that assumption is necessarily going to be true in the future.
For that reason, I think the requirement of encoding the specific OS where the library was built *on* is not just a proxy for glibc version, but an important record of what ABI stability is being promised by the vendors.
I think it becomes necessary to encode that more strictly, such that a solver can reject a binary built on RHEL5 when trying to be consumed in a RHEL8 box, for those that need to stay within the scope of what is formally supported by the vendor, even in the cases where it would work.
These are all really good points. The thing we have struggled with here is what is “the OS” — there is often not just one configuration of an OS, and it’s hard to pick a good identifier or ABI version for it. We currently use this library on Linux: https://github.com/python-distro/distro, which gives us identifiers like ubuntu18.04 or rhel7. Sometime people expect that to mean that some default package is installed that actually isn’t in, say, the base image.
I guess if I had my way, the vendors would publish a set of rules or something that say what ABI guarantees they’re promising *in terms of* core libs*, or at least someone would maintain a set of rules like that the same way people maintain rules for regular packages. I think coarse rules like “RHEL5 binaries can’t be consumed by RHEL8” are kind of hard to maintain because the reason for them gets lost in the vagueness of the OS identifier.
The other way the OS identifier is vague is that sometimes you may build on a particular OS, but with a completely different user land stack. e.g., if you build with Nix on a macOS system, you’re using Nix’s glibc (or maybe even one of many Nix glibc builds). Same thing is true for binaries built by Guix or Gentoo prefix. If you wanted to know where you could consume them (and you probably do, because people sometimes use environments like this to create lowest-common-denominator glibc builds of things to work across OS’s) then you can’t really represent that with a regular OS identifier. The OS ID is really a proxy for these ABI decisions.
As you point out, on other OS’s, like macOS, the ABI surface isn’t even glibc or the kernel ABI — it’s lib system and some set of core libraries. Apple is good about backward compatibility guarantees, so it’s not as much of an issue there. But for different musl versions, incompatible BSD/glibc versions, or if, say, clang actually made a libc — I think you need the finer granularity to really judge compatibility.
-Todd
On Wed, Oct 27, 2021 at 4:16 PM Gamblin, Todd via SG15 <sg15_at_[hidden]<mailto:sg15_at_[hidden]>> wrote:
The glibc version used is not a complete description of the dependency, because it presumes that glibc will never drop versioned symbols that are only needed for older binaries. RedHat has been communicating on their ABI guidelines that even glibc should only be expected to be compatible across three major versions. Likewise, while it's true that most Linux distributions have a binary-compatible glibc, I don't think that assumption is necessarily going to be true in the future.
For that reason, I think the requirement of encoding the specific OS where the library was built *on* is not just a proxy for glibc version, but an important record of what ABI stability is being promised by the vendors.
I think it becomes necessary to encode that more strictly, such that a solver can reject a binary built on RHEL5 when trying to be consumed in a RHEL8 box, for those that need to stay within the scope of what is formally supported by the vendor, even in the cases where it would work.
These are all really good points. The thing we have struggled with here is what is “the OS” — there is often not just one configuration of an OS, and it’s hard to pick a good identifier or ABI version for it. We currently use this library on Linux: https://github.com/python-distro/distro, which gives us identifiers like ubuntu18.04 or rhel7. Sometime people expect that to mean that some default package is installed that actually isn’t in, say, the base image.
I guess if I had my way, the vendors would publish a set of rules or something that say what ABI guarantees they’re promising *in terms of* core libs*, or at least someone would maintain a set of rules like that the same way people maintain rules for regular packages. I think coarse rules like “RHEL5 binaries can’t be consumed by RHEL8” are kind of hard to maintain because the reason for them gets lost in the vagueness of the OS identifier.
The other way the OS identifier is vague is that sometimes you may build on a particular OS, but with a completely different user land stack. e.g., if you build with Nix on a macOS system, you’re using Nix’s glibc (or maybe even one of many Nix glibc builds). Same thing is true for binaries built by Guix or Gentoo prefix. If you wanted to know where you could consume them (and you probably do, because people sometimes use environments like this to create lowest-common-denominator glibc builds of things to work across OS’s) then you can’t really represent that with a regular OS identifier. The OS ID is really a proxy for these ABI decisions.
As you point out, on other OS’s, like macOS, the ABI surface isn’t even glibc or the kernel ABI — it’s lib system and some set of core libraries. Apple is good about backward compatibility guarantees, so it’s not as much of an issue there. But for different musl versions, incompatible BSD/glibc versions, or if, say, clang actually made a libc — I think you need the finer granularity to really judge compatibility.
-Todd
Received on 2021-10-28 15:30:18