1.
Module-specific file vs. Wider "library manifest" file -
I prefer the specific as it is simple and no discussion was made on the
wider topic.
Also a specific one can always be easily extended.

2.
I feel that having a file manifest per module would be more
maintainable once we add more modules beside std.
It feels that having the manifest file named after the logical-name it
represents would make it easier to locate and work with it.
e.g.
std.module
std.compat.module
foo.module
etc

3.
A question:
Should "is-std-library" be set to true when injecting custom
specializations to std (e.g. hash
<https://en.cppreference.com/w/cpp/utility/hash>)?


Ran.


On Wed, Dec 13, 2023 at 9:36 AM Iain Sandoe via SG15 <sg15_at_[hidden]>
wrote:

> Hi Folks
>
> > On 13 Dec 2023, at 07:17, Olga Arkhipova via SG15 <sg15_at_[hidden]>
> wrote:
> >
> > Thanks, Daniel, for writing all this down.
> >
> > Looks like there is not much difference between library manifest and
> module metadata currently. So I think it should not be much harm in threat
> it as a library manifest v0, we can always ignore fields later if we decide
> otherwise.
> >
> > If we do it as a library manifest, we have to have the library name
> there.
> > Besides being useful in diagnostic messages and output location (and we
> do use it in current MS implementation), it is needed to allow future
> library reference resolution. At least on Windows the location of the
> manifest does not give any information about the library it represents, so
> we need it in the manifest itself.
> >
> > The library name has to be unique to allow unambiguous referencing in
> other libraries. I don’t know if there are other libraries in the world
> which provide the same functionality besides STL, but to allow the
> replacement, we should also have something like “library-type” or
> “library-common-name” – something similar to is-std-library, but more
> generic. For std libs it will be set to “std” so referencing libraries can
> just use “std” if they allow the substitution. I also think that this Is a
> library and not a module attribute, but if there are some nuances I am not
> aware of, we can sure have additional attributes for each module overriding
> the general library one.
> >
> > So, to make it visual, I am proposing to have
> >
> > {
> > "version": 1,
> > "revision": 0,
>
> In order to support interoperability of tools supporting different
> revisions of one of these files, I think we need:
>
> - A "current version" and a “minimum supported version", which means that
> this manifest would be a correct representation for a tool supporting
> “minimum supported version” (but might have some improved capability for
> later tools, supporting "current version").
>
> - A clear rule that says how a tool supporting minimum but not current is
> to skip the data it does not recognise (and a mechanism for organising
> content that allows this).
>
> thanks
> Iain
>
> > “library”: “unique name like Microsoft.STL",
> > “library-common-name”:”std”
> > "c++": {
> > "modules": []
> > }
> > “referenced-libraries”:[] ß not present for std libs, but other
> libraries can reference “std” and other libs.
> > }
> >
> > Thanks,
> > Olga
> >
> > From: SG15 <sg15-bounces_at_[hidden]> On Behalf Of Daniel Ruoso
> via SG15
> > Sent: Tuesday, December 12, 2023 13:57
> > To: sg15_at_[hidden]
> > Cc: Daniel Ruoso <daniel_at_[hidden]>
> > Subject: [SG15] Proposal for module metadata format to be used by the
> std library and others
> >
> > As discussed in the meeting on 2023-12-12, I'm putting together a
> proposal for the metadata format to be used both when discovering the std
> modules as well as for pre-built libraries in general.
> >
> > There was general agreement that while the std library always needs
> special cases, we will need some metadata format to describe the std
> modules, and it would be unfortunate if the format used for the std library
> was different than that used in other scenarios.
> >
> > There was some desire to have this be a starting point for a wider
> metadata format to cover other aspects beyond modules, so this proposal
> will be split in two. First covering modules, and later offering an option
> on how that information could be embedded into a wider format.
> >
> > Before jumping into the format itself, here's the list of requirements
> that guided the design:
> >
> > * Module discovery should be subordinate to ABI decisions having been
> made already. The outcome is that we don't expect the module metadata to be
> used to discover which ABI settings are available for a given standard
> library or how to choose between them. The way that this manifests is that,
> following the lead from P2577R2, we expect the libraries to ship those
> metadata files on a one-to-one mapping with the library binary used as
> input to the linker.
> >
> > * The path to the module source files should be discovered via the
> meatadata file. The outcome is that we don't expect a mechanism to search
> for those module files in a directory, nor a specific standard on how they
> should be named. but rather the build system will be given that information
> directly. This also allows the choice of the standard library splitting its
> implementation into interface partitions for ease of maintainability.
> >
> > * The compiler should offer a mechanism to introspect the path to the
> metadata file of the standard library, with the settings that the compiler
> will be used. That means build systems don't need to perform the lookup of
> the metadata.
> > The format should be extensible to cover vendor-specific settings.
> > The format should be versioned to allow future backward-compatible
> changes.
> >
> > * The format should allow indicating when the intention is to build the
> std module, since the compiler should reject a module accidentally named in
> a way that conflicts with the std module.
> >
> > Requirements that are likely to become important in the future, but that
> are not otherwise included in this proposal:
> >
> > * Pre-built libraries, including the std library, may want to advertise
> pre-built BMIs to be reused by the build system. This requires more
> convergence on the mechanisms to identify BMI compatibility, which have
> been discussed in P2581R2, but not yet supported by implementors.
> >
> > * Future integration with more general package management facilities.
> Although my expectation is that this is a step in that direction, where it
> may be possible to either make the data described here embedded there, or
> have the path to this metadata referenced instead. Particularly this
> proposal is not trying to specify requirements that would allow discovering
> when using this would be appropriate or not.
> >
> > Some a-priori decisions that are assumed in this format:
> >
> > * JSON: while there are many competing alternatives for the
> serialization format for this metadata, various other parts of the tooling
> ecosystem (such as compile_commands.json and the dependency scanning
> output) are already using this format, therefore I chose to just stick to
> it, rather than consider introducing a new serialization format.
> >
> > * Relative file paths: Any non-absolute path described in this file
> will be presumed to have the directory where the metadata file was found as
> the base for the lookup.
> >
> > Since the goal of this proposal is to evaluate specific usage, I'll will
> prioritize describing the file with examples, rather than writing a JSON
> schema for it. The final design should still be encoded that way, but I
> feel the format of json schema would make this conversation harder to
> maintain.
> >
> > # Envelope
> >
> > The first thing I want to address is the envelope that will contain the
> module-specific metadata. For that there are two options:
> >
> > ## Module-specific file
> >
> > If we decide to go with a file that refers only to module metadata, the
> envelope could look like:
> >
> > {
> > "version": 1,
> > "revision": 0,
> > "modules": []
> > }
> >
> > ## Wider "library manifest" file
> >
> > If we decide we should go with a wider format for future extensibility,
> the envelope could look like:
> >
> > {
> > "version": 1,
> > "revision": 0,
> > "c++": {
> > "modules": []
> > }
> > }
> >
> > # The modules value
> >
> > In both possible envelopes, module metadata is represented as an array
> of objects, where all the importable module units provided by this library
> that may be reachable by a consumer in this library will be described.
> >
> > While we currently don't expect the std module to depend on any module
> not provided by the std library itself at this point, there are already
> situations where the std library has external dependencies (e.g.: tbb for
> libstdc++). This is an area that needs further exploration in the future,
> and it may be the case that the compiler may need to report several module
> metadata files, rather than just one.
> >
> > ## Describing a module
> >
> > The following keys and values are expected in the object in the modules
> array:
> >
> > * logical-name (mandatory): This includes the name of the module being
> provided, the same semantics of P1689 applies.
> >
> > * is-interface (optional, default to true): This describes whether this
> contributes to the external interface of the module, the same semantics of
> P1689 applies.
> >
> > * source-path (mandatory): The path to the source code of the
> importable unit. If expressed as a relative path, lookup is done from the
> directory where the module metadata was found.
> >
> > * is-std-library (optional, default to false): Indicates that the
> module is allowed to use names that are reserved to the standard library.
> >
> > * local-arguments (optional), an object describing arguments that
> should be applied for translating this particular importable unit, but that
> doesn't need to be in the compilation of the translation unit importing
> this module:
> >
> > ** include-directories (optional): an array of paths that need to be
> appended to the compilation include search path, same semantics as
> appending -I in gcc and clang.
> >
> > ** system-include-directories (optional): an array of paths that need
> to be appended to the compilation include path as system locations, same
> semantics as appending -isystem in gcc and clang.
> >
> > ** definitions: an array of objects to be appended in order.
> >
> > *** name (mandatory): the name of the definition to be used
> >
> > *** value (optional): the value to be set. If missing, it is the
> equivalent of -DFOO in gcc and clang.
> >
> > *** undef (optional, defaults to false, incompatible with value): The
> equivalent of -UFOO in gcc and clang.
> >
> > *** vendor (optional): extension point for vendor-specific
> configurations. This is an object where the key is the name of the vendor,
> and the value is implementation-defined.
> >
> > * vendor (optional): extension point for vendor-specific information.
> This is an object where the key is the name of the vendor and the value is
> implementation-defined.
> >
> > # Example
> >
> > Here's how I would expect that would look like for a standard library
> (assuming the modules file for now), such as libc++:
> >
> > {
> > "version": 1,
> > "revision": 1,
> > "modules": [
> > {
> > "logical-name": "std",
> > "source-path": "modules/std.cppm",
> > "is-standard-library": true
> > },
> > {
> > "logical-name": "std.compat",
> > "source-path": "modules/std.compat.cppm"
> > "is-std-library": true
> > },
> > {
> > "logical-name": "std:someinterfacepartition",
> > "source-path": "modules/std-someinterfacepartition.cppm"
> > "is-std-library": true
> > }
> > ]
> > }
> >
> > Note that this specifically doesn't use any of the local arguments,
> because I don't really think that's going to be needed for the standard
> library case. The only special case is the is-standard-library key, to
> allow the build system to know this is not an accidental collision with the
> reserved names. We may decide not to settle the local-arguments part of the
> proposal now for that reason.
> >
> > Daniel
> >
> > _______________________________________________
> > SG15 mailing list
> > SG15_at_[hidden]
> > https://lists.isocpp.org/mailman/listinfo.cgi/sg15
>
> _______________________________________________
> SG15 mailing list
> SG15_at_[hidden]
> https://lists.isocpp.org/mailman/listinfo.cgi/sg15
>