I cannot connect to the meeting - is it running?

From: David Blaikie <dblaikie_at_[hidden]>
Sent: Friday, March 4, 2022 9:07 AM
To: Gabriel Dos Reis <gdr_at_[hidden]>
Cc: Olga Arkhipova <olgaark_at_[hidden]>; David Blaikie via SG15 <sg15_at_[hidden]>
Subject: Re: [SG15] Meeting on Friday March 4th at 9AM Pacific.

I'm using them as examples of libraries - I expect/understand that they will include multiple modules in some modular future, as they include multiple headers today.

On Fri, Mar 4, 2022 at 9:00 AM Gabriel Dos Reis <gdr_at_[hidden]<mailto:gdr_at_[hidden]>> wrote:
If we pick Boost of LLVM as example, are you thinking of them as libraries or as modules?

-- Gaby

________________________________
From: SG15 <sg15-bounces_at_[hidden]<mailto:sg15-bounces_at_[hidden]>> on behalf of David Blaikie via SG15 <sg15_at_[hidden]<mailto:sg15_at_[hidden]>>
Sent: Friday, March 4, 2022 12:52:17 AM
To: Olga Arkhipova <olgaark_at_[hidden]<mailto:olgaark_at_[hidden]>>
Cc: David Blaikie <dblaikie_at_[hidden]<mailto:dblaikie_at_[hidden]>>; David Blaikie via SG15 <sg15_at_[hidden]<mailto:sg15_at_[hidden]>>
Subject: Re: [SG15] Meeting on Friday March 4th at 9AM Pacific.

On Thu, Mar 3, 2022 at 10:49 PM Olga Arkhipova <olgaark_at_[hidden]<mailto:olgaark_at_[hidden]>> wrote:

>>. Is my understanding correct that MSVC can't consume external modules that don't include an MSVC Project file? (is this true for external dependencies pre-modules, or is this a new/tighter constraint for consuming external libraries?)



As there is no "well defined" libraries location on Windows, all locations for includes, libs, binaries, whatever have to be specified explicitly for the build there either by the user or by package managers (via build extensions)

OK, so if I picked up LLVM or Boost, etc, from an open source build without a MSVC project file - I'd probably add some include and library paths to my MSVC project?

looks like the boost docs document something like this: https://www.boost.org/doc/libs/1_65_0/more/getting_started/windows.html<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.boost.org%2Fdoc%2Flibs%2F1_65_0%2Fmore%2Fgetting_started%2Fwindows.html&data=04%7C01%7Colgaark%40microsoft.com%7C7f3df669f1a64c9fd4fe08d9fe017cb6%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637820104615041171%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=sDDpm%2F%2B2bk0T9WzquTl7bgOhV34n%2BVhCt7jAH%2BXdxwA%3D&reserved=0>

Looks like it's a total of 3 steps:
1) Add an include path
2) Add a library path
3) Add a library name to link to (oh, this is optional if your linker supports auto-linking - I guess the headers embed some metadata into the objects, and the linker reads that metadata to determine what to link)


MSVC can consume prebuilt modules if somebody specifies the BMI locations as build options either in the form of

/reference "ModuleName="Path/To/Module/BMI"

or the BMI search path.

What about not-prebuilt-modules - where there's a static/shared library, and some cppm files, no (or not usable-by-this-tool) BMI? How does MSVC consume those, when they're external to the MSVC projects?


In the last case the expectation is that ifc name would exactly match the module and it is not working very well due to build not producing what compiler expects there - user intervention is needed to change the default ifc name to match the module name.



In any case is it quite tedious work for the user as all module dependencies must be fould by the compiler and this is why we'd like to have some additional info provided by the libraries which would allow us to get "ModuleName=BMI location" info automatically and to reduce user's work. It would be great if we could use the same mechanism that we use today for getting the BMI dependencies, i.e. to be able to find json with additional info from the BMI location.

Right, though for other folks/use cases, they'd like to be able to go from ModuleName to cppm location and build commands (since they might not have a usable BMI - different compiler, build configuration, tool-that-isn't-a-compiler/doesn't-embed-one, etc).




Thanks,

Olga





From: David Blaikie <dblaikie_at_[hidden]<mailto:dblaikie_at_[hidden]>>
Sent: Thursday, March 3, 2022 20:47
To: David Blaikie via SG15 <sg15_at_[hidden]<mailto:sg15_at_[hidden]>>
Cc: Olga Arkhipova <olgaark_at_[hidden]<mailto:olgaark_at_[hidden]>>
Subject: Re: [SG15] Meeting on Friday March 4th at 9AM Pacific.



You don't often get email from dblaikie_at_[hidden]<mailto:dblaikie_at_[hidden]>. Learn why this is important<http://aka.ms/LearnAboutSenderIdentification>

On Thu, Mar 3, 2022 at 4:34 PM Olga Arkhipova via SG15 <sg15_at_[hidden]<mailto:sg15_at_[hidden]>> wrote:

I think it would be useful to finish the discussion about scenarios and assumptions before we go to solutions



Here is what I believe we've got so far



Suppose we have:



Library A: contains Module A

Library B: contains Module B which imports Module A

Code C: imports Module B



The libraries can come in different forms:



  1. Binary Libraries - libraries that contain prebuilt BMIs and static lib (as well as at least some sources)
  2. Source Libraries - libraries that don't contain any binaries, most of the popular libraries today. There are two main forms today

These two scenarios don't seem to include what I think is the more common scenario that, so far as I understand it, is at the core one being discussed in the proposal:

Code that would currently ship as a binary library - headers and precompiled (static and/or shared) libraries. I think "most of the popular libraries today" ship this way (see LLVM, boost, etc).

The scenario that Bloomberg folks are trying to figure out is what those libraries look like/how they ship in a modules world - which means shipping module interface source and precompiled static/shared libraries (/maybe/ some BMIs that cover some compilers and build configurations, but the more challenging cases/motivation for the proposal are where those BMIs aren't applicable to the consumer's (C's) build) - and probably some kind of build instructions for the module interface source. The hope/desire is to standardize these build instructions, to some degree. Not for building the whole source of library A or B, only the BMIs.

  1.

b.1. Header only libraries - headers are built as a part of the user's code TUs.

   Note: This form is not applicable to module interfaces as they are different TUs. The module interface source will have to be built explicitly (i.e. it has to be b.2).

b.2. Sources and (some) build instructions - sources are built on the user's machine with user's build tools (producing static or dynamic lib) which then consumed by the user's build.

  The build is performed either by the package manager or by user's build explicitly. The result and further usage is similar to Binary Libraries.



Scenarios and assumptions



  1. Main build of C.

1.1 The build needs to have Module B's and Module A's BMIs to be able to compile the source that import Module B.

The build needs to

1.1.1. find BMIs in the (assuming it can read them) using the information available to that build system

or

1.1.2. somehow re-build the BMIs (assuming it cannot read them).



1.2. The build also needs to find and link static lib of Library B (which contains implementation of Module B), as well as static lib of Library A (which contains implementation of Module A).



  1. Tools outside of the main build (IDE, static analysis, etc.) need to compile/parse C.

Assumption: the main build of C succeeds on the same machine.

For that they need to be able to rebuild Module A and Module B (the assumption is that they never can read BMIs).

Right - including the case where this "tool" might also be another compiler. (eg: someone ships code with GCC BMIs and needs to compile it with Clang - or a different build mode that's not BMI compatible (clang's BMIs are much more constrained than their ABI - there are different flags you can use that would make the BMI not valid to share, but would still be usable with some exsiting shared/static library that the module was an interface to))





What we've agreed on:



  1. Any library which contains modules must include module interface sources
  2. If a library contains BMIs, it should also contain information (probably in some json form) sufficient to be able to rebuilt them.

Could we remove the "IF a library contains BMIs" from this statement? "A shipped library should include some portable-ish* means of defining how to build BMIs from its module interface definitions"

* portable-ish in that it might have parts that lets it specify compiler-specific flags (maybe general ones - GCC-like or MSVC-like, etc - enable or disable certain warnings, or define certain preprocessor directives, etc).



  1.





I also promised a description of what VS is doing for module's build, please see attached.

Thanks! Had a bit of a look. Is my understanding correct that MSVC can't consume external modules that don't include an MSVC Project file? (is this true for external dependencies pre-modules, or is this a new/tighter constraint for consuming external libraries?)