You may be assuming that the worker machines in a distributed build environment have access to the original source. In many cases they do not. Not everything is pointed at a shared network server. When they include <boost/log.h> or import abseil.foo, they have no access to a file system that contains the backing files. Also, if one of the backing files changes, then that should generally cause a cache miss in the caching tools.

Mathias's suggestion of a "single stream" format works too. If the compiler can emit (and then later consume) a .zip or some other kind of archive file format, then those binary file formats can be transmitted across the network for distribution, and stored (and hashed) in caching tools.

> -----Original Message-----
> From: Gabriel Dos Reis <gdr_at_microsoft.com>
> Sent: Tuesday, March 5, 2019 11:19 AM
> To: Ben Craig <ben.craig_at_ni.com>
> Cc: modules_at_lists.isocpp.org; WG21 Tooling Study Group SG15
> <tooling_at_[hidden]rg>; Nathan Sidwell <nathan_at_acm.org>
> Subject: [EXTERNAL] Re: [isocpp-modules] Round2: Path to modules with old
> bad build systems
>
>
>
> > On Mar 5, 2019, at 9:10 AM, Ben Craig <ben.craig_at_ni.com> wrote:
> >
> > Here are two use cases (that I've mentioned before) for a flattened
> > file format that doesn't need to be parse-able by third party tools
> > 1) Distributed build systems like distcc and FASTBuild that currently
> preprocess a file and send the file over the network to be compiled. These
> tools don't currently have a way to reach back over the network to pick up
> artifacts on the host machine.
> > 2) Object caching tools, like ccache that use the preprocessed output of a
> file as a key in a map from preprocessed output to built object file.
>
> I acknowledge all those scenarios. My real question is why the compiler has
> to go through the trouble of emitting such text, as opposed to being invoked
> on the other end with say -ftreat-imports-as-glorified headers? Tools like
> ccache couldn’t care less where the preprocessed file contains these magical
> pragmas or not.
>
> — Gaby
>
> >
> >> -----Original Message-----
> >> From: Gabriel Dos Reis <gdr_at_microsoft.com>
> >> Sent: Tuesday, March 5, 2019 11:04 AM
> >> To: Ben Craig <ben.craig_at_ni.com>
> >> Cc: modules_at_[hidden]; WG21 Tooling Study Group SG15
> >> <tooling_at_[hidden]>; Nathan Sidwell <nathan_at_acm.org>
> >> Subject: [EXTERNAL] Re: [isocpp-modules] Round2: Path to modules with
> >> old bad build systems
> >>
> >> If the flattened file format doesn’t need to be parse-able by third
> >> party tools, then why go through the trouble of generating it in the
> >> first place - requiring additional language extensions at the source level?
> >>
> >> — Gaby
> >>
> >>> On Mar 5, 2019, at 8:43 AM, Ben Craig <ben.craig_at_ni.com> wrote:
> >>>
> >>> If the flattened file format doesn't need to be parse-able by third
> >>> party
> >> tools, then the compiler can do whatever it wants with the text
> >> format, so long as the compiler itself can consume the resulting file
> >> on a different machine. Abandoning the parse-able requirement means
> >> that you could instead do...
> >>>
> >>> #pragma module begin(foo)
> >>> /* base64 encoded implicitly generated BMI */ #pragma module
> >>> end(foo)
> >>>
> >>> No cross-vendor spec would be necessary for such an approach.
> >>>
> >>>> -----Original Message-----
> >>>> From: Modules <modules-bounces_at_lists.isocpp.org> On Behalf Of
> >> Gabriel
> >>>> Dos Reis via Modules
> >>>> Sent: Tuesday, March 5, 2019 10:19 AM
> >>>> To: modules_at_[hidden]
> >>>> Cc: Gabriel Dos Reis <gdr_at_[hidden]>; WG21 Tooling Study Group
> >>>> SG15 <tooling_at_[hidden]>; Nathan Sidwell <nathan_at_acm.org>
> >>>> Subject: [EXTERNAL] Re: [isocpp-modules] Round2: Path to modules
> >>>> with old bad build systems
> >>>>
> >>>>
> >>>>
> >>>>> On Mar 5, 2019, at 8:12 AM, Ben Craig <ben.craig_at_ni.com> wrote:
> >>>>>
> >>>>> I will concede that static analysis tools and other tools that try
> >>>>> to parse C++
> >>>> probably don't need a textual inclusion format, since they most
> >>>> likely need to be able to parse and understand the pragmas anyway.
> >>>> If module mapping is sufficiently straightforward, then those tools
> >>>> can do module lookup the same as a compiler. Those tools already
> >>>> need to do include lookup in the same way that compilers do.
> >>>>
> >>>> I haven’t yet seen a mapping that isn’t straightforward :-)
> >>>>
> >>>>>
> >>>>> I think the textual inclusion format will still be very useful to
> >>>>> distribution
> >>>> and caching tools though, as they don't need to understand the code.
> >>>> Those tools frequently lean on the compiler's preprocessor today,
> >>>> and don't know how to do include lookups.
> >>>>
> >>>> I don’t know there is an actual spec of such flattened text file
> >>>> that isn’t at least as involved as the module spec - if not more involved.
> >>>>
> >>>> I worry that we may be creating a more complex problem than the
> >>>> issue we are set to address.
> >>>>
> >>>> — Gaby
> >>>>
> >>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Nathan Sidwell <nathanmsidwell_at_gmail.com> On Behalf Of
> >> Nathan
> >>>>>> Sidwell
> >>>>>> Sent: Tuesday, March 5, 2019 6:04 AM
> >>>>>> To: Ben Craig <ben.craig_at_ni.com>; modules_at_lists.isocpp.org; WG21
> >>>>>> Tooling Study Group SG15 <tooling_at_open-std.org>
> >>>>>> Subject: [EXTERNAL] Re: [isocpp-modules] Round2: Path to modules
> >>>>>> with old bad build systems
> >>>>>>
> >>>>>>> On 3/4/19 10:02 AM, Ben Craig wrote:
> >>>>>>> I do mean textual inclusion, though I can be convinced otherwise.
> >>>>>>> Textual
> >>>>>> inclusion (with extra generated pragmas) should make it much
> >>>>>> easier to keep tools like distcc and cppcheck happy in the short
> >>>>>> term. I suspect that those tools don't want to crack open a BMI
> >>>>>> to figure out which other BMIs need to be found.
> >>>>>>>
> >>>>>>> Tools that (think they can) parse C++ will still need to
> >>>>>>> understand these
> >>>>>> pragmas in order to provide the right macro, visibility, and
> >>>>>> reachability behaviors, so some work will still be required on
> >>>>>> their part, but at least they won't need to understand new binary
> >> formats.
> >>>>>>
> >>>>>> Correct, tools consuming such #pragma-marked flattened source
> >>>>>> will need to understand modules at a fundamental level. As such,
> >>>>>> why not implement the same mechanisms to find module source as
> >>>>>> the
> >> compiler?
> >>>>>> That'll give them more information to perform code analysis with.
> >>>>>>
> >>>>>> nathan
> >>>>>>
> >>>>>>>>> On 3/2/19 1:03 PM, Ben Craig wrote:
> >>>>>>>>>
> >>>>>>>>> Some quick notes on this implementation strategy:
> >>>>>>>>> * Uses TEXTUAL inclusion
> >>>>>>>>> * Compiler assumes that the build system knows nothing of
> BMIs
> >>>>>>>>> * Compiler needs to be able to do module mapping with minimal
> >>>>>>>>> input from users.
> >>>>>>>>
> >>>>>>>> Do you literally mean textual inclusion or do you really mean
> >>>>>>>> dynamically produce an internal-only BMI object?
> >>>>>>>>
> >>>>>>>> nathan
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Nathan Sidwell
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Nathan Sidwell
> >>>>> _______________________________________________
> >>>>> Modules mailing list
> >>>>> Modules_at_[hidden]pp.org
> >>>>> Subscription:
> >>>>> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__nam06.safelin
> >>>>> ks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-
> 252Furldefense.
> >>>>> proofpoint.com-252Fv2-252Furl-253Fu-253Dhttps-2D-26amp-3Bdata-
> 3D02
> >>>>> -257C01-257Cgdr-2540microsoft.com-
> 257Cbb5ce952bffc482e87b108d6a18d
> >>>>> 66a7-257C72f988bf86f141af91ab2d7cd011db47-257C1-257C0-
> 257C63687402
> >>>>> 6040110157-26amp-3Bsdata-3DB7tMwaCrtynz2Hru8w-252B-
> 252FilQOK7FRTTF
> >>>>> RGCvUnzQRBvI-253D-26amp-3Breserved-
> 3D0&d=DwIGaQ&c=I_0YwoKy7z5LMTVd
> >>>>> yO6YCiE2uzI1jjZZuIPelcSjixA&r=y8mub81SfUi-
> UCZRX0Vl1g&m=zRBB7ngX1sZ
> >>>>> WaksjZq9yWSoeKaf6vLIPPSby0m8osco&s=Gt0wubq23-
> ZhAHWBXMgBTcVDBWoGbTe
> >>>>> -xkSdPsBTv4U&e=
> >> 3A__nam06.safelinks
> >>>>> .protection.outlook.com_-3Furl-3Dhttps-253A-252F-
> >> 252Furldefense.proo
> >>>>> fpoint.com-252Fv2-252Furl-253Fu-253Dhttps-2D3A-5F-
> >> 5Fnam06.safelinks.
> >>>>> p-26amp-3Bdata-3D02-257C01-257Cgdr-2540microsoft.com-
> >> 257Cb10248c40d1
> >>>>> 04406a58008d6a1899f80-257C72f988bf86f141af91ab2d7cd011db47-
> >> 257C1-257
> >>>>> C0-257C636874009800024740-26amp-3Bsdata-
> >> 3Dp7A5pBCX07tWsIhQL6SmfcP6iu
> >>>>> nVFWqvUY1vKEw69t8-253D-26amp-3Breserved-
> >> 3D0&d=DwIGaQ&c=I_0YwoKy7z5LM
> >>>>> TVdyO6YCiE2uzI1jjZZuIPelcSjixA&r=y8mub81SfUi-
> >> UCZRX0Vl1g&m=OVMNvDsyTU
> >>>>> HgfVyzp-kJjPXybT2k8OAlWfIf-It-W20&s=ceh-
> >> yGW5IKak5viXjbJplZ1eOuUh8QuG
> >>>>> -ECUU_y4QqY&e=
> >>>>> rotection.outlook.com_-3Furl-3Dhttp-253A-252F-252Flists.isocpp.org
> >>>>> -2
> >>>>> 52
> >>>>> Fmailman-252Flistinfo.cgi-252Fmodules-26amp-3Bdata-3D02-257C01-
> >>>> 257Cgdr
> >>>>> -2540microsoft.com-257Ca5f3d89c3d7d4346707f08d6a1854be8-
> >>>> 257C72f988bf86
> >>>>> f141af91ab2d7cd011db47-257C1-257C0-257C636873991217371177-
> >> 26amp-
> >>>> 3Bsdat
> >>>>> a-3DRwfoGFvU8Df-252F2llvXjbpAiIIG-252B0RxL4Su8ewJeoypCw-
> 253D-
> >>>> 26amp-3Br
> >>>>> eserved-
> >>>>
> 3D0&d=DwIGaQ&c=I_0YwoKy7z5LMTVdyO6YCiE2uzI1jjZZuIPelcSjixA&r=y
> >>>>> 8mub81SfUi-
> >>>>
> >>
> UCZRX0Vl1g&m=AdhXKMaBKONGion_X4H9mmw08XjCtqqGoeW9bs7cVuQ&
> >>>> s=
> >>>>> ofQHenjonX7d85yaK2-6F3x_myxVyMcitwJo9rUo8gI&e=
> >>>>> Link to this post:
> >>>>> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__nam06.safelin
> >>>>> ks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-
> 252Furldefense.
> >>>>> proofpoint.com-252Fv2-252Furl-253Fu-253Dhttps-2D-26amp-3Bdata-
> 3D02
> >>>>> -257C01-257Cgdr-2540microsoft.com-
> 257Cbb5ce952bffc482e87b108d6a18d
> >>>>> 66a7-257C72f988bf86f141af91ab2d7cd011db47-257C1-257C0-
> 257C63687402
> >>>>> 6040110157-26amp-3Bsdata-3DB7tMwaCrtynz2Hru8w-252B-
> 252FilQOK7FRTTF
> >>>>> RGCvUnzQRBvI-253D-26amp-3Breserved-
> 3D0&d=DwIGaQ&c=I_0YwoKy7z5LMTVd
> >>>>> yO6YCiE2uzI1jjZZuIPelcSjixA&r=y8mub81SfUi-
> UCZRX0Vl1g&m=zRBB7ngX1sZ
> >>>>> WaksjZq9yWSoeKaf6vLIPPSby0m8osco&s=Gt0wubq23-
> ZhAHWBXMgBTcVDBWoGbTe
> >>>>> -xkSdPsBTv4U&e=
> >> 3A__nam06.safelinks
> >>>>> .protection.outlook.com_-3Furl-3Dhttps-253A-252F-
> >> 252Furldefense.proo
> >>>>> fpoint.com-252Fv2-252Furl-253Fu-253Dhttps-2D3A-5F-
> >> 5Fnam06.safelinks.
> >>>>> p-26amp-3Bdata-3D02-257C01-257Cgdr-2540microsoft.com-
> >> 257Cb10248c40d1
> >>>>> 04406a58008d6a1899f80-257C72f988bf86f141af91ab2d7cd011db47-
> >> 257C1-257
> >>>>> C0-257C636874009800024740-26amp-3Bsdata-
> >> 3Dp7A5pBCX07tWsIhQL6SmfcP6iu
> >>>>> nVFWqvUY1vKEw69t8-253D-26amp-3Breserved-
> >> 3D0&d=DwIGaQ&c=I_0YwoKy7z5LM
> >>>>> TVdyO6YCiE2uzI1jjZZuIPelcSjixA&r=y8mub81SfUi-
> >> UCZRX0Vl1g&m=OVMNvDsyTU
> >>>>> HgfVyzp-kJjPXybT2k8OAlWfIf-It-W20&s=ceh-
> >> yGW5IKak5viXjbJplZ1eOuUh8QuG
> >>>>> -ECUU_y4QqY&e=
> >>>>> rotection.outlook.com_-3Furl-3Dhttp-253A-252F-252Flists.isocpp.org
> >>>>> -2
> >>>>> 52
> >>>>> Fmodules-252F2019-252F03-252F0145.php-26amp-3Bdata-3D02-
> 257C01-
> >>>> 257Cgdr
> >>>>> -2540microsoft.com-257Ca5f3d89c3d7d4346707f08d6a1854be8-
> >>>> 257C72f988bf86
> >>>>> f141af91ab2d7cd011db47-257C1-257C0-257C636873991217371177-
> >> 26amp-
> >>>> 3Bsdat
> >>>>> a-3DOuOtS6D49Sy5EhRl8VUpv-252ByE5esxGZ7dvPvf-252Be1zr1c-
> 253D-
> >>>> 26amp-3Br
> >>>>> eserved-
> >>>>
> 3D0&d=DwIGaQ&c=I_0YwoKy7z5LMTVdyO6YCiE2uzI1jjZZuIPelcSjixA&r=y
> >>>>> 8mub81SfUi-
> >>>>
> >>
> UCZRX0Vl1g&m=AdhXKMaBKONGion_X4H9mmw08XjCtqqGoeW9bs7cVuQ&
> >>>> s=
> >>>>> sg8E8Gx-jIbRDIRalv20OlbIkHaNJbyC3TwX8-ueRDU&e=
> >>>> _______________________________________________
> >>>> Modules mailing list
> >>>> Modules_at_lists.isocpp.org
> >>>> Subscription:
> >>>> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__nam06.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-
> 252Furldefense.proofpoint.com-252Fv2-252Furl-253Fu-253Dhttps-2D3A-5F-
> 5Fnam06.safelinks-26amp-3Bdata-3D02-257C01-257Cgdr-2540microsoft.com-
> 257Cbb5ce952bffc482e87b108d6a18d66a7-
> 257C72f988bf86f141af91ab2d7cd011db47-257C1-257C0-
> 257C636874026040110157-26amp-3Bsdata-
> 3Dypcf1hy9wb9Y5IsJTmgg0OJiClBGm2rlS4ryjKxaEiI-253D-26amp-3Breserved-
> 3D0&d=DwIGaQ&c=I_0YwoKy7z5LMTVdyO6YCiE2uzI1jjZZuIPelcSjixA&r=y8m
> ub81SfUi-
> UCZRX0Vl1g&m=zRBB7ngX1sZWaksjZq9yWSoeKaf6vLIPPSby0m8osco&s=ohj
> R14bgYnHsi79fNDd_iWjs7fY8OBX3nyJI8zxEU00&e=.
> >>>> protection.outlook.com_-3Furl-3Dhttps-253A-252F-
> >> 252Furldefense.proofp
> >>>> oint.com-252Fv2-252Furl-253Fu-253Dhttp-2D-26amp-3Bdata-3D02-
> >> 257C01-25
> >>>> 7Cgdr-2540microsoft.com-257Cb10248c40d104406a58008d6a1899f80-
> >> 257C72f9
> >>>> 88bf86f141af91ab2d7cd011db47-257C1-257C0-
> 257C636874009800024740-
> >> 26amp
> >>>> -3Bsdata-3DvJxYGAlfPqfe-252BB6HOBD4JCjOZSEP9Sc1fKDJt8ehEu4-
> 253D-
> >> 26amp
> >>>> -3Breserved-
> >> 3D0&d=DwIGaQ&c=I_0YwoKy7z5LMTVdyO6YCiE2uzI1jjZZuIPelcSjix
> >>>> A&r=y8mub81SfUi-UCZRX0Vl1g&m=OVMNvDsyTUHgfVyzp-
> >> kJjPXybT2k8OAlWfIf-It-
> >>>> W20&s=LGVTU9QiQe6-6qXpL9XYoNaa4f0qPN5ydsuaoCSBxQU&e=
> >>>>
> >>
> 3A__lists.isocpp.org_mailman_listinfo.cgi_modules&d=DwIGaQ&c=I_0YwoK
> >>>> y7z5LMTVdyO6YCiE2uzI1jjZZuIPelcSjixA&r=y8mub81SfUi-
> >>>>
> >>
> UCZRX0Vl1g&m=AdhXKMaBKONGion_X4H9mmw08XjCtqqGoeW9bs7cVuQ&
> >>>> s=gmG6dYi-y2pNIFyGrUsRGlDKCKRKWf4WQQlvzQDWM-s&e=
> >>>> Link to this post:
> >>>> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__nam06.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-
> 252Furldefense.proofpoint.com-252Fv2-252Furl-253Fu-253Dhttps-2D3A-5F-
> 5Fnam06.safelinks-26amp-3Bdata-3D02-257C01-257Cgdr-2540microsoft.com-
> 257Cbb5ce952bffc482e87b108d6a18d66a7-
> 257C72f988bf86f141af91ab2d7cd011db47-257C1-257C0-
> 257C636874026040110157-26amp-3Bsdata-
> 3Dypcf1hy9wb9Y5IsJTmgg0OJiClBGm2rlS4ryjKxaEiI-253D-26amp-3Breserved-
> 3D0&d=DwIGaQ&c=I_0YwoKy7z5LMTVdyO6YCiE2uzI1jjZZuIPelcSjixA&r=y8m
> ub81SfUi-
> UCZRX0Vl1g&m=zRBB7ngX1sZWaksjZq9yWSoeKaf6vLIPPSby0m8osco&s=ohj
> R14bgYnHsi79fNDd_iWjs7fY8OBX3nyJI8zxEU00&e=.
> >>>> protection.outlook.com_-3Furl-3Dhttps-253A-252F-
> >> 252Furldefense.proofp
> >>>> oint.com-252Fv2-252Furl-253Fu-253Dhttp-2D-26amp-3Bdata-3D02-
> >> 257C01-25
> >>>> 7Cgdr-2540microsoft.com-257Cb10248c40d104406a58008d6a1899f80-
> >> 257C72f9
> >>>> 88bf86f141af91ab2d7cd011db47-257C1-257C0-
> 257C636874009800034748-
> >> 26amp
> >>>> -3Bsdata-3D123o9Z3tNcVY-
> 252FTsR6VbHITcMCHsPMk3459OhqCqpGwk-
> >> 253D-26amp
> >>>> -3Breserved-
> >> 3D0&d=DwIGaQ&c=I_0YwoKy7z5LMTVdyO6YCiE2uzI1jjZZuIPelcSjix
> >>>> A&r=y8mub81SfUi-UCZRX0Vl1g&m=OVMNvDsyTUHgfVyzp-
> >> kJjPXybT2k8OAlWfIf-It-
> >>>> W20&s=Kjdm4tjilk-j7xq9maoXFFiZ0PeL2iBPixAyNRRBc4s&e=
> >>>>
> >>
> 3A__lists.isocpp.org_modules_2019_03_0147.php&d=DwIGaQ&c=I_0YwoKy
> >>>> 7z5LMTVdyO6YCiE2uzI1jjZZuIPelcSjixA&r=y8mub81SfUi-
> >>>>
> >>
> UCZRX0Vl1g&m=AdhXKMaBKONGion_X4H9mmw08XjCtqqGoeW9bs7cVuQ&
> >>>> s=51GVUSm-w3aQdhBiPoTvrNnOE6QlopXYmzwNAYDSU5o&e=