C++ Logo

sg15

Advanced search

Re: [Tooling] [isocpp-modules] Round2: Path to modules with old bad build systems

From: Ben Craig <ben.craig_at_[hidden]>
Date: Tue, 5 Mar 2019 17:09:49 +0000
Here are two use cases (that I've mentioned before) for a flattened file format that doesn't need to be parse-able by third party tools
1) Distributed build systems like distcc and FASTBuild that currently preprocess a file and send the file over the network to be compiled. These tools don't currently have a way to reach back over the network to pick up artifacts on the host machine.
2) Object caching tools, like ccache that use the preprocessed output of a file as a key in a map from preprocessed output to built object file.

> -----Original Message-----
> From: Gabriel Dos Reis <gdr_at_microsoft.com>
> Sent: Tuesday, March 5, 2019 11:04 AM
> To: Ben Craig <ben.craig_at_[hidden].com>
> Cc: modules_at_[hidden]; WG21 Tooling Study Group SG15
> <tooling_at_[hidden]>; Nathan Sidwell <nathan_at_acm.org>
> Subject: [EXTERNAL] Re: [isocpp-modules] Round2: Path to modules with old
> bad build systems
>
> If the flattened file format doesn’t need to be parse-able by third party tools,
> then why go through the trouble of generating it in the first place - requiring
> additional language extensions at the source level?
>
> — Gaby
>
> > On Mar 5, 2019, at 8:43 AM, Ben Craig <ben.craig_at_ni.com> wrote:
> >
> > If the flattened file format doesn't need to be parse-able by third party
> tools, then the compiler can do whatever it wants with the text format, so
> long as the compiler itself can consume the resulting file on a different
> machine. Abandoning the parse-able requirement means that you could
> instead do...
> >
> > #pragma module begin(foo)
> > /* base64 encoded implicitly generated BMI */ #pragma module end(foo)
> >
> > No cross-vendor spec would be necessary for such an approach.
> >
> >> -----Original Message-----
> >> From: Modules <modules-bounces_at_[hidden]p.org> On Behalf Of
> Gabriel
> >> Dos Reis via Modules
> >> Sent: Tuesday, March 5, 2019 10:19 AM
> >> To: modules_at_lists.isocpp.org
> >> Cc: Gabriel Dos Reis <gdr_at_microsoft.com>; WG21 Tooling Study Group
> >> SG15 <tooling_at_open-std.org>; Nathan Sidwell <nathan_at_acm.org>
> >> Subject: [EXTERNAL] Re: [isocpp-modules] Round2: Path to modules with
> >> old bad build systems
> >>
> >>
> >>
> >>> On Mar 5, 2019, at 8:12 AM, Ben Craig <ben.craig_at_[hidden]i.com> wrote:
> >>>
> >>> I will concede that static analysis tools and other tools that try
> >>> to parse C++
> >> probably don't need a textual inclusion format, since they most
> >> likely need to be able to parse and understand the pragmas anyway.
> >> If module mapping is sufficiently straightforward, then those tools
> >> can do module lookup the same as a compiler. Those tools already
> >> need to do include lookup in the same way that compilers do.
> >>
> >> I haven’t yet seen a mapping that isn’t straightforward :-)
> >>
> >>>
> >>> I think the textual inclusion format will still be very useful to
> >>> distribution
> >> and caching tools though, as they don't need to understand the code.
> >> Those tools frequently lean on the compiler's preprocessor today, and
> >> don't know how to do include lookups.
> >>
> >> I don’t know there is an actual spec of such flattened text file that
> >> isn’t at least as involved as the module spec - if not more involved.
> >>
> >> I worry that we may be creating a more complex problem than the issue
> >> we are set to address.
> >>
> >> — Gaby
> >>
> >>>
> >>>> -----Original Message-----
> >>>> From: Nathan Sidwell <nathanmsidwell_at_gmail.com> On Behalf Of
> Nathan
> >>>> Sidwell
> >>>> Sent: Tuesday, March 5, 2019 6:04 AM
> >>>> To: Ben Craig <ben.craig_at_[hidden]m>; modules_at_lists.isocpp.org; WG21
> >>>> Tooling Study Group SG15 <tooling_at_open-std.org>
> >>>> Subject: [EXTERNAL] Re: [isocpp-modules] Round2: Path to modules
> >>>> with old bad build systems
> >>>>
> >>>>> On 3/4/19 10:02 AM, Ben Craig wrote:
> >>>>> I do mean textual inclusion, though I can be convinced otherwise.
> >>>>> Textual
> >>>> inclusion (with extra generated pragmas) should make it much easier
> >>>> to keep tools like distcc and cppcheck happy in the short term. I
> >>>> suspect that those tools don't want to crack open a BMI to figure
> >>>> out which other BMIs need to be found.
> >>>>>
> >>>>> Tools that (think they can) parse C++ will still need to
> >>>>> understand these
> >>>> pragmas in order to provide the right macro, visibility, and
> >>>> reachability behaviors, so some work will still be required on
> >>>> their part, but at least they won't need to understand new binary
> formats.
> >>>>
> >>>> Correct, tools consuming such #pragma-marked flattened source will
> >>>> need to understand modules at a fundamental level. As such, why
> >>>> not implement the same mechanisms to find module source as the
> compiler?
> >>>> That'll give them more information to perform code analysis with.
> >>>>
> >>>> nathan
> >>>>
> >>>>>>> On 3/2/19 1:03 PM, Ben Craig wrote:
> >>>>>>>
> >>>>>>> Some quick notes on this implementation strategy:
> >>>>>>> * Uses TEXTUAL inclusion
> >>>>>>> * Compiler assumes that the build system knows nothing of BMIs
> >>>>>>> * Compiler needs to be able to do module mapping with minimal
> >>>>>>> input from users.
> >>>>>>
> >>>>>> Do you literally mean textual inclusion or do you really mean
> >>>>>> dynamically produce an internal-only BMI object?
> >>>>>>
> >>>>>> nathan
> >>>>>>
> >>>>>> --
> >>>>>> Nathan Sidwell
> >>>>
> >>>>
> >>>> --
> >>>> Nathan Sidwell
> >>> _______________________________________________
> >>> Modules mailing list
> >>> Modules_at_[hidden]p.org
> >>> Subscription:
> >>> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__nam06.safelinks
> >>> .protection.outlook.com_-3Furl-3Dhttps-253A-252F-
> 252Furldefense.proo
> >>> fpoint.com-252Fv2-252Furl-253Fu-253Dhttps-2D3A-5F-
> 5Fnam06.safelinks.
> >>> p-26amp-3Bdata-3D02-257C01-257Cgdr-2540microsoft.com-
> 257Cb10248c40d1
> >>> 04406a58008d6a1899f80-257C72f988bf86f141af91ab2d7cd011db47-
> 257C1-257
> >>> C0-257C636874009800024740-26amp-3Bsdata-
> 3Dp7A5pBCX07tWsIhQL6SmfcP6iu
> >>> nVFWqvUY1vKEw69t8-253D-26amp-3Breserved-
> 3D0&d=DwIGaQ&c=I_0YwoKy7z5LM
> >>> TVdyO6YCiE2uzI1jjZZuIPelcSjixA&r=y8mub81SfUi-
> UCZRX0Vl1g&m=OVMNvDsyTU
> >>> HgfVyzp-kJjPXybT2k8OAlWfIf-It-W20&s=ceh-
> yGW5IKak5viXjbJplZ1eOuUh8QuG
> >>> -ECUU_y4QqY&e=
> >>> rotection.outlook.com_-3Furl-3Dhttp-253A-252F-252Flists.isocpp.org-2
> >>> 52
> >>> Fmailman-252Flistinfo.cgi-252Fmodules-26amp-3Bdata-3D02-257C01-
> >> 257Cgdr
> >>> -2540microsoft.com-257Ca5f3d89c3d7d4346707f08d6a1854be8-
> >> 257C72f988bf86
> >>> f141af91ab2d7cd011db47-257C1-257C0-257C636873991217371177-
> 26amp-
> >> 3Bsdat
> >>> a-3DRwfoGFvU8Df-252F2llvXjbpAiIIG-252B0RxL4Su8ewJeoypCw-253D-
> >> 26amp-3Br
> >>> eserved-
> >> 3D0&d=DwIGaQ&c=I_0YwoKy7z5LMTVdyO6YCiE2uzI1jjZZuIPelcSjixA&r=y
> >>> 8mub81SfUi-
> >>
> UCZRX0Vl1g&m=AdhXKMaBKONGion_X4H9mmw08XjCtqqGoeW9bs7cVuQ&
> >> s=
> >>> ofQHenjonX7d85yaK2-6F3x_myxVyMcitwJo9rUo8gI&e=
> >>> Link to this post:
> >>> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__nam06.safelinks
> >>> .protection.outlook.com_-3Furl-3Dhttps-253A-252F-
> 252Furldefense.proo
> >>> fpoint.com-252Fv2-252Furl-253Fu-253Dhttps-2D3A-5F-
> 5Fnam06.safelinks.
> >>> p-26amp-3Bdata-3D02-257C01-257Cgdr-2540microsoft.com-
> 257Cb10248c40d1
> >>> 04406a58008d6a1899f80-257C72f988bf86f141af91ab2d7cd011db47-
> 257C1-257
> >>> C0-257C636874009800024740-26amp-3Bsdata-
> 3Dp7A5pBCX07tWsIhQL6SmfcP6iu
> >>> nVFWqvUY1vKEw69t8-253D-26amp-3Breserved-
> 3D0&d=DwIGaQ&c=I_0YwoKy7z5LM
> >>> TVdyO6YCiE2uzI1jjZZuIPelcSjixA&r=y8mub81SfUi-
> UCZRX0Vl1g&m=OVMNvDsyTU
> >>> HgfVyzp-kJjPXybT2k8OAlWfIf-It-W20&s=ceh-
> yGW5IKak5viXjbJplZ1eOuUh8QuG
> >>> -ECUU_y4QqY&e=
> >>> rotection.outlook.com_-3Furl-3Dhttp-253A-252F-252Flists.isocpp.org-2
> >>> 52
> >>> Fmodules-252F2019-252F03-252F0145.php-26amp-3Bdata-3D02-257C01-
> >> 257Cgdr
> >>> -2540microsoft.com-257Ca5f3d89c3d7d4346707f08d6a1854be8-
> >> 257C72f988bf86
> >>> f141af91ab2d7cd011db47-257C1-257C0-257C636873991217371177-
> 26amp-
> >> 3Bsdat
> >>> a-3DOuOtS6D49Sy5EhRl8VUpv-252ByE5esxGZ7dvPvf-252Be1zr1c-253D-
> >> 26amp-3Br
> >>> eserved-
> >> 3D0&d=DwIGaQ&c=I_0YwoKy7z5LMTVdyO6YCiE2uzI1jjZZuIPelcSjixA&r=y
> >>> 8mub81SfUi-
> >>
> UCZRX0Vl1g&m=AdhXKMaBKONGion_X4H9mmw08XjCtqqGoeW9bs7cVuQ&
> >> s=
> >>> sg8E8Gx-jIbRDIRalv20OlbIkHaNJbyC3TwX8-ueRDU&e=
> >> _______________________________________________
> >> Modules mailing list
> >> Modules_at_lists.isocpp.org
> >> Subscription:
> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__nam06.safelinks.
> >> protection.outlook.com_-3Furl-3Dhttps-253A-252F-
> 252Furldefense.proofp
> >> oint.com-252Fv2-252Furl-253Fu-253Dhttp-2D-26amp-3Bdata-3D02-
> 257C01-25
> >> 7Cgdr-2540microsoft.com-257Cb10248c40d104406a58008d6a1899f80-
> 257C72f9
> >> 88bf86f141af91ab2d7cd011db47-257C1-257C0-257C636874009800024740-
> 26amp
> >> -3Bsdata-3DvJxYGAlfPqfe-252BB6HOBD4JCjOZSEP9Sc1fKDJt8ehEu4-253D-
> 26amp
> >> -3Breserved-
> 3D0&d=DwIGaQ&c=I_0YwoKy7z5LMTVdyO6YCiE2uzI1jjZZuIPelcSjix
> >> A&r=y8mub81SfUi-UCZRX0Vl1g&m=OVMNvDsyTUHgfVyzp-
> kJjPXybT2k8OAlWfIf-It-
> >> W20&s=LGVTU9QiQe6-6qXpL9XYoNaa4f0qPN5ydsuaoCSBxQU&e=
> >>
> 3A__lists.isocpp.org_mailman_listinfo.cgi_modules&d=DwIGaQ&c=I_0YwoK
> >> y7z5LMTVdyO6YCiE2uzI1jjZZuIPelcSjixA&r=y8mub81SfUi-
> >>
> UCZRX0Vl1g&m=AdhXKMaBKONGion_X4H9mmw08XjCtqqGoeW9bs7cVuQ&
> >> s=gmG6dYi-y2pNIFyGrUsRGlDKCKRKWf4WQQlvzQDWM-s&e=
> >> Link to this post:
> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__nam06.safelinks.
> >> protection.outlook.com_-3Furl-3Dhttps-253A-252F-
> 252Furldefense.proofp
> >> oint.com-252Fv2-252Furl-253Fu-253Dhttp-2D-26amp-3Bdata-3D02-
> 257C01-25
> >> 7Cgdr-2540microsoft.com-257Cb10248c40d104406a58008d6a1899f80-
> 257C72f9
> >> 88bf86f141af91ab2d7cd011db47-257C1-257C0-257C636874009800034748-
> 26amp
> >> -3Bsdata-3D123o9Z3tNcVY-252FTsR6VbHITcMCHsPMk3459OhqCqpGwk-
> 253D-26amp
> >> -3Breserved-
> 3D0&d=DwIGaQ&c=I_0YwoKy7z5LMTVdyO6YCiE2uzI1jjZZuIPelcSjix
> >> A&r=y8mub81SfUi-UCZRX0Vl1g&m=OVMNvDsyTUHgfVyzp-
> kJjPXybT2k8OAlWfIf-It-
> >> W20&s=Kjdm4tjilk-j7xq9maoXFFiZ0PeL2iBPixAyNRRBc4s&e=
> >>
> 3A__lists.isocpp.org_modules_2019_03_0147.php&d=DwIGaQ&c=I_0YwoKy
> >> 7z5LMTVdyO6YCiE2uzI1jjZZuIPelcSjixA&r=y8mub81SfUi-
> >>
> UCZRX0Vl1g&m=AdhXKMaBKONGion_X4H9mmw08XjCtqqGoeW9bs7cVuQ&
> >> s=51GVUSm-w3aQdhBiPoTvrNnOE6QlopXYmzwNAYDSU5o&e=

Received on 2019-03-05 18:10:03